I have looked around at dcraw_common.ccp and I find no instance of setting data_size... ever. I looked at the latest dcraw.c source and I see the LibRaw version is doing something interesting in copying the entire JPEG data area and byte-flipping it whereas this is apparently done in the input buffer in dcraw.c. I can certainly understand a need/desire to keep the input buffer as "const", even at the expense of a huge malloc to hold the copy.
But this doesn't explain how data_size (which controls the malloc and byte swapping) gets set in things like RawDigger.
I put in some code to set data size at the beginning of kodak_jpeg_load_raw and it works flawlessly now. This isn't something I am comfortable with. I would very much like to understand how data_size is supposed to be set, given that it is buried in an "internal" struct, I am sure it is not an external parameter. Clearly, data_size is completely new to LibRaw and doesn't appear in the original dcraw.c source. But when I search on data_size in dcraw_common.cpp the only occurrences I find are reading it. Actually, I find no occurrence anywhere in LibRaw where this value is set.
positive exposure shift may be non-linear (highlights compression), while auto-bright is linear.
BTW, LibRaw's dcraw_process() is only 'sample' processing. It is not ideal and not in development. We hope developers will implement own processing on top of LibRaw::open_file/unpack.
But if exp_shift works when auto brightness is enabled, why doesn't it darken the image again when using negativ exposure shift - think of it as means to overwrite the auto brightness functionality - or shift it :-)
Most likely, this is some 'approximation' if 1st word is Canon, than second is model name.
This camera was produces ~20years ago, current canon models do not use 'Inc' in maker/model
I find it interesting that the latest EXIFTool incorrectly reports the camera manufacturer as "Canon" and the model as "Inc.". Any reference to the actual camera model (which appears to be properly encoded in the data) is missing. I looked (briefly) at the EXIFTool source and it is far, far beyond what I want to get into.
If you have an EXIFTool which reports the name of the camera model correctly, can you please share? (if nothing else, you can upload it to ftp.infinadyne.com - it takes anonymous uploads in the root.
I could run EXIFTool against all the raw files I have from various sources, but if they do not properly reflect the camera model, it isn't really all that useful.
But what about the totally wrong colors ?
If it is just scaling, I would assume that the output which utilizes only fraction of the full data-width ( i.e. only 10 of 16 bits ) has "correct" colors but is way to dark.
My output just looks mostly like gray pixel dust with false colored spots !?
I'll try to attach a picture...
1) no_auto_bright will do not do any auto-brigten, so for 10-bit images (for example) you'll get 0...1023 data range whilt 16-bit data range is 0..64k. Most likely you need some data scaling
2) gamm[0]=gamm[1] = 1.0 is NOT 'gamma corrected' image. It is gamma=1.0 with slope (linear part) 1.0
BTW, the sample you sent is very rare PowerShot 600 file. Unfortunately, this is not complete file, but cutted (it is easily visible by exiftool: metadata structure is not full, jpeg preview is also missing)
Could you please send us all files seems problematic? Hope, there is complete file from Canon PS 600 (this is 1996 model) i need to check PS600 route in dcraw/libraw (this is very different from other canon files)
OK, magic is completely resolved:
- on 64-bit compile everything is OK because of 64-bit size_t
- on unix 32 bit compile all is OK because fseek's offset is long, so 32 bit
- on 32-bit LibRaw with file interface, LibRaw_file_datastream is used by default. It is based on 32-bit std::streambuf, so out-of-range offset is converted back to 32-bit '-53' (for the sample).
So, only bigfile_datastream (not used for small files by default) and buffer_datastream (read from memory) are affected.
Fortunately, this is the only place where strlen is used as fseek() parameter.
BTW, best way to fix looks like (in the beginning of dcraw_common.cpp):
Yes, this data type conversion problem looks like cause:
- model reads incorrectly
- decoder do not set right (canon_load_raw() remains) because it set by make/model name
- and, so, data do not decoded right.
With VC 2013 (and probably many others) the return value from strlen() is size_t, which is unsigned. What we want here is to back up the data pointer with a signed offset. The fseek memory buffer implementation wants a signed 64-bit value, but the result of (10u - 63) promoted as an unsigned value to 64 bits is 0x00000000ffffffcb which doesn't work at all.
I suspect there are a few other places where a signed offset for fseek is used as well and anywhere it occurs with an unsigned 32-bit value being promoted to 64 bits is going to have problems.
By casting the size_t to an int the promotion of the value from 32 to 64 bits works properly. Unfortunately, this is in the base dcraw code. I suspect there might be a way to fix this in the LibRaw code. I do not believe that the define:
#define strlen(x) (INT64)strlen(x)
is the right way to go, but it would at least get the cast out of the dcraw code. Alex, I'll let you figure out a way to accomplish this with the least amount of work.
I have looked around at dcraw_common.ccp and I find no instance of setting data_size... ever. I looked at the latest dcraw.c source and I see the LibRaw version is doing something interesting in copying the entire JPEG data area and byte-flipping it whereas this is apparently done in the input buffer in dcraw.c. I can certainly understand a need/desire to keep the input buffer as "const", even at the expense of a huge malloc to hold the copy.
But this doesn't explain how data_size (which controls the malloc and byte swapping) gets set in things like RawDigger.
I put in some code to set data size at the beginning of kodak_jpeg_load_raw and it works flawlessly now. This isn't something I am comfortable with. I would very much like to understand how data_size is supposed to be set, given that it is buried in an "internal" struct, I am sure it is not an external parameter. Clearly, data_size is completely new to LibRaw and doesn't appear in the original dcraw.c source. But when I search on data_size in dcraw_common.cpp the only occurrences I find are reading it. Actually, I find no occurrence anywhere in LibRaw where this value is set.
What am I missing here?
And 32-bit version works fine too: https://www.dropbox.com/s/4azsj2qecpl4af9/Screenshot%202015-11-10%2019.1...
Sorry. looked under wrong #ifdef.
And (looked in RawDigger in debugger) data_size is correct. Looks like I need to switch to 32-bit version again?
What I am seeing at the beginning of kodak_jpeg_load_raw is the following:
// LibRaw's Kodak_jpeg_load_raw void CLASS kodak_jpeg_load_raw() { if(data_size < 1) throw LIBRAW_EXCEPTION_DECODE_JPEG;This is the source of the -100008 error I am getting. This is in dcraw_common.cpp.
I am building this with NO_JPEG not defined.
ok. thanks for clarifying this.
positive exposure shift may be non-linear (highlights compression), while auto-bright is linear.
BTW, LibRaw's dcraw_process() is only 'sample' processing. It is not ideal and not in development. We hope developers will implement own processing on top of LibRaw::open_file/unpack.
...kind of expected.
But if exp_shift works when auto brightness is enabled, why doesn't it darken the image again when using negativ exposure shift - think of it as means to overwrite the auto brightness functionality - or shift it :-)
We use LibRaw in 64-bit windows projects (Visual Studio 2010 and 2013) without any problem.
Could you please describe the problem in more details?
'negative' exposure shift darkens image (in linear fashion).
Auto-bright - brightens it.
It is expected, is not it?
Seems like this is due to
Using no_auto_bright = 1 seems to allow using values of 0.25 <= exp_shift < 1.0 !?
API docu doesn't indicate this constraint.
Sorry: I meant "Everything works fine when platform and target machine are x86"
data_size is not used in kodak_jpeg_load_raw(), so it is not critical.
All Kodak files from your link are processed OK by our FastRawViewer and RawDigger (LibRaw used internally).
dcraw_emu.exe -4 -T -w -o 0 looks OK too.
-4: equal to gamm[0]=gamm[1] = no_auto_bright = 1; output_bps=16
-w - use camera WB
-o 0 - RAW color
You can download the RAW here.
The output of the following command looks ok.
Find a thumbnail of the output here
Seems like I don't use the correct options on LibRaw or GraphicsMagick is causing the problem.
Most likely, this is some 'approximation' if 1st word is Canon, than second is model name.
This camera was produces ~20years ago, current canon models do not use 'Inc' in maker/model
I find it interesting that the latest EXIFTool incorrectly reports the camera manufacturer as "Canon" and the model as "Inc.". Any reference to the actual camera model (which appears to be properly encoded in the data) is missing. I looked (briefly) at the EXIFTool source and it is far, far beyond what I want to get into.
If you have an EXIFTool which reports the name of the camera model correctly, can you please share? (if nothing else, you can upload it to ftp.infinadyne.com - it takes anonymous uploads in the root.
I could run EXIFTool against all the raw files I have from various sources, but if they do not properly reflect the camera model, it isn't really all that useful.
Please provide raw file for inspection
See a downscaled version of the output here.
This used to be a picture of a beach incl. bright sun light.
But what about the totally wrong colors ?
If it is just scaling, I would assume that the output which utilizes only fraction of the full data-width ( i.e. only 10 of 16 bits ) has "correct" colors but is way to dark.
My output just looks mostly like gray pixel dust with false colored spots !?
I'll try to attach a picture...
1) no_auto_bright will do not do any auto-brigten, so for 10-bit images (for example) you'll get 0...1023 data range whilt 16-bit data range is 0..64k. Most likely you need some data scaling
2) gamm[0]=gamm[1] = 1.0 is NOT 'gamma corrected' image. It is gamma=1.0 with slope (linear part) 1.0
BTW, the sample you sent is very rare PowerShot 600 file. Unfortunately, this is not complete file, but cutted (it is easily visible by exiftool: metadata structure is not full, jpeg preview is also missing)
Could you please send us all files seems problematic? Hope, there is complete file from Canon PS 600 (this is 1996 model) i need to check PS600 route in dcraw/libraw (this is very different from other canon files)
OK, magic is completely resolved:
- on 64-bit compile everything is OK because of 64-bit size_t
- on unix 32 bit compile all is OK because fseek's offset is long, so 32 bit
- on 32-bit LibRaw with file interface, LibRaw_file_datastream is used by default. It is based on 32-bit std::streambuf, so out-of-range offset is converted back to 32-bit '-53' (for the sample).
So, only bigfile_datastream (not used for small files by default) and buffer_datastream (read from memory) are affected.
Fortunately, this is the only place where strlen is used as fseek() parameter.
BTW, best way to fix looks like (in the beginning of dcraw_common.cpp):
int my_strlen(const char *str) { return (int)strnlen(str,0x7fffffff); } #define strlen(a) my_strlen((a))Yes, this data type conversion problem looks like cause:
- model reads incorrectly
- decoder do not set right (canon_load_raw() remains) because it set by make/model name
- and, so, data do not decoded right.
Yes, I tracked it down to that as well.
There is the odd construct (odd? For dcraw? Whatever!) that is making some interesting assumptions:
fread (make, 64, 1, ifp);
fseek (ifp, strlen(make) - 63, SEEK_CUR);
fread (model, 64, 1, ifp);
With VC 2013 (and probably many others) the return value from strlen() is size_t, which is unsigned. What we want here is to back up the data pointer with a signed offset. The fseek memory buffer implementation wants a signed 64-bit value, but the result of (10u - 63) promoted as an unsigned value to 64 bits is 0x00000000ffffffcb which doesn't work at all.
I suspect there are a few other places where a signed offset for fseek is used as well and anywhere it occurs with an unsigned 32-bit value being promoted to 64 bits is going to have problems.
One way to get this working is:
fread (make, 64, 1, ifp);
fseek (ifp, (INT64)strlen(make) - 63, SEEK_CUR);
fread (model, 64, 1, ifp);
By casting the size_t to an int the promotion of the value from 32 to 64 bits works properly. Unfortunately, this is in the base dcraw code. I suspect there might be a way to fix this in the LibRaw code. I do not believe that the define:
#define strlen(x) (INT64)strlen(x)
is the right way to go, but it would at least get the cast out of the dcraw code. Alex, I'll let you figure out a way to accomplish this with the least amount of work.
Mystery solved: 64-bit LibRaw is OK, while 32-bit builds gives this error.
To be continued tomorrow morning....
Pages