OK we get into canon_load_raw with "maximum" set to 4095 (fff), discover that "canon_has_lowbits" returns true so maximum is not changed to 1023. I find this curious since the pixel value evaluation that is going on later is checking against a fixed value of 1023 rather than comparing against "maximum".
So, should the derror check actually be if the pixel value exceeds maximum? I am wondering if in release mode this right-shift operation is actually just being optimized away because there result really isn't being used. One of the things with serious C obfuscation is that today's optimizing compilers tend to interpret fancy side-effects as being irrelevant and just throw them away. This differs compiler-to-compiler, so if you are using GCC (some version?) and I am using VC 2013 we might be seeing vastly different results in optimized code.
Especially when someone is using a right-shift as a value test.
That does make things tough to figure out. I will pull the error count to check for things like this - I can see the counter at 255110 after unpack() with what I have. I think any non-zero value is likely to be fatal.
I am not at all sure I understand this line of code. If one separates out things in parentheses, we have an assignment to pixel[block << 6> + I] which makes sense, but the condition being tested is, at best, a little odd. This does not appear in older versions of dcraw, so I have nothing to compare against. But why would anyone form a conditional expression which comes down to (x >> 10)???
I suppose this could be a rather strange way of saying (x > 1023), but why make it so cryptic? I know, that is the dcraw/dave coffin way of things. But I can complain, can't I? It has been a long time since I actually found obfuscated C to be all that funny.
Does this pixel-content test actually make sense in this context?
One thing occurs to me with this - I am thinking that it is a typical debug/release sort of problem. I am working with DEBUG builds right now and building the library as a static library in DEBUG mode. I am wondering if the problem might disappear with a release build - perhaps because of initializing memory with zeros instead of CDCDCDCD, or just because of different stack layout.
I will do some more digging since I now have a much better handle on why it is failing. Maybe it should fail, but there is an image buried in there that can be rendered, so it would be nice to be able to get it out.
I was trying to understand how an error could be being returned and not being caught. Turns out, I have callbacks turned off (as in LIBRAW_OPIONS_NO_MEMERR_CALLBACK | LIBRAW_OPIONS_NO_DATAERR_CALLBACK) and this error is being reported with the dcraw_emu program via a callback!
So, it would seem that this error is not being reported or returned when the callback is turned off. That is an important note as to the importance of the callbacks which I did not understand originally.
Turns out for the image in question "derror()" is being called from canon_load_raw(). Not exactly clear what is going on with this, but it seems like it is a problem with pixel values. What isn't at all clear is how the image is being processed correctly when this is not asserted.
I am not so sure I am missing something in my use of libraw any longer, other than it should probably pick up data error callbacks and treat the image as unsupported if this occurs.
Well, that is the problem. No error codes. It is difficult to describe the contents of the image, but what I am getting is some fairly random noise.
Here is a link to everything: https://dl.dropboxusercontent.com/u/43288013/Photos.zip
There are three files: 600_0001.CRW (original), 600_0001.CRW.ppm (what it is supposed to show as) and 600_0001.CRW.jpg (what I am seeing).
It seems most of the failures are .CRW (Canon) images, at least so far. Some of the .CRW are working and .NEF is working fine for all of them.
I will see what I can do with the prebuilt binary for dcraw_emu and experiment with the -mem switch. My use of the library has the image loaded into memory already which is why I am using open_buffer. Basically where I am in this is that I took an older dcraw-conversion and ripped it out replacing it with libraw. It was mostly working before (and did work with .CRW files) but it was a dcraw from around 2009. I wanted to pick up more current camera support. Two weeks into the project... here I am.
LibRaw's internal/* files are generated from (patched) dcraw.c source.
If the warnings are from this code (dcraw_common.cpp, dcraw_fileio.cpp), your changes will *not* be merged into main LibRaw source because changing of these files will broke dcraw auto-import.
Thank you for clarification, that makes sense now. I'm using LibRaw under Mac with Xcode/clang and have lots of warnings (>150), so I'm considering to fix them or not. It is worth doing only if fixes are mainlined somehow.
0.17 is latest public code.
We do not push 'work in progress' code to github, but use own private repo for it. Only big pieces, known to work (in our RawDigger/FastRawViewer) are pushed.
(I answer anyway: this comments are public and may be helpful for other users)
LibRaw extract full visible area, 5202x3465 pixels. Canon's suggestion of smaller area, '5184x3456', is ignored.
Probably, Canon's suggestion is smaller to make some room for demosaic, do not know.
Can you think of any other ways I could attempt to get around this this with LibRaw? My application is running on a Rasberry Pi 2. Unfortunately my processing pipeline seems to be bottlenecked by the time it takes to write to the Pi's SD card. Despite having the fasted SD card available, writing is quite slow with the Pi 2. With 16bit TIFF's and then having to write out another PNG or JPEG to disk I think the cumulative effect is adding roughly 25-30% extra processing time to each photo.
Since my application converts 100s of pictures per job this can add quite few more minutes for each capture session. Perhaps I need to look in to modifying one of the source code examples and trying to figure out how to modify the code to do this myself. Do you think this would be possible? I'm not an experienced C++ programmer though and so any pointers would be most welcome. If you think it's a non-trivial task then perhaps I'd be best to just live with the extra time this steps takes.
LibRaw is library, it is not targeted to be full dcraw replacement. So, there is no such thing as 'document mode' (-D or -d in dcraw), programmer have full access to unprocessed raw data, but no ready samples for direct emulation of dcraw -D (may be unprocessed_raw sample will solve your problem)
Second one (-o1 is sRGB output, so the default) is emulated by dcraw_emu -T
Ok. Thank you for your advice. After the first feedback I hacked a small batch processing for preview images which works quite well. Even if I wondered that there is no Cam_XYZ populated with Blackmagic Production Camera raw files. But this should be another thread maybe later.
Pixels outside of image area are not image area (in most cases).
Unlike Adobe tools (e.g. DNG Converter), LibRaw tries to save as much image area as possible, without any space for demosaic to run. So, you need to crop slightly more to get demosaic some space.
There is small exclusion from 'as much as possible rule': border size is usually even, to keep bayer patern the same from both edges (uncropped and 'active area' cropped). This is not general rule, but I hold on to it when image format do not dictates other borders (such as DNG ActiveArea or so)
This sample is from PowerShot 600 camera, so decoder is canon_600_load_raw()
Not trying to be annoying here, but ...
OK we get into canon_load_raw with "maximum" set to 4095 (fff), discover that "canon_has_lowbits" returns true so maximum is not changed to 1023. I find this curious since the pixel value evaluation that is going on later is checking against a fixed value of 1023 rather than comparing against "maximum".
So, should the derror check actually be if the pixel value exceeds maximum? I am wondering if in release mode this right-shift operation is actually just being optimized away because there result really isn't being used. One of the things with serious C obfuscation is that today's optimizing compilers tend to interpret fancy side-effects as being irrelevant and just throw them away. This differs compiler-to-compiler, so if you are using GCC (some version?) and I am using VC 2013 we might be seeing vastly different results in optimized code.
Especially when someone is using a right-shift as a value test.
That does make things tough to figure out. I will pull the error count to check for things like this - I can see the counter at 255110 after unpack() with what I have. I think any non-zero value is likely to be fatal.
What is failing is the line:
if ((pixel[(block << 6) + i] = base[i & 1] += diffbuf[i]) >> 10)
derror();
I am not at all sure I understand this line of code. If one separates out things in parentheses, we have an assignment to pixel[block << 6> + I] which makes sense, but the condition being tested is, at best, a little odd. This does not appear in older versions of dcraw, so I have nothing to compare against. But why would anyone form a conditional expression which comes down to (x >> 10)???
I suppose this could be a rather strange way of saying (x > 1023), but why make it so cryptic? I know, that is the dcraw/dave coffin way of things. But I can complain, can't I? It has been a long time since I actually found obfuscated C to be all that funny.
Does this pixel-content test actually make sense in this context?
One thing occurs to me with this - I am thinking that it is a typical debug/release sort of problem. I am working with DEBUG builds right now and building the library as a static library in DEBUG mode. I am wondering if the problem might disappear with a release build - perhaps because of initializing memory with zeros instead of CDCDCDCD, or just because of different stack layout.
I will do some more digging since I now have a much better handle on why it is failing. Maybe it should fail, but there is an image buried in there that can be rendered, so it would be nice to be able to get it out.
Things become stranger: recompiled LibRaw 0,17 and the problem gone. Now I cannot reproduce the problem.
Anyway, the file is broken (metadata incomplete).
I still do not know why it not unpacked by 0.17-WIn32-binary from this site and no way to know it because the problem looks un-reproducible.
BTW, derror() count call is returned by LibRaw::error_count() (after unpack() ), so it is better to check it in production code.
callbacks are optional.
You'll receive error code for call and, also, bits in imgdata.process_warnings for non-fatal errors (data error is not fatal).
BTW, the image is indeed broken, but raw data in place. Still investigating.
I was trying to understand how an error could be being returned and not being caught. Turns out, I have callbacks turned off (as in LIBRAW_OPIONS_NO_MEMERR_CALLBACK | LIBRAW_OPIONS_NO_DATAERR_CALLBACK) and this error is being reported with the dcraw_emu program via a callback!
So, it would seem that this error is not being reported or returned when the callback is turned off. That is an important note as to the importance of the callbacks which I did not understand originally.
Turns out for the image in question "derror()" is being called from canon_load_raw(). Not exactly clear what is going on with this, but it seems like it is a problem with pixel values. What isn't at all clear is how the image is being processed correctly when this is not asserted.
I am not so sure I am missing something in my use of libraw any longer, other than it should probably pick up data error callbacks and treat the image as unsupported if this occurs.
More likely, it means that some of internal tags pointed outside of data buffer.
This is not an error for file case (seek out of file will not read anything, so tag value is ignored), but will raise error in buffer case.
BTW, it means you *have* error code at unpack()
Thank you for file sample, to be investigated in depth.
dcraw_emu (prebuilt) says "unknown file: data corrupted at 127149" with the -mem switch. Without it, works fine.
Not sure what this means, but it is different.
Well, that is the problem. No error codes. It is difficult to describe the contents of the image, but what I am getting is some fairly random noise.
Here is a link to everything: https://dl.dropboxusercontent.com/u/43288013/Photos.zip
There are three files: 600_0001.CRW (original), 600_0001.CRW.ppm (what it is supposed to show as) and 600_0001.CRW.jpg (what I am seeing).
It seems most of the failures are .CRW (Canon) images, at least so far. Some of the .CRW are working and .NEF is working fine for all of them.
I will see what I can do with the prebuilt binary for dcraw_emu and experiment with the -mem switch. My use of the library has the image loaded into memory already which is why I am using open_buffer. Basically where I am in this is that I took an older dcraw-conversion and ripped it out replacing it with libraw. It was mostly working before (and did work with .CRW files) but it was a dcraw from around 2009. I wanted to pick up more current camera support. Two weeks into the project... here I am.
Could you please explain what is 'not working':
- error codes (on which call exactly, what code)
- error codes are OK, but resulting image is broken?
Also, dcraw_emu has -mem command-line switch to test memory I/O instead of File I/O. It it working for your case, or not?
Then I think it's not worth it :)
LibRaw's internal/* files are generated from (patched) dcraw.c source.
If the warnings are from this code (dcraw_common.cpp, dcraw_fileio.cpp), your changes will *not* be merged into main LibRaw source because changing of these files will broke dcraw auto-import.
Thank you for clarification, that makes sense now. I'm using LibRaw under Mac with Xcode/clang and have lots of warnings (>150), so I'm considering to fix them or not. It is worth doing only if fixes are mainlined somehow.
0.17 is latest public code.
We do not push 'work in progress' code to github, but use own private repo for it. Only big pieces, known to work (in our RawDigger/FastRawViewer) are pushed.
Expect big github update in 2-3 weeks or so.
(I answer anyway: this comments are public and may be helpful for other users)
LibRaw extract full visible area, 5202x3465 pixels. Canon's suggestion of smaller area, '5184x3456', is ignored.
Probably, Canon's suggestion is smaller to make some room for demosaic, do not know.
I think I have it figured out. Thanks.
So is (50,158) in 5202 x 3465 the same pixel as (56,168)
Thanks
Canon stores sensor active area in Makernotes (vendor specific), not in EXIF fields.
LibRaw do not use this data, so for Canon 7D active area (reported by LibRaw) is 5202x3465, while left-top corner is at 50,158
LibRaw::dcraw_ppm_tiff_writer(char* fn) accepts filename as parameter.
It looks like you need to create own image writing code with ability to write to stdout
Thanks for replying so quickly Alex.
Can you think of any other ways I could attempt to get around this this with LibRaw? My application is running on a Rasberry Pi 2. Unfortunately my processing pipeline seems to be bottlenecked by the time it takes to write to the Pi's SD card. Despite having the fasted SD card available, writing is quite slow with the Pi 2. With 16bit TIFF's and then having to write out another PNG or JPEG to disk I think the cumulative effect is adding roughly 25-30% extra processing time to each photo.
Since my application converts 100s of pictures per job this can add quite few more minutes for each capture session. Perhaps I need to look in to modifying one of the source code examples and trying to figure out how to modify the code to do this myself. Do you think this would be possible? I'm not an experienced C++ programmer though and so any pointers would be most welcome. If you think it's a non-trivial task then perhaps I'd be best to just live with the extra time this steps takes.
LibRaw is library, it is not targeted to be full dcraw replacement. So, there is no such thing as 'document mode' (-D or -d in dcraw), programmer have full access to unprocessed raw data, but no ready samples for direct emulation of dcraw -D (may be unprocessed_raw sample will solve your problem)
Second one (-o1 is sRGB output, so the default) is emulated by dcraw_emu -T
dcraw_emu only writes to files now, so you need to create temporary .tif file
use cmatrix (this is camera raw values to output rgb space matrix) if you need to use file built-in color profile
cam_xyz is filled in by internal LibRaw color profile, but no such profile is set for BlackMagic cameras.
Ok. Thank you for your advice. After the first feedback I hacked a small batch processing for preview images which works quite well. Even if I wondered that there is no Cam_XYZ populated with Blackmagic Production Camera raw files. But this should be another thread maybe later.
Pixels outside of image area are not image area (in most cases).
Unlike Adobe tools (e.g. DNG Converter), LibRaw tries to save as much image area as possible, without any space for demosaic to run. So, you need to crop slightly more to get demosaic some space.
There is small exclusion from 'as much as possible rule': border size is usually even, to keep bayer patern the same from both edges (uncropped and 'active area' cropped). This is not general rule, but I hold on to it when image format do not dictates other borders (such as DNG ActiveArea or so)
Pages