0.17 is latest public code.
We do not push 'work in progress' code to github, but use own private repo for it. Only big pieces, known to work (in our RawDigger/FastRawViewer) are pushed.
(I answer anyway: this comments are public and may be helpful for other users)
LibRaw extract full visible area, 5202x3465 pixels. Canon's suggestion of smaller area, '5184x3456', is ignored.
Probably, Canon's suggestion is smaller to make some room for demosaic, do not know.
Can you think of any other ways I could attempt to get around this this with LibRaw? My application is running on a Rasberry Pi 2. Unfortunately my processing pipeline seems to be bottlenecked by the time it takes to write to the Pi's SD card. Despite having the fasted SD card available, writing is quite slow with the Pi 2. With 16bit TIFF's and then having to write out another PNG or JPEG to disk I think the cumulative effect is adding roughly 25-30% extra processing time to each photo.
Since my application converts 100s of pictures per job this can add quite few more minutes for each capture session. Perhaps I need to look in to modifying one of the source code examples and trying to figure out how to modify the code to do this myself. Do you think this would be possible? I'm not an experienced C++ programmer though and so any pointers would be most welcome. If you think it's a non-trivial task then perhaps I'd be best to just live with the extra time this steps takes.
LibRaw is library, it is not targeted to be full dcraw replacement. So, there is no such thing as 'document mode' (-D or -d in dcraw), programmer have full access to unprocessed raw data, but no ready samples for direct emulation of dcraw -D (may be unprocessed_raw sample will solve your problem)
Second one (-o1 is sRGB output, so the default) is emulated by dcraw_emu -T
Ok. Thank you for your advice. After the first feedback I hacked a small batch processing for preview images which works quite well. Even if I wondered that there is no Cam_XYZ populated with Blackmagic Production Camera raw files. But this should be another thread maybe later.
Pixels outside of image area are not image area (in most cases).
Unlike Adobe tools (e.g. DNG Converter), LibRaw tries to save as much image area as possible, without any space for demosaic to run. So, you need to crop slightly more to get demosaic some space.
There is small exclusion from 'as much as possible rule': border size is usually even, to keep bayer patern the same from both edges (uncropped and 'active area' cropped). This is not general rule, but I hold on to it when image format do not dictates other borders (such as DNG ActiveArea or so)
thank you for your reply Alex! Cleared up some stuff and I'm somehow happy that unpack already applies the curve.
Regarding 1): How much can be relied on the pixels just outside of the active pixel area? Most algorithms I have currently implemented need 2-3 pixels on the outside to get the pixels at the edge. Therefore I currently copy the 2 rows/cols at the edge to the outside but would like to save the processing time of this step. Or is the usability of this pixels mostly depending on the raw format?
1)Yes, raw_image is raw_width*raw_height 1D array. It is uncropped, full image (including 'masked areas') is read by unpack().
Black level is not subtracted, phase one postprocessing is not performed, but linearization curve is applied to raw_image data (see below)
2) Yes, curve is populated on the early stage of open_file() with curve[i] = i; (for full range loop, from 0 to 65535)
After open_file(), the curve[] will contain linearization curve read from file metadata.
Unfortunately, some formats (e.g. Nikon NEF) store curve not in metadata, but in raw image data itself. For these formats curve is read on unpack() stage.
curve[] size is not set anywhere: curve usage is specific for specific data format/unpacker code, curve is applied at unpack() stage.
Note: same curve[] array is used for gamma correction of output.
3) Use LibRaw::COLOR(row,col) call to get color at row,col.
Please note, that row,col are 'visible area' (cropped) coordinates (i.e. uncropped at top_margin/left_margin is COLOR(0,0))
Is the camera color matrix you're talking about rgb_cam?
Since it gets only used when use_camera_matrix=1, I guess that this matrix is a "camera as shot" to rgb6500K... Is that Right?
In the case of my image, there is quite a big difference between use_camera_matrix=1 and LibRaw built in... LibRaw built in produces a much warmer image -I like it better on that specific picture-...
Any idea about the reason behind that? Are Sony raw files known to have differences between the camera_matrix and the cam_mul they include? Or is cam_mul not in the raw file but calculated from camera_matrix for Sony raws?
Still, it worries me that use_camera_wb=1 and user_mul=copyof(cam_mul) -not touching anything else- create 2 different pictures.
I had the idea to put use_camera_wb=1 AFTER open() instead of before. If I use it that way, then I get the same picture as with user_mul=copyof(cam_mul).
Do you confirm that this picture I get using use_camera_wb=1 AFTER open() or user_mul=copyof(cam_mul) is the correct "as shot" one, and that use_camera_wb=1 should be used after open() and not before?
I was referring to that page telling that use_camera_wb could affect open(), and therefore was affecting it before open(), but maybe it's wrong...
Things are complicated because unpack() saves all color data into rawdata.color (and raw2image restores it). This is made to restore postprocessing initial state to run dcraw_process() multiple times with single unpack().
So, just use recommended way:
- default settings (nothing touched in params) to get daylight WB
- use_camera_wb=1 to get As shot
- use_auto_wb - to get automatic WB
- user_mul[] to get custom WB
If you mix these 3 customs settings, result is undefined (precedence is not specified, take look into source if you interested)
If I copy cam_mul to pre_mul BEFORE unpack(), I get the same picture (picture1) as if I copy cam_mul to user_mul.
If I copy cam_mul to pre_mul AFTER unpack(), I get picture2, different from the first one. However, copying cam_mul to user_mul after unpack() still produces picture1.
That may be simply because the right way to do is to set user_mul, not pre_mul... ;)
What worries me is that if I use_camera_wb=1, and do not touch user_mul nor pre_mul, I get picture3 (different from picture1). I was expecting to get picture1...
As written on my pictures, I put either:
a copy of all 4 cam_mul values in user_mul
a copy of all 4 cam_mul values in pre_mul
I found some LibRaw code (scale colors) where either user_mul or cam_mul are being copied to pre_mul, which seem to confirm what I thought...
I just cannot understand why my pictures are different... I'll have to re-check...
0.17 is latest public code.
We do not push 'work in progress' code to github, but use own private repo for it. Only big pieces, known to work (in our RawDigger/FastRawViewer) are pushed.
Expect big github update in 2-3 weeks or so.
(I answer anyway: this comments are public and may be helpful for other users)
LibRaw extract full visible area, 5202x3465 pixels. Canon's suggestion of smaller area, '5184x3456', is ignored.
Probably, Canon's suggestion is smaller to make some room for demosaic, do not know.
I think I have it figured out. Thanks.
So is (50,158) in 5202 x 3465 the same pixel as (56,168)
Thanks
Canon stores sensor active area in Makernotes (vendor specific), not in EXIF fields.
LibRaw do not use this data, so for Canon 7D active area (reported by LibRaw) is 5202x3465, while left-top corner is at 50,158
LibRaw::dcraw_ppm_tiff_writer(char* fn) accepts filename as parameter.
It looks like you need to create own image writing code with ability to write to stdout
Thanks for replying so quickly Alex.
Can you think of any other ways I could attempt to get around this this with LibRaw? My application is running on a Rasberry Pi 2. Unfortunately my processing pipeline seems to be bottlenecked by the time it takes to write to the Pi's SD card. Despite having the fasted SD card available, writing is quite slow with the Pi 2. With 16bit TIFF's and then having to write out another PNG or JPEG to disk I think the cumulative effect is adding roughly 25-30% extra processing time to each photo.
Since my application converts 100s of pictures per job this can add quite few more minutes for each capture session. Perhaps I need to look in to modifying one of the source code examples and trying to figure out how to modify the code to do this myself. Do you think this would be possible? I'm not an experienced C++ programmer though and so any pointers would be most welcome. If you think it's a non-trivial task then perhaps I'd be best to just live with the extra time this steps takes.
LibRaw is library, it is not targeted to be full dcraw replacement. So, there is no such thing as 'document mode' (-D or -d in dcraw), programmer have full access to unprocessed raw data, but no ready samples for direct emulation of dcraw -D (may be unprocessed_raw sample will solve your problem)
Second one (-o1 is sRGB output, so the default) is emulated by dcraw_emu -T
dcraw_emu only writes to files now, so you need to create temporary .tif file
use cmatrix (this is camera raw values to output rgb space matrix) if you need to use file built-in color profile
cam_xyz is filled in by internal LibRaw color profile, but no such profile is set for BlackMagic cameras.
Ok. Thank you for your advice. After the first feedback I hacked a small batch processing for preview images which works quite well. Even if I wondered that there is no Cam_XYZ populated with Blackmagic Production Camera raw files. But this should be another thread maybe later.
Pixels outside of image area are not image area (in most cases).
Unlike Adobe tools (e.g. DNG Converter), LibRaw tries to save as much image area as possible, without any space for demosaic to run. So, you need to crop slightly more to get demosaic some space.
There is small exclusion from 'as much as possible rule': border size is usually even, to keep bayer patern the same from both edges (uncropped and 'active area' cropped). This is not general rule, but I hold on to it when image format do not dictates other borders (such as DNG ActiveArea or so)
Hey,
thank you for your reply Alex! Cleared up some stuff and I'm somehow happy that unpack already applies the curve.
Regarding 1): How much can be relied on the pixels just outside of the active pixel area? Most algorithms I have currently implemented need 2-3 pixels on the outside to get the pixels at the edge. Therefore I currently copy the 2 rows/cols at the edge to the outside but would like to save the processing time of this step. Or is the usability of this pixels mostly depending on the raw format?
1)Yes, raw_image is raw_width*raw_height 1D array. It is uncropped, full image (including 'masked areas') is read by unpack().
Black level is not subtracted, phase one postprocessing is not performed, but linearization curve is applied to raw_image data (see below)
2) Yes, curve is populated on the early stage of open_file() with curve[i] = i; (for full range loop, from 0 to 65535)
After open_file(), the curve[] will contain linearization curve read from file metadata.
Unfortunately, some formats (e.g. Nikon NEF) store curve not in metadata, but in raw image data itself. For these formats curve is read on unpack() stage.
curve[] size is not set anywhere: curve usage is specific for specific data format/unpacker code, curve is applied at unpack() stage.
Note: same curve[] array is used for gamma correction of output.
3) Use LibRaw::COLOR(row,col) call to get color at row,col.
Please note, that row,col are 'visible area' (cropped) coordinates (i.e. uncropped at top_margin/left_margin is COLOR(0,0))
Being very similar to Adobe is surely not bad at all! ;)
LibRaw's embedded color profile (usually very similar to Adobe's) may differ from camera provided color data, so result difference is expected.
Is the camera color matrix you're talking about rgb_cam?
Since it gets only used when use_camera_matrix=1, I guess that this matrix is a "camera as shot" to rgb6500K... Is that Right?
In the case of my image, there is quite a big difference between use_camera_matrix=1 and LibRaw built in... LibRaw built in produces a much warmer image -I like it better on that specific picture-...
Any idea about the reason behind that? Are Sony raw files known to have differences between the camera_matrix and the cam_mul they include? Or is cam_mul not in the raw file but calculated from camera_matrix for Sony raws?
Thanks,
Sylvain.
Looks like Sony file (ARW extension).
For Sony files (with color profile data embedded in file), if use_camera_wb is on, then camera color matrix is used instead of LibRaw built in.
To get same results for use_camera_wb and user_mul set to camera WB values, set use_camera_matrix to 0
You can download my file here
Sylvain.
File from what camera you use for testing?
Understood...
Still, it worries me that use_camera_wb=1 and user_mul=copyof(cam_mul) -not touching anything else- create 2 different pictures.
I had the idea to put use_camera_wb=1 AFTER open() instead of before. If I use it that way, then I get the same picture as with user_mul=copyof(cam_mul).
Do you confirm that this picture I get using use_camera_wb=1 AFTER open() or user_mul=copyof(cam_mul) is the correct "as shot" one, and that use_camera_wb=1 should be used after open() and not before?
I was referring to that page telling that use_camera_wb could affect open(), and therefore was affecting it before open(), but maybe it's wrong...
Things are complicated because unpack() saves all color data into rawdata.color (and raw2image restores it). This is made to restore postprocessing initial state to run dcraw_process() multiple times with single unpack().
So, just use recommended way:
- default settings (nothing touched in params) to get daylight WB
- use_camera_wb=1 to get As shot
- use_auto_wb - to get automatic WB
- user_mul[] to get custom WB
If you mix these 3 customs settings, result is undefined (precedence is not specified, take look into source if you interested)
So, after re-checking:
If I copy cam_mul to pre_mul BEFORE unpack(), I get the same picture (picture1) as if I copy cam_mul to user_mul.
If I copy cam_mul to pre_mul AFTER unpack(), I get picture2, different from the first one. However, copying cam_mul to user_mul after unpack() still produces picture1.
That may be simply because the right way to do is to set user_mul, not pre_mul... ;)
What worries me is that if I use_camera_wb=1, and do not touch user_mul nor pre_mul, I get picture3 (different from picture1). I was expecting to get picture1...
As written on my pictures, I put either:
a copy of all 4 cam_mul values in user_mul
a copy of all 4 cam_mul values in pre_mul
I found some LibRaw code (scale colors) where either user_mul or cam_mul are being copied to pre_mul, which seem to confirm what I thought...
I just cannot understand why my pictures are different... I'll have to re-check...
What values you use in user_mul ?
Pages