To match JPEG colors you need not dcraw.c color profiles, but profiles used by camera vendors for in-camera processing (unavailable in most cases).
Also, the main problem is to repeat vendor's tone/contrast curve (which, in turn, changes when photographer change camera contrast/saturation/color mode) to match midtone brightness and highlights compression.
So, if your target is 'embedded JPEG match', than you (very likely) need own RAW postprocessing code instead of LibRaw::dcraw_process()
I'm sorry I do not understand why you can't use this solution? On a side note, why use colour processing from dcraw at all, why not to use quality colour transforms, since you develop anyway?
If that means that an upcoming version of dcraw_emu will use the dcraw color profile(s) that would be really good. Mainly, I just need the colors to match up with the colors in the embedded JPEGs. So far that seems to be the case, except for this one example.
dcraw.c has not changed for more than one year now (also, may-2016 change was minor, so real freeze is about 2 years).
Because of that, maintaining full dcraw.c compatibility is obsolete goal :)
Meanwhile, for cameras supported in (last) dcraw.c we (very likely) will change color profile to dcraw's even if we think our profile is better.
Since I'm using dcraw_emu, I can't use that solution. If the intent with dcraw_emu was to keep it consistent with dcraw, wouldn't it make sense to use that (revised) profile in dcraw_emu?
auto brightness calculates image histogram, than adjust (increase) values to fit full 16-bit range (with 1% of pixels at saturation with default auto_bright_thr). So, if you use linear exposure correction (w/o highlights compression), than your changes will be ruined (indeed), because brightness is adjusted to use full data range.
Do not know what 'color adjustment' you use,because no such mechanics exists in dcraw_process() options (excluding white balance).
Without auto-bright, 8-bit images use only 0..255 range (to be correct,it may be more if white balance is used). This is 8 stop below full range (0..65535), so resulting image will look 'near black' (not completely black).
Just rebuilt raw.dll (only ended up having to edit dcraw_common.cpp, thanks for the tips), and openbayer_sample() works with the patch! Thank you so much for your fast help. This is awesome!
I have found that to get what I want from LibRaw, I need to set no_auto_bright = 0 for 8bit images, and no_auto_bright = 1 for 16bit images. This is no problem to me, but the behavior is a little bizarre. The rest of this comment is to help you figure out if this is an unintentional error within LibRaw.
I do some precise exposure compensation and color correction before using LibRaw. After some testing, I have found a bizarre result when I set imgdata.params.no_auto_bright:
__________________|___8bit image____|__16bit image___ no_auto_bright = 0 |___100% correct___|__Earlier exposure / color adjustments are ruined no_auto_bright = 1 |_Completely black_|_100% correct___
(By the way, when I say "100% correct," I mean that my earlier exposure adjustments were conserved :) )
The results for 16bit images in the table above make perfect sense.
The results for 8bit images don't make any sense to me! I would expect results exactly like 16bit. Instead, in 8bit, no_auto_bright = 0 behaves like I would expect no_auto_bright = 1 to behave, and no_auto_bright = 1 makes the entire image all-black no matter what.
The difference is in camera color profiles used in LibRaw and dcraw.
To get the same color as in dcraw, please go to adobe_coeff() function in (LibRaw)/internal/dcraw_common.cpp
and replace our color profile (two lines:
{ "Fujifilm X-E2", 0, 0,
{ 12066,-5927,-367,-1969,9878,1503,-721,2034,5453 } },
to dcraw.c profile:
{ "Fujifilm X-E2", 0, 0,
{ 8458,-2451,-855,-4597,12447,2407,-1475,2482,6526 } },
The sample file in question was created with my X-E2 camera. I uploaded some screenshots and a copy of the file. The converted files are both a bit brighter than the embedded JPEG because I used a brightness of 1.
Hi Alex,
First of all, this specific problem was my own doing. I was using the -4 DCRaw parm that apparently does create some issues. Some of the files that I compared with had been converted without that parm. When I removed that, DCRaw_emu worked the way I had hoped and in a very similar way to DCRaw 9.27 (but faster).
There are still a couple of small differences from DCRaw 9.27:
* Fuji X files - these are coming out with the green a sort of yellow-green. This is different from the result with DCRAW 9.27 and different from the embedded JPEG.
* The resulting file name when I convert to a Tiff file is in the format filename.raf.tiff instead of filename.tiff
Otherwise, DCRaw_emu seems to do what I need it to do.
Chris
PS: I looked on the Data Structures page for -c so that's why I couldn't find it.
Hi Alex,
Thank you for the quick reply. I tried it with -c 0. I had the brightness set to 2.3 (leaving it at 1 made everything too dark). I did not see any difference using -c 0. Some images came out darker than the result from DCRaw 9.27, some came out lighter and one came out very similar. Obviously, changing the brightness level would change which was most similar. For most files, the version from DCRaw 9.27 (with brightness set to 1) was pretty consistent with the brightness level of the embedded JPEG from the RAW files (the version from DCRaw_emu was not).
Any other thoughts? I'd really like to use your version because of its other benefits but this complicates it.
NOTE: I did not see - c mentioned in your documentation. In the DCRAW 9.27 documentation, it says that -c is used to: "Write decoded images or thumbnails to standard output."
Yes, this is bug in open_bayer() code, the standard tone curve (color.curve[] array) was not initialized to linear curve, but it is used in 8-bit decoder.
Just add these lines:
for (int i = 0; i < 0x10000; i++)
curve[i] = i;
To void CLASS initdata() (anywhere in the function body). This call is in
* dcraw/dcraw.c if you're using snapshot (and re-generate script/makefile)
* internal/dcraw_common.cpp (if you're do not regenerate this file from LibRaw's dcraw.c).
(you may patch both source files if you're not sure, if re-generate script is used, than dcraw_common.cpp to be generated from dcraw.c if needed)
I found a workaround to make LibRaw accept the file (and other 8bit files from the same camera)... I store each 8bit pixel in memory as if it were a 16bit pixel, then have LibRaw output 8bits. I cannot keep using this because it is exactly twice the amount of IO that should be done!
Hope it's an easy fix
P.S. might be helpful -
This one does not have 255-level pixels at the beginning, they are quite dark in the upper left.
It is in the "RGGB" RGRGRG.../GBGBGB.../ Bayer pattern
LibRaw contains 'automatic adjust maximum' code to not get 'pink clouds' problem when maximum data level reported by camera is higher than real data maximum (this is common problem with canon files esp. if intermediate iso are used).
To disable this feature use -c 0 option of dcraw_emu (imgdata.params.adjust_maximum_thr = 0 if you use LibRaw directly from your code, not dcraw_emu.exe)
We do not have Sinar 4-shot anfd 16-shot samples, so we cannot test this support (and hope dcraw-derived code works right).
Could you please provide several 4-shot and 16-shot samples for testing (use any file sharing service and send link to info@libraw.org or post link in comments here)
Sorry for delayed publishing of this topic, summer is vacation time even for libraw.org moderator :)
Could you please provide image sample to test it with. open_bayer should work with 8-bit images, but it looks like it needs additional testing.
version 0.19 expected to be released this year or in Q1-2019 (we definitely will publish 1-2 snapshots before that). Snapshots are used in our applications (RawDigger, FastRawViewer) and stable for production use.
To match JPEG colors you need not dcraw.c color profiles, but profiles used by camera vendors for in-camera processing (unavailable in most cases).
Also, the main problem is to repeat vendor's tone/contrast curve (which, in turn, changes when photographer change camera contrast/saturation/color mode) to match midtone brightness and highlights compression.
So, if your target is 'embedded JPEG match', than you (very likely) need own RAW postprocessing code instead of LibRaw::dcraw_process()
> I just need the colors to match up with the colors in the embedded JPEGs. So far that seems to be the case
Ummm.... "Match" means, in a very relaxed environment, "within maximum of 4..6 dE".
I'm sorry I do not understand why you can't use this solution? On a side note, why use colour processing from dcraw at all, why not to use quality colour transforms, since you develop anyway?
If that means that an upcoming version of dcraw_emu will use the dcraw color profile(s) that would be really good. Mainly, I just need the colors to match up with the colors in the embedded JPEGs. So far that seems to be the case, except for this one example.
dcraw.c has not changed for more than one year now (also, may-2016 change was minor, so real freeze is about 2 years).
Because of that, maintaining full dcraw.c compatibility is obsolete goal :)
Meanwhile, for cameras supported in (last) dcraw.c we (very likely) will change color profile to dcraw's even if we think our profile is better.
Since I'm using dcraw_emu, I can't use that solution. If the intent with dcraw_emu was to keep it consistent with dcraw, wouldn't it make sense to use that (revised) profile in dcraw_emu?
These results are expected (more or less).
auto brightness calculates image histogram, than adjust (increase) values to fit full 16-bit range (with 1% of pixels at saturation with default auto_bright_thr). So, if you use linear exposure correction (w/o highlights compression), than your changes will be ruined (indeed), because brightness is adjusted to use full data range.
Do not know what 'color adjustment' you use,because no such mechanics exists in dcraw_process() options (excluding white balance).
Without auto-bright, 8-bit images use only 0..255 range (to be correct,it may be more if white balance is used). This is 8 stop below full range (0..65535), so resulting image will look 'near black' (not completely black).
Just rebuilt raw.dll (only ended up having to edit dcraw_common.cpp, thanks for the tips), and openbayer_sample() works with the patch! Thank you so much for your fast help. This is awesome!
I have found that to get what I want from LibRaw, I need to set no_auto_bright = 0 for 8bit images, and no_auto_bright = 1 for 16bit images. This is no problem to me, but the behavior is a little bizarre. The rest of this comment is to help you figure out if this is an unintentional error within LibRaw.
.............................................................................................................................
I do some precise exposure compensation and color correction before using LibRaw. After some testing, I have found a bizarre result when I set imgdata.params.no_auto_bright:
__________________|___8bit image____|__16bit image___
no_auto_bright = 0 |___100% correct___|__Earlier exposure / color adjustments are ruined
no_auto_bright = 1 |_Completely black_|_100% correct___
(By the way, when I say "100% correct," I mean that my earlier exposure adjustments were conserved :) )
The results for 16bit images in the table above make perfect sense.
The results for 8bit images don't make any sense to me! I would expect results exactly like 16bit. Instead, in 8bit, no_auto_bright = 0 behaves like I would expect no_auto_bright = 1 to behave, and no_auto_bright = 1 makes the entire image all-black no matter what.
.............................................................................................................................
Hope that was helpful for moving on with Release 0.19!
The difference is in camera color profiles used in LibRaw and dcraw.
To get the same color as in dcraw, please go to adobe_coeff() function in (LibRaw)/internal/dcraw_common.cpp
and replace our color profile (two lines:
{ "Fujifilm X-E2", 0, 0,
{ 12066,-5927,-367,-1969,9878,1503,-721,2034,5453 } },
to dcraw.c profile:
{ "Fujifilm X-E2", 0, 0,
{ 8458,-2451,-855,-4597,12447,2407,-1475,2482,6526 } },
Than re-compile/rebuild libraw/dcraw_emu
Thanks.
dcraw processed file (with -w to get camera balance) is definitely more green.
We'll investigate the difference.
Here is another link directly to the file:
https://1drv.ms/u/s!AhvHWo83K_oNjxRpT9I4Xzo00Z1X
Sorry, your link allows only download of screeenshot_embedded_JPEG.JPG, no RAW to play with
The sample file in question was created with my X-E2 camera. I uploaded some screenshots and a copy of the file. The converted files are both a bit brighter than the embedded JPEG because I used a brightness of 1.
Link
dcraw's -4 option results in
- linear gamma (-g 1 1)
- no automatic brigtening (-W)
- 16 bit output (-6)
There is no -4 option for dcraw_emu, but you may use -g 1 1 -W -6 instead.
Fuji X: what exact camera and what Libraw version?
.rawext.tiff - yes, this is intended option to allow convert filename.cr2 and filename.dng in the same folder.
Hi Alex,
First of all, this specific problem was my own doing. I was using the -4 DCRaw parm that apparently does create some issues. Some of the files that I compared with had been converted without that parm. When I removed that, DCRaw_emu worked the way I had hoped and in a very similar way to DCRaw 9.27 (but faster).
There are still a couple of small differences from DCRaw 9.27:
* Fuji X files - these are coming out with the green a sort of yellow-green. This is different from the result with DCRAW 9.27 and different from the embedded JPEG.
* The resulting file name when I convert to a Tiff file is in the format filename.raf.tiff instead of filename.tiff
Otherwise, DCRaw_emu seems to do what I need it to do.
Chris
PS: I looked on the Data Structures page for -c so that's why I couldn't find it.
-с is, indeed, documented: https://www.libraw.org/docs/Samples-LibRaw.html
-c float-value
This key sets params.adjust_maximum_thr parameter.
Use -c 0 to completely disable automatic maximum calculation.
Default value: 0.75
Could you please provide some samples to test against dcraw/dcraw_emu. With -c 0 it should be the same, at least with default settings.
Hi Alex,
Thank you for the quick reply. I tried it with -c 0. I had the brightness set to 2.3 (leaving it at 1 made everything too dark). I did not see any difference using -c 0. Some images came out darker than the result from DCRaw 9.27, some came out lighter and one came out very similar. Obviously, changing the brightness level would change which was most similar. For most files, the version from DCRaw 9.27 (with brightness set to 1) was pretty consistent with the brightness level of the embedded JPEG from the RAW files (the version from DCRaw_emu was not).
Any other thoughts? I'd really like to use your version because of its other benefits but this complicates it.
NOTE: I did not see - c mentioned in your documentation. In the DCRAW 9.27 documentation, it says that -c is used to: "Write decoded images or thumbnails to standard output."
Thanks for the sample.
Yes, this is bug in open_bayer() code, the standard tone curve (color.curve[] array) was not initialized to linear curve, but it is used in 8-bit decoder.
Just add these lines:
for (int i = 0; i < 0x10000; i++) curve[i] = i;To
void CLASS initdata()(anywhere in the function body). This call is in* dcraw/dcraw.c if you're using snapshot (and re-generate script/makefile)
* internal/dcraw_common.cpp (if you're do not regenerate this file from LibRaw's dcraw.c).
(you may patch both source files if you're not sure, if re-generate script is used, than dcraw_common.cpp to be generated from dcraw.c if needed)
Link to full patch: https://www.dropbox.com/s/xqi4ox7ui4r5cqg/8bit-openbayer.patch?dl=0
(sorry, our forum does not allow attachments to topic comment, it is easier to put it on dropbox than re-tune our forum).
This full patch also patches openbayer_sample.cpp to get correct parameters to this call.
Hi Alex, no worries! Hope it was good vacation time.
Here is my 8bit raw file on Google Drive:
https://drive.google.com/file/d/0B41mEzFmIzubcmxKMnVEcE14MEE/view?usp=sh...
Here it is demosaiced to a .tif using other software (with worse algorithms!):
https://drive.google.com/file/d/0B41mEzFmIzubZW0wMXE3YWo3eFk/view?usp=sh...
I found a workaround to make LibRaw accept the file (and other 8bit files from the same camera)... I store each 8bit pixel in memory as if it were a 16bit pixel, then have LibRaw output 8bits. I cannot keep using this because it is exactly twice the amount of IO that should be done!
Hope it's an easy fix
P.S. might be helpful -
This one does not have 255-level pixels at the beginning, they are quite dark in the upper left.
It is in the "RGGB" RGRGRG.../GBGBGB.../ Bayer pattern
LibRaw contains 'automatic adjust maximum' code to not get 'pink clouds' problem when maximum data level reported by camera is higher than real data maximum (this is common problem with canon files esp. if intermediate iso are used).
To disable this feature use -c 0 option of dcraw_emu (imgdata.params.adjust_maximum_thr = 0 if you use LibRaw directly from your code, not dcraw_emu.exe)
Thank you for your request!!
We do not have Sinar 4-shot anfd 16-shot samples, so we cannot test this support (and hope dcraw-derived code works right).
Could you please provide several 4-shot and 16-shot samples for testing (use any file sharing service and send link to info@libraw.org or post link in comments here)
The file you've shared is only 772415 bytes size (unless I downloaded it wrong, please check too).
The original RAW_CANON_40D_SRAW_V103.CR2 from Rawsamples.ch (same preview, so I'm sure it is the same file) is 6800865 bytes.
Looks like your sample is truncated.,
Sorry for delayed publishing of this topic, summer is vacation time even for libraw.org moderator :)
Could you please provide image sample to test it with. open_bayer should work with 8-bit images, but it looks like it needs additional testing.
version 0.19 expected to be released this year or in Q1-2019 (we definitely will publish 1-2 snapshots before that). Snapshots are used in our applications (RawDigger, FastRawViewer) and stable for production use.
Indeed: https://www.libraw.org/news/libraw-0-18-released
I agree, aligned scanlines are good.
We're open to patches/contribution.
Pages