For X-Trans sensors (imgdata.idata.filters == 9), color pattern is contained in
imgdata.idata.xtrans (6x6 array, indexed by row%6,col%6, row and col are relative to visible area)
and in
imgdata.idata.xtrans_abs (same as above, but row/col are relative to full sensor)
Hi - to anyone else who wants to use open_bayer() before anything later than 0.18.2 is stable-released, this is a link to the Github page at the earliest (least risky) version which supports open_bayer(). https://github.com/LibRaw/LibRaw/tree/d4c4f5c8e4a2c5db56287f805d44b590ac...
This works perfectly for 16bit (EDIT: and now 8bit) RAW images without any header or metadata.
(wish I could delete comments but I can only edit them so I must look really silly with two comments in a row)
I was serious that I do color, whitebalance, exposure, etc corrections to the raw data before putting it into LibRaw. I wrote the adjustments, and only use LibRaw to demosaic. No dcraw or LibRaw emulation of dcraw involved for color-correction / whitebalance / etc.
(Output with output_bps = 8, no_auto_bright = 1, no_auto_scale = 1, user_black = 0.) Every pixel is 0, and this was an overexposed 8bit raw. Do what you will with this information; I was just trying to repay a favor, thinking I might have found unintended behavior. I see now that LibRaw uses 16bit internally, even if the input buffer was only 8 BPP.
So, what seems to be happening...
1. 8 BPP data in the buffer has max value of 255
2. The buffer is read into LibRaw, which uses 16 BPP data internally
3. no_auto_bright = 1, so values are never scaled up to 65535
4. output_bps = 8, so values are all divided by 257 to scale down to 255 maxval... but at most, we had (255 / 257), which with integer rounding down equals 0. Anything less than 255 will of course also become 0.
5. Every pixel is 0 in the resulting image!
If that's correct (?), let this post just be a warning to any 8bit - LibRaw users who read this!
Hi Iliah,
I am using dcraw_emu (as I used dcraw 9.27 previously) as a simple tool to convert a raw file to a tiff. I'm trying to keep it very simple and use existing tools created by people such as yourself who are experts in many of these things. My program acts as a controller and uses these existing tools (such as dcraw_emu) to perform specific tasks. I'm not a C programmer and thus I don't have the ability to rebuild a custom version of dcraw_emu.
My need is simple: I just need dcraw_emu to create tiff files that are good quality (color-accurate) versions of the image that was captured in the raw file. The result should be quite similar to the original embedded JPEG and to the results produced by other raw converters. In almost all cases, dcraw_emu is doing that. The greens in my X-E2 sample pictures seem to be an exception.
As a further comparison, I converted my sample file with Lightroom and with Affinity Photo (I can provide samples if you wish). The result was:
* The original embedded JPEG and the tiff file produced by dcraw 9.27 were the brightest green (almost the same).
* The version produced by Lightroom was close to the embedded JPEG but the greens weren't quite as bright.
* The Affinity Photo version was quite a bit yellower than the previous versions. I don't think that it provided a very accurate version of the greens.
* The Dcraw_emu version came out similar to the Affinity Photo version (but lighter). Converting it with the brightness turned down further might produce a very similar version to Affinity Photo.
Going back to the dcraw 9.27 color profiles could be an unnecessary step backward. In my testing of the Sony ARW files, I found that the version produced by dcraw_emu has more accurate colors than the dcraw 9.27 version (which was much yellower). Thus, for my purposes, a small change to the color profile for the X-E2 (and the other X cameras if they are the same) should give me everything that I need from dcraw_emu.
To match JPEG colors you need not dcraw.c color profiles, but profiles used by camera vendors for in-camera processing (unavailable in most cases).
Also, the main problem is to repeat vendor's tone/contrast curve (which, in turn, changes when photographer change camera contrast/saturation/color mode) to match midtone brightness and highlights compression.
So, if your target is 'embedded JPEG match', than you (very likely) need own RAW postprocessing code instead of LibRaw::dcraw_process()
I'm sorry I do not understand why you can't use this solution? On a side note, why use colour processing from dcraw at all, why not to use quality colour transforms, since you develop anyway?
If that means that an upcoming version of dcraw_emu will use the dcraw color profile(s) that would be really good. Mainly, I just need the colors to match up with the colors in the embedded JPEGs. So far that seems to be the case, except for this one example.
dcraw.c has not changed for more than one year now (also, may-2016 change was minor, so real freeze is about 2 years).
Because of that, maintaining full dcraw.c compatibility is obsolete goal :)
Meanwhile, for cameras supported in (last) dcraw.c we (very likely) will change color profile to dcraw's even if we think our profile is better.
Since I'm using dcraw_emu, I can't use that solution. If the intent with dcraw_emu was to keep it consistent with dcraw, wouldn't it make sense to use that (revised) profile in dcraw_emu?
auto brightness calculates image histogram, than adjust (increase) values to fit full 16-bit range (with 1% of pixels at saturation with default auto_bright_thr). So, if you use linear exposure correction (w/o highlights compression), than your changes will be ruined (indeed), because brightness is adjusted to use full data range.
Do not know what 'color adjustment' you use,because no such mechanics exists in dcraw_process() options (excluding white balance).
Without auto-bright, 8-bit images use only 0..255 range (to be correct,it may be more if white balance is used). This is 8 stop below full range (0..65535), so resulting image will look 'near black' (not completely black).
Just rebuilt raw.dll (only ended up having to edit dcraw_common.cpp, thanks for the tips), and openbayer_sample() works with the patch! Thank you so much for your fast help. This is awesome!
I have found that to get what I want from LibRaw, I need to set no_auto_bright = 0 for 8bit images, and no_auto_bright = 1 for 16bit images. This is no problem to me, but the behavior is a little bizarre. The rest of this comment is to help you figure out if this is an unintentional error within LibRaw.
I do some precise exposure compensation and color correction before using LibRaw. After some testing, I have found a bizarre result when I set imgdata.params.no_auto_bright:
__________________|___8bit image____|__16bit image___ no_auto_bright = 0 |___100% correct___|__Earlier exposure / color adjustments are ruined no_auto_bright = 1 |_Completely black_|_100% correct___
(By the way, when I say "100% correct," I mean that my earlier exposure adjustments were conserved :) )
The results for 16bit images in the table above make perfect sense.
The results for 8bit images don't make any sense to me! I would expect results exactly like 16bit. Instead, in 8bit, no_auto_bright = 0 behaves like I would expect no_auto_bright = 1 to behave, and no_auto_bright = 1 makes the entire image all-black no matter what.
The difference is in camera color profiles used in LibRaw and dcraw.
To get the same color as in dcraw, please go to adobe_coeff() function in (LibRaw)/internal/dcraw_common.cpp
and replace our color profile (two lines:
{ "Fujifilm X-E2", 0, 0,
{ 12066,-5927,-367,-1969,9878,1503,-721,2034,5453 } },
to dcraw.c profile:
{ "Fujifilm X-E2", 0, 0,
{ 8458,-2451,-855,-4597,12447,2407,-1475,2482,6526 } },
Followup:
if you need to free file handle w/o releasing metadata, there is recycle_datastream() call, this call will close file handle and nothing more.
Indeed, you need only open_file() (or open_datastream() if you use it) to extract metadata.
unpack() only read and unpacks raw pixels, you do not need to use it if you need only metadata.
Also, if you need thumbnail too, you may use open_file() + unpack_thumb() calls only.
Looks like rawpy incompleteness.
For X-Trans sensors (imgdata.idata.filters == 9), color pattern is contained in
imgdata.idata.xtrans (6x6 array, indexed by row%6,col%6, row and col are relative to visible area)
and in
imgdata.idata.xtrans_abs (same as above, but row/col are relative to full sensor)
Could you please provide link to some samples?
(modern) Canon files has EXIF.Orientation tag, indeed
The patch works. Thanks!
Hi Alex, thanks for the patch! Will try it out.
Here is the patch:
diff --git a/dcraw/dcraw.c b/dcraw/dcraw.c
index fb78663..87915d9 100644
--- a/dcraw/dcraw.c
+++ b/dcraw/dcraw.c
@@ -2768,7 +2768,7 @@ void CLASS unpacked_load_raw()
while (1 << ++bits < maximum)
;
read_shorts(raw_image, raw_width * raw_height);
- if (maximum < 0xffff)
+ if (maximum < 0xffff || load_flags)
for (row = 0; row < raw_height; row++)
{
#ifdef LIBRAW_LIBRARY_BUILD
diff --git a/internal/dcraw_common.cpp b/internal/dcraw_common.cpp
index 168a095..d8e22e6 100644
--- a/internal/dcraw_common.cpp
+++ b/internal/dcraw_common.cpp
@@ -2472,7 +2472,7 @@ void CLASS unpacked_load_raw()
while (1 << ++bits < maximum)
;
read_shorts(raw_image, raw_width * raw_height);
- if (maximum < 0xffff)
+ if (maximum < 0xffff || load_flags)
for (row = 0; row < raw_height; row++)
{
#ifdef LIBRAW_LIBRARY_BUILD
but the data range is 64k
> LibRaw 0.19 says that the black level is 205 and white level is 4095
This part is as it should be, the camera is 12-bit.
Thanks a lot, issue confirmed
Meanwhile, I could not confirm the problem.
Here is the changes made for openbayer_sample.cpp:
--- a/samples/openbayer_sample.cpp
+++ b/samples/openbayer_sample.cpp
@@ -45,7 +45,9 @@ int main(int ac, char *av[])
return 3;
LibRaw rp;
rp.imgdata.params.output_tiff = 1;
- int ret = rp.open_bayer(buffer, fsz, 640, 480, 0, 0, 0, 0, 0, LIBRAW_OPENBAYER_RGGB, 0, 0, 1400);
+ rp.imgdata.params.no_auto_bright = 1;
+ //int ret = rp.open_bayer(buffer, fsz, 640, 480, 0, 0, 0, 0, 0, LIBRAW_OPENBAYER_RGGB, 0, 0, 1400);
+ int ret = rp.open_bayer(buffer, fsz,4112,3008,0,0,0,0,0,LIBRAW_OPENBAYER_RGGB,0,0,0);
if (ret != LIBRAW_SUCCESS)
return 4;
if ((ret = rp.unpack()) != LIBRAW_SUCCESS)
(no_auto_bright=1 added to default settings).
Here is TIFF produced from your sample (screenshot in viewer): https://www.dropbox.com/s/alm7jqumy7fmf12/Screenshot%202017-08-02%2019.1...
Hi - to anyone else who wants to use open_bayer() before anything later than 0.18.2 is stable-released, this is a link to the Github page at the earliest (least risky) version which supports open_bayer().
https://github.com/LibRaw/LibRaw/tree/d4c4f5c8e4a2c5db56287f805d44b590ac...
This works perfectly for 16bit (EDIT: and now 8bit) RAW images without any header or metadata.
(wish I could delete comments but I can only edit them so I must look really silly with two comments in a row)
Alex fixed the problem with open_bayer() 8-bit images. Fix is documented here https://www.libraw.org/comment/4678#comment-4678 :)
I was serious that I do color, whitebalance, exposure, etc corrections to the raw data before putting it into LibRaw. I wrote the adjustments, and only use LibRaw to demosaic. No dcraw or LibRaw emulation of dcraw involved for color-correction / whitebalance / etc.
I wish it were the case that the resulting 8bit images without auto_bright were 'near black,' but take a histogram of this result and see for yourself:
https://drive.google.com/file/d/0B41mEzFmIzubTkhfSk9zMmtVVnc/view?usp=sh...
(Output with output_bps = 8, no_auto_bright = 1, no_auto_scale = 1, user_black = 0.) Every pixel is 0, and this was an overexposed 8bit raw. Do what you will with this information; I was just trying to repay a favor, thinking I might have found unintended behavior. I see now that LibRaw uses 16bit internally, even if the input buffer was only 8 BPP.
So, what seems to be happening...
1. 8 BPP data in the buffer has max value of 255
2. The buffer is read into LibRaw, which uses 16 BPP data internally
3. no_auto_bright = 1, so values are never scaled up to 65535
4. output_bps = 8, so values are all divided by 257 to scale down to 255 maxval... but at most, we had (255 / 257), which with integer rounding down equals 0. Anything less than 255 will of course also become 0.
5. Every pixel is 0 in the resulting image!
If that's correct (?), let this post just be a warning to any 8bit - LibRaw users who read this!
Thank you for the explanation, and we hear you. It is just that the word "accurate" is not the one I would use.
Hi Iliah,
I am using dcraw_emu (as I used dcraw 9.27 previously) as a simple tool to convert a raw file to a tiff. I'm trying to keep it very simple and use existing tools created by people such as yourself who are experts in many of these things. My program acts as a controller and uses these existing tools (such as dcraw_emu) to perform specific tasks. I'm not a C programmer and thus I don't have the ability to rebuild a custom version of dcraw_emu.
My need is simple: I just need dcraw_emu to create tiff files that are good quality (color-accurate) versions of the image that was captured in the raw file. The result should be quite similar to the original embedded JPEG and to the results produced by other raw converters. In almost all cases, dcraw_emu is doing that. The greens in my X-E2 sample pictures seem to be an exception.
As a further comparison, I converted my sample file with Lightroom and with Affinity Photo (I can provide samples if you wish). The result was:
* The original embedded JPEG and the tiff file produced by dcraw 9.27 were the brightest green (almost the same).
* The version produced by Lightroom was close to the embedded JPEG but the greens weren't quite as bright.
* The Affinity Photo version was quite a bit yellower than the previous versions. I don't think that it provided a very accurate version of the greens.
* The Dcraw_emu version came out similar to the Affinity Photo version (but lighter). Converting it with the brightness turned down further might produce a very similar version to Affinity Photo.
Going back to the dcraw 9.27 color profiles could be an unnecessary step backward. In my testing of the Sony ARW files, I found that the version produced by dcraw_emu has more accurate colors than the dcraw 9.27 version (which was much yellower). Thus, for my purposes, a small change to the color profile for the X-E2 (and the other X cameras if they are the same) should give me everything that I need from dcraw_emu.
To match JPEG colors you need not dcraw.c color profiles, but profiles used by camera vendors for in-camera processing (unavailable in most cases).
Also, the main problem is to repeat vendor's tone/contrast curve (which, in turn, changes when photographer change camera contrast/saturation/color mode) to match midtone brightness and highlights compression.
So, if your target is 'embedded JPEG match', than you (very likely) need own RAW postprocessing code instead of LibRaw::dcraw_process()
> I just need the colors to match up with the colors in the embedded JPEGs. So far that seems to be the case
Ummm.... "Match" means, in a very relaxed environment, "within maximum of 4..6 dE".
I'm sorry I do not understand why you can't use this solution? On a side note, why use colour processing from dcraw at all, why not to use quality colour transforms, since you develop anyway?
If that means that an upcoming version of dcraw_emu will use the dcraw color profile(s) that would be really good. Mainly, I just need the colors to match up with the colors in the embedded JPEGs. So far that seems to be the case, except for this one example.
dcraw.c has not changed for more than one year now (also, may-2016 change was minor, so real freeze is about 2 years).
Because of that, maintaining full dcraw.c compatibility is obsolete goal :)
Meanwhile, for cameras supported in (last) dcraw.c we (very likely) will change color profile to dcraw's even if we think our profile is better.
Since I'm using dcraw_emu, I can't use that solution. If the intent with dcraw_emu was to keep it consistent with dcraw, wouldn't it make sense to use that (revised) profile in dcraw_emu?
These results are expected (more or less).
auto brightness calculates image histogram, than adjust (increase) values to fit full 16-bit range (with 1% of pixels at saturation with default auto_bright_thr). So, if you use linear exposure correction (w/o highlights compression), than your changes will be ruined (indeed), because brightness is adjusted to use full data range.
Do not know what 'color adjustment' you use,because no such mechanics exists in dcraw_process() options (excluding white balance).
Without auto-bright, 8-bit images use only 0..255 range (to be correct,it may be more if white balance is used). This is 8 stop below full range (0..65535), so resulting image will look 'near black' (not completely black).
Just rebuilt raw.dll (only ended up having to edit dcraw_common.cpp, thanks for the tips), and openbayer_sample() works with the patch! Thank you so much for your fast help. This is awesome!
I have found that to get what I want from LibRaw, I need to set no_auto_bright = 0 for 8bit images, and no_auto_bright = 1 for 16bit images. This is no problem to me, but the behavior is a little bizarre. The rest of this comment is to help you figure out if this is an unintentional error within LibRaw.
.............................................................................................................................
I do some precise exposure compensation and color correction before using LibRaw. After some testing, I have found a bizarre result when I set imgdata.params.no_auto_bright:
__________________|___8bit image____|__16bit image___
no_auto_bright = 0 |___100% correct___|__Earlier exposure / color adjustments are ruined
no_auto_bright = 1 |_Completely black_|_100% correct___
(By the way, when I say "100% correct," I mean that my earlier exposure adjustments were conserved :) )
The results for 16bit images in the table above make perfect sense.
The results for 8bit images don't make any sense to me! I would expect results exactly like 16bit. Instead, in 8bit, no_auto_bright = 0 behaves like I would expect no_auto_bright = 1 to behave, and no_auto_bright = 1 makes the entire image all-black no matter what.
.............................................................................................................................
Hope that was helpful for moving on with Release 0.19!
The difference is in camera color profiles used in LibRaw and dcraw.
To get the same color as in dcraw, please go to adobe_coeff() function in (LibRaw)/internal/dcraw_common.cpp
and replace our color profile (two lines:
{ "Fujifilm X-E2", 0, 0,
{ 12066,-5927,-367,-1969,9878,1503,-721,2034,5453 } },
to dcraw.c profile:
{ "Fujifilm X-E2", 0, 0,
{ 8458,-2451,-855,-4597,12447,2407,-1475,2482,6526 } },
Than re-compile/rebuild libraw/dcraw_emu
Pages