White level (data maximum) is not as easy as it should be.
LibRaw provides (estimated) data maximum in imgdata.color.maximum. This value is
- either hardcoded (this is derived from dcraw.c)
- or last item of linearization curve
- or determined using camera bit count
For many formats/vendors/settings it is correct, for others it is overestimated (for example, Panasonic low ISO w/ full-well limited sensor, also Canon's intermediate ISOs)
Also, there is imgdata.color.linear_max[]. If filled (nonzero) this value(s) represents vendor suggested white point parsed from metadata. This value is usually way too low (there are a lot of pixels above this threshold), but it is (yes) suggested by vendor.
Also, there are vendor-specific values in parsed metadata fields (e.g. canon.NormalWhiteLevel/canon.SpecularWhiteLevel)
If you're ready to subclass LibRaw to own class to access protected fields (libraw_internal_data.unpacker_data) , use this method:
position to libraw_internal_data.unpacker_data.data_offset and read header (16 bytes), data version in 3rd byte of header (header[2]), it is 1 for lossless compression (supported) and 0 for unsupported lossy.
Alex,
Apologies for jumping the gun and reporting the issue in 0.20.0 as a bug. I just saw that my tests had failed and assumed a regression. But I agree that 0.20.0 behaviour is better.
I will upgrade to 0.20.2 for the updated raw-identify behaviour.
Alex,
This file has four distinct sensors. However, when the multiplier for E is marked as 0, this would indicate that the E component is being discarded. Same with the rgb_cam having 0's for the E component.
I noticed that the cam_xyz is the same as the F828 entry in the adobe_coeff table is the same. This indicates there is a change in how the rgb_cam and hence the Daylight multipliers are derived in the code.
That being said, the 0.19.5 version has a very strong yellow cast on the image but the image definitely looks better in 0.20.0. Please find attached screenshots in the same drive location as above. I assume the 0.20.0 is the expected behaviour.
Also, the filter pattern reported appears to be different (REGB vs ERBG) but I guess this is expected because the visible dimensions are different too.
Alex,
Thanks for this. Once I applied this fix, the rgb_cam appears to be populated as you expected. The value is different than 0.19.5 but that appears to be expected based on the DNG fixes you made.
There does appear to be a bug with raw-identify though. When I dumped out information about this file, no information gets printed past "MakerNotes WB data". The executable appears to exit. Even the line "fprintf(outfile, "\nXYZ->CamRGB matrix:\n");" is not executed. The last few lines displayed are:
Alex,
Thanks for sharing those. The colors in the 0.20 version do appear muted compared to 0.19 in my opinion. The output_params settings I am using the camera white balance and not d65. The use_camera_matrix setting is the same as before.
I am getting very similar results to libraw when I attempt to generate an sRGB image using my custom pipeline and the metadata that libraw has returned.
I wanted to confirm that the new behaviour is expected and getting rgb_cam as an identity matrix is valid because it does not seem intuitive to me.
RawDigger works with Wine.
Hello,
Seems it's supported since last release, I could not try since I work in linux heheh
Changelog
RawDigger 1.4.2 (2020-09-24)
Camera support:
Canon EOS R5, R6
Please check if RawDigger https://www.rawdigger.com/download supports the files. If it doesn't, please e-mail us - info@libraw.org
Can I help somehow? providing samples or similar? let me know thanks
EOS R5 is not officially supported by LibRaw 0.20, here is supported camera list: https://www.libraw.org/supported-cameras
OK, thank you.
There is no 'Document mode' in dcraw_emu
unprocessed_raw and/or 4channels sample(s) may solve your task.
White level (data maximum) is not as easy as it should be.
LibRaw provides (estimated) data maximum in imgdata.color.maximum. This value is
- either hardcoded (this is derived from dcraw.c)
- or last item of linearization curve
- or determined using camera bit count
For many formats/vendors/settings it is correct, for others it is overestimated (for example, Panasonic low ISO w/ full-well limited sensor, also Canon's intermediate ISOs)
Also, there is imgdata.color.linear_max[]. If filled (nonzero) this value(s) represents vendor suggested white point parsed from metadata. This value is usually way too low (there are a lot of pixels above this threshold), but it is (yes) suggested by vendor.
Also, there are vendor-specific values in parsed metadata fields (e.g. canon.NormalWhiteLevel/canon.SpecularWhiteLevel)
There is no direct/accurate way in LibRaw 0.20
If you're ready to subclass LibRaw to own class to access protected fields (libraw_internal_data.unpacker_data) , use this method:
position to libraw_internal_data.unpacker_data.data_offset and read header (16 bytes), data version in 3rd byte of header (header[2]), it is 1 for lossless compression (supported) and 0 for unsupported lossy.
Alex,
Apologies for jumping the gun and reporting the issue in 0.20.0 as a bug. I just saw that my tests had failed and assumed a regression. But I agree that 0.20.0 behaviour is better.
I will upgrade to 0.20.2 for the updated raw-identify behaviour.
Regards,
Dinesh
BTW, I strongly suggest to update to 0.20.2. raw-identify in 0.20.0 may display rgb_cam incorrect.
I also think the new version is better.
That is why I do not understand why it was called incorrect.
Alex,
This file has four distinct sensors. However, when the multiplier for E is marked as 0, this would indicate that the E component is being discarded. Same with the rgb_cam having 0's for the E component.
I noticed that the cam_xyz is the same as the F828 entry in the adobe_coeff table is the same. This indicates there is a change in how the rgb_cam and hence the Daylight multipliers are derived in the code.
That being said, the 0.19.5 version has a very strong yellow cast on the image but the image definitely looks better in 0.20.0. Please find attached screenshots in the same drive location as above. I assume the 0.20.0 is the expected behaviour.
Regards,
Dinesh
Your message subject: libraw 0.20.0 returns incorrect D65 multipliers and rgb_cam for ERBG file
Could you please clarify why you call 0.20.x data 'incorrect'?
What values do you think are correct and why?
The file can be found below:
https://drive.google.com/drive/folders/1Bm-rfEpa6Ql-CtJFXJehtP51L9sjES9i...
Also, the filter pattern reported appears to be different (REGB vs ERBG) but I guess this is expected because the visible dimensions are different too.
Dinesh
Thanks,
Dinesh
Сould you please reformulate your question?
I cannot understand what exactly confuses you.
Removing the WB_Coeff printing results in the rest of the raw-identify code working.
Dinesh
raw-identify is already fixed by this patch: https://github.com/LibRaw/LibRaw/commit/1274768a3353ca408e60314133201cb9...
Alex,
Thanks for this. Once I applied this fix, the rgb_cam appears to be populated as you expected. The value is different than 0.19.5 but that appears to be expected based on the DNG fixes you made.
There does appear to be a bug with raw-identify though. When I dumped out information about this file, no information gets printed past "MakerNotes WB data". The executable appears to exit. Even the line "fprintf(outfile, "\nXYZ->CamRGB matrix:\n");" is not executed. The last few lines displayed are:
Filter pattern: RGGBRGGBRGGBRGGB
Highlight linearity limits: 3827 3827 3827 3827
Makernotes WB data: coeffs EVs
As shot 495 256 324 256 0.95 0.00 0.34 0.00
Happens even with NEF files. Appears to be an issue with printing out WB_Coeffs. I am on Windows.
Followup: yes, there is a bug here introduced by compiler warning elimination on Jul 02
Here is the fix: https://github.com/LibRaw/LibRaw/commit/349935f416b83069aefb286a5d8ff0f5...
The difference: in our code we always set use_camera_matrix to either 0 or 2 (do not use/forced), default value (it depends) is not used.
cmatrix is extracted
Could you please verify that the code snippet (memcpy from cmatrix to rgb_cam) is executed (or not)?
Alex,
I am not really seeing this behaviour on Windows. I am attaching three screenshots from VS debugger.
color struct after unpack()
+ cam_mul 0x000000fbf3763464 {2.08979583, 1.00000000, 1.78397214, 0.00000000} float[4] + pre_mul 0x000000fbf3763474 {2.35568857, 1.00068939, 1.55817533, 0.00000000} float[4] + cmatrix 0x000000fbf3763484 {0x000000fbf3763484 {1.60388803, -0.595357001, -0.00853104331, 0.00000000}, 0x000000fbf3763494 {...}, ...} float[3][4] + ccm 0x000000fbf37634b4 {0x000000fbf37634b4 {0.00000000, 0.00000000, 0.00000000, 0.00000000}, 0x000000fbf37634c4 {...}, ...} float[3][4] - rgb_cam 0x000000fbf37634e4 {0x000000fbf37634e4 {1.00000000, 0.00000000, 0.00000000, 0.00000000}, 0x000000fbf37634f4 {...}, ...} float[3][4] + [0] 0x000000fbf37634e4 {1.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] + [1] 0x000000fbf37634f4 {0.00000000, 1.00000000, 0.00000000, 0.00000000} float[4] + [2] 0x000000fbf3763504 {0.00000000, 0.00000000, 1.00000000, 0.00000000} float[4] - cam_xyz 0x000000fbf3763514 {0x000000fbf3763514 {0.00000000, 0.00000000, 0.00000000}, 0x000000fbf3763520 {0.00000000, ...}, ...} float[4][3] + [0] 0x000000fbf3763514 {0.00000000, 0.00000000, 0.00000000} float[3] + [1] 0x000000fbf3763520 {0.00000000, 0.00000000, 0.00000000} float[3] + [2] 0x000000fbf376352c {0.00000000, 0.00000000, 0.00000000} float[3] + [3] 0x000000fbf3763538 {0.00000000, 0.00000000, 0.00000000} float[3]params struct before dcraw_process
+ gamm 0x000000fbf373f288 {0.41666666666666669, 12.920000000000000, 0.0000000000000000, 0.0000000000000000, ...} double[6] + user_mul 0x000000fbf373f2b8 {0.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] shot_select 0 unsigned int bright 1.00000000 float threshold 0.00000000 float half_size 0 int four_color_rgb 0 int highlight 0 int use_auto_wb 0 int use_camera_wb 1 int use_camera_matrix 1 int output_color 1 int + output_profile 0x0000000000000000 <NULL> char * + camera_profile 0x0000000000000000 <NULL> char *color struct after dcraw_process()
+ cam_mul 0x000000fbf3763464 {2.08979583, 1.00000000, 1.78397214, 0.00000000} float[4] + pre_mul 0x000000fbf3763474 {2.08979583, 1.00000000, 1.78397214, 1.00000000} float[4] + cmatrix 0x000000fbf3763484 {0x000000fbf3763484 {1.60388803, -0.595357001, -0.00853104331, 0.00000000}, 0x000000fbf3763494 {...}, ...} float[3][4] - ccm 0x000000fbf37634b4 {0x000000fbf37634b4 {0.00000000, 0.00000000, 0.00000000, 0.00000000}, 0x000000fbf37634c4 {...}, ...} float[3][4] + [0] 0x000000fbf37634b4 {0.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] + [1] 0x000000fbf37634c4 {0.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] + [2] 0x000000fbf37634d4 {0.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] - rgb_cam 0x000000fbf37634e4 {0x000000fbf37634e4 {1.00000000, 0.00000000, 0.00000000, 0.00000000}, 0x000000fbf37634f4 {...}, ...} float[3][4] + [0] 0x000000fbf37634e4 {1.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] + [1] 0x000000fbf37634f4 {0.00000000, 1.00000000, 0.00000000, 0.00000000} float[4] + [2] 0x000000fbf3763504 {0.00000000, 0.00000000, 1.00000000, 0.00000000} float[4] - cam_xyz 0x000000fbf3763514 {0x000000fbf3763514 {0.00000000, 0.00000000, 0.00000000}, 0x000000fbf3763520 {0.00000000, ...}, ...} float[4][3] + [0] 0x000000fbf3763514 {0.00000000, 0.00000000, 0.00000000} float[3] + [1] 0x000000fbf3763520 {0.00000000, 0.00000000, 0.00000000} float[3] + [2] 0x000000fbf376352c {0.00000000, 0.00000000, 0.00000000} float[3] + [3] 0x000000fbf3763538 {0.00000000, 0.00000000, 0.00000000} float[3]My workflow is:
open_file()
unpack()
// Set params
dcraw_process()
dcraw_make_mem_image()
Is this something to do with how the library is compiled?
Regards,
Dinesh
This code is (indeed) executed and rgb_cam becomes:
- rgb_cam 0x00000128c200b044 {0x00000128c200b044 {1.60388803, -0.595357001, -0.00853104331, 0.00000000}, 0x00000128c200b054 {...}, ...} float[3][4] + [0] 0x00000128c200b044 {1.60388803, -0.595357001, -0.00853104331, 0.00000000} float[4] + [1] 0x00000128c200b054 {-0.145714581, 1.32941270, -0.183698103, 0.00000000} float[4] + [2] 0x00000128c200b064 {0.0402970426, -0.722867191, 1.68257010, 0.00000000} float[4]It is not identity if params are set correctly, this code piece propagates cmatrix to rgb_cam:
if ((use_camera_matrix & (((use_camera_wb || dng_version)?0:1) | 0x2)) && cmatrix[0][0] > 0.125) { memcpy(rgb_cam, cmatrix, sizeof cmatrix); ...Alex,
Thanks for sharing those. The colors in the 0.20 version do appear muted compared to 0.19 in my opinion. The output_params settings I am using the camera white balance and not d65. The use_camera_matrix setting is the same as before.
I am getting very similar results to libraw when I attempt to generate an sRGB image using my custom pipeline and the metadata that libraw has returned.
I wanted to confirm that the new behaviour is expected and getting rgb_cam as an identity matrix is valid because it does not seem intuitive to me.
Regards
Dinesh
Pages