Also, the filter pattern reported appears to be different (REGB vs ERBG) but I guess this is expected because the visible dimensions are different too.
Alex,
Thanks for this. Once I applied this fix, the rgb_cam appears to be populated as you expected. The value is different than 0.19.5 but that appears to be expected based on the DNG fixes you made.
There does appear to be a bug with raw-identify though. When I dumped out information about this file, no information gets printed past "MakerNotes WB data". The executable appears to exit. Even the line "fprintf(outfile, "\nXYZ->CamRGB matrix:\n");" is not executed. The last few lines displayed are:
Alex,
Thanks for sharing those. The colors in the 0.20 version do appear muted compared to 0.19 in my opinion. The output_params settings I am using the camera white balance and not d65. The use_camera_matrix setting is the same as before.
I am getting very similar results to libraw when I attempt to generate an sRGB image using my custom pipeline and the metadata that libraw has returned.
I wanted to confirm that the new behaviour is expected and getting rgb_cam as an identity matrix is valid because it does not seem intuitive to me.
Alex,
So is the old DNG processing behaviour a bug which has been fixed in the new code? Because the colours in the rendered sRGB image with 0.20 appear duller than 0.19.5.
But, I am surprised why the rgb_cam is being returned as an identity matrix. There was a bunch of changes made to DNG Frame Selection. I want to ensure that I do not have to add any additional code to ensure the correct frame and the properties corresponding to it are selected.
Is there a way to fallback on 0.19.5 behaviour? I see there is ifdef'ed code to fallback to use auto_ptr vs unique_ptr. Is there similar for old DNG processing?
1) Many smartphone-produced DNG files has 'unusual' order of color data, the 1st one is daylight matrix, the 2nd one is for incandescent.
The old code (LibRaw 0.19/dcraw.c) always uses the last colordata (ColorMatrix) seen in the file. This works fine for Adobe-DNG-SDK-produced files (daylight colormatrix is the last one), but not with files where ColorMatrix1/ColorMatrix2 order is reversed.
Also, if several different sub-images are present in the DNG file with separate (own) colordata, the old code (again) uses the last one ColorMatrix specified
In the LibRaw 0.20 correct color matrix selection is implemented in both cases.
2) Visible size (or margins size) has changed in 0.20 for many cameras:
- for some cameras, vendor specified size is used instead of hardcoded one
- for all Bayer cameras margins are rounded to multiple of 2
- for all X-Trans cameras margins are rounded to multiple of 6
The last two changes is because of ambiguity in previous version: since LibRaw 0.20 bayer(/X-Trans) pattern is the same for both possible coordinate systems (entire area/visible area)
> So, you are suggesting I create CFLAGS_ZLIB ....
There are multiple ways to change variables in makefile. I think, it is much faster to try and see what happens, than wait for approval....
> On a related note. Where do I specify USE_X3FTOOLS and USE_6BY9RPI
Again: there are multiple ways.The easiest one is to modify COPT=... line
>Do, I need download the sources
No, both X3F_TOOLS and RPI-related code is already in place. It is just not compiled until user explicitly selects it (via specific defines, so /D.... statements in msvc case)
I noticed there were a bunch of changes made to DNG processing. I am not sure if there are any changes I need to be making to the source code for this.
So, you are suggesting I create CFLAGS_ZLIB and LDFLAGS_ZLIB and add them to COPT and LINKLIB lines?
On a related note. Where do I specify USE_X3FTOOLS and USE_6BY9RPI on the Makefile.msvc? Can I update the COPT line to say /DUSE_X3FTOOLS?
Also, I am a little confused on the imported code policy? Do, I need download the sources from the suitable github repo in addition to the libraw source OR is it sufficient to specify the macros enable them as the sources are already present?
It is not possible to add getter(s) for every field in LibRaw structure (while every field may be needed in some specific case).
We provide very limited C-API that solves most common need(s): open RAW file, render it, provide the rendering.
We do not have any plans to extend it much for every need that may occur.
C-API is simple enough, so any not very experienced programmer may extend it using provided C-API code as a sample. Although, there is no guarantee that we'll not reorder internals in future.
Alex, what do you think, makes it sense to provide a getter method in C API only for this field (libraw_data_t->image)?
I just checked the definition of that field and when I am correct, it is a pointer to a whole 4 element array of ushorts.
I guess that array will be resized internally by whatever call. What is the final size then?
Unlike unix systems (Linux, macos, etc), there is no standard path to zlib (headers, library), also there is no standard name for the library, it may be called zlib(.lib,.dll) zlib.lib+zlib1.dll, zlib_static.lib, etc.
So, I'm unable to provide specific and precise instructions, only general direction:
Easiest way to re-build libraw under Windows is to call
nmake -f Makefile.msvc
(for example, In Visual Studio command prompt)
So, modify Makefile.msvc to add /DUSE_ZLIB, path to zlib includes in compiler flags, flags to link with zlib to link flags
Alternatively, you may use Visual Studio solution/project provided and add the same to project properties via property settings.
Hi Alex,
I am having trouble adding ZLIB support on Windows. I believe the Windows build using NMAKE. Not very clear on how to specify the USE_ZLIB and the supporting include and lib paths to nmake on Windows.
That's because I did not directly define the libraw_data_t structure (a record in pascal) in the wrapper. I am simply using a general void* (a Pointer type in pascal) instead. Means any pascal program using this wrapper has access only to the pointer itself (to the libraw_data_t structure) but not the structure fields directly.
Reason is simple: All subsequent calls after libraw_init simply accepts the pointer (to structure) and they just do their job.
There was no reason yet to directly define the libraw_data_t record in the wrapper because no direct access to any field was required.
The file can be found below:
https://drive.google.com/drive/folders/1Bm-rfEpa6Ql-CtJFXJehtP51L9sjES9i...
Also, the filter pattern reported appears to be different (REGB vs ERBG) but I guess this is expected because the visible dimensions are different too.
Dinesh
Thanks,
Dinesh
Сould you please reformulate your question?
I cannot understand what exactly confuses you.
Removing the WB_Coeff printing results in the rest of the raw-identify code working.
Dinesh
raw-identify is already fixed by this patch: https://github.com/LibRaw/LibRaw/commit/1274768a3353ca408e60314133201cb9...
Alex,
Thanks for this. Once I applied this fix, the rgb_cam appears to be populated as you expected. The value is different than 0.19.5 but that appears to be expected based on the DNG fixes you made.
There does appear to be a bug with raw-identify though. When I dumped out information about this file, no information gets printed past "MakerNotes WB data". The executable appears to exit. Even the line "fprintf(outfile, "\nXYZ->CamRGB matrix:\n");" is not executed. The last few lines displayed are:
Filter pattern: RGGBRGGBRGGBRGGB
Highlight linearity limits: 3827 3827 3827 3827
Makernotes WB data: coeffs EVs
As shot 495 256 324 256 0.95 0.00 0.34 0.00
Happens even with NEF files. Appears to be an issue with printing out WB_Coeffs. I am on Windows.
Followup: yes, there is a bug here introduced by compiler warning elimination on Jul 02
Here is the fix: https://github.com/LibRaw/LibRaw/commit/349935f416b83069aefb286a5d8ff0f5...
The difference: in our code we always set use_camera_matrix to either 0 or 2 (do not use/forced), default value (it depends) is not used.
cmatrix is extracted
Could you please verify that the code snippet (memcpy from cmatrix to rgb_cam) is executed (or not)?
Alex,
I am not really seeing this behaviour on Windows. I am attaching three screenshots from VS debugger.
color struct after unpack()
+ cam_mul 0x000000fbf3763464 {2.08979583, 1.00000000, 1.78397214, 0.00000000} float[4] + pre_mul 0x000000fbf3763474 {2.35568857, 1.00068939, 1.55817533, 0.00000000} float[4] + cmatrix 0x000000fbf3763484 {0x000000fbf3763484 {1.60388803, -0.595357001, -0.00853104331, 0.00000000}, 0x000000fbf3763494 {...}, ...} float[3][4] + ccm 0x000000fbf37634b4 {0x000000fbf37634b4 {0.00000000, 0.00000000, 0.00000000, 0.00000000}, 0x000000fbf37634c4 {...}, ...} float[3][4] - rgb_cam 0x000000fbf37634e4 {0x000000fbf37634e4 {1.00000000, 0.00000000, 0.00000000, 0.00000000}, 0x000000fbf37634f4 {...}, ...} float[3][4] + [0] 0x000000fbf37634e4 {1.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] + [1] 0x000000fbf37634f4 {0.00000000, 1.00000000, 0.00000000, 0.00000000} float[4] + [2] 0x000000fbf3763504 {0.00000000, 0.00000000, 1.00000000, 0.00000000} float[4] - cam_xyz 0x000000fbf3763514 {0x000000fbf3763514 {0.00000000, 0.00000000, 0.00000000}, 0x000000fbf3763520 {0.00000000, ...}, ...} float[4][3] + [0] 0x000000fbf3763514 {0.00000000, 0.00000000, 0.00000000} float[3] + [1] 0x000000fbf3763520 {0.00000000, 0.00000000, 0.00000000} float[3] + [2] 0x000000fbf376352c {0.00000000, 0.00000000, 0.00000000} float[3] + [3] 0x000000fbf3763538 {0.00000000, 0.00000000, 0.00000000} float[3]params struct before dcraw_process
+ gamm 0x000000fbf373f288 {0.41666666666666669, 12.920000000000000, 0.0000000000000000, 0.0000000000000000, ...} double[6] + user_mul 0x000000fbf373f2b8 {0.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] shot_select 0 unsigned int bright 1.00000000 float threshold 0.00000000 float half_size 0 int four_color_rgb 0 int highlight 0 int use_auto_wb 0 int use_camera_wb 1 int use_camera_matrix 1 int output_color 1 int + output_profile 0x0000000000000000 <NULL> char * + camera_profile 0x0000000000000000 <NULL> char *color struct after dcraw_process()
+ cam_mul 0x000000fbf3763464 {2.08979583, 1.00000000, 1.78397214, 0.00000000} float[4] + pre_mul 0x000000fbf3763474 {2.08979583, 1.00000000, 1.78397214, 1.00000000} float[4] + cmatrix 0x000000fbf3763484 {0x000000fbf3763484 {1.60388803, -0.595357001, -0.00853104331, 0.00000000}, 0x000000fbf3763494 {...}, ...} float[3][4] - ccm 0x000000fbf37634b4 {0x000000fbf37634b4 {0.00000000, 0.00000000, 0.00000000, 0.00000000}, 0x000000fbf37634c4 {...}, ...} float[3][4] + [0] 0x000000fbf37634b4 {0.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] + [1] 0x000000fbf37634c4 {0.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] + [2] 0x000000fbf37634d4 {0.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] - rgb_cam 0x000000fbf37634e4 {0x000000fbf37634e4 {1.00000000, 0.00000000, 0.00000000, 0.00000000}, 0x000000fbf37634f4 {...}, ...} float[3][4] + [0] 0x000000fbf37634e4 {1.00000000, 0.00000000, 0.00000000, 0.00000000} float[4] + [1] 0x000000fbf37634f4 {0.00000000, 1.00000000, 0.00000000, 0.00000000} float[4] + [2] 0x000000fbf3763504 {0.00000000, 0.00000000, 1.00000000, 0.00000000} float[4] - cam_xyz 0x000000fbf3763514 {0x000000fbf3763514 {0.00000000, 0.00000000, 0.00000000}, 0x000000fbf3763520 {0.00000000, ...}, ...} float[4][3] + [0] 0x000000fbf3763514 {0.00000000, 0.00000000, 0.00000000} float[3] + [1] 0x000000fbf3763520 {0.00000000, 0.00000000, 0.00000000} float[3] + [2] 0x000000fbf376352c {0.00000000, 0.00000000, 0.00000000} float[3] + [3] 0x000000fbf3763538 {0.00000000, 0.00000000, 0.00000000} float[3]My workflow is:
open_file()
unpack()
// Set params
dcraw_process()
dcraw_make_mem_image()
Is this something to do with how the library is compiled?
Regards,
Dinesh
This code is (indeed) executed and rgb_cam becomes:
- rgb_cam 0x00000128c200b044 {0x00000128c200b044 {1.60388803, -0.595357001, -0.00853104331, 0.00000000}, 0x00000128c200b054 {...}, ...} float[3][4] + [0] 0x00000128c200b044 {1.60388803, -0.595357001, -0.00853104331, 0.00000000} float[4] + [1] 0x00000128c200b054 {-0.145714581, 1.32941270, -0.183698103, 0.00000000} float[4] + [2] 0x00000128c200b064 {0.0402970426, -0.722867191, 1.68257010, 0.00000000} float[4]It is not identity if params are set correctly, this code piece propagates cmatrix to rgb_cam:
if ((use_camera_matrix & (((use_camera_wb || dng_version)?0:1) | 0x2)) && cmatrix[0][0] > 0.125) { memcpy(rgb_cam, cmatrix, sizeof cmatrix); ...Alex,
Thanks for sharing those. The colors in the 0.20 version do appear muted compared to 0.19 in my opinion. The output_params settings I am using the camera white balance and not d65. The use_camera_matrix setting is the same as before.
I am getting very similar results to libraw when I attempt to generate an sRGB image using my custom pipeline and the metadata that libraw has returned.
I wanted to confirm that the new behaviour is expected and getting rgb_cam as an identity matrix is valid because it does not seem intuitive to me.
Regards
Dinesh
I do not see that processing result is dull
Here is the file processed with LibRaw 0.19.5 (dcraw_emu -w -T): https://www.dropbox.com/s/efh2f1rkukunuif/screenshot%202020-10-13%2016.2...
Here is LibRaw 0.20 processing (with daylight WB): https://www.dropbox.com/s/6x7l71j1e9f90s1/screenshot%202020-10-13%2016.2...
Please note that 'camera matrix' is used for DNG files if use_camera_matrix is either default (1) or greater.
Alex,
So is the old DNG processing behaviour a bug which has been fixed in the new code? Because the colours in the rendered sRGB image with 0.20 appear duller than 0.19.5.
But, I am surprised why the rgb_cam is being returned as an identity matrix. There was a bunch of changes made to DNG Frame Selection. I want to ensure that I do not have to add any additional code to ensure the correct frame and the properties corresponding to it are selected.
Is there a way to fallback on 0.19.5 behaviour? I see there is ifdef'ed code to fallback to use auto_ptr vs unique_ptr. Is there similar for old DNG processing?
Regards,
Dinesh
I will give this a shot and get back if I have any trouble.
Dinesh
1) Many smartphone-produced DNG files has 'unusual' order of color data, the 1st one is daylight matrix, the 2nd one is for incandescent.
The old code (LibRaw 0.19/dcraw.c) always uses the last colordata (ColorMatrix) seen in the file. This works fine for Adobe-DNG-SDK-produced files (daylight colormatrix is the last one), but not with files where ColorMatrix1/ColorMatrix2 order is reversed.
Also, if several different sub-images are present in the DNG file with separate (own) colordata, the old code (again) uses the last one ColorMatrix specified
In the LibRaw 0.20 correct color matrix selection is implemented in both cases.
2) Visible size (or margins size) has changed in 0.20 for many cameras:
- for some cameras, vendor specified size is used instead of hardcoded one
- for all Bayer cameras margins are rounded to multiple of 2
- for all X-Trans cameras margins are rounded to multiple of 6
The last two changes is because of ambiguity in previous version: since LibRaw 0.20 bayer(/X-Trans) pattern is the same for both possible coordinate systems (entire area/visible area)
> So, you are suggesting I create CFLAGS_ZLIB ....
There are multiple ways to change variables in makefile. I think, it is much faster to try and see what happens, than wait for approval....
> On a related note. Where do I specify USE_X3FTOOLS and USE_6BY9RPI
Again: there are multiple ways.The easiest one is to modify COPT=... line
>Do, I need download the sources
No, both X3F_TOOLS and RPI-related code is already in place. It is just not compiled until user explicitly selects it (via specific defines, so /D.... statements in msvc case)
I noticed there were a bunch of changes made to DNG processing. I am not sure if there are any changes I need to be making to the source code for this.
So, you are suggesting I create CFLAGS_ZLIB and LDFLAGS_ZLIB and add them to COPT and LINKLIB lines?
On a related note. Where do I specify USE_X3FTOOLS and USE_6BY9RPI on the Makefile.msvc? Can I update the COPT line to say /DUSE_X3FTOOLS?
Also, I am a little confused on the imported code policy? Do, I need download the sources from the suitable github repo in addition to the libraw source OR is it sufficient to specify the macros enable them as the sources are already present?
Regards,
Dinesh
(and second answer)
imgdata.image array is allocated either in raw2image() call or in unpack() call.
array size is no less than imgata.sizes.iwidth * imgdata.sizes.iheight
It is not possible to add getter(s) for every field in LibRaw structure (while every field may be needed in some specific case).
We provide very limited C-API that solves most common need(s): open RAW file, render it, provide the rendering.
We do not have any plans to extend it much for every need that may occur.
C-API is simple enough, so any not very experienced programmer may extend it using provided C-API code as a sample. Although, there is no guarantee that we'll not reorder internals in future.
Alex, what do you think, makes it sense to provide a getter method in C API only for this field (libraw_data_t->image)?
I just checked the definition of that field and when I am correct, it is a pointer to a whole 4 element array of ushorts.
I guess that array will be resized internally by whatever call. What is the final size then?
Unlike unix systems (Linux, macos, etc), there is no standard path to zlib (headers, library), also there is no standard name for the library, it may be called zlib(.lib,.dll) zlib.lib+zlib1.dll, zlib_static.lib, etc.
So, I'm unable to provide specific and precise instructions, only general direction:
Easiest way to re-build libraw under Windows is to call
nmake -f Makefile.msvc
(for example, In Visual Studio command prompt)
So, modify Makefile.msvc to add /DUSE_ZLIB, path to zlib includes in compiler flags, flags to link with zlib to link flags
Alternatively, you may use Visual Studio solution/project provided and add the same to project properties via property settings.
Hi Alex,
I am having trouble adding ZLIB support on Windows. I believe the Windows build using NMAKE. Not very clear on how to specify the USE_ZLIB and the supporting include and lib paths to nmake on Windows.
Regards
Dinesh
It looks like Han needs this access for his code.
That's because I did not directly define the libraw_data_t structure (a record in pascal) in the wrapper. I am simply using a general void* (a Pointer type in pascal) instead. Means any pascal program using this wrapper has access only to the pointer itself (to the libraw_data_t structure) but not the structure fields directly.
Reason is simple: All subsequent calls after libraw_init simply accepts the pointer (to structure) and they just do their job.
There was no reason yet to directly define the libraw_data_t record in the wrapper because no direct access to any field was required.
Pages