So, if I understand correctly, dcraw's -D -W -g11 command maps DNs of 12 or 14bit raws as such to a 16bit space because -D does not scale(restructure) DNs and the 16bit space is universal for all the available raw files. On the other hand, -d -W -g 11 command scales DNs to be mapped(fit) to the 16bit space. Taking 'scale_colors' into account, result from -D-W-g11-b16 would be roughly similar with one from -d -W -g11-r1111 when decoding a 14bit with 0-16383DNs. Is this the story ? In that case, "a fixed white level" of -W command would mean simply no setting of white level, isnt it ?
dcraw does all processing until convert_to_rgb() in camera native color space and in 16-bit integers (no conversion to 0-1 float).
dcraw -d is completely diferent from dcraw -D because 'scale_colors()' call is used with -d. It scales input data range (12 or 14 bit, for example) to full 0..65535 scale (based on camera theoretical maximum, so underexposed images are not brightened more than needed).
dcraw -d -W will brighten output to get 1% of pixels in saturation.
Hi lexa, thanks again for your commitment !
" Irreversible " was employed to paraphrase " compressed with loss ". I believe that DN values of a compressed raw look like curved and truncated as if an inverse gamma of 0.45 for example was put in place. ( so you need a gamma to decode and linearize compressed values). Secondly, yes too, I wanted to say a floating point range of 0.0 to 1.0 with logically infinite intermediates. When you say that dcraw does not map DN values to this kind of 0-1 range, does dcraw map them directly to a working color space ? Thirdly, I meant dcraw's "-D -W -g 1 1" instead of "-D -W". Finally, the result of a 12bit or 14bit raw file decoded with "-D -W -g 1 1" is completely different from one with "-d -W -k 0 -S 4095 -g 1 1". "-D -W -g 1 1" makes image darker as if its "fixed white level" seems to be at around 65535. Why is this ?
Two notes:
1) Not sure what 'irreversible raw file' is. If you mean 'highlights compression' used in many lossy formats (Nikon, Sony, DNGs), than yes, first step is to make data from file back to linear space.
5) Mapping to logical 0-1 values: from developer (programmer) point of view, 'logical 0-1' means 'zeroes and ones only'. If I understand you correctly, in our (programmer) terms, you mean 0...1.0 'floating point' range with intermediate values.
Practically, this conversion is not mandatory unless 'possible data maximum' is correct. One may use, for example, 0...16383 range for 14 bit data.
dcraw -D -W
-D means 'document mode 2' so: unchanged raw values (linearized on your step 1) are contained in dcraw's image[] array after processing. Black level is not subtracted.
-W means 'no automatic brightening', so image histogram is not calculated and data are converted to 2.2 gamma and scaled to 8 bits (if no other keys are passed to dcraw)
Without -W, image histogram is calculated and output is scaled to have 1% of pixels in saturation.
I was absent for some days but now I have tested the temperature patch and can report that for my EOS 1100D the result is equal to what I extracted myself from the maker notes.
An additional information I can supply is that at least for the 1100D, but probably also for the other, more recent EOS models, this EXIF temperature is the true sensor temperature. For the 1100D I'm very sure about this and also for the 5D-MkII there is strong indication therefore.
OpenMP is a easy way to use multiple cores/cpus from program. It requires compiler and runtime support, also program code needs to be modified (#pragma omp ...)
DHT demosaic code already uses these pragmas, so if compiled witf -fopenmp (this is command line key for gcc compiler) it may run much faster on multicore/multicpu.
This patch supports Canon and Olympus cameras, more is on the way.
Camera/Sensor themperatures are in (added) imgdata.other.*Temperature fields (in Celsius)
These fields are inited to -1000 at open_file() start, so check values against >-1000 (or >= -273 :) before use.
Yes, RawSpeed is LGPL2 licensed, that means (at least, in common interpretation) that you need it as separate DLL (or LibRaw+RawSpeed as single DLL).
lossless_jpeg_load_raw() in LibRaw/dcraw is completely different from RawSpeeds. It handles all ljpeg formats at once, while RawSpeed has separate code (subclassing/using own common decoder) for different vendors. LibRaw+RawSpeed combo works perfect now (although we've blacklisted some cameras in LibRaw::unpack()), so why do backport?
'RawSpeed2' testing is in progress now (just managed to get it working under Windows), so I expect support in next snapshot/release.
Sensor temp: could you please provide your patch to backport into LibRaw (my E-mail: lexa@lexa.ru)
Looks like I would need to include RawSpeed as a DLL, right? Has anybody ever tried to replace the DCRaw CR2 decompressor (lossless_jpeg_load_raw(), I guess) by the one from RawSpeed?
Another thing: I noticed that I couldn't readout the Canon sensor temperature (which is important for long exposure dark subtraction) by use of set_exifparser_handler(), because the sensor temp appears in parse_makernote(), where the EXIF callback is not called. For the moment I solved this by inserting this callback in parse_makernote() in dcraw_common. May I suggest to either do so too in future versions, or to readout the sensor temperature to imgdata.makernotes.canon?
We've reworked imgdata.thumbnails.tcolors field
- correct handling for bitmap (based on SubIFD SamplesPerPixel field)
- if compiled with JPEG, jpeg header parsed for color count too (so for BW JPEG previews this field is correct).
I have now ported my code to work with LibRaw and most things worked fine instantly.
Up to now there have been only 2 issues I'd like to report:
1.) I had to
#define _WINSOCK2API_
before
#include "libraw\libraw.h"
in order to prevent winsock.h and winsock2.h being included both. (Looks like winsock.h is included somewhere else in one of my headers, but I couldn't locate where.)
2.) On calling convert_to_rgb() right after scale_colors() and pre_interpolate() an exception was thrown because the histogram buffer was not allocated.
I'm no going to build LibRaw_static.lib (with VS 2017). Thereby, I would like to include RawSpeed, since I often work with Canon CR2 Raws. Is there a recipe how to do this on Windows?
Agree, it would be better to initialize this field into correct value, at least if preview IFD contains SamplesPerPixel tag.
I need to collect more samples (at least color DNG + BW preview) to implement it in correct way (unpack_thumb assumes 3 colors for PPM bitmaps, this may be not correct and will result into read beyond EOF).
As I already wrote above:
Here is the link to annotated scale_colors(): http://ninedegreesbelow.com/files/dcraw-c-code-annotated-code.html#G2
OK, ignore my former post. I might not fully undestand what you said.
>dcraw does all processing until convert_to_rgb() in camera native color space and in 16-bit integers (no conversion to 0-1 float).
Do you mean that dcraw converts 12 or 14bit DNs to 16bit, with scaling in -d command & without scaling in -D command ?
Sorry, to much switch combinations to comment each variant separately (and, also, ensure the terms used have same meaning for both sides).
If you're interested in dcraw.c internals and each command line switch effect, the only reliable source is dcraw.c itself (annotated or not).
So, if I understand correctly, dcraw's -D -W -g11 command maps DNs of 12 or 14bit raws as such to a 16bit space because -D does not scale(restructure) DNs and the 16bit space is universal for all the available raw files. On the other hand, -d -W -g 11 command scales DNs to be mapped(fit) to the 16bit space. Taking 'scale_colors' into account, result from -D-W-g11-b16 would be roughly similar with one from -d -W -g11-r1111 when decoding a 14bit with 0-16383DNs. Is this the story ? In that case, "a fixed white level" of -W command would mean simply no setting of white level, isnt it ?
dcraw does all processing until convert_to_rgb() in camera native color space and in 16-bit integers (no conversion to 0-1 float).
dcraw -d is completely diferent from dcraw -D because 'scale_colors()' call is used with -d. It scales input data range (12 or 14 bit, for example) to full 0..65535 scale (based on camera theoretical maximum, so underexposed images are not brightened more than needed).
dcraw -d -W will brighten output to get 1% of pixels in saturation.
Here is 'dcraw annotated and outlined': http://ninedegreesbelow.com/files/dcraw-c-code-annotated-code.html
(and preface: http://ninedegreesbelow.com/photography/dcraw-c-code-annotated.html ), hope it will help you with dcraw internals.
Hi lexa, thanks again for your commitment !
" Irreversible " was employed to paraphrase " compressed with loss ". I believe that DN values of a compressed raw look like curved and truncated as if an inverse gamma of 0.45 for example was put in place. ( so you need a gamma to decode and linearize compressed values). Secondly, yes too, I wanted to say a floating point range of 0.0 to 1.0 with logically infinite intermediates. When you say that dcraw does not map DN values to this kind of 0-1 range, does dcraw map them directly to a working color space ? Thirdly, I meant dcraw's "-D -W -g 1 1" instead of "-D -W". Finally, the result of a 12bit or 14bit raw file decoded with "-D -W -g 1 1" is completely different from one with "-d -W -k 0 -S 4095 -g 1 1". "-D -W -g 1 1" makes image darker as if its "fixed white level" seems to be at around 65535. Why is this ?
Also: dcraw does not do '0-1' mapping. It does black subtraction (in camera data range) and works with subtracted values w/o any scaling.
Two notes:
1) Not sure what 'irreversible raw file' is. If you mean 'highlights compression' used in many lossy formats (Nikon, Sony, DNGs), than yes, first step is to make data from file back to linear space.
5) Mapping to logical 0-1 values: from developer (programmer) point of view, 'logical 0-1' means 'zeroes and ones only'. If I understand you correctly, in our (programmer) terms, you mean 0...1.0 'floating point' range with intermediate values.
Practically, this conversion is not mandatory unless 'possible data maximum' is correct. One may use, for example, 0...16383 range for 14 bit data.
dcraw -D -W
-D means 'document mode 2' so: unchanged raw values (linearized on your step 1) are contained in dcraw's image[] array after processing. Black level is not subtracted.
-W means 'no automatic brightening', so image histogram is not calculated and data are converted to 2.2 gamma and scaled to 8 bits (if no other keys are passed to dcraw)
Without -W, image histogram is calculated and output is scaled to have 1% of pixels in saturation.
I was absent for some days but now I have tested the temperature patch and can report that for my EOS 1100D the result is equal to what I extracted myself from the maker notes.
An additional information I can supply is that at least for the 1100D, but probably also for the other, more recent EOS models, this EXIF temperature is the true sensor temperature. For the 1100D I'm very sure about this and also for the 5D-MkII there is strong indication therefore.
Erwin
Thank you for the info.
OpenMP is a easy way to use multiple cores/cpus from program. It requires compiler and runtime support, also program code needs to be modified (#pragma omp ...)
DHT demosaic code already uses these pragmas, so if compiled witf -fopenmp (this is command line key for gcc compiler) it may run much faster on multicore/multicpu.
I do not know what OpenMP is. My project didn't use it :)
Thank you for quick fix!
Konrad
Thank you for sharing your analysis results.
Could you please note:
- was OpenMP used, or not?
- if yes, what is computer CPU/core count?
Here is the patch: https://www.dropbox.com/s/6ku6scc9ar33nr0/canon-oly-temperature.patch?dl=0
(against 0.19-20170812 snapshot, https://www.libraw.org/news/libraw-0-19-snapshot-20170812)
This patch supports Canon and Olympus cameras, more is on the way.
Camera/Sensor themperatures are in (added) imgdata.other.*Temperature fields (in Celsius)
These fields are inited to -1000 at open_file() start, so check values against >-1000 (or >= -273 :) before use.
RawSpeed2 support in next snapshot/release sounds good - I'll wait for it.
Many thanks, Erwin
Yes, RawSpeed is LGPL2 licensed, that means (at least, in common interpretation) that you need it as separate DLL (or LibRaw+RawSpeed as single DLL).
lossless_jpeg_load_raw() in LibRaw/dcraw is completely different from RawSpeeds. It handles all ljpeg formats at once, while RawSpeed has separate code (subclassing/using own common decoder) for different vendors. LibRaw+RawSpeed combo works perfect now (although we've blacklisted some cameras in LibRaw::unpack()), so why do backport?
'RawSpeed2' testing is in progress now (just managed to get it working under Windows), so I expect support in next snapshot/release.
Sensor temp: could you please provide your patch to backport into LibRaw (my E-mail: lexa@lexa.ru)
Looks like I would need to include RawSpeed as a DLL, right? Has anybody ever tried to replace the DCRaw CR2 decompressor (lossless_jpeg_load_raw(), I guess) by the one from RawSpeed?
Another thing: I noticed that I couldn't readout the Canon sensor temperature (which is important for long exposure dark subtraction) by use of set_exifparser_handler(), because the sensor temp appears in parse_makernote(), where the EXIF callback is not called. For the moment I solved this by inserting this callback in parse_makernote() in dcraw_common. May I suggest to either do so too in future versions, or to readout the sensor temperature to imgdata.makernotes.canon?
LibRaw 0.19-20170812 just published: https://www.libraw.org/news/libraw-0-19-snapshot-20170812
We've reworked imgdata.thumbnails.tcolors field
- correct handling for bitmap (based on SubIFD SamplesPerPixel field)
- if compiled with JPEG, jpeg header parsed for color count too (so for BW JPEG previews this field is correct).
2: yes, histogram is dynamically allocated (in dcraw_process()), to reduce size of LibRaw class sizeof()
RawSpeed:
- use 'master' (very old!) RawSpeed from https://github.com/klauspost/rawspeed/tree/master
(not the current one from darktable, current version does not compile with MSVC)
- Working recipe described in https://github.com/LibRaw/LibRaw/blob/master/README.RawSpeed.txt
I have now ported my code to work with LibRaw and most things worked fine instantly.
Up to now there have been only 2 issues I'd like to report:
1.) I had to
#define _WINSOCK2API_
before
#include "libraw\libraw.h"
in order to prevent winsock.h and winsock2.h being included both. (Looks like winsock.h is included somewhere else in one of my headers, but I couldn't locate where.)
2.) On calling convert_to_rgb() right after scale_colors() and pre_interpolate() an exception was thrown because the histogram buffer was not allocated.
I'm no going to build LibRaw_static.lib (with VS 2017). Thereby, I would like to include RawSpeed, since I often work with Canon CR2 Raws. Is there a recipe how to do this on Windows?
Erwin
Agree, it would be better to initialize this field into correct value, at least if preview IFD contains SamplesPerPixel tag.
I need to collect more samples (at least color DNG + BW preview) to implement it in correct way (unpack_thumb assumes 3 colors for PPM bitmaps, this may be not correct and will result into read beyond EOF).
Expect fix in 0.19-snapshot coming soon.
Currently, no plans to change present data layout.
You can re-use internal/var_defines.h with your code, so it will remain in agreement with new versions even if data fields will change.
Ok, "internal/var_defines.h" exactly is what I was looking for.
Can I presume that the meaning of the defined variables is unchanged?
Thanks a lot, Erwin
To be precise, this is data for last (largest) preview, which is incomplete too.
raw section is completely not present in file (because of truncation)
Thank you for the detailed answer.
Konrad
Pages