Speaking of photographyblog samples and crazy histograms, I've tested my libraw code on samples from other cameras and I haven't seen anything crazier than a Sony RX100 V's histogram. It's coded on 14-bits (actually perhaps more since it says the maximum is 17220), but at the lower end it only needs 12-bits, and it goes down in steps as you go up in values! So there's a 11-bit section, then 10-bit, then 9-bit in the highlights! It probably helps with the compression? Anyway I thought you'd be interested to know about this in case you didn't already.
Auto-bright is performed on dcraw_make_mem_image() stage and changes the range of input data that mapped into output (gamma-corrected) image, based on image histogram (in linear space).
So, yes, it is done in linear space, but on output.
So for every 4 values in the histogram you have one with green (not sure which but it's one of the two), the next one has both blue and the other green, the next one has red only, and finally there's one with no values at all. This would make sense if they were all gathered in the same bins but instead they're spread differently, like they're being digitised differently maybe?
It might be worth noting that the G7 X Mark I (which preceded the related G9 X) only had 12-bit values, so maybe the bump to 14-bit was done with cutting a corner somewhere.
I agree, it seems that there are pretty consistent biases, so I have the answer to my original question, thank you all!
One thing I'm wondering though is why is there a difference across channels? Because electronically, they should be the same, just with different filters on top, right? But then again comparing histograms there are clearly differences in which values which channel have lots of or none of. I guess I'll do some tests to figure this out, comparing different regions.
A side note: in LibRaw we read Canon's metadata black level too, have a look at libraw_types.h, libraw_canon_makernotes_t -- it is in ChannelBlackLevel[4]. You can add something like
if (Canon.ChannelBlackLevel[0])
printf("ChannelBlackLevel (from makernotes): %d %d %d %d\n",
Canon.ChannelBlackLevel[0],
Canon.ChannelBlackLevel[1],
Canon.ChannelBlackLevel[2],
Canon.ChannelBlackLevel[3]);
to raw-identify and it will print those levels if called with '-v'.
I had a look at full raw picture histograms with daylight pictures at ISO 125 or even darker shots at ISO 6400 and we've got the same pattern around 2048.
In fact I just looked at a much older daylight ISO 125 shot taken with my old G9 X (I currently have both a G9 X and a G9 X II) and it's no better: http://i.imgur.com/sfMDaKr.png < the histogram is perfectly antialiased, it's no visualisation artifact, that's exactly how it is, you've got the strict 12-bit comb pattern that gradually fades as you go up in values.
As for any noise reduction I just looked and I do have "High ISO speed NR" set to "Standard", but I don't think it (and sure hope it doesn't) affects the RAW. According to this it doesn't http://www.learn.usa.canon.com/resources/articles/2011/high_iso_noise_re.... But again I get the same histogram pattern with ISO 125 shots.
As for the noise level yep there's a lot of it but I stack bursts (usually at least 20 frames, though I'm thinking of going higher) which reduces noise quite a bit, so at that point getting the offset right can make a difference I suppose, mostly if like I said added gain and WB end up boosting that offset quite a bit. But yeah I suppose a bit of extra quantisation noise from this histogram quirk doesn't do much.
Wow you're right, but look at the histogram for the whole picture, it's very strange, around 2048 you have values that never occur which gives these huge peaks and troughs but the rest of the histogram is much more normal or at least has a much tamer version of the problem. I knew that ADCs could have such quirks but it still looks pretty odd.
Hi,
I print out the code values after "libraw" processing. They are all within the range of "0 - 65535".
The settings that I use before "dcraw_make_mem_image (...)" are:
"
output_color = 5;
output_bps = 16;
highlight = 0;
use_camera_matrix = 0;
gamm[0] = 1.0;
gamm[1] = 1.0;
auto_bright_thr = 0;
no_auto_bright = 1;
"
I tried to upload the RAW file, but the file size is big and the format cannot be recognized.
Thanks a lot!
Data range is 0...65535 for 16-bit output, what is '1.0' mean?
Also, could you please upload some clipped file to test with?
Hi,
I actually set "no_auto_bright" to 1, it actually clipped the white to 1.0.
Thanks a lot!
Try auto_bright_thr set to 0, or no_auto_bright set to 1
Yes, Sony ARW 2.3 uses 11-bit data (lossy compressed and highlights compressed): https://www.rawdigger.com/howtouse/sony-craw-arw2-posterization-detection (see 'inside Sony cRAW format' chapter)
Speaking of photographyblog samples and crazy histograms, I've tested my libraw code on samples from other cameras and I haven't seen anything crazier than a Sony RX100 V's histogram. It's coded on 14-bits (actually perhaps more since it says the maximum is 17220), but at the lower end it only needs 12-bits, and it goes down in steps as you go up in values! So there's a 11-bit section, then 10-bit, then 9-bit in the highlights! It probably helps with the compression? Anyway I thought you'd be interested to know about this in case you didn't already.
http://i.imgur.com/4GdKyxk.png
Auto-bright is performed on dcraw_make_mem_image() stage and changes the range of input data that mapped into output (gamma-corrected) image, based on image histogram (in linear space).
So, yes, it is done in linear space, but on output.
Okay I did some tests with a black frame by doing averages on various non-overlapping windows and the biases across channels are consistent.
Here's an histogram of a black frame at ISO 125, 1/2000 sec, it's also pretty weird: http://i.imgur.com/Q7kgaLU.png
line 6479, yep I see it, thanks!
Indeed, and look at what happens when you superimpose all 4: http://i.imgur.com/EovPwIF.png
So for every 4 values in the histogram you have one with green (not sure which but it's one of the two), the next one has both blue and the other green, the next one has red only, and finally there's one with no values at all. This would make sense if they were all gathered in the same bins but instead they're spread differently, like they're being digitised differently maybe?
It might be worth noting that the G7 X Mark I (which preceded the related G9 X) only had 12-bit values, so maybe the bump to 14-bit was done with cutting a corner somewhere.
If you are working with archive, edit dcraw_common.cpp in /internal, after the line 5479
I'm looking at the source for LibRaw 0.18.2 and in dcraw/dcraw.c I don't see this at all at line 7662. Lines 7662 and 7663 read this:
{ "Olympus E-PL7", 0, 0, { 9197,-3190,-659,-2606,10830,2039,-458,1250,5458 } },In LibRaw/dcraw.c after the line 7662 you have:
if ((unique_id == 0x03970000) || // G7 X Mark II
(unique_id == 0x80000394)) // EOS M5
make it:
if ((unique_id == 0x03970000) || // G7 X Mark II
(unique_id == 0x04100000) || // G9 X Mark II
(unique_id == 0x80000394)) // EOS M5
Here is histogam of visible area: https://www.dropbox.com/s/09p4dcys46ldx86/Screenshot%202017-03-26%2017.3...
Black subtraction is turned off, log vertical scale.
Entire histogram looks very strange.
I agree, it seems that there are pretty consistent biases, so I have the answer to my original question, thank you all!
One thing I'm wondering though is why is there a difference across channels? Because electronically, they should be the same, just with different filters on top, right? But then again comparing histograms there are clearly differences in which values which channel have lots of or none of. I guess I'll do some tests to figure this out, comparing different regions.
I've got 0.18.2, I literally just got started with LibRaw a few days ago after looking at the dcraw code made me despair.
I'm tempted to think it might be something more upstream, something with the ADC or a sort of calibration curve applied at some point around the ADC.
Maybe your LibRaw version is not current?
I tried these (on my dark frame shot) and I get 10000 0 0 389. and AverageBlackLevel says 2597, these values look strange.
Btw I was gonna printf everything but turns out it's much quicker to use your debugger to have a quick look at everything ;)
Yes, histogram of your _0298 shot looks strange, I suspect some kind of raw data processing applied in camera.
Thank you for the dark frame.
I've averaged central part of the image using RawDigger: https://www.dropbox.com/s/ciftfhp8zgfkxfp/Screenshot%202017-03-26%2008.4...
Black subtraction is off, averaged values are 2044.5, 2045.6, 2047.7, 2046.2 (rounded in red).
Entire image is purple because black subtraction is turned off.
Next screenshot shows black values determined by LibRaw using black frame integration: https://www.dropbox.com/s/fcxybepvflf0ec9/Screenshot%202017-03-26%2008.5...
(bottom left corner): 2044, 2045, 2047, 2046
Looks like masked pixels averaging is right (may be 1 level off for G2), it is in very good agreement with black frame ('real black') data.
I do not see any reason to switch from masked pixels averaging to metadata value.
Thank you for the samples, most interesting.
A side note: in LibRaw we read Canon's metadata black level too, have a look at libraw_types.h, libraw_canon_makernotes_t -- it is in ChannelBlackLevel[4]. You can add something like
if (Canon.ChannelBlackLevel[0])
printf("ChannelBlackLevel (from makernotes): %d %d %d %d\n",
Canon.ChannelBlackLevel[0],
Canon.ChannelBlackLevel[1],
Canon.ChannelBlackLevel[2],
Canon.ChannelBlackLevel[3]);
to raw-identify and it will print those levels if called with '-v'.
I had a look at full raw picture histograms with daylight pictures at ISO 125 or even darker shots at ISO 6400 and we've got the same pattern around 2048.
In fact I just looked at a much older daylight ISO 125 shot taken with my old G9 X (I currently have both a G9 X and a G9 X II) and it's no better: http://i.imgur.com/sfMDaKr.png < the histogram is perfectly antialiased, it's no visualisation artifact, that's exactly how it is, you've got the strict 12-bit comb pattern that gradually fades as you go up in values.
Here's the dark frame http://www.mediafire.com/file/1823atsyamr1xx5/IMG_1554.CR2 , I'll spare you any suspense, here's the histogram: http://i.imgur.com/pSVFFLK.png
As for any noise reduction I just looked and I do have "High ISO speed NR" set to "Standard", but I don't think it (and sure hope it doesn't) affects the RAW. According to this it doesn't http://www.learn.usa.canon.com/resources/articles/2011/high_iso_noise_re.... But again I get the same histogram pattern with ISO 125 shots.
As for the noise level yep there's a lot of it but I stack bursts (usually at least 20 frames, though I'm thinking of going higher) which reduces noise quite a bit, so at that point getting the offset right can make a difference I suppose, mostly if like I said added gain and WB end up boosting that offset quite a bit. But yeah I suppose a bit of extra quantisation noise from this histogram quirk doesn't do much.
Wow you're right, but look at the histogram for the whole picture, it's very strange, around 2048 you have values that never occur which gives these huge peaks and troughs but the rest of the histogram is much more normal or at least has a much tamer version of the problem. I knew that ADCs could have such quirks but it still looks pretty odd.
http://i.imgur.com/iqXfe1E.png
To be honest it kind of makes you want to reconsider relying on values in the vicinity of 2048 to determine anything.
Pages