Auto-bright is performed on dcraw_make_mem_image() stage and changes the range of input data that mapped into output (gamma-corrected) image, based on image histogram (in linear space).
So, yes, it is done in linear space, but on output.
So for every 4 values in the histogram you have one with green (not sure which but it's one of the two), the next one has both blue and the other green, the next one has red only, and finally there's one with no values at all. This would make sense if they were all gathered in the same bins but instead they're spread differently, like they're being digitised differently maybe?
It might be worth noting that the G7 X Mark I (which preceded the related G9 X) only had 12-bit values, so maybe the bump to 14-bit was done with cutting a corner somewhere.
I agree, it seems that there are pretty consistent biases, so I have the answer to my original question, thank you all!
One thing I'm wondering though is why is there a difference across channels? Because electronically, they should be the same, just with different filters on top, right? But then again comparing histograms there are clearly differences in which values which channel have lots of or none of. I guess I'll do some tests to figure this out, comparing different regions.
A side note: in LibRaw we read Canon's metadata black level too, have a look at libraw_types.h, libraw_canon_makernotes_t -- it is in ChannelBlackLevel[4]. You can add something like
if (Canon.ChannelBlackLevel[0])
printf("ChannelBlackLevel (from makernotes): %d %d %d %d\n",
Canon.ChannelBlackLevel[0],
Canon.ChannelBlackLevel[1],
Canon.ChannelBlackLevel[2],
Canon.ChannelBlackLevel[3]);
to raw-identify and it will print those levels if called with '-v'.
I had a look at full raw picture histograms with daylight pictures at ISO 125 or even darker shots at ISO 6400 and we've got the same pattern around 2048.
In fact I just looked at a much older daylight ISO 125 shot taken with my old G9 X (I currently have both a G9 X and a G9 X II) and it's no better: http://i.imgur.com/sfMDaKr.png < the histogram is perfectly antialiased, it's no visualisation artifact, that's exactly how it is, you've got the strict 12-bit comb pattern that gradually fades as you go up in values.
As for any noise reduction I just looked and I do have "High ISO speed NR" set to "Standard", but I don't think it (and sure hope it doesn't) affects the RAW. According to this it doesn't http://www.learn.usa.canon.com/resources/articles/2011/high_iso_noise_re.... But again I get the same histogram pattern with ISO 125 shots.
As for the noise level yep there's a lot of it but I stack bursts (usually at least 20 frames, though I'm thinking of going higher) which reduces noise quite a bit, so at that point getting the offset right can make a difference I suppose, mostly if like I said added gain and WB end up boosting that offset quite a bit. But yeah I suppose a bit of extra quantisation noise from this histogram quirk doesn't do much.
Wow you're right, but look at the histogram for the whole picture, it's very strange, around 2048 you have values that never occur which gives these huge peaks and troughs but the rest of the histogram is much more normal or at least has a much tamer version of the problem. I knew that ADCs could have such quirks but it still looks pretty odd.
G9X-II samples from photographyblog all have Black Level about 2047-2050. Do you use some special mode (noise reduction??) on your camera?
UPD: also, these files do not show 'one of 4' histogram even for higher ISO (3200-6400-12800). Unfortunately, no samples with intermediate ISO (2000) in this site.
standard deviation value for black masked pixels is about 37 for this shot. So 5 lower bits (for 14-bit scale) are useless, difference between black at 2048 and 2044 to be masked under the noise.
Please note, that only one value of 4 is used (so camera effectively 12 bit at this ISO).
With such high noise (screenshot #1, see histogram bell width) difference in one 'real' level is very hard to see in practice (and I do not see any visible difference between automatic black level and manual set to 2047 or 2048).
Strange, that would explain why only I thought this was a problem. Here's the result from a burst I took (Canon G9 X II):
IMG_0293.CR2: black 2044, cblack 2 0 1 0
IMG_0294.CR2: black 2044, cblack 1 0 1 0
IMG_0295.CR2: black 2044, cblack 1 0 2 0
IMG_0296.CR2: black 2045, cblack 1 1 0 0
IMG_0297.CR2: black 2044, cblack 0 1 1 2
IMG_0298.CR2: black 2043, cblack 1 2 2 0
IMG_0299.CR2: black 2044, cblack 1 0 2 2
IMG_0300.CR2: black 2044, cblack 1 2 1 0
IMG_0301.CR2: black 2043, cblack 3 2 2 0
IMG_0302.CR2: black 2044, cblack 1 1 3 0
IMG_0303.CR2: black 2045, cblack 0 1 1 1
IMG_0304.CR2: black 2043, cblack 2 0 3 1
IMG_0305.CR2: black 2044, cblack 1 0 3 0
IMG_0306.CR2: black 2044, cblack 1 1 1 0
IMG_0307.CR2: black 2044, cblack 0 0 3 1
IMG_0308.CR2: black 2045, cblack 0 0 0 0
IMG_0309.CR2: black 2042, cblack 3 0 4 3
IMG_0310.CR2: black 2044, cblack 1 1 1 0
IMG_0311.CR2: black 2044, cblack 1 1 0 0
IMG_0312.CR2: black 2044, cblack 1 2 1 0
IMG_0313.CR2: black 2042, cblack 3 5 4 0
For what it's worth I took another burst just a couple of minutes later (same settings except higher ISO) and all these files in that burst have exactly "black 2049, cblack 0 0 0 0"
I've examined several samples (from latest cameras with approx. black level 2048) and I could not find any sample with black level below 2047 (LibRaw's value)
Auto-bright is performed on dcraw_make_mem_image() stage and changes the range of input data that mapped into output (gamma-corrected) image, based on image histogram (in linear space).
So, yes, it is done in linear space, but on output.
Okay I did some tests with a black frame by doing averages on various non-overlapping windows and the biases across channels are consistent.
Here's an histogram of a black frame at ISO 125, 1/2000 sec, it's also pretty weird: http://i.imgur.com/Q7kgaLU.png
line 6479, yep I see it, thanks!
Indeed, and look at what happens when you superimpose all 4: http://i.imgur.com/EovPwIF.png
So for every 4 values in the histogram you have one with green (not sure which but it's one of the two), the next one has both blue and the other green, the next one has red only, and finally there's one with no values at all. This would make sense if they were all gathered in the same bins but instead they're spread differently, like they're being digitised differently maybe?
It might be worth noting that the G7 X Mark I (which preceded the related G9 X) only had 12-bit values, so maybe the bump to 14-bit was done with cutting a corner somewhere.
If you are working with archive, edit dcraw_common.cpp in /internal, after the line 5479
I'm looking at the source for LibRaw 0.18.2 and in dcraw/dcraw.c I don't see this at all at line 7662. Lines 7662 and 7663 read this:
{ "Olympus E-PL7", 0, 0, { 9197,-3190,-659,-2606,10830,2039,-458,1250,5458 } },In LibRaw/dcraw.c after the line 7662 you have:
if ((unique_id == 0x03970000) || // G7 X Mark II
(unique_id == 0x80000394)) // EOS M5
make it:
if ((unique_id == 0x03970000) || // G7 X Mark II
(unique_id == 0x04100000) || // G9 X Mark II
(unique_id == 0x80000394)) // EOS M5
Here is histogam of visible area: https://www.dropbox.com/s/09p4dcys46ldx86/Screenshot%202017-03-26%2017.3...
Black subtraction is turned off, log vertical scale.
Entire histogram looks very strange.
I agree, it seems that there are pretty consistent biases, so I have the answer to my original question, thank you all!
One thing I'm wondering though is why is there a difference across channels? Because electronically, they should be the same, just with different filters on top, right? But then again comparing histograms there are clearly differences in which values which channel have lots of or none of. I guess I'll do some tests to figure this out, comparing different regions.
I've got 0.18.2, I literally just got started with LibRaw a few days ago after looking at the dcraw code made me despair.
I'm tempted to think it might be something more upstream, something with the ADC or a sort of calibration curve applied at some point around the ADC.
Maybe your LibRaw version is not current?
I tried these (on my dark frame shot) and I get 10000 0 0 389. and AverageBlackLevel says 2597, these values look strange.
Btw I was gonna printf everything but turns out it's much quicker to use your debugger to have a quick look at everything ;)
Yes, histogram of your _0298 shot looks strange, I suspect some kind of raw data processing applied in camera.
Thank you for the dark frame.
I've averaged central part of the image using RawDigger: https://www.dropbox.com/s/ciftfhp8zgfkxfp/Screenshot%202017-03-26%2008.4...
Black subtraction is off, averaged values are 2044.5, 2045.6, 2047.7, 2046.2 (rounded in red).
Entire image is purple because black subtraction is turned off.
Next screenshot shows black values determined by LibRaw using black frame integration: https://www.dropbox.com/s/fcxybepvflf0ec9/Screenshot%202017-03-26%2008.5...
(bottom left corner): 2044, 2045, 2047, 2046
Looks like masked pixels averaging is right (may be 1 level off for G2), it is in very good agreement with black frame ('real black') data.
I do not see any reason to switch from masked pixels averaging to metadata value.
Thank you for the samples, most interesting.
A side note: in LibRaw we read Canon's metadata black level too, have a look at libraw_types.h, libraw_canon_makernotes_t -- it is in ChannelBlackLevel[4]. You can add something like
if (Canon.ChannelBlackLevel[0])
printf("ChannelBlackLevel (from makernotes): %d %d %d %d\n",
Canon.ChannelBlackLevel[0],
Canon.ChannelBlackLevel[1],
Canon.ChannelBlackLevel[2],
Canon.ChannelBlackLevel[3]);
to raw-identify and it will print those levels if called with '-v'.
I had a look at full raw picture histograms with daylight pictures at ISO 125 or even darker shots at ISO 6400 and we've got the same pattern around 2048.
In fact I just looked at a much older daylight ISO 125 shot taken with my old G9 X (I currently have both a G9 X and a G9 X II) and it's no better: http://i.imgur.com/sfMDaKr.png < the histogram is perfectly antialiased, it's no visualisation artifact, that's exactly how it is, you've got the strict 12-bit comb pattern that gradually fades as you go up in values.
Here's the dark frame http://www.mediafire.com/file/1823atsyamr1xx5/IMG_1554.CR2 , I'll spare you any suspense, here's the histogram: http://i.imgur.com/pSVFFLK.png
As for any noise reduction I just looked and I do have "High ISO speed NR" set to "Standard", but I don't think it (and sure hope it doesn't) affects the RAW. According to this it doesn't http://www.learn.usa.canon.com/resources/articles/2011/high_iso_noise_re.... But again I get the same histogram pattern with ISO 125 shots.
As for the noise level yep there's a lot of it but I stack bursts (usually at least 20 frames, though I'm thinking of going higher) which reduces noise quite a bit, so at that point getting the offset right can make a difference I suppose, mostly if like I said added gain and WB end up boosting that offset quite a bit. But yeah I suppose a bit of extra quantisation noise from this histogram quirk doesn't do much.
Wow you're right, but look at the histogram for the whole picture, it's very strange, around 2048 you have values that never occur which gives these huge peaks and troughs but the rest of the histogram is much more normal or at least has a much tamer version of the problem. I knew that ADCs could have such quirks but it still looks pretty odd.
http://i.imgur.com/iqXfe1E.png
To be honest it kind of makes you want to reconsider relying on values in the vicinity of 2048 to determine anything.
Followup #3:
G9X-II samples from photographyblog all have Black Level about 2047-2050. Do you use some special mode (noise reduction??) on your camera?
UPD: also, these files do not show 'one of 4' histogram even for higher ISO (3200-6400-12800). Unfortunately, no samples with intermediate ISO (2000) in this site.
And another request: could you please provide dark frame shot (lens caps on) with same exposure (1/80, ISO 2000)?
Followup:
standard deviation value for black masked pixels is about 37 for this shot. So 5 lower bits (for 14-bit scale) are useless, difference between black at 2048 and 2044 to be masked under the noise.
Thank you for the sample (I've checked only EOS cameras, will inspect G9X2 in more depth).
Here is histogram of masked pixels: https://www.dropbox.com/s/06nyv7dhu56n4i1/Screenshot%202017-03-25%2022.4...
(black subtraction turned off).
And here is the same histogram, but zoomed in: https://www.dropbox.com/s/d1grqdcl343tncj/Screenshot%202017-03-25%2022.5...
Please note, that only one value of 4 is used (so camera effectively 12 bit at this ISO).
With such high noise (screenshot #1, see histogram bell width) difference in one 'real' level is very hard to see in practice (and I do not see any visible difference between automatic black level and manual set to 2047 or 2048).
Strange, that would explain why only I thought this was a problem. Here's the result from a burst I took (Canon G9 X II):
For what it's worth I took another burst just a couple of minutes later (same settings except higher ISO) and all these files in that burst have exactly "black 2049, cblack 0 0 0 0"
Here is IMG_0298.CR2 http://www.mediafire.com/file/b2xzd3tit3gww85/IMG_0298.CR2
I've examined several samples (from latest cameras with approx. black level 2048) and I could not find any sample with black level below 2047 (LibRaw's value)
Could you please provide one?
Pages