Problem with dng file from Pentax K5-II

I'm seeing some odd results when processing a DNG from a Pentax K5-II. Exiftool reports:

======== IMGP1593.DNG
Black Point                     : 513 513 515 516
Black Level                     : 513 513 515 516
Exposure Time                   : 1/8000
ISO                             : 800

Yet after calling LibRaw.unpack() I see

+		cblack	0xcadc10d8 {0, 0, 0, 0, 2, 2, 513, 513, 515, 516, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...}	unsigned int[4102]
		black	0	unsigned int

So my code didn't do black subtraction (because black was reported as 0).

When processing an equivalent PEF file

======== IMGP1596.PEF
Black Point                     : 513 513 516 515
Exposure Time                   : 1/8000
ISO                             : 800
 
+		cblack	0xc4ecb0d8 {0, 0, 2, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...}	unsigned int[4102]
		black	513	unsigned int

Odd to see that both black, cblack[2] and cblack[3] are filled in this case so I'd much appreciate some explanation of how cblack really works and how I should process these when operating on the raw image data?

Thanks

Forums: 

However variable black is

However variable black is ZERO and I don't believe it should be

David Partridge

> I'd much appreciate some

> I'd much appreciate some explanation of how cblack really works and how I should process these when operating on the raw image data?

You also didn't address that at all.

David Partridge

I'm now using adjust_bl()

I'm now using adjust_bl() before doing my black level subtraction, and I'm seeing:

00000094 2019/06/24 14:33:18.354 018788 00002900             >Before adjust_bl() C.black = 0.
00000095 2019/06/24 14:33:18.363 018788 00002900             >First 10 C.cblack elements
00000095 2019/06/24 14:33:18.363 018788 00002900             >0, 0, 0, 0
00000095 2019/06/24 14:33:18.363 018788 00002900             >2, 2
00000095 2019/06/24 14:33:18.363 018788 00002900             >513, 513, 515, 516
00000096 2019/06/24 14:33:18.372 018788 00002900             >Subtracting black level of C.black = 513 from raw_image data.
00000097 2019/06/24 14:33:18.382 018788 00002900             >First 10 C.cblack elements
00000097 2019/06/24 14:33:18.382 018788 00002900             >516, 515, 513, 513
00000097 2019/06/24 14:33:18.382 018788 00002900             >0, 0
00000097 2019/06/24 14:33:18.382 018788 00002900             >513, 513, 515, 516

should I expect the order of cblack[0]-cblack[4] to be the reverse of cblack[7]-cblack[9]???

I'd have expected it to be in the same order as the levels reported by exiftool (513, 513, 515, 516)

David Partridge

Still seeking clarification

Still seeking clarification here ... What order are the first four cblack elements in? RGBG, RGGB?

What order they in for cblack[6]-[9]?

What order are they supposed to be in for the exif tags?

Is there any clear write up explaining the cblack value array other than the code and the rather terse decription in the documentation which doesn't explain much ...?

Thanks

David Partridge

Dear Sir:

Dear Sir:
I don't find the description lacking.
For parsing cblack in DNG and for order used in EXIF tags please see DNG specification: "This tag specifies the zero light ... encoding level, as a repeating pattern. The origin of this pattern is the top-left corner of the ActiveArea rectangle. The values are stored in row-column-sample scan order."
Reading DNG specification may help answering your questions.
For DNG, cblack is filled under "case 0xc619" in parse_tiff_ifd.

--
Iliah Borg

You know the purpose of

You know the purpose of documentation is so other people can read use and understand your stuff. It's no great surprise that you understand it!

You still didn't answer my question about the order of values in cblack[0-3] versus cblack[6-9]. I *think* that it is RGBG for cblack[0-4] and RGGB for cblack[6-9].

Of course if you had deigned to answer my questions in the first place instead of almost going out of your way to avoid doing so - saying along the way "Well I understand it ..." that would have saved me a lot of time which is what this forum is supposed to be about.

David Partridge

Dear Sir:

Dear Sir:

In my previous post I quoted Adobe documentation on the matter. I'm afraid you are not paying attention. Your question is about EXIF, not LibRaw.

--
Iliah Borg

Well, actually I am asking

Well, actually I am asking about the color.cblack array provided by YOU to my application.

Clearly you think you're justified in saying that isn't something you answer questions on.

If I gave answers like that to my customers when I was providing tech support I would have been fired (quite rightly) by my management.

D.

David Partridge

Dear Sir:

Dear Sir:

Have you already looked at processing of pertinent EXIF tags under "case 0xc619" and "case 0xc61a"? They simply read the EXIF tags, and thus my reference to Adobe DNG Specification is quite justified. Please read Adobe documentation and try to understand it.

I don't see the reason for your anger, but in any case please refrain from showing it here in future.

--
Iliah Borg

Oh let me see: Could it be

(edited to remove flame - ib)
yes I did look at the code, but you could have maybe said something like:

Yes the order is different (or not), as you will see if you compare the processing for DNG black level at case xxxxx and the one for Black Point at case yyyyy

And then pointed me at the DNG spec to add more information. I still don't know for certain that cblack[0-3] is in a different order to cblack[6-9]

David Partridge

Dear Sir:

Dear Sir:

Quoting Adobe again, from a previous, "The origin of this pattern is the top-left corner of the ActiveArea rectangle. The values are stored in row-column-sample scan order". How do you interpret this? Please draw two different Bayer patterns and suggest how the tag should be filled based on the description above.

--
Iliah Borg

I totally get that this tells

I believe that it tells me the order in the DNG file is RGGB (if I read it right) and this this maps directly into cblack[6-9]. It doesn't confirm the order for cblack[0-3] In fact I'm not even sure now about that order given what I see in the cblack array before and after calling adjust_bl() as per my post on 24 June, 2019 - 15:39 which I repeat the data of here:

00000094 2019/06/24 14:33:18.354 018788 00002900             >Before adjust_bl() C.black = 0.
00000095 2019/06/24 14:33:18.363 018788 00002900             >First 10 C.cblack elements
00000095 2019/06/24 14:33:18.363 018788 00002900             >0, 0, 0, 0
00000095 2019/06/24 14:33:18.363 018788 00002900             >2, 2
00000095 2019/06/24 14:33:18.363 018788 00002900             >513, 513, 515, 516
00000096 2019/06/24 14:33:18.372 018788 00002900             >Subtracting black level of C.black = 513 from raw_image data.
00000097 2019/06/24 14:33:18.382 018788 00002900             >First 10 C.cblack elements
00000097 2019/06/24 14:33:18.382 018788 00002900             >516, 515, 513, 513
00000097 2019/06/24 14:33:18.382 018788 00002900             >0, 0
00000097 2019/06/24 14:33:18.382 018788 00002900             >513, 513, 515, 516

should I expect the order of cblack[0]-cblack[3] to be the reverse of cblack[6]-cblack[9]???

If I'm not understanding what's going on here please correct my misunderstandings

David

David Partridge

The CFA for the K5 is BGGR

The CFA for the K5 is BGGR format (silly me for assuming RGGB). So I think the coefficients in cblack[6-9] are in that order. Is the order for cblack[0-3] RGGB - if so that explains the reversal I see - or am I still confused??

David Partridge

Depending on the offset of

Depending on the offset of the active zone, camera model (obviously), and firmware - useful Bayer patterns can be different. Answering pattern questions, specific to model, firmware, or active zone, makes sense only if there is a bug in the library, and only with a "thank you!".

From LibRaw docs:

unsigned black;
Black level. Depending on the camera, it may be zero (this means that black has been subtracted at the unpacking stage or by the camera itself), calculated at the unpacking stage, read from the RAW file, or hardcoded.

my comment: if only one black level value is found / hardcoded / calculated, it is in this field. Before using it, you may want to check the content of cblack.

More from docs:
unsigned cblack[4102];
Per-channel black level correction. First 4 values are per-channel correction, next two are black level pattern block size, than cblack[4]*cblack[5] correction values (for indexes [6....6+cblack[4]*cblack[5]).

my comment: if 4 (per channel) black level values are found / calculated from sources other than Adobe EXIF tag (with one exception that is irrelevant for the matter at hand), for the camera you are interested in they are read into cblack[0 .. 3] in 0 1 3 2 index order (see code, permutations
FORC4 cblack[c ^ c >> 1] = get2();
do that. That's the way Mr. Coffin did it in dcraw, and we preserved it in LibRaw for compatibility.)

if cblack[0] is 0, and / or cblack[4 .. 5] are present (DNG), you can use cblack [6 .. ...]. You can follow the code under "case 0xc61a" to see the order cblack is populated in this case. For a regular DNG, the cycle in essence is
FORC(cblack[4] * cblack[5]) cblack[6+c] = getreal(type);
meaning, the order coming from Adobe EXIF tag is preserved. Again, preserved from dcraw for compatibility.

--
Iliah Borg

Thank you

Thank you

David Partridge