the pipeline of raw decoding

I have difficulty in clarifying how the pipeline of decoding a raw file goes. First, DNs of a raw file are scaled. I suppose that scaling consists of 5 elements: 1) linearization of curved values as far as an irreversible raw file is concerned, 2)setting of black level, 3) setting of white level, 4) subtraction of black level values from each pixel, and 5) mapping modified values to a logical range of 0-1. The logicalized values are mapped by way of matrix into a working color space before a final output space. My question is linked to dcraw's command -W -D. -W is described to kill brightening and scaling. You will see a pretty dark image result from -D -W, though with less contrast, which means no black level applied. Very dark in particular when a 12bit raw file is decoded. A14 bit raw would be brighter rhan the 12bit one, even though an almost identical scene was taken. Why is this ? A potential answer to this case might be that dcraw boost values in default mode, making 12 and 14bit as if 16bit in terms of pixel values, then execute scaling. But such an intervention would be quite unnecessary because all the values are mapped to the logical 0-1 in th end, regardless of that intervention. I hope responses from somebody who knows about it.

Forums: 

Two notes:

Two notes:
1) Not sure what 'irreversible raw file' is. If you mean 'highlights compression' used in many lossy formats (Nikon, Sony, DNGs), than yes, first step is to make data from file back to linear space.

5) Mapping to logical 0-1 values: from developer (programmer) point of view, 'logical 0-1' means 'zeroes and ones only'. If I understand you correctly, in our (programmer) terms, you mean 0...1.0 'floating point' range with intermediate values.
Practically, this conversion is not mandatory unless 'possible data maximum' is correct. One may use, for example, 0...16383 range for 14 bit data.

dcraw -D -W

-D means 'document mode 2' so: unchanged raw values (linearized on your step 1) are contained in dcraw's image[] array after processing. Black level is not subtracted.

-W means 'no automatic brightening', so image histogram is not calculated and data are converted to 2.2 gamma and scaled to 8 bits (if no other keys are passed to dcraw)

Without -W, image histogram is calculated and output is scaled to have 1% of pixels in saturation.

-- Alex Tutubalin @LibRaw LLC

Also: dcraw does not do '0-1'

Also: dcraw does not do '0-1' mapping. It does black subtraction (in camera data range) and works with subtracted values w/o any scaling.

-- Alex Tutubalin @LibRaw LLC

Hi lexa, thanks again for

Hi lexa, thanks again for your commitment !
" Irreversible " was employed to paraphrase " compressed with loss ". I believe that DN values of a compressed raw look like curved and truncated as if an inverse gamma of 0.45 for example was put in place. ( so you need a gamma to decode and linearize compressed values). Secondly, yes too, I wanted to say a floating point range of 0.0 to 1.0 with logically infinite intermediates. When you say that dcraw does not map DN values to this kind of 0-1 range, does dcraw map them directly to a working color space ? Thirdly, I meant dcraw's "-D -W -g 1 1" instead of "-D -W". Finally, the result of a 12bit or 14bit raw file decoded with "-D -W -g 1 1" is completely different from one with "-d -W -k 0 -S 4095 -g 1 1". "-D -W -g 1 1" makes image darker as if its "fixed white level" seems to be at around 65535. Why is this ?

dcraw does all processing

dcraw does all processing until convert_to_rgb() in camera native color space and in 16-bit integers (no conversion to 0-1 float).

dcraw -d is completely diferent from dcraw -D because 'scale_colors()' call is used with -d. It scales input data range (12 or 14 bit, for example) to full 0..65535 scale (based on camera theoretical maximum, so underexposed images are not brightened more than needed).
dcraw -d -W will brighten output to get 1% of pixels in saturation.

Here is 'dcraw annotated and outlined': http://ninedegreesbelow.com/files/dcraw-c-code-annotated-code.html
(and preface: http://ninedegreesbelow.com/photography/dcraw-c-code-annotated.html ), hope it will help you with dcraw internals.

-- Alex Tutubalin @LibRaw LLC

So, if I understand correctly

So, if I understand correctly, dcraw's -D -W -g11 command maps DNs of 12 or 14bit raws as such to a 16bit space because -D does not scale(restructure) DNs and the 16bit space is universal for all the available raw files. On the other hand, -d -W -g 11 command scales DNs to be mapped(fit) to the 16bit space. Taking 'scale_colors' into account, result from -D-W-g11-b16 would be roughly similar with one from -d -W -g11-r1111 when decoding a 14bit with 0-16383DNs. Is this the story ? In that case, "a fixed white level" of -W command would mean simply no setting of white level, isnt it ?

Sorry, to much switch

Sorry, to much switch combinations to comment each variant separately (and, also, ensure the terms used have same meaning for both sides).

If you're interested in dcraw.c internals and each command line switch effect, the only reliable source is dcraw.c itself (annotated or not).

-- Alex Tutubalin @LibRaw LLC

OK, ignore my former post. I

OK, ignore my former post. I might not fully undestand what you said.

>dcraw does all processing until convert_to_rgb() in camera native color space and in 16-bit integers (no conversion to 0-1 float).

Do you mean that dcraw converts 12 or 14bit DNs to 16bit, with scaling in -d command & without scaling in -D command ?

scale_colors means white

scale_colors means white balance, doesnt it ?

Or scaling itself ?

As regards -D, it maps DNs of

As regards -D, it maps DNs of 12 or 14 bit as they are into the 16 "working bit". Am I right ?

I did not understand the

I did not understand the question. What is 'maps' in this context?

-- Alex Tutubalin @LibRaw LLC

Sorry for ambiguity. I mean

Sorry for ambiguity. I mean by "maps" " places without changing values". For example, 16384th DN of a 14 bit raw is placed without change to 16384th of 16bit by -D. Is it ok?

Yes, values are (mostly)

Yes, values are (mostly) unchanged (excluding PhaseOne data correction and, may be, something like it)

-- Alex Tutubalin @LibRaw LLC

Thanks. Finally, I want to

Thanks. Finally, I want to clarify the key reason why an image derived from -D -W looks so dark. Because the command does not scale DNs. Yes. But which element of scaling enables conversion (boost) of 12 or 14 bit DNs so as to fit to the 16 working bit ?

I did not understand the

I did not understand the question (again). What 'element of scaling' is?

-- Alex Tutubalin @LibRaw LLC

I put it at the beginning of

I put it at the beginning of my question:

>I suppose that scaling consists of 5 elements: 1) linearization of curved values as far as an irreversible raw file is concerned, 2)setting of black level, 3) setting of white level, 4) subtraction of black level values from each pixel, and 5) mapping modified values to a logical range of 0-1.

In the case of dcraw, 5th element is not appropriate.

Still unable to understand

Still unable to understand what is 'setting of black level' if 'subtraction of black level values' is a separate step.

-- Alex Tutubalin @LibRaw LLC

If you think it strange, lets

If you think it strange, lets join them together in the name of black level application. For me, setting was fixing and a state before implementation. In this regard, I had forgotten to note another element of white level implementation.

Hope, this is my last answer

Hope, this is my last answer in this thread.

dcraw's scale_colors() does
- black subtraction
- white balance
- and data scaling (looks like it is 'setting of white level' in your terms?)

in one action.

-- Alex Tutubalin @LibRaw LLC

Hi lexa

Hi lexa
>dcraw's scale_colors() does
> - black subtraction
> - white balance
> - and data scaling (looks like it is 'setting of white level' in your terms?)
>in one action.

In your wording, scaling of dcraw looks like "mapping DN values so as to fit to the 16 working bit". Is it ?

Sorry, I'm really tired of

Sorry, I'm really tired of this discussion.

Yes. Or no. I do not know is my 'data scaling' is the same of your 'mapping to fit'. The problem is with the terms, as usual.

-- Alex Tutubalin @LibRaw LLC

Also, dcraw.c is very well

Also, dcraw.c is very well documented (in C language :) and annotated (see links above).

It is very hard to translate from "C" to human language if all terms are not very well defined before.

-- Alex Tutubalin @LibRaw LLC

I am very sorry if you felt

I am very sorry if you felt weary of our conversations. As I said, I have difficulty in clarifying dcraw’s raw processing, and its key seems to be scaling at last. Given scaling is the setting of white level as you said, how are the values between black and white levels then converted to the 16 working bit ? Is there any rule of the conversion as adobe describes in the specification of DNG ? https://wwwimages.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs...

dcraw.c is self-documented

dcraw.c is self-documented (in computer code). The function you need to analyze is scale_colors()

-- Alex Tutubalin @LibRaw LLC

Thanks for your response. I'm

Thanks for your response. I'm gonna dig it for a while, though I am not a programmer but only a user. Best,