What to do with rgb_cam matrix

I'm confused about how to apply the conversion matrix given by rd.color.rgb_cam. Here's the sequence of what I do:

-Load the raw pixel values with libraw_unpack(rd)
-Subtract the black levels and normalise to 1.0
-Multiply each pixel by its bayer filter colour times 2 for green and times 4 for red/blue
-Debayer roughly but ultra simply using a half pixel by half pixel shift
-Do the matrix multiplication with rd->color.rgb_cam
-Apply the camera's rd->color.cam_mul white balance correction, or whatever white balance the user sees fit

At that point I'd expect good colours, but instead I get a very yellow image. What's strange is that if I reverse the last two steps by applying the white balance before the matrix multiplication then I get good colours except in the clipped highlights that turn dark purple. But it doesn't make sense that something arbitrary like white balance should be done before something fixed like colour space conversion. But still it seems like there should be something somewhat similar to white balance being done before the matrix multiplication to bring the colours to the illuminant expected by the colour space conversion, but what exactly?

I tried figuring out what LibRaw does about that when running libraw_dcraw_process() but it's a bit hard to follow what it does with matrices, explanations would be most welcome.

Forums: 

1) White balance should be

1) White balance should be applied before interpolation (half and bilinear demosaics could handle non-balanced data, but others are not).
2) I could not understand this step:
"Multiply each pixel by its bayer filter colour times 2 for green and times 4 for red/blue"
Is it 'poor man approximate white balance'? If so, WB values should be adjusted to 0.5/0.25 :)

3) rgb_cam is 'camera to sRGB' matrix. If your output space is not sRGB, color profile matrix (rgb_cam) should be adjusted to output space (see convert_to_rgb() source for an example)

-- Alex Tutubalin @LibRaw LLC

2) It's a pretty sensible

2) It's a pretty sensible step if you think about interpreting the bayer image from a signal processing point of view, each red or blue channel has 1 pixel set for every 4 pixels, so if you're going to use any kind of low pass filtering then in order to have the correct energy you need to multiply everything by 4 (or 2 for green) to create a valid intermediary representation so that the average pixel values for the whole image would be the same before and after such low pass filter-based demosaicing, like blurring, halving or interpolating, because each set pixel will have to be averaged with 3 black pixels. I wouldn't call that white balance at all, the image still comes off as mostly green and applying the white balance coefs makes it look right albeit desaturated if I don't do the colour space conversion.

1 & 3) So you are saying that white balance should be done before the matrix multiplication? This perplexes me because usually white balance correction in an editor is done by the user towards the end of the processing chain, as in your editor gives you a debayered colour space-converted defringed rectilinearised image and then it applies white balance but if you don't like it you can do your own additional white balance which is a simple rgb multiplication, or so it seems to be since it seems WB adjustment is a lightweight operation. But to be consistent any WB change by the user in an editor would have to be done before even debayering and everything else would have to be done all over again for each user adjustment of WB. It sounds like that's what should correctly be done, right?

The reason I made this thread is that I first tried doing WB correction followed by the matrix multiplication, but in some cases it gives me rather dark purple highlights, so I thought there must be something wrong.

Here's my processing with WB followed by matrix multiplication: https://i.imgur.com/F3FjABw.png
Here's the libraw_dcraw_process() output with use_camera_wb set to 1 and highlight set to 0 (clip, so that means no reconstruction, right?): https://i.imgur.com/Y2uxhWl.png

If all 3 channels are maxed out but WB correction boosts red and blue and the matrix multiplication essentially increases the saturation then I guess it makes sense that it would turn out that way, but clearly I'm missing something rather important to avoid getting purple street lamps.

You are multiplying _after_

You are multiplying _after_ normalization, and you are using wb coefficients that are valid before your multiplication.
You have purple highlights because you are not applying proper clipping. To operate in energy domain you need to have a better picture of spectral response, before and after demosaicking.

--
Iliah Borg

"You are multiplying _after_

"You are multiplying _after_ normalization"
I'm not sure what you mean and I don't understand the relevance, since normalisation doesn't do much besides scaling values.

"you are not applying proper clipping"
Well, what is proper clipping? I thought I'm supposed to do everything and then clip at the very end, but clearly that won't solve the dark purple highlights. Am I rather supposed to boost the values of maxed out pixels?

Why are you normalizing to 1

Why are you normalizing to 1 and after that ruining normalization with multiplication? Not to mention that your justification for multiplication the way you do it is not convincing.
If you have green at clipping already in raw data, and white balance promotes red and blue channels, you have minus green, that's magenta / purple, as an magentish highlights. So the question is: what clipping value to use? Suppose you have a 14-bit raw, and the green channel clips at the value of 14000 (that's before subtracting black).

--
Iliah Borg

> Why are you normalizing to

> Why are you normalizing to 1 and after that ruining normalization with multiplication?
As opposed to doing what? It's hard for me to explain what I'm doing wrong if no one explains how to do things right. The colour space matrix multiplication is kind of an afterthought to the way I'm doing this, what I do makes more sense if you exclude it. Every stage of my processing is meant to be visible with a rather consistent approach to gain, so naturally it starts with subtracting black levels and normalisation. I divide the WB coefs from cam_mul by their minimum (green, because I don't understand why the green coef should ever be less than 1.0), so that green stays normalised while red and blue can go higher than 1.0 but are clipped on screen (but not in the data) so that at that stage highlights are white on screen but purple in the data.

Is it after the white balance multiplication that I should clip the data to 1.0? I don't like that it means I'm throwing data away, but I guess that makes sense if I'm not going to do highlight recovery and that the matrix multiplication will make the higher red and blue values push green down. So I guess I should clip, do the colour space conversion, then make things normalised again by dividing everything by the lowest value from the result of (1 , 1 , 1) × the matrix (and then clip again?).

1.0 is after normalization,

1.0 is after normalization, so if you read my reply above you should be asking yourself, "what is the correct denominator for normalization?"
How to do it right? - but there is a lot of open source code available, including in libraw. Have you looked at the code in our samples folder? Also, there are books and papers available.

--
Iliah Borg

I'm not sure why you were

I'm not sure why you were cryptic about whether my description of what I should do is right, but anyway I tried it and it seems about right, see https://i.imgur.com/gidxryK.png (mine seems to have slightly brighter highlights compared with the rest of the picture, maybe my normalisation is better. I didn't have to do anything after the matrix multiplication btw, I guess the matrix is normalised). I also don't understand why you talk so much about normalisation given that it wasn't relevant to my problem, I didn't have to change anything about normalisation at all, knowing the correct order of operations and knowing to clip (and when) was what was crucial.

I didn't look at sample code, maybe I should have, I just didn't think of doing that because I'd expect such examples to show how to call functions, not how to replicate them, which is what I'm trying to do in my own mostly GPGPU-based way. Instead I debugged through libraw_dcraw_process() to see what it does, and let's just say that it's rather difficult to learn anything doing that given that the 21,420 lines of code in draw_common.cpp are hardly self-documenting, not to mention there are some things with which even the debugger is lost, like accessing the elements of variables (in libraw_cxx.cpp) such as O. or C. or S..

I don't blame you guys for the lack of explanatory comments in the source since I know dcraw was the same way, but while it's good that you documented the API, I think that libraw, as the main successor of dcraw, deserves more because it's so important, your work will outlive you, it's the go-to thing for processing images from any camera, and when we're all dead of old age digital archaeologists will be looking into whatever succeeds libraw to know how to decipher ancient digital photographs, not just how to use it because even now the way it does things shows its age. And so the problems I'm facing they will face too unless the people who understand the code document what it does. Just look at a function like recover_highlights(), I'm sure it does something quite good, but how would anyone know what it does? It doesn't have a single comment explaining what's going on, how is anyone supposed to maintain that? Look at it https://github.com/LibRaw/LibRaw/blob/master/internal/dcraw_common.cpp#L... and see how much you need to use your memory to remember what's the purpose of each of the blocks of loops. Even I am not fond of adding comments, but for blocks of code I'll put a small comment that says what the block of code below does, it just makes everything seem instantly clearer and less daunting.

Let's say someone (e.g. me) thinks "libraw is good, but I want to do most of the processing in floating-point arithmetic in real time OpenCL. I don't want to do original research into every aspect of processing, I just want to do what dcraw/libraw does for every phase of processing", then I need to understand what the code does, not just how to use it. I looked into a couple of the better demosaicing algorithms that libraw provides and it seems like there's literally nothing about at least one of these anywhere on the Internet. Some of them have a few comments, but only in Russian! All we have is the code itself and little to help us understand what it does let alone why it does it. I don't mean to criticise you guys, I really appreciate your hard work and how you guys answer all our questions, we'd have a much harder time without your help, I'm just saying that ideally you should write down what you understand about the code as comments while you still remember what it does. As it is the inner workings of libraw could hardly be more arcane.

LibRaw is not about

LibRaw is not about processing, it is about decoding raw and some of the relevant metadata. If you don't want to do your own research it's your choice. I was doing mine for more than 15 years, and still continue to do it.

--
Iliah Borg

"Do your own research like I

"Do your own research like I did mine" is not a good reason to keep large chunks of code absolutely undocumented to the point of inscrutability, mostly not for code that went through a couple of decades and will go through many more. None of us are going to devote 15 years of our lives towards reinventing the wheel and figuring out how you guys did some particular demosaicing algorithm that only you have and works great but doesn't begin to explain what it does and isn't documented anywhere. If you put so much work into researching and making something and shared it you might as well enable people to understand what was done and learn from it rather than telling them to disregard the work you present and to painstakingly retrace your steps or telling them "these large parts of our library are outside of the scope of our library so whatever".

It's not like the parts of the library that are within the core scope of the library are documented either, maybe you're all just too used to dealing with a practically comment-free 21 kLoC file (btw that's at least 20 times too long for a single file) to see what's the problem with that approach. But if you guys aren't interested in documenting 22 years of work beyond telling people how to use it on a basic level then I'm not going to convince you of anything. I'm not telling you you should write an encyclopedia about every function either, and I understand that as a commercial developer you have better things to do than to document an open source project, but a few comments here and there whenever you get the chance to help clarify why things are being done would go a long way, and it might just help you in the future when the meaning of tens of thousands of lines of esoteric code isn't fresh in your mind anymore. It's probably too late for that anyway, most of the people who wrote that stuff have probably long since moved on or forgotten how everything they wrote works.

Dear Sir:

Dear Sir:

Please don't put words into my mouth. We put a lot of effort into saving people some time, but we assume that libraw users have a good idea of what raw files are and how to process them, or use our samples, or learn as they go - if they want to implement their own processing but have no experience and theoretical knowledge necessary. Libraw by itself is not meant to be a textbook on raw decoding or raw conversion.

Meanwhile I answered your question right away, pointing out that your problem with magentish highlights here https://imgur.com/F3FjABw are because of how you normalize (denominator) and clip.

I will leave it at it.

--
Iliah Borg

I understand that you may be

I understand that you may be feeling "got at" here, but to be fair the OP does have a point in that the code definitely does suffer from a distinct paucity of comments - and while you say the libraw focus is on extracting data from raw files, there's still a very high proportion of the code dedicated to post-processing the raw image.

I truly don't expect you to drop everything and add reams of comments but if the next time you go near any code that lacks comments, adding some lines to explain how/why it's doing what it does would be most helpful to the rest of us.

Cheers

David Partridge