Thank you very much for your help! I did some more "testing" today and set the exposure time to 5 seconds so I would basically get a white picture and calculated the maximum of each channel. This is what I got:
ISO R G B
400 15760 15761 15761
500 15760 15761 15761
640 12810 12811 12811
800 15760 15761 15761
1000 15760 15762 15760
1250 12810 12813 12810
1600 15764 15767 15763
Now at least I know what value I have to normalize with. I still have one technical question: Why are the values for 640 and 1250 different than the others?
First. ph1_black is the data field for PhaseOne black data. It is not useful for any other formats.
Second. If you want to scale data up to saturation, you should scan for maximum value in image[] field, than scale as you want.
Also, for real image processing you should subtract black value first.
thank you very much for your answer. I implemented an averaging of entries for the fields top and left. Oddly, it is the same for ISO400 and ISO800 (1058). I am not sure if I understood something wrong, but I expected a much larger value to account for the difference in maximum pixel value for the ISO800 picture. My results so far are: The laserpointer is bright enough and the exposure time high enough to drive at least the green channel into saturation.
When I use the imgdata.color.maximum field, I get the value 15600 for ISO400 and for ISO800, which is about what I expected and not a problem for my application.
When I scan for the maximum value in any channel, I get 15760 for the ISO400 (close enough to 15600 I think) but only ~12811 for the ISO800. This is what I do not understand because masked_pixels buffer contain almost the same values.
Also, I tried to access the ph1_black but I think it is not allocated in my case, at least something like
threw an error. If the 12811 for ISO800 is the other maximum value as you suggested then it is a problem because when my application works, my goal is to not drive the channel into saturation but to use maybe 80% of the maximum saturation. Do you have any idea about why the black levels are the same and if the 12811 is the maximum value you suggested is used for high ISO settings or what I can do to solve this issue?
Yes, amplifier+ADC combo works this way. On my 5Dmk2 I've two different 'maximum' values, one for normal ISOs and one for 'High' ones.
Anyway, there is two different questions
- maximum possible value (2^(bit depth) minus dark current). This value is data format/camera dependend, you may rely on imgdata.color.maximum field
- maximum value for given shot. In general, you should scan entire image are to calculate it.
Of course I set these before calling open_file and unpack. In addition, the reason I found out about this was that I used two sets of pictures, one shot with ISO400 and one with ISO800 and basically illuminated a spot with a damped laserpointer (in a dark room). I wanted to calculate the exposure time that I need to saturate one of the channels. I noticed that the maximum value I could get for ISO400 was above 14500, while for the ISO800 I couldn't get the value above 13000 which doesn't make sense to me, since the wavelength of the laser didn't change and only the ISO setting was increased.
Good evening Dan!
I'm a moderator of Russian CG Forum Arttower.ru. Could you give us the permisson to publish the Russian translation of this article (http://www.libraw.su/articles/raw-module.html) on our site?
As for profiles. There is not a profile as such. You need to determine your own cameras RGB to XYZ response. If you have a D70s I could send you some matrixes that I made earlier. I am in the process of publishing a paper on the subject. Ill keep you posted.
I successfully applied CIECAM02 to a RAW imaging workflow. The main problem was to identify the adopted white from scene colorimerty as an input to CIECAM02. I ended up using the cameras white balance for ease in order to complete the experiment. The scene illumination was calculated from ISO and Exposure and gave favorable results. The main part of the process was to characterise the camera using a double monochromator and calculate the cameras spectral response. This was then used to determine the device RGB to XYZ matrix by reducing errors in JMH space for different values of La and white point.
The resulting matrices were used to transform RAW RGB to XYZ according to scene white and La and then passed through CIECAM02.
The internal coordinates used were JMH and output to standard sRGB viewing conditions through the reverse CIECAM02.
I got some pretty good results, (although maybe not appearance perfect) compared to the sRGB JPEG produced by the Nikon D70s.
The problem is in MacOS X fseek() call. It takes 32-bit offset (long int). There is fseeko() function which takes off_t offset and so will work with files larger than 2Gb.
The quick fix:
in libraw/libraw_datastream.h change:
return substream?substream->seek(o,whence):fseek(f,o,whence);
to
return substream?substream->seek(o,whence):fseeko(f,o,whence);
This will break compatibility with Windows (no fseeko) and, possible, Linux. I'll prepare more universal solution (with #ifdef) today or tomorrow (to appear in 0.8-Beta5)
1. Frame offset is stored as 64 bit in this format, so there is no *theoretical* problems.
2. There is two possible sources of problems under Mac OS X:
a) one possible source is 32-bit off_t type (i.e. some special compilation flags are needed because 64-bit offsets are possible)
b) FILE * interface under Mac OS X does not support large files.
I've MacBook with MacOS 10.5 on hands, so will continue investigations. Hope, I'll find problem source today.
Hi, many apologies for the long delay, I got food poisoning after the last post,
and I've been trying to catch up ever since.
I've put a test file on http://www.modern-industry.com/test.cine Its about 49MB.
It is the same case with the brightness contrast: for the majority of street scenes, there is an empirical desire to increase the brightness. This effect was measured by Stevens and Stevens 45 years ago, and the results can be quantitatively used. The reason why you need to increase the contrast of the printouts is the same: you look at them at a much dimmer and duller light than the one that was present during the shooting and presumed by the viewer.
I'm afraid it may not be very easy to repeat, as the file I'm using
is a .cine file of around 16G! I know this sounds an extreme case, but
some professional movie cameras shoot raw...
I didn't use any special switches, just something of the form of
dcraw_emu -s 700 mytestfile.cine
I'm afraid that I'm going to be very busy for the next couple of days, but
a good experiment I might perform would be to create a file with very small frames, so I
can find out if the frame count or the file size is the issue. That might
also create a test case that can be loaded via the Internet.
After much experiment, I tried the same code on an Ubuntu 9.04 box, and it works perfectly. Backporting the same code back to OS X gives me the original problem.
Finally, I just tried dcraw_emu from the samples directory, and that suffers the
same problem, so I've eliminated my application code.
There are 2 items I would like to see enhanced in the current Raw conversion libraries out there:
1. I would like extra output 32-bit float instead of kist 16-bit integer.
2. The ability to quickly extract a section of an image. Lets say I want to create a zoom tool that zooms into a section of a Raw file I would like to have a fast Raw library that only process the pixels involved in the area you are zooming in. I'm aware that raw is a stream of data but at leas a fast way of walking through the stream instead of interpolating all the pixels on its way.
The typical shape of the curve is lighter in the midtone and darker in the three-quartertone. However, it would be easy to incorporate some presets that would permit users to choose the type of color enhancement.
Looks like 640 and 1250 are 'software' ISO on your camera. So 1250 is shot at 1600, than values are divided by 1.25 or so.
It also looks like you've not subtracted black value (about 1000 on your camera).
Thank you very much for your help! I did some more "testing" today and set the exposure time to 5 seconds so I would basically get a white picture and calculated the maximum of each channel. This is what I got:
ISO R G B
400 15760 15761 15761
500 15760 15761 15761
640 12810 12811 12811
800 15760 15761 15761
1000 15760 15762 15760
1250 12810 12813 12810
1600 15764 15767 15763
Now at least I know what value I have to normalize with. I still have one technical question: Why are the values for 640 and 1250 different than the others?
bye, Peter
First. ph1_black is the data field for PhaseOne black data. It is not useful for any other formats.
Second. If you want to scale data up to saturation, you should scan for maximum value in image[] field, than scale as you want.
Also, for real image processing you should subtract black value first.
Hey,
thank you very much for your answer. I implemented an averaging of entries for the fields top and left. Oddly, it is the same for ISO400 and ISO800 (1058). I am not sure if I understood something wrong, but I expected a much larger value to account for the difference in maximum pixel value for the ISO800 picture. My results so far are: The laserpointer is bright enough and the exposure time high enough to drive at least the green channel into saturation.
When I use the imgdata.color.maximum field, I get the value 15600 for ISO400 and for ISO800, which is about what I expected and not a problem for my application.
When I scan for the maximum value in any channel, I get 15760 for the ISO400 (close enough to 15600 I think) but only ~12811 for the ISO800. This is what I do not understand because masked_pixels buffer contain almost the same values.
Also, I tried to access the ph1_black but I think it is not allocated in my case, at least something like
rawProcessor.imgdata.masked_pixels.ph1_black[1][0]
threw an error. If the 12811 for ISO800 is the other maximum value as you suggested then it is a problem because when my application works, my goal is to not drive the channel into saturation but to use maybe 80% of the maximum saturation. Do you have any idea about why the black levels are the same and if the 12811 is the maximum value you suggested is used for high ISO settings or what I can do to solve this issue?
bye, Peter
Yes, amplifier+ADC combo works this way. On my 5Dmk2 I've two different 'maximum' values, one for normal ISOs and one for 'High' ones.
Anyway, there is two different questions
- maximum possible value (2^(bit depth) minus dark current). This value is data format/camera dependend, you may rely on imgdata.color.maximum field
- maximum value for given shot. In general, you should scan entire image are to calculate it.
So far I tried setting the following parameters, similar to unprocessed_raw.cpp:
OUT.document_mode=2;
OUT.output_bps=16;
OUT.user_flip=0;
OUT.no_auto_bright = 1;
OUT.filtering_mode=LIBRAW_FILTERING_NONE;
Of course I set these before calling open_file and unpack. In addition, the reason I found out about this was that I used two sets of pictures, one shot with ISO400 and one with ISO800 and basically illuminated a spot with a damped laserpointer (in a dark room). I wanted to calculate the exposure time that I need to saturate one of the channels. I noticed that the maximum value I could get for ISO400 was above 14500, while for the ISO800 I couldn't get the value above 13000 which doesn't make sense to me, since the wavelength of the laser didn't change and only the ISO setting was increased.
bye, Peter
Good evening Dan!
I'm a moderator of Russian CG Forum Arttower.ru. Could you give us the permisson to publish the Russian translation of this article (http://www.libraw.su/articles/raw-module.html) on our site?
You may rely on imgdata.color.maximum field.
LibRaw need to seek to arbitrary position in input stream. So, there is no way to read from stdin.
You may implement you own LibRaw_datastreeam class with full buffering.
http://www.digitalcolour.org/toolbox.htm
Dr Green has matlab code for CIECAM02.
As for profiles. There is not a profile as such. You need to determine your own cameras RGB to XYZ response. If you have a D70s I could send you some matrixes that I made earlier. I am in the process of publishing a paper on the subject. Ill keep you posted.
Looks very promising.
Is there some code (plus profile) to play with?
Hi
I successfully applied CIECAM02 to a RAW imaging workflow. The main problem was to identify the adopted white from scene colorimerty as an input to CIECAM02. I ended up using the cameras white balance for ease in order to complete the experiment. The scene illumination was calculated from ISO and Exposure and gave favorable results. The main part of the process was to characterise the camera using a double monochromator and calculate the cameras spectral response. This was then used to determine the device RGB to XYZ matrix by reducing errors in JMH space for different values of La and white point.
The resulting matrices were used to transform RAW RGB to XYZ according to scene white and La and then passed through CIECAM02.
The internal coordinates used were JMH and output to standard sRGB viewing conditions through the reverse CIECAM02.
I got some pretty good results, (although maybe not appearance perfect) compared to the sRGB JPEG produced by the Nikon D70s.
DCRAW and Matlab used for computation.
The problem is in MacOS X fseek() call. It takes 32-bit offset (long int). There is fseeko() function which takes off_t offset and so will work with files larger than 2Gb.
The quick fix:
in libraw/libraw_datastream.h change:
return substream?substream->seek(o,whence):fseek(f,o,whence);
to
return substream?substream->seek(o,whence):fseeko(f,o,whence);
This will break compatibility with Windows (no fseeko) and, possible, Linux. I'll prepare more universal solution (with #ifdef) today or tomorrow (to appear in 0.8-Beta5)
There is two news, good one and not-so-good :)
1. Frame offset is stored as 64 bit in this format, so there is no *theoretical* problems.
2. There is two possible sources of problems under Mac OS X:
a) one possible source is 32-bit off_t type (i.e. some special compilation flags are needed because 64-bit offsets are possible)
b) FILE * interface under Mac OS X does not support large files.
I've MacBook with MacOS 10.5 on hands, so will continue investigations. Hope, I'll find problem source today.
Oh, sorry.
I've downloaded this file several weeks ago, than make LibRaw 0.8-b4, than forgot about file.
Will work with it tomorrow.
gentle ping....
M
Hi, many apologies for the long delay, I got food poisoning after the last post,
and I've been trying to catch up ever since.
I've put a test file on http://www.modern-industry.com/test.cine Its about 49MB.
Mike
It is the same case with the brightness contrast: for the majority of street scenes, there is an empirical desire to increase the brightness. This effect was measured by Stevens and Stevens 45 years ago, and the results can be quantitatively used. The reason why you need to increase the contrast of the printouts is the same: you look at them at a much dimmer and duller light than the one that was present during the shooting and presumed by the viewer.
Sorry, link to warez site deleted by LibRaw admin
You say '16Gb' ?
For many file types data_offset of frame data is read as 32-bit integer (via get4()). So, maximum frame offset is 2Gb.
To check this I need some sample with several frames. I don't need full 16-Gb file multi-multi-frame file, but file with 3-4 frames will be enough.
I'm afraid it may not be very easy to repeat, as the file I'm using
is a .cine file of around 16G! I know this sounds an extreme case, but
some professional movie cameras shoot raw...
I didn't use any special switches, just something of the form of
dcraw_emu -s 700 mytestfile.cine
I'm afraid that I'm going to be very busy for the next couple of days, but
a good experiment I might perform would be to create a file with very small frames, so I
can find out if the frame count or the file size is the issue. That might
also create a test case that can be loaded via the Internet.
Mike
Could you please describe how to repeat the problem with dcraw_emu? What command-line switches used?
After much experiment, I tried the same code on an Ubuntu 9.04 box, and it works perfectly. Backporting the same code back to OS X gives me the original problem.
Finally, I just tried dcraw_emu from the samples directory, and that suffers the
same problem, so I've eliminated my application code.
Any further input welcome...
There are 2 items I would like to see enhanced in the current Raw conversion libraries out there:
1. I would like extra output 32-bit float instead of kist 16-bit integer.
2. The ability to quickly extract a section of an image. Lets say I want to create a zoom tool that zooms into a section of a Raw file I would like to have a fast Raw library that only process the pixels involved in the area you are zooming in. I'm aware that raw is a stream of data but at leas a fast way of walking through the stream instead of interpolating all the pixels on its way.
The typical shape of the curve is lighter in the midtone and darker in the three-quartertone. However, it would be easy to incorporate some presets that would permit users to choose the type of color enhancement.
erreauk
Unfortunately, I'm not an Matlab user, so cannot provide significant help.
I guess, Matlab must have some interface to external libraries. LibRaw is a library, you can compile it according to Matlab requirements and use.