Thank you very much for your fast response, looking at the data structure documentation for imgdata.idata.filters, I see this:
unsigned filters;
Bit mask describing the order of color pixels in the matrix (0 for full-color images). 32 bits of this field describe 16 pixels (8 rows with two pixels in each, from left to right and from top to bottom). Each two bits have values 0 to 3, which correspond to four possible colors. Convenient work with this field is ensured by the COLOR(row,column) function, which returns the number of the active color for a given pixel.
values less than 1000 are reserved as special cases:
1 - Leaf Catchlight with 16x16 bayer matrix;
9 - Fuji X-Trans (6x6 matrix)
3..8 and 10..999 - are unused.
Some points are not very clear to me:
-32 bits for 16 pixels, 2bits per pixels to code color, but what is the mapping ? (0->red, 1->green, 2->blue)
-What does this 1-->999 code means ?
Current libraw snapshot can run user-specified demosaic routine, just set interpolate_bayer pointer to point to your function.
Look into other demosaic functions code (e.g. lin_interpolate()) to see how to loop through all pixels.
To get bayer pattern, use imgdata.idata.filters variable, or COLOR(r,c) call.
I'm also a little irritated. I basically changed nothing and used the latest git (just from today). The only thing I changed for Android is to manually implement the "swab" function, but the implementation I use should be the same as the original from c. I could also see no calls on this function when the xtrans decoding is going on.
C++ standard defines arrays as dense arrays, no spacing or extra rows padding.
Also, because xtrans/_abs arrays are defined in LibRaw class definition, and it is possible that we compile two source files with different -O.. switches, array layout should not changed from -O0 (no warning) to -O2 (warning issued).
So, I think this is gcc problem: it should either show this warning with any -O.. setting, or do not show it with -O2
Anyway, this loop was already changed to to 6-wide loops in master branch with commit message 'to make gcc happy' :)
That is a very common optimization and is not likely to be the issue. The problem is usually found in the expression itself, commonly either the right-side array element is undefined or the result overflows the variable.
On the 6th iteration of that loop the result of this expression is undefined:
Can you point me to a matrix that "libraw" will use to convert DNG file? (to XYZ) The "cam_xyz" contain all zeros, there should be some matrix being used.
gcc preprocessor does not count blank and comment lines, and I gave the wrong code section. sorry.
I pulled the rest of the warnings from the build log:
/tmp/portage/portage/media-libs/libraw-0.18.5/work/LibRaw-0.18.5/src/libraw_cxx.cpp: In member function ‘virtual int LibRaw::open_datastream(LibRaw_abstract_datastream*)’:
We always welcome raw samples.
To send us files, you may use any file sharing service (WeTransfer, Dropbox, Google Drive, etc) and send link to it to info@libraw.org.
Please use simple public links (available for anyone, who knows the URL), but not 'Share for account' feature
0...999 range is reserved for future custom layouts, like Fuji 6x6.
32 bits describes 16 pixels, 4x4 block. For all today cameras it is repeated 2x2 blocks.
00 - red, 01 - green, 10 - blue, 11 - green2 channels
(just use LibRaw::COLOR(row,col) call to get layout)
Thank you very much for your fast response, looking at the data structure documentation for imgdata.idata.filters, I see this:
unsigned filters;
Bit mask describing the order of color pixels in the matrix (0 for full-color images). 32 bits of this field describe 16 pixels (8 rows with two pixels in each, from left to right and from top to bottom). Each two bits have values 0 to 3, which correspond to four possible colors. Convenient work with this field is ensured by the COLOR(row,column) function, which returns the number of the active color for a given pixel.
values less than 1000 are reserved as special cases:
1 - Leaf Catchlight with 16x16 bayer matrix;
9 - Fuji X-Trans (6x6 matrix)
3..8 and 10..999 - are unused.
Some points are not very clear to me:
-32 bits for 16 pixels, 2bits per pixels to code color, but what is the mapping ? (0->red, 1->green, 2->blue)
-What does this 1-->999 code means ?
I will take a look at the source code tomorrow
Current libraw snapshot can run user-specified demosaic routine, just set interpolate_bayer pointer to point to your function.
Look into other demosaic functions code (e.g. lin_interpolate()) to see how to loop through all pixels.
To get bayer pattern, use imgdata.idata.filters variable, or COLOR(r,c) call.
Is there any way to get exact code point that fails with exception? maybe coredump or so?
Hi Alex,
yes, it's the line, I put it behind the bracket:
{__android_log_print(ANDROID_LOG_INFO,"libraw","decode_block %d %d %d",g_even_pos,g_odd_pos,line_width); ...This is the output I see:
I'm also a little irritated. I basically changed nothing and used the latest git (just from today). The only thing I changed for Android is to manually implement the "swab" function, but the implementation I use should be the same as the original from c. I could also see no calls on this function when the xtrans decoding is going on.
Regards
Torsten
Run the file through valgrind (under FreeBSD), do not see any fuji-decoder problems (buffer overrun, etc), so no idea.
BTW, what exact version do you use (in my working version line 712 is { bracket after while operator)?
1) If use_camera_matrix value suggests to use camera (raw metadata) color, than this color data is copied over rgb_cam array at open_file() stage.
2) If camera matrix is not available, embedded color data (from adobe_coeff() large array) is used.
Camera color matrix is parsed into imgdata.color.cmatrix
Sorry, wrong. If use_camera_matrix is set, so camera matrix should be used, camera color matrix is copied to rgb_cam
C++ standard defines arrays as dense arrays, no spacing or extra rows padding.
Also, because xtrans/_abs arrays are defined in LibRaw class definition, and it is possible that we compile two source files with different -O.. switches, array layout should not changed from -O0 (no warning) to -O2 (warning issued).
So, I think this is gcc problem: it should either show this warning with any -O.. setting, or do not show it with -O2
Anyway, this loop was already changed to to 6-wide loops in master branch with commit message 'to make gcc happy' :)
You need to apply white balance (usually, before demosaic stage)
That is a very common optimization and is not likely to be the issue. The problem is usually found in the expression itself, commonly either the right-side array element is undefined or the result overflows the variable.
On the 6th iteration of that loop the result of this expression is undefined:
imgdata.idata.xtrans
Hi Alex,
Thanks for the quick reply!
Can you point me to a matrix that "libraw" will use to convert DNG file? (to XYZ) The "cam_xyz" contain all zeros, there should be some matrix being used.
Regards,
Mio
what is wrong with
loop if both xtrans and xtrans_abs are [6][6] arrays, so xtrans[0][35] is equal to xtrans[5][5]?
"libraw" does not average/weight DNG color data
cmatrix is camera color data, it used according to use_camera_matrix value.
gcc preprocessor does not count blank and comment lines, and I gave the wrong code section. sorry.
I pulled the rest of the warnings from the build log:
/tmp/portage/portage/media-libs/libraw-0.18.5/work/LibRaw-0.18.5/src/libraw_cxx.cpp: In member function ‘virtual int LibRaw::open_datastream(LibRaw_abstract_datastream*)’:
/tmp/portage/portage/media-libs/libraw-0.18.5/work/LibRaw-0.18.5/src/libraw_cxx.cpp:1784:64: warning: iteration 6 invokes undefined behavior [-Waggressive-loop-optimizations]
imgdata.idata.xtrans[0]
= imgdata.idata.xtrans_abs[0][c]; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ /tmp/portage/portage/media-libs/libraw-0.18.5/work/LibRaw-0.18.5/src/libraw_cxx.cpp:1783:22: note: within this loop for(int c = 0; c < 36; c++) ~~^~~~ So it appears the actual code section begins at line 1999: // XTrans Compressed? if (!imgdata.idata.dng_version && !strcasecmp(imgdata.idata.make, "Fujifilm") && (load_raw == &LibRaw::unpacked_load_raw)) { if (imgdata.sizes.raw_width * imgdata.sizes.raw_height * 2 != libraw_internal_data.unpacker_data.data_size) { if (imgdata.sizes.raw_width * imgdata.sizes.raw_height * 7 / 4 == libraw_internal_data.unpacker_data.data_size) load_raw = &LibRaw::fuji_14bit_load_raw; else parse_fuji_compressed_header(); } if (imgdata.idata.filters == 9) { // Adjust top/left margins for X-Trans int newtm = imgdata.sizes.top_margin % 6 ? (imgdata.sizes.top_margin / 6 + 1) * 6 : imgdata.sizes.top_margin; int newlm = imgdata.sizes.left_margin % 6 ? (imgdata.sizes.left_margin / 6 + 1) * 6 : imgdata.sizes.left_margin; if (newtm != imgdata.sizes.top_margin || newlm != imgdata.sizes.left_margin) { imgdata.sizes.height -= (newtm - imgdata.sizes.top_margin); imgdata.sizes.top_margin = newtm; imgdata.sizes.width -= (newlm - imgdata.sizes.left_margin); imgdata.sizes.left_margin = newlm; for (int c1 = 0; c1 < 6; c1++) for (int c2 = 0; c2 < 6; c2++) imgdata.idata.xtrans[c1][c2] = imgdata.idata.xtrans_abs[c1][c2]; } } } gcc optimizes the nested 6x6 loop to: for(int c = 0; c < 36; c++) on the 6th iteration of that loop the result of imgdata.idata.xtrans[c] = imgdata.idata.xtrans_abs[c]; is undefined.what 'iteration 6' mean in this context?
Hi, I'm getting the following warning from gcc-6.4.0 in libraw_cpp.cxx:
* /tmp/portage/portage/media-libs/libraw-0.18.5/work/LibRaw-0.18.5/src/libraw_cxx.cpp:1784:64: warning: iteration 6 invokes undefined behavior [-Waggressive-loop-optimizations]
CFLAGS="-O2 -march=native -ftree-vectorize -mprefer-avx128 -mvzeroupper -fstack-protector -ftree-loop-distribution -ftree-loop-distribute-patterns -pipe"
Code:
static inline void unpack28bytesto16x16ns(unsigned char *src, unsigned short *dest)
{
dest[0] = (src[3] << 6) | (src[2] >> 2);
dest[1] = ((src[2] & 0x3) << 12) | (src[1] << 4) | (src[0] >> 4);
dest[2] = (src[0] & 0xf) << 10 | (src[7] << 2) | (src[6] >> 6);
dest[3] = ((src[6] & 0x3f) << 8) | src[5];
dest[4] = (src[4] << 6) | (src[11] >> 2);
dest[5] = ((src[11] & 0x3) << 12) | (src[10] << 4) | (src[9] >> 4);
dest[6] = (src[9] & 0xf) << 10 | (src[8] << 2) | (src[15] >> 6);
dest[7] = ((src[15] & 0x3f) << 8) | src[14];
dest[8] = (src[13] << 6) | (src[12] >> 2);
dest[9] = ((src[12] & 0x3) << 12) | (src[19] << 4) | (src[18] >> 4);
dest[10] = (src[18] & 0xf) << 10 | (src[17] << 2) | (src[16] >> 6);
dest[11] = ((src[16] & 0x3f) << 8) | src[23]; ## line 1784
dest[12] = (src[22] << 6) | (src[21] >> 2);
dest[13] = ((src[21] & 0x3) << 12) | (src[20] << 4) | (src[27] >> 4);
dest[14] = (src[27] & 0xf) << 10 | (src[26] << 2) | (src[25] >> 6);
dest[15] = ((src[25] & 0x3f) << 8) | src[24];
}
use samples/unprocessed_raw.cpp as a starting point
Thanks for the quick response.
I found the answer to my question. It's very simple. For L *, use gamm [0] = 1/3 and gamm [1] = 9.033.
With best wishes, Konstantin
Output curve is used only on output phase (dcraw_make_mem_image or dcraw_ppm_tiff_writer)
So, you may modify source and replace calls to gamma_curve(...) by own output curve creation.
No direct way in LibRaw to do that, we'll consider to add some interface for user-defined curve in 0.19 release.
all postprocessing is done at dcraw_process() step.
Unpack only unpacks (uncompress for compressed formats) raw data and stores it without any modification
You mean demosaic the X-Trans CFA is done at the dcraw_process() step?
unpack only unpacks raw data ('as is')
interpolation is performed on dcraw_process() step
To use bilinear set
half_size to 0
user_qual to 0
So how do I specify bilinear filter?
When I execute unpack(), does it perform demosaic of the X-Trans CFA automatically?
Pages