That is a very common optimization and is not likely to be the issue. The problem is usually found in the expression itself, commonly either the right-side array element is undefined or the result overflows the variable.

On the 6th iteration of that loop the result of this expression is undefined:

imgdata.idata.xtrans

 = imgdata.idata.xtrans_abs[c];  

what is wrong with

for(int c = 0; c < 36; c++) 

loop if both xtrans and xtrans_abs are [6][6] arrays, so xtrans[0][35] is equal to xtrans[5][5]?

gcc preprocessor does not count blank and comment lines, and I gave the wrong code section. sorry.

I pulled the rest of the warnings from the build log:

/tmp/portage/portage/media-libs/libraw-0.18.5/work/LibRaw-0.18.5/src/libraw_cxx.cpp: In member function ‘virtual int LibRaw::open_datastream(LibRaw_abstract_datastream*)’:

/tmp/portage/portage/media-libs/libraw-0.18.5/work/LibRaw-0.18.5/src/libraw_cxx.cpp:1784:64: warning: iteration 6 invokes undefined behavior [-Waggressive-loop-optimizations]

imgdata.idata.xtrans[0]

 = imgdata.idata.xtrans_abs[0][c];
                                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
 
 
/tmp/portage/portage/media-libs/libraw-0.18.5/work/LibRaw-0.18.5/src/libraw_cxx.cpp:1783:22: note: within this loop
 
     for(int c = 0; c < 36; c++)
                    ~~^~~~
 
So it appears the actual code section begins at line 1999:
 
 
    // XTrans Compressed?
    if (!imgdata.idata.dng_version && !strcasecmp(imgdata.idata.make, "Fujifilm") &&
        (load_raw == &LibRaw::unpacked_load_raw))
    {
      if (imgdata.sizes.raw_width * imgdata.sizes.raw_height * 2 != libraw_internal_data.unpacker_data.data_size)
      {
        if (imgdata.sizes.raw_width * imgdata.sizes.raw_height * 7 / 4 == libraw_internal_data.unpacker_data.data_size)
          load_raw = &LibRaw::fuji_14bit_load_raw;
        else
          parse_fuji_compressed_header();
      }
      if (imgdata.idata.filters == 9)
      {
        // Adjust top/left margins for X-Trans
        int newtm = imgdata.sizes.top_margin % 6 ? (imgdata.sizes.top_margin / 6 + 1) * 6 : imgdata.sizes.top_margin;
        int newlm = imgdata.sizes.left_margin % 6 ? (imgdata.sizes.left_margin / 6 + 1) * 6 : imgdata.sizes.left_margin;
        if (newtm != imgdata.sizes.top_margin || newlm != imgdata.sizes.left_margin)
        {
          imgdata.sizes.height -= (newtm - imgdata.sizes.top_margin);
          imgdata.sizes.top_margin = newtm;
          imgdata.sizes.width -= (newlm - imgdata.sizes.left_margin);
          imgdata.sizes.left_margin = newlm;
          for (int c1 = 0; c1 < 6; c1++)
            for (int c2 = 0; c2 < 6; c2++)
              imgdata.idata.xtrans[c1][c2] = imgdata.idata.xtrans_abs[c1][c2];
        }
      }
}
 
 
gcc optimizes the nested 6x6 loop to:   for(int c = 0; c < 36; c++)
 
on the 6th iteration of that loop the result of  imgdata.idata.xtrans[c] = imgdata.idata.xtrans_abs[c];  
is undefined. 

Hi, I'm getting the following warning from gcc-6.4.0 in libraw_cpp.cxx:

* /tmp/portage/portage/media-libs/libraw-0.18.5/work/LibRaw-0.18.5/src/libraw_cxx.cpp:1784:64: warning: iteration 6 invokes undefined behavior [-Waggressive-loop-optimizations]

CFLAGS="-O2 -march=native -ftree-vectorize -mprefer-avx128 -mvzeroupper -fstack-protector -ftree-loop-distribution -ftree-loop-distribute-patterns -pipe"

Code:

static inline void unpack28bytesto16x16ns(unsigned char *src, unsigned short *dest)
{
dest[0] = (src[3] << 6) | (src[2] >> 2);
dest[1] = ((src[2] & 0x3) << 12) | (src[1] << 4) | (src[0] >> 4);
dest[2] = (src[0] & 0xf) << 10 | (src[7] << 2) | (src[6] >> 6);
dest[3] = ((src[6] & 0x3f) << 8) | src[5];
dest[4] = (src[4] << 6) | (src[11] >> 2);
dest[5] = ((src[11] & 0x3) << 12) | (src[10] << 4) | (src[9] >> 4);
dest[6] = (src[9] & 0xf) << 10 | (src[8] << 2) | (src[15] >> 6);
dest[7] = ((src[15] & 0x3f) << 8) | src[14];
dest[8] = (src[13] << 6) | (src[12] >> 2);
dest[9] = ((src[12] & 0x3) << 12) | (src[19] << 4) | (src[18] >> 4);
dest[10] = (src[18] & 0xf) << 10 | (src[17] << 2) | (src[16] >> 6);
dest[11] = ((src[16] & 0x3f) << 8) | src[23]; ## line 1784
dest[12] = (src[22] << 6) | (src[21] >> 2);
dest[13] = ((src[21] & 0x3) << 12) | (src[20] << 4) | (src[27] >> 4);
dest[14] = (src[27] & 0xf) << 10 | (src[26] << 2) | (src[25] >> 6);
dest[15] = ((src[25] & 0x3f) << 8) | src[24];
}

Thanks for the quick response.
I found the answer to my question. It's very simple. For L *, use gamm [0] = 1/3 and gamm [1] = 9.033.

With best wishes, Konstantin

all postprocessing is done at dcraw_process() step.

Unpack only unpacks (uncompress for compressed formats) raw data and stores it without any modification

unpack only unpacks raw data ('as is')
interpolation is performed on dcraw_process() step

To use bilinear set
half_size to 0
user_qual to 0

Use bilinear. It is fast enough (but not perfect too)

What is criteria for 'best'?

I finally found the culprit of the crash: the thread executing LibRaw ran out of stack space! I observed LibRaw crashing internally when I single step into the actual decoder for DNG.

By replacing "LibRaw rawProcessor;" with "LibRaw *rawProcessor = new LibRaw;", the crash went away :)

Thank you Alex,

What about "imgdata.params.output_color" ? I noticed it is producing different outputs if I set it to 0 (RAW) or 1 (sRGB). I would like the output to be linear as I am already setting gamm[0] = gamm[1] = 1, but looks like this parameter is override some options..

I don't think it is the file, because when use_rawspeed = 0 , LibRaw has no problem loading this file.

Would you be so kindly post links to the libraw_r.a and librawspeed.a compiled for macOS so I can download them to try out? PM me if it is not convenient to do it in the forum post.

Thanks!