switch (imgdata.sizes.flip)
{
case 3://0011
case 180:
metaData.flip = Meta_data::Flip_180;
break;
case 5://0101
case 270:
metaData.flip = Meta_data::Flip_90CCW;
break;
case 6://0110
case 90:
metaData.flip = Meta_data::Flip_90CW;
break;
From first lines of raw2image_start (called by _ex):
if (O.user_flip >= 0)
S.flip = O.user_flip;
switch ((S.flip+3600) % 360)
{
case 270: S.flip = 5; break;
case 180: S.flip = 3; break;
case 90: S.flip = 6; break;
}
So:
flip/user_filp are bit-field. But if someone has specified user_flip in degrees it will work OK.
Specifically for CIFF (CRW) format camera sets rotation angle, not Orientation tag. This value is preserved after open_datastream (looks like we need to fix it for all cases in 0.18?)
For most camera which I have used the flip field means "bit-field" before actually processing (unpack()/raw2image_start()). But for EOS 10D this flag means the rotation angle. So, I would prefer not to start processing but get meaning of this field: is it bit-field or angle. Is it possible to know if it was corrected before?
imgdata.image (and rawdata.*) and imgdata.sizes are always in 'sensor dimensions', so it is alwasy unrotated and rotation/mirroring is read/set via flip/user_flip.
Rotation/mirroring is performed on output phase:
dcraw_ppm_tiff_writer()
or
dcraw_make_mem_image()
I've got 3 more images from the photographer. They were all reported as being corrupted and after the patch you sent, libraw works with all of them. Thanks!
Thanks for your email too. That would be useful for the future because some photographers do not want to post their images publicly.
It seems that for rotated images (.imgdata.sizes.flip&4) you need to use height for stride calculation. And keep in mind that you will receive rotated image with width and height swapped.
The only real recommendation is RawSpeed (if you use it): LibRaw is tested with RawSpeed/master branch, not with develop branch.
Other libs are used in very standard way, so you need jpeg8+ (or any replacement with jpeg_mem_src), latest DNG SDK (if you use it), any modern version of zlib
Thanks for the response (apologies if you got my request twice due to me thinking the forum one got lost)
That reminds me, whilst looking at the current changelog in github, I wondered if you have a recommended set of supporting library versions to build against ?
If you'll find another problem(s), please share the files.
You may send it to info@libraw.org (or to personal E-mail lexa@lexa.ru) if you do not wish to expose these files to public.
Looks like we can safely ignore 'not enough data problem' for last line(s) of last block (it contains masked pixels, not needed for data display).
It may be Firmware issue(??)
Here is the patch (for last line of last block only):
diff --git a/src/libraw_xtrans_compressed.cpp b/src/libraw_xtrans_compressed.cpp
index b6555d3..ec6dbb0 100644
--- a/src/libraw_xtrans_compressed.cpp
+++ b/src/libraw_xtrans_compressed.cpp
@@ -46,6 +46,7 @@ struct xtrans_block {
unsigned max_read_size; // Amount of data to be read
int cur_buf_size; // buffer size
uchar *cur_buf; // currently read block
+ bool lastlast;
LibRaw_abstract_datastream *input;
struct int_pair grad_even[3][41]; // tables of gradients
struct int_pair grad_odd[3][41];
@@ -143,7 +144,7 @@ static inline void fuji_fill_buffer(struct xtrans_block *info)
#ifndef LIBRAW_USE_OPENMP
info->input->unlock();
#endif
- if(info->cur_buf_size<1) // nothing read
+ if(info->cur_buf_size<1 && info->max_read_size && !info->lastlast) // nothing read
throw LIBRAW_EXCEPTION_IO_EOF;
info->max_read_size -= info->cur_buf_size;
@@ -159,6 +160,7 @@ void LibRaw::init_xtrans_block(struct xtrans_block* info, const struct xtrans_pa
INT64 fsize = libraw_internal_data.internal_data.input->size();
info->max_read_size = _min(unsigned(fsize-raw_offset),dsize+16); // Data size may be incorrect?
+ info->lastlast = false;
info->input = libraw_internal_data.internal_data.input;
info->linebuf[_R0] = info->linealloc;
for(int i = _R1; i<=_B4;i++)
@@ -668,6 +670,7 @@ void LibRaw::xtrans_decode_strip(const struct xtrans_params* info_common, int cu
ztable[3]= {{_R2,3},{_G2,6},{_B2,3}};
for (cur_line = 0; cur_line < libraw_internal_data.unpacker_data.fuji_total_lines; cur_line++)
{
+ info.lastlast = (cur_block == libraw_internal_data.unpacker_data.fuji_total_blocks-1) && (cur_line ==libraw_internal_data.unpacker_data.fuji_total_lines-1);
xtrans_decode_block(&info, info_common, cur_line);
// copy data from line buffers and advance
I have only one sample. But the photographer that sent me that sample showed me a case where out of 36 images he had, 4 of them appeared corrupted. I'll try to get more samples.
Thank you for sharing the sample.
In this sample
data offset (of raw data) 805904
data size 25251744
sum is 26057648
but file size is 26057632, so 16 bytes missing somewhere
Do you have only one sample, or some camera produces many such files?
If only one, this looks like some data error (camera, card, transfer). If this is systematic error it should be handled in software (e.g. ignore last bytes without error raise)
And in reverse direction in LibRaw master: https://github.com/LibRaw/LibRaw/commit/f9a69610748abe1a07362696129123eb...
:)
Now I have my fix looks like:
switch (imgdata.sizes.flip)
{
case 3://0011
case 180:
metaData.flip = Meta_data::Flip_180;
break;
case 5://0101
case 270:
metaData.flip = Meta_data::Flip_90CCW;
break;
case 6://0110
case 90:
metaData.flip = Meta_data::Flip_90CW;
break;
From first lines of raw2image_start (called by _ex):
if (O.user_flip >= 0) S.flip = O.user_flip; switch ((S.flip+3600) % 360) { case 270: S.flip = 5; break; case 180: S.flip = 3; break; case 90: S.flip = 6; break; }So:
flip/user_filp are bit-field. But if someone has specified user_flip in degrees it will work OK.
Specifically for CIFF (CRW) format camera sets rotation angle, not Orientation tag. This value is preserved after open_datastream (looks like we need to fix it for all cases in 0.18?)
For most camera which I have used the flip field means "bit-field" before actually processing (unpack()/raw2image_start()). But for EOS 10D this flag means the rotation angle. So, I would prefer not to start processing but get meaning of this field: is it bit-field or angle. Is it possible to know if it was corrected before?
Also, flip is 'bit-field', it converted from degree rotation to bits in raw2image_start() if not corrected before
imgdata.image (and rawdata.*) and imgdata.sizes are always in 'sensor dimensions', so it is alwasy unrotated and rotation/mirroring is read/set via flip/user_flip.
Rotation/mirroring is performed on output phase:
dcraw_ppm_tiff_writer()
or
dcraw_make_mem_image()
I just used
dcraw_emu -T sample.DNG
to get sample.DNG.tiff
and tiff is OK.
Have not tried python bindings, i'm not a python programmer (even at small fraction)
I used 0.17 and 0.18-beta, both provide the same (wrong) output.
When you run your test, did you also use the Python interface?
I've got 3 more images from the photographer. They were all reported as being corrupted and after the patch you sent, libraw works with all of them. Thanks!
Thanks for your email too. That would be useful for the future because some photographers do not want to post their images publicly.
Current LibRaw (0.18-beta1) works fine with this image.
What version do you use?
Thanks!
To be included in beta-2
Hi, compilation fails when openmp is enabled. My fix below:
--- a/src/libraw_xtrans_compressed.cpp
+++ b/src/libraw_xtrans_compressed.cpp
@@ -738,10 +738,11 @@ void LibRaw::xtrans_compressed_load_raw()
void LibRaw::xtrans_decode_loop(const struct xtrans_params* common_info, int count, INT64* raw_block_offsets, unsigned *block_sizes)
{
+ int cur_block;
#ifdef LIBRAW_USE_OPENMP
#pragma omp parallel for private(cur_block)
#endif
- for (int cur_block = 0; cur_block < count ; cur_block++)
+ for (cur_block = 0; cur_block < count ; cur_block++)
{
xtrans_decode_strip(common_info, cur_block, raw_block_offsets[cur_block], block_sizes[cur_block]);
}
And here it is: http://www.libraw.org/news/libraw-0-18-beta1
dcraw_make_mem_image() returns correct width/height,
It seems that for rotated images (.imgdata.sizes.flip&4) you need to use height for stride calculation. And keep in mind that you will receive rotated image with width and height swapped.
The only real recommendation is RawSpeed (if you use it): LibRaw is tested with RawSpeed/master branch, not with develop branch.
Other libs are used in very standard way, so you need jpeg8+ (or any replacement with jpeg_mem_src), latest DNG SDK (if you use it), any modern version of zlib
Thanks for the response (apologies if you got my request twice due to me thinking the forum one got lost)
That reminds me, whilst looking at the current changelog in github, I wondered if you have a recommended set of supporting library versions to build against ?
Thanks
Kevin
If you'll find another problem(s), please share the files.
You may send it to info@libraw.org (or to personal E-mail lexa@lexa.ru) if you do not wish to expose these files to public.
We plan to release 0.18-beta in a few days. Only one item in TODO remains: changelog
Thanks for the patch, Alex. I'm waiting for more images so we can properly test the patch. I'll send another update when I've done that.
diff --git a/src/libraw_xtrans_compressed.cpp b/src/libraw_xtrans_compressed.cpp index b6555d3..ec6dbb0 100644 --- a/src/libraw_xtrans_compressed.cpp +++ b/src/libraw_xtrans_compressed.cpp @@ -46,6 +46,7 @@ struct xtrans_block { unsigned max_read_size; // Amount of data to be read int cur_buf_size; // buffer size uchar *cur_buf; // currently read block + bool lastlast; LibRaw_abstract_datastream *input; struct int_pair grad_even[3][41]; // tables of gradients struct int_pair grad_odd[3][41]; @@ -143,7 +144,7 @@ static inline void fuji_fill_buffer(struct xtrans_block *info) #ifndef LIBRAW_USE_OPENMP info->input->unlock(); #endif - if(info->cur_buf_size<1) // nothing read + if(info->cur_buf_size<1 && info->max_read_size && !info->lastlast) // nothing read throw LIBRAW_EXCEPTION_IO_EOF; info->max_read_size -= info->cur_buf_size; @@ -159,6 +160,7 @@ void LibRaw::init_xtrans_block(struct xtrans_block* info, const struct xtrans_pa INT64 fsize = libraw_internal_data.internal_data.input->size(); info->max_read_size = _min(unsigned(fsize-raw_offset),dsize+16); // Data size may be incorrect? + info->lastlast = false; info->input = libraw_internal_data.internal_data.input; info->linebuf[_R0] = info->linealloc; for(int i = _R1; i<=_B4;i++) @@ -668,6 +670,7 @@ void LibRaw::xtrans_decode_strip(const struct xtrans_params* info_common, int cu ztable[3]= {{_R2,3},{_G2,6},{_B2,3}}; for (cur_line = 0; cur_line < libraw_internal_data.unpacker_data.fuji_total_lines; cur_line++) { + info.lastlast = (cur_block == libraw_internal_data.unpacker_data.fuji_total_blocks-1) && (cur_line ==libraw_internal_data.unpacker_data.fuji_total_lines-1); xtrans_decode_block(&info, info_common, cur_line); // copy data from line buffers and advanceBTW, ignoring this error for the last bytes of last line looks OK.
I've digged this data in depth: decoder really wants to see these extra bytes for the last line, EOF error looks correct for this data.
I have only one sample. But the photographer that sent me that sample showed me a case where out of 36 images he had, 4 of them appeared corrupted. I'll try to get more samples.
Thank you for sharing the sample.
In this sample
data offset (of raw data) 805904
data size 25251744
sum is 26057648
but file size is 26057632, so 16 bytes missing somewhere
Do you have only one sample, or some camera produces many such files?
If only one, this looks like some data error (camera, card, transfer). If this is systematic error it should be handled in software (e.g. ignore last bytes without error raise)
Pages