Thanks for quick reply.
So can I assume that for 16bit RGB data the order of bytes will be:
pixel0-R-LSbyte, pixel0-R-MSbyte, pixel0-G-LSbyte, pixel0-G-MSbyte, pixel0-B-LSbyte, pixel0-B-MSbyte, pixel1-R-LSbyte, and so on?
You need to ensure you use same data types definitions (.h files) and same structure alignment in both
- your code (that calculates offsets of imgdata.rawdata members during your code compilation)
- and LibRaw library (offsets calculated by compiler at library build stage).
Thanks for the prompt reply. Sorry, I should have provided a more complete problem statement (but where do you stop?) My code tests all the different data raw data pointers in libraw_rawdata_t and they are all null. I'm testing with Canon CR2 and Panasonic RW2. I assume unpack has done its job since it returned no error and libraw_image_sizes_t.raw_pitch has (what seems like) valid data (which it didn't have before the unpack).
For anyone that reaches this topic at some point, and are facing the exact same problem, here is what seems to be the fix. At least, it seems to fixed the problem for me. As Alex described, the problem seems to be in how I built the Libraw library in my machine. I just made a brand new clone of the Libraw repo, and I built the library from source again, following the steps described in the INSTALL file: https://github.com/LibRaw/LibRaw/blob/master/INSTALL
Then, I payed close attention in the compiler flags that were used in the Makefile that was generated by automake. Then, I changed the build command in my Makefile to use the same compiler flags that were used in the Makefile that was generated by the automake. As a result, I got this Makefile:
Please make sure
1) you use same LibRaw version (.h files) for your code and for linking
2) you use same compilation options (esp. structure aligning) in libraw build and in your code build.
The symptoms you describe seem to be either 1) different versions of header files 2) or different calculated offsets due to different structure fields alignment.
Now, to answer your 2nd point, I understand that libraw_unpack should be enough. However, for some reason that I also don't understand, when I try to access the libraw->rawdata.raw_image array, or literally any of the arrays that are present inside the libraw->rawdata structure, I get a "Cannot access memory at address" error message. So apparently, they are all "not-valid pointers", even though they are not-null pointers. You can see this by looking at the screenshot that I took from a GDB session: https://github.com/pedropark99/libraw-example/blob/main/screenshot.png
1st:It is quite difficult to discuss the results of code running on a file without having that file in hand. Please share sample file you're testing with.
The library is intended (primarily) for decoding/decompressing RAW and RAW metadata. Compressed RAW data is arranged (compressed) in such a way that no gain can be obtained from using highly parallel computing (except for partial bulk operations, such as tone conversion on the curve/LUT, but for these operations there will be no gain either, since the transfer of data to/from the accelerator is slower than the LUT conversion directly on the CPU (esp. today multi-core CPU).
Postprocessing can certainly gain a lot from using parallel computing, but postprocessing is not the task of the library and the corresponding code is included there only as a minimal and imperfect example.
However, you can try to do parallel decompression of RAW data. If the result obtained is many times faster than on the CPU and you agree to provide your code for our library, we will gladly accept this contribution.
Everything related to post-processing (debayering and so on) should be done outside of LibRaw: we do not plan to extend this part of our library
Thank you so much for the quick reply and the helpful information!
I still shoot uncompressed raw on my old Sony cameras as the compressed raw format isn't lossless. I do convert everything to DNG eventually as the format I'm working with, but I do like to archive the original out-of-camera files, probably out of paranoia. :)
I'll take a look at the fields you've mentioned and see if I can build something on top of that!
Unfortunately, the information you are asking about and in the form you are asking about is not available in the library (in its direct form).
You can query the (function) name of the decoder used by calling LibRaw:get_decoder_info() and create your own table indexed by the decoder name and containing the properties you need.
In practice, there are very few uncompressed formats now: photographers prefer to save space on flash cards and camera manufacturers follow this wish. Such uncompressed files may still present in archives, but it is hard to find an uncompressed file produced by modern camera (even if camera supports it, photographer will, most likely, switch to compressed format).
If you still want to handle uncompressed formats separately: they can be tiled/striped, so you'll need to add own code that interprets this correctly.
For uncompressed/not tiled data, data starts at libraw_internal_data.unpacker_data.data_offset and
data size is libraw_internal_data.unpacker_data.data_size
These fields are protected (in C++ terms), you may need to subclass LibRaw to access these fields.
I'm not a huge fan of my steps (a few of the stages feel like major hacks), so I might re-approach this with your CMake file and diff when I have time in the future. Thanks again for sharing!
our release policy is published on this site frontpage: https://www.libraw.org/#updatepolicy
Hello LibRaw,
Could you advise when Sony ILCE-7CM2 (A7C II) will be supported in release?
We cannot find it in 0.21.3 release but in 202403 snapshot.
Thank you very much in advance.
Thank you!
You can assume that this is pixel0R, pixel0G, pixel0B.... each value is 16-bit unsigned integer datatype in your CPU byte order.
Thanks for quick reply.
So can I assume that for 16bit RGB data the order of bytes will be:
pixel0-R-LSbyte, pixel0-R-MSbyte, pixel0-G-LSbyte, pixel0-G-MSbyte, pixel0-B-LSbyte, pixel0-B-MSbyte, pixel1-R-LSbyte, and so on?
Yes, data[1] is to make it 'in place array' (no additional indirection level)
It is allocated via this code:
Sorry, that turned out to be a weird JNA issue rather than with LibRaw.
All LibRaw::dcraw_process() return codes are negative.
A quick look at the source code showed no deviations from this rule.
Could you please provide raw file you're testing with
Just to conclude this thread: I found my bug - a missing member in one of the makernotes classes. Now getting correct raw data in raw_image.
Also you may start with simple C-language sample (just print non-zero pointer) to make sure it works on C side.
You need to ensure you use same data types definitions (.h files) and same structure alignment in both
- your code (that calculates offsets of imgdata.rawdata members during your code compilation)
- and LibRaw library (offsets calculated by compiler at library build stage).
Thanks for the prompt reply. Sorry, I should have provided a more complete problem statement (but where do you stop?) My code tests all the different data raw data pointers in libraw_rawdata_t and they are all null. I'm testing with Canon CR2 and Panasonic RW2. I assume unpack has done its job since it returned no error and libraw_image_sizes_t.raw_pitch has (what seems like) valid data (which it didn't have before the unpack).
Sorry, know nothing about JAVA, JNA, etc.
As mentioned in documentation: 'After call to unpack() only one of these fields is non-NULL.': https://www.libraw.org/docs/API-datastruct-eng.html#libraw_rawdata_t
Which one: depends of raw file you're unpacking.
For anyone that reaches this topic at some point, and are facing the exact same problem, here is what seems to be the fix. At least, it seems to fixed the problem for me. As Alex described, the problem seems to be in how I built the Libraw library in my machine. I just made a brand new clone of the Libraw repo, and I built the library from source again, following the steps described in the INSTALL file: https://github.com/LibRaw/LibRaw/blob/master/INSTALL
Then, I payed close attention in the compiler flags that were used in the Makefile that was generated by automake. Then, I changed the build command in my Makefile to use the same compiler flags that were used in the Makefile that was generated by the automake. As a result, I got this Makefile:
There are still some issues in the results, but it is already much better with this change.
Ok, thank you very much!
Please make sure
1) you use same LibRaw version (.h files) for your code and for linking
2) you use same compilation options (esp. structure aligning) in libraw build and in your code build.
The symptoms you describe seem to be either 1) different versions of header files 2) or different calculated offsets due to different structure fields alignment.
Sorry, my bad. To make the files easily available, I just published the code and the image file inside a GitHub repository: https://github.com/pedropark99/libraw-example
Now, to answer your 2nd point, I understand that libraw_unpack should be enough. However, for some reason that I also don't understand, when I try to access the
libraw->rawdata.raw_image
array, or literally any of the arrays that are present inside thelibraw->rawdata
structure, I get a "Cannot access memory at address" error message. So apparently, they are all "not-valid pointers", even though they are not-null pointers. You can see this by looking at the screenshot that I took from a GDB session: https://github.com/pedropark99/libraw-example/blob/main/screenshot.pngThank you anyway for your attention!
1st:It is quite difficult to discuss the results of code running on a file without having that file in hand. Please share sample file you're testing with.
2nd: generally, libraw_init and libraw_unpack is enough. libraw_raw2image copies pixels from imgdata.rawdata....array to the imgdata.image array.
rawdata is described here: https://www.libraw.org/docs/API-datastruct-eng.html#libraw_rawdata_t
We have no requirements or wishes for how applications display the version of our library they are using.
Version defines for 202403 snapshot are:
#define LIBRAW_MAJOR_VERSION 0
#define LIBRAW_MINOR_VERSION 22
#define LIBRAW_PATCH_VERSION 0
#define LIBRAW_VERSION_TAIL Devel202403
The library is intended (primarily) for decoding/decompressing RAW and RAW metadata. Compressed RAW data is arranged (compressed) in such a way that no gain can be obtained from using highly parallel computing (except for partial bulk operations, such as tone conversion on the curve/LUT, but for these operations there will be no gain either, since the transfer of data to/from the accelerator is slower than the LUT conversion directly on the CPU (esp. today multi-core CPU).
Postprocessing can certainly gain a lot from using parallel computing, but postprocessing is not the task of the library and the corresponding code is included there only as a minimal and imperfect example.
However, you can try to do parallel decompression of RAW data. If the result obtained is many times faster than on the CPU and you agree to provide your code for our library, we will gladly accept this contribution.
Everything related to post-processing (debayering and so on) should be done outside of LibRaw: we do not plan to extend this part of our library
Hi Alex,
Thank you so much for the quick reply and the helpful information!
I still shoot uncompressed raw on my old Sony cameras as the compressed raw format isn't lossless. I do convert everything to DNG eventually as the format I'm working with, but I do like to archive the original out-of-camera files, probably out of paranoia. :)
I'll take a look at the fields you've mentioned and see if I can build something on top of that!
Cheers,
Marcus
Unfortunately, the information you are asking about and in the form you are asking about is not available in the library (in its direct form).
You can query the (function) name of the decoder used by calling LibRaw:get_decoder_info() and create your own table indexed by the decoder name and containing the properties you need.
In practice, there are very few uncompressed formats now: photographers prefer to save space on flash cards and camera manufacturers follow this wish. Such uncompressed files may still present in archives, but it is hard to find an uncompressed file produced by modern camera (even if camera supports it, photographer will, most likely, switch to compressed format).
If you still want to handle uncompressed formats separately: they can be tiled/striped, so you'll need to add own code that interprets this correctly.
For uncompressed/not tiled data, data starts at libraw_internal_data.unpacker_data.data_offset and
data size is libraw_internal_data.unpacker_data.data_size
These fields are protected (in C++ terms), you may need to subclass LibRaw to access these fields.
No worries Alex, thanks for the reply and thanks joostn for your CMake steps!
I had started on a process from scratch last week after seeing Alex's reply and put up a gist for the macOS side here before I saw joostn's reply: https://gist.github.com/mikamikem/4e826bb9cf5beacd86ba00e42ba0b115#file-...
Linux steps here: https://gist.github.com/mikamikem/6cf1530b2fa7b011eadd46e19ed9aab7#file-...
I'm not a huge fan of my steps (a few of the stages feel like major hacks), so I might re-approach this with your CMake file and diff when I have time in the future. Thanks again for sharing!
Not sure it will be available in the next snapshot, sorry.
Ah yes, the problem is fixed!
Will this be fixed in LibRaw? I guess I'll have to wait for the next public snapshot?
Thanks
Joost
Pages