Yeah, the MLV format is designed to have frames appear out-of-order to be able to optimize memory block and write sizes.
Nevertheless it is simple to overcome this - there is a .idx file which contains the file numbers and offsets to VIDF blocks.
These file can be generated using mlv_dump or with the in-camera player.
If you can send me the direct link to the API the mlv routines have to fulfill, i can prepare what is necessary to index the files on startup etc.
typedef struct {
uint16_t fileNumber; /* the logical file number as specified in header */
uint16_t empty; /* for future use. set to zero. */
uint64_t frameOffset; /* the file offset at which the frame is stored (VIDF/AUDF) */
} PACKED mlv_xref_t;
typedef struct {
uint8_t blockType[4]; /* can be added in post processing when out of order data is present */
uint32_t blockSize; /* this can also be placed in a separate file with only file header plus this block */
uint64_t timestamp;
uint32_t frameType; /* bitmask: 1=video, 2=audio */
uint32_t entryCount; /* number of xrefs that follow here */
//mlv_xref_t xrefEntries; /* this structure refers to the n'th video/audio frame offset in the files */
} PACKED mlv_xref_hdr_t;
Thanks for info.
I've take a quick look into the specs. It looks like, there is no way to locate Nth frame without reading all previous frames, right?
In LibRaw paradigm it will result into very inefficient applications: for movie apps one need API like get_next_frame(), while photography-oriented API (with one-two frame(s) per file) is entirely different.
Without stream-oriented API, support for MLV in LibRaw will be useless.
The binary libraw.dll is compiled using nmake -f Makefile.msvc (this Makefile is also included into distribution) using
MSVC 2010 with current patches/serivice packs.
You may easily recompile this DLL with compiler flags you want to use
I use the pre-compiled DLL and I have settled with calling FNINIT right before I call the procedure with LibRaw code and it works perfectly.
I guess the compiler made some assumptions (that it shouldn't have?) and the generated code is probably MMX because that and FPU do no mix well without clearing flags in between.
Not too many people will experience this problem as assembly is falling out of use so I don't know if and where this should be documented in LibRaw or if VS2010 can be told to optimize for SSE (it does not have this incompatibility) instead for MMX which I suspect is the case.
There is no ASM/intrinsinc code in LibRaw. Just pure C++.
The problem may relate to different MMX/SSE conventions (compiler settings) in your app and in LibRaw.
Do you recompile LibRaw or use pre-compiled DLLs from this site?
As you are familiar with DCRAW and changes you've made, does it use any MMX or FPU asm code?
In my code I use FPU which is correctly initialized using the FNINIT instruction. I have found it that if I write
FNINIT or EMMS at the end of my FPU block the problem goes away and LibRaw functions as intended.
This is why I suspect some LibRaw or more likely dcraw block is not initialized with FNINIT for FPU or EMMS for MMX.
I have noticed this because the first time I use LibRaw everything works fine but the second time I try to load an image that is when everything goes green or red. Then I started commenting out code suspecting of some memory corruption until I got to FPU.
Do you use same compiler settings (esp. structure fields alignment) in LibRaw compilation and your app compilation.
If structures are aligned differently, you may think you use params.half_size, but really change other structure field.
Try to use default settings (i.e. no raw.imgdata.params.... assignment).
I've tried both your samples without any problem (using dcraw_emu LibRaw sample with -mem switch to use open_buffer() interface)
Initially I thought the false color is my wrong and that is why I have put in the dcraw_ppm_tiff_writer("test.ppm") line in the code and the output is corrupted.
I am now trying the C interface and MingW. I will soon know if the problem persists.
P.S. Please consider exposing copy_mem_image() in C API, it will save a lot of memory copying from one buffer to another.
I later discovered that my problem was due to colour space. I got more vibrant colors once I specified the color space to sRGB (My camera is set to sRGB) - This matched the colours reproduced from LightRoom / Photoshop.
And another thing was that my pointers were incorrect. imgD.params.use_camera_wb=1; was not finding it's way back into the LibRaw RawProcessor structure.
Nikon's HIGH-ISOs are specified not in 'common EXIF' tag (tag id=2), but in special tag (tag id = 37).
LibRaw 0.16-beta1 already contains this change, so it should work fine).
Could you please upload sample file to somewhere (Dropbox or so)?
According to gcc 4.1 manual, __sync_add_and fetch is builtin for this version, so it should works on 4.2 too.
I have not old Xcode on hands, only Xcode4/Xcode5 (llvm based), but this code compiles and links fine on my FreeBSD with gcc 4.2.1 (in both 32-bit and 64-bit mode):
int main()
{
int _exitflag=0;
if( __sync_add_and_fetch(&_exitflag,0))
return 1;
return 0;
}
May be, you need to specify processor target (-march=core2duo because OS X requires at least core2duo)
Yes, in Canon data these pixels used for black level calibration.
On other cameras (e.g. Nikons) these pixels may be used for banding noise elimination
Yeah, the MLV format is designed to have frames appear out-of-order to be able to optimize memory block and write sizes.
Nevertheless it is simple to overcome this - there is a .idx file which contains the file numbers and offsets to VIDF blocks.
These file can be generated using mlv_dump or with the in-camera player.
If you can send me the direct link to the API the mlv routines have to fulfill, i can prepare what is necessary to index the files on startup etc.
see https://bitbucket.org/hudson/magic-lantern/src/tip/modules/mlv_rec/mlv.h...
typedef struct {
uint16_t fileNumber; /* the logical file number as specified in header */
uint16_t empty; /* for future use. set to zero. */
uint64_t frameOffset; /* the file offset at which the frame is stored (VIDF/AUDF) */
} PACKED mlv_xref_t;
typedef struct {
uint8_t blockType[4]; /* can be added in post processing when out of order data is present */
uint32_t blockSize; /* this can also be placed in a separate file with only file header plus this block */
uint64_t timestamp;
uint32_t frameType; /* bitmask: 1=video, 2=audio */
uint32_t entryCount; /* number of xrefs that follow here */
//mlv_xref_t xrefEntries; /* this structure refers to the n'th video/audio frame offset in the files */
} PACKED mlv_xref_hdr_t;
Thanks for info.
I've take a quick look into the specs. It looks like, there is no way to locate Nth frame without reading all previous frames, right?
In LibRaw paradigm it will result into very inefficient applications: for movie apps one need API like get_next_frame(), while photography-oriented API (with one-two frame(s) per file) is entirely different.
Without stream-oriented API, support for MLV in LibRaw will be useless.
Great to see that Blackmagic raw files are now supported.
How do you perceive a possible implementation of the Magic Lantern MLV file format?
http://www.magiclantern.fm/forum/index.php?topic=7122.0
1. dcraw is utility, LibRaw is library. Generally, you need to change your source code
2a. Yes, if compiled with OpenMP support
2b. Yes in some cases
Hi Iliah,
I'm new here. :-) Some questions about LibRaw:
1. It has calling compatibility with DCRaw? (IOW can I switch from DCraw to LibRaw without any modifications in code?)
2.a. It uses all the cores of a multi-core CPU?
2.b. It is faster than DCRaw?
Thanks in advance
gpsdata is just dump of TIFF(EXIF) sub-IFD structure.
In that case, I presume that the
unsigned[32]holds the [tag ID]-[type]-[length]-[data]. Am I right?Thank you :-)
GPS data is not parsed by LibRaw in any way. It just stored and copied into output image.
GPS block is described in Exiftool docs: http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/GPS.html
The binary libraw.dll is compiled using nmake -f Makefile.msvc (this Makefile is also included into distribution) using
MSVC 2010 with current patches/serivice packs.
You may easily recompile this DLL with compiler flags you want to use
I use the pre-compiled DLL and I have settled with calling FNINIT right before I call the procedure with LibRaw code and it works perfectly.
I guess the compiler made some assumptions (that it shouldn't have?) and the generated code is probably MMX because that and FPU do no mix well without clearing flags in between.
Not too many people will experience this problem as assembly is falling out of use so I don't know if and where this should be documented in LibRaw or if VS2010 can be told to optimize for SSE (it does not have this incompatibility) instead for MMX which I suspect is the case.
There is no ASM/intrinsinc code in LibRaw. Just pure C++.
The problem may relate to different MMX/SSE conventions (compiler settings) in your app and in LibRaw.
Do you recompile LibRaw or use pre-compiled DLLs from this site?
Hey Alex, I have found the problem!
As you are familiar with DCRAW and changes you've made, does it use any MMX or FPU asm code?
In my code I use FPU which is correctly initialized using the FNINIT instruction. I have found it that if I write
FNINIT or EMMS at the end of my FPU block the problem goes away and LibRaw functions as intended.
This is why I suspect some LibRaw or more likely dcraw block is not initialized with FNINIT for FPU or EMMS for MMX.
I have noticed this because the first time I use LibRaw everything works fine but the second time I try to load an image that is when everything goes green or red. Then I started commenting out code suspecting of some memory corruption until I got to FPU.
Do you use same compiler settings (esp. structure fields alignment) in LibRaw compilation and your app compilation.
If structures are aligned differently, you may think you use params.half_size, but really change other structure field.
Try to use default settings (i.e. no raw.imgdata.params.... assignment).
I've tried both your samples without any problem (using dcraw_emu LibRaw sample with -mem switch to use open_buffer() interface)
Hi Alex,
Initially I thought the false color is my wrong and that is why I have put in the dcraw_ppm_tiff_writer("test.ppm") line in the code and the output is corrupted.
I am now trying the C interface and MingW. I will soon know if the problem persists.
P.S. Please consider exposing copy_mem_image() in C API, it will save a lot of memory copying from one buffer to another.
Could you please specify, where you get incorrect colors: in .ppm-file written by LibRaw or in your Bitmap?
I discovered that the line
imgD.params.user_flip = 0;was an unnecessary line. Things worked after removing it from my code.Also I forgot to include
and
in the original code.
I later discovered that my problem was due to colour space. I got more vibrant colors once I specified the color space to sRGB (My camera is set to sRGB) - This matched the colours reproduced from LightRoom / Photoshop.
And another thing was that my pointers were incorrect.
imgD.params.use_camera_wb=1;was not finding it's way back into theLibRaw RawProcessorstructure.Thank you :-)
Bharath
use_camera_wb uses camera White Balance ('As Shot').
If you need to specify your own white balance, use user_mul[] coeffs
P.S. All user comment/posts are published only after approval due to heavy comment spam
Comment update: use our RawDigger application (shareware)
Sure, raw analyzing application is already planned.
Due to vacation schedule, it is unrealistic to expect such application before August.
Oh, you react really fast. Very much thanks.
So, can I get the right iso values from LibRaw?
I don't have a dropbox account, I upload 2 files to baidu and the link is below.
http://pan.baidu.com/share/link?shareid=1813243432&uk=1865652979
These 2 files are taken by Nikon D7100 @iso12800 and iso25600. Hope they are helpful.
Nikon's HIGH-ISOs are specified not in 'common EXIF' tag (tag id=2), but in special tag (tag id = 37).
LibRaw 0.16-beta1 already contains this change, so it should work fine).
Could you please upload sample file to somewhere (Dropbox or so)?
?
Use nmake -f Makefile.msvc
According to gcc 4.1 manual, __sync_add_and fetch is builtin for this version, so it should works on 4.2 too.
I have not old Xcode on hands, only Xcode4/Xcode5 (llvm based), but this code compiles and links fine on my FreeBSD with gcc 4.2.1 (in both 32-bit and 64-bit mode):
int main() { int _exitflag=0; if( __sync_add_and_fetch(&_exitflag,0)) return 1; return 0; }May be, you need to specify processor target (-march=core2duo because OS X requires at least core2duo)
Yes, in Canon data these pixels used for black level calibration.
On other cameras (e.g. Nikons) these pixels may be used for banding noise elimination
Pages