LibRaw Project Goals and Objectives

Contents

  1. Who May Be Interested in This Project
  2. Goals and Objectives
  3. LibRaw Support Principles
  4. Implemented Improvements of dcraw
  5. Planned Improvements

Who May Be Interested in This Project

The proposed project and the software products generated within this project are intended for

  • Developers of RAW converters, including current and new developments
  • Those willing to write their own graphic interface for RAW file processing
  • Developers and enthusiasts creating their own primary and auxiliary data processing algorithms, including
    • Interpolation (de-Bayer),
    • Noise reduction
    • White balance
    • Correction of aberrations and distortions
    • Color conversions (e.g., creation, editing and application of camera profiles )
    • RAW data analysis
    • Comparison of cameras and lenses
    • and so on...

Among the few existing implementations of RAW converters, those based on the dcraw utility by Dave Coffin are the most used. It is hard to find a more or less popular camera that is not supported by this utility; while the implementation of RAW data extraction in dcraw is of a very high quality.

However, developers and enthusiasts who use or are going to use dcraw for fulfilling the above-listed and similar objectives encounter a number of difficulties. First, the author of dcraw refuses to turn his product into a handy library, yet permitting anybody else to do so. Besides, dcraw contains a number of questionable features which may hinder its use without modifications, as well as instances of distortions in the photographic sense of it. As a result, once every several months, virtually all developers, part of them listed on the dcraw Web site, independently convert each subsequent release of the software into the library.

As for enthusiasts, the "entry cost" of verifying their own ideas and implementing their own algorithms is often unreasonably high: they have to either use the dcraw command line, thus being forced to use the unavoidable early processing stages, or understand the source code and maintain their own library based on it.

Thus, the inconveniences of dcraw make the developers' community quite small and halt further evolution and improvement of RAW format converters.

Goals and Objectives

We are going to create LibRaw in order to "get a library based on dcraw, only better". Thus:

  1. To "librarize" dcraw, i.e., to develop a stable and consistent API suitable for other applications (RAW converters, data analyzers, panorama stitchers, etc.).
  2. To divide processing into independent parts (groups of API calls)
    • Reading, decoding, and unpacking of RAW data: this is the main functionality of LibRaw
    • Data conversions: interpolation, white balance, etc.
    • File output of the processing results.
    The latter two groups of functions are maintained primarily for dcraw compatibility testing.
  3. To improve the procedures of RAW data retrieval and decoding (see below for details)
  4. To supply other developers with a "framework" (freeware and open-source), e.g., for experimenting with their own methods of RAW data processing (interpolation, noise reduction, white balance, etc.; some directions of the possible efforts are listed above), so that they could create their own GUI programs and interfaces without developing the entire RAW converter.
  5. To ensure easy modification for code synchronization with dcraw releases.

LibRaw Support Principles

  1. To reproduce the functionality of dcraw using its source code as the basis for our work; to achieve binary identity of results generated by dcraw and by LibRaw-based utilities, provided the same processing settings are used.
  2. To support new cameras ASAP, faster than most commercial software.
  3. To monitor future improvements in dcraw (support of new cameras, error correction, use of better algorithms) and import them from dcraw to LibRaw.

Implemented Improvements of dcraw

As of the time of writing, additional improvements to the dcraw source code have been introduced into LibRaw:

  • All global variables have been removed (while linking the thread-safe version, static variables of functions have been removed as well).
  • Thread safety. An example of its use in the multithreaded model is included into the library distribution package.
  • Data extracted from a RAW file is already somewhat structured: geometry is separated from color data. This work has not yet been completed; it will be continued simultaneously with the work on improvement of EXIF processing.
  • Retrieval of RAW data and thumbnail can be performed in two or three API calls with a very simple program interface.
  • Work with color information: color data (white balance coefficients, tone curve, etc.) has flags referencing the source of this information: retrieved from RAW data, calculated from the image itself, or taken from the code as constants.
  • ICC profile is extracted (for those RAW files that contain it).
  • The image itself and the thumbnail can be retrieved by successive calls, without reopening the file or relaunching the library.
  • The required amount of RAM is somewhat lower.
  • Work with masked pixels and black subtraction:: it is possible to turn off all data filtering and black level subtraction.
  • Processing of the black frame: data read from the black frame (or two black frames, for cameras that have two) is accessible from the calling application (this may be necessary, e.g., for suppression of certain artifacts, including banding).
  • Processing the extremes of the value range: minimum values (often referred to as black point) and maximum values (saturation point).
    • The black point is not subtracted from the data at the unpacking stage, in contrast to the implementation found in dcraw: the calling application may have its own plans, e.g., channel-wise subtraction of the minimum.
    • The maximum values are calculated on per-channel basis. Calculation of the maximum values for a given camera sample at a specified sensitivity (converter calibration) is to be done in the application that calls LibRaw.

Planned Improvements

The dcraw code needs significant modifications and additions:

  • Single-pass unpacking of RAW files containing several image variants/planes: for applicable cameras (Fuji cameras, cameras with 4-shot/16-shot modes), several planes of the same image will be extracted.
  • Processing of the EXIF/Makernote data: retrieval of large data amounts, including
    • Extraction of the maximum possible amount of color data (white balance settings, profile, tone curve, contrast settings, etc.)
    • Color data generalization: reduction of color data from different cameras "to a common denominator" in order to facilitate color processing (without loss of quality)
    • Extraction of camera data (firmware version, serial number)
    • Extraction of photographic data (lens used, focusing distance, focus points and their coordinates, etc.)

At the same time, we are not planning any modifications or extensions of the dcraw data processing code (de-Bayer, color conversions, etc.); all of this is up to the calling application. Standard processing modes included in dcraw will be preserved for an indefinite time in the set of dcraw-emulating API calls, with the exception of LCMS and libjpeg support, as well as certain processing stages of minor importance.

Comments

LibRaw to DNG

Hi there,
Will it be possible for you to provide some pointers on how to use LibRaw to get a DNG output.

Raw documentation is very sparse and its very hard to understand anything from DCRaw code.

Thanks,

I have been trying to find a

I have been trying to find a windows port of digikam and it seems like it's not there.

Is it possible to get digiKam install for windows.

Are there any other applications that user LibRaw to convert Raw files to DNG?

Thanks

Do you really want DNG

Do you really want DNG converter under Windows? AFAIK, Adobe DNG Converter is free...

-- Alex Tutubalin

The problem is with Adobe

The problem is with Adobe slow release of releases. And I bet this will remain a problem every year after photokina. New cameras will be available in the market and Adobe will pick and choose which one to support when.

An open source alternative to get to DNG would be very helpful.

Right now, I have Panasonic LX3 and its not supported. The same goes for G10 and I am sure there are lot more.

And unfortunately, a lot of ppl( including me) are tied to Adobe Photoshop.

Any help to get digiKam running on windows would be appreciated.

Thanks

LibRaw is based on dcraw

LibRaw is based on dcraw sources. In turn, dcraw uses RGB-XYZ tables from Adobe (DNG converter? Or camera raw?). So, do not expect to get good color on digikam/LibRaw until dcraw and-or Adobe converter arrives

-- Alex Tutubalin

LibRaw->DNG

Thanks. I think not having good color is a secondary issue. The first problem is to inteface LibRaw with DNGSdk. If possible, could you provide some simple sample or write-up on how to interface LibRaw with DNG SDK.

Also, by any chance is it possible to provide a Visual Studio Solution for building LibRaw.

I looked around and even though DigiKam is very promising, but running KDE is not possible for everybody. Unfortunately, a lot of ppl are hooked on Windows.

Thanks for any help you can provide.

You can use digiKam DNG

You can use digiKam DNG convertor as a reference. It uses both libraw and dng sdk.

Also, I'm not great VS2005/2008 guru. So, it is possible to provide Solution file, but I cannot be sure that it working. Makefile is much easier to test.

All CMake/Visual Studio/configure/automake and so on tools are scheduled to LibRaw 1.0. Too much things to do before

-- Alex Tutubalin

Support for Camera

Hi There,
Thanks for the pointers. I am able to compile the code in VC++ 2008. Samples are also running fine.
I noticed in the list of cameras it says panasonic lx3 but the tiff conversion results in a file with only vertical line.
The same program works fine for Pentax K10D.

Is it because of dcraw only recently supported the LX3.
If so, will it be possible to reimport DCRaw latest version. It will be a while before new cameras will come out and this step may not be needed very soon again.

Thanks,

I've tried LX3 samples from

I've tried LX3 samples from photographyblog and process them without any problems on LibRaw 0.6.1.
Anyway, there is some strange problems when processing Panasonic files (in my tests - files from FZ28) on multi-threaded LibRaw version, so I'll check it.

You also may check new LibRaw release (sorry, source code only): http://www.libraw.org/blog/libraw-062-beta1.html

-- Alex Tutubalin

We've found and fixed bug in

We've found and fixed bug in Panasonic .RW2 files processing. This bug affects only thread-safe version of library.

Could you please test LibRaw 0.6.2 on your files?

-- Alex Tutubalin

It Works!!

Yes, It works now with LX3 files. For first time I am seeing my true raw as it should be seen.
I will update you once I have a decent UI that uses libraw and it's up on sourceforge.

Thanks,

Native tool in Windows for DNG conversion with LibRaw

Hi! I've been looking for such a tool as well. Lately digikam makers pulled together the DNG writer for their solution. However, you can very well use the HDRMerge tool and it's command-line interface to achieve good quality. I did some testing, and to me it looks like the DNGs produced with this combo are faster in Lightroom to load than those produced by Adobe.

http://jcelaya.github.io/hdrmerge/
http://jcelaya.github.io/hdrmerge/documentation/2014/07/11/user-manual.html

As to my own motivation, I'm interested in getting the best possible results out of Sony A7, and currently the best results sharpness wise still come from TIFF files produced by ImageDataConverter. I hope the best result at some point will come from LibRaw. I will be switching to DNGs by LibRaw for performance now.

Unfortunately, I'm not an

Unfortunately, I'm not an Matlab user, so cannot provide significant help.

I guess, Matlab must have some interface to external libraries. LibRaw is a library, you can compile it according to Matlab requirements and use.

-- Alex Tutubalin

MATLAB and mex

You can integrate C-code in MATLAB using the mex-API. I looked at this for dcraw, but got stuck, and I found that it was simply less work to make a thin MATLAB wrapper calling dcraw from the command-line.

I remember some frustrating aspects of doing anything to the dcraw code, hope that this project makes it easier.

(the spam-filter here really hurts!)

I was able to associate the

I was able to associate the libraw.dll and libraw.h with matlab using loadlibrary function. It builds a function table and I am able to open a raw file and so on. However, I haven't gotten to the point where I can open the bayer pixel data. It has something to do with how pointers are used. In any case, I wanted to let you know it is possible to interface with Matlab.

Interesting to see in next version

There are 2 items I would like to see enhanced in the current Raw conversion libraries out there:

1. I would like extra output 32-bit float instead of kist 16-bit integer.
2. The ability to quickly extract a section of an image. Lets say I want to create a zoom tool that zooms into a section of a Raw file I would like to have a fast Raw library that only process the pixels involved in the area you are zooming in. I'm aware that raw is a stream of data but at leas a fast way of walking through the stream instead of interpolating all the pixels on its way.

32bit float data output would

32bit float data output would be very beneficial.
Export of full scene referred linear data with no clipping from 14bit chips would be great.

We hacked dcraw about a bit to reverse the scaling that's done with full highlight retention to push superbrights back in to the >1 range and export as exr.

I can see a need for that kind of a feature.

For float data output we need

For float data output we need to change entire processing pipeline to floating point.

There are no such plans for LibRaw team: we're open to contributions, but will not spend much own effort on postprocessing phase.

-- Alex Tutubalin

then

I don't think there is much point in the whole project if its not in Floating point. There are lots of good enough alternatives.

ADU values

Good afternoon. I have downloaded the version of RAWSHECK to 64-bit Windows 7, but I do not know to use the programme, because there is not any exe file. In the zipped box folder, there are files with unknown formats and I do not know to open them. I have created pictures in CR2 format and I want to view the ADU values. Help me, please.

Thank you.

could you please explain your

could you please explain your problem in more detail. Google and I knows nothing about RAWSHECK

-- Alex Tutubalin

Converter to jpg thumbnail

Hi, I have downloaded VS project and succesfully comiled with VS desktop version 2012.
I am not C programmer but VB.NET
After compilation I got a bunch of exe files.
My mission is to create small thumbnail jpg image file from raw file. That is all.
How do I do that.

Rgards,
Dusan

I'm not VB.NET programmer.

I'm not VB.NET programmer.
use simple_dcraw.cpp sample as guide (you need open_file then extract_thumb, then write thumb to memory buffer).

But I do not now how to call C++ library from VB.NET

-- Alex Tutubalin

read, average, write DNG files.

I am looking for the simplest possible solution to a simple problem: to compute the average of a number of images stored as dng files, and save it as a dng file. I have not found any ready-made app and have just spent a frustrating afternoon failing to build Adobe's dng sdk. Can you suggest something practical?

DNG is just a TIFF file with

DNG is just a TIFF file with custom tags. So, you may modify any tiff-writing code to write DNG.

-- Alex Tutubalin

Add new comment