Two Paths Leading Nowhere

Why are photographers compelled to suffer the discord of RAW formats?!

During the last 10 to 15 years, digital photography forced the film out of nearly all the domains. End users purchased hundreds of millions of digital cameras; and that is not including the cameras sold integrated with cellular phones. Such a huge industry can't exist without standards and such standards appear to exist. They cover the storage media (various flash cards), and image format which happens to be JPEG. Currently JPEG is the most widely used image format and its image quality and size satisfy the overwhelming majority of users.

However it is not always what professionals want. By professionals we do not mean just professional photographers. The list includes designers, pre-press staff, archivists, photo banks and many others. It often happens that JPEG format is also deemed less than appropriate by advanced amateurs. That is why nearly all digital cameras that are positioned by manufacturers as professional or semi-professionals models (as well as all current dSLR cameras) suggest an alternative format, the so-called RAW. For a casual onlooker it may appear that RAW is also some kind of a standard format that delivers better quality quality for pros.

This small article is to show that the matter is much more complicated. At the current stage the situation with RAW format is not just bad but really dreadful and continues to spiral downwards rapidly. This affects mostly professionals while less demanding amateurs simply enjoy the progress of digital.

RAW and JPEG: what is the difference?

When a photo is saved in JPEG all the processing is done in the camera before the image file is written to the storage media. On the contrary, when we shoot in RAW mode the data recorded is pretty much unchanged data read from the camera sensor.

Using classical photography terminology JPEG is pretty much a finished product, like an exposed Polaroid film is; while RAW is more like a latent, hidden image on a film before the film is developed. Such development for RAW files is often called RAW conversion. The programs used to perform such RAW conversion are often referred to as RAW converters. However there is a huge difference between undeveloped film and RAW data. The film can be processed only once. RAW data can be processed as many times as necessary using different RAW converters and experimenting with settings until the desired result is achieved. Of course in order to record an image in JPEG format the camera itself performs the necessary RAW conversion. The RAW converter is a part of camera intellect. In case the conversion performed in the camera itself turns to be less than satisfactory because of a poor contrast, bad colour, plugged shadows, blown out highlights, moir or any other reason it is very difficult and often impossible to recover the image. This is because the image, after conversion to 8-bit bitmap suitable for viewing, is already stripped of a lot of initial image data that was captured by the camera in RAW. This information is gone, lost past recovery. On top of that, saving it in a lossy format such as JPEG further complicates adjusting the image during post-processing.

While recording data in RAW format no digital processing is normally applied. The conversion of RAW data for viewing and printing is performed after the shooting using a powerful computer with a decent monitor. This allows to employ much more complex algorithms of RAW conversions, to monitor the process of the conversion, and to verify the results of the conversion visually. As a rule, this allows substantially better quality of the image; moreover, the conversion parameters can be tuned to a good degree without running into posterization, excessive noise levels, and other artefacts, that is without destroying the image.

Processing of RAW files demands certain additional skills and habits as well as additional time. This is why RAW format is used by a minority of photographers, mostly by advanced armatures and professionals and mostly when the quality of the result is more important that the speed of getting to it; or, when the shooting conditions do not allow the use of JPEG because of its limited ability to maintain dynamic range as compared to RAW. Another important case for RAW is when the customer explicitly demands for it. Quite often, RAW is used as a kind of a safety net. However it should be mentioned that the minority we referred to above are millions of users around the world.

The benefits that that are presented when using RAW format are by much overshadowed by the absence of well thought-out standards for the format. Because of this we deal with tens (or even hundreds depends on how you count) of varieties of this format.

Data formats and compatibility

Photography is one of those industries where much depends on the transparency and one-to-one biunic data exchange between the players. Sometimes these expectations are met, especially when a well-established formats such as JPEG, TIFF, EPS and others are used in a consistent workflow. But that is definitely not the case with RAW. For instance, take a photographer who has completed the shoot in RAW, and then browsed through the results and finally selected the keepers. Obviously this photographer is using the RAW converter of choice to evaluate the images. Unfortunately, selected RAW files can not be blindly submitted to the customer or to the pre-press bureau. The other party may be using a different RAW converter or a different version of the same converter, hence, the results of their conversion may be different to the point of being unacceptable and that is even if the photographer submits conversion parameters with the RAW file.

As of today we do not have any RAW standards that are recognized by all parties involved. Historically, the RAW format is defined by the camera s manufacturer. In most cases some extension of TIFF format is used. All camera manufacturers add their own extensions. Some manufacturers use several incompatible RAW formats. Sometimes it goes as far as one camera can record data in several incompatible formats.

Conveniently, the formats are not documented publicly. Some manufacturers enforce additional measures to protect data, for example, Nikon ciphers some data fields but this is not the only case of data concealment.

The manufacturers explain why they prefer not to disclose the RAW formats. Usually their explanations add up to the following simple reasons:

  • Reserved and new data fields can reveal some trade secrets to competition.
  • Quite often the data fields are added just in case ; as soon as the camera design allows to get this information it should be preserved and it may be used later to improve RAW conversion. Such fields include for example diagnostics and service fields. To document all these fields means to take responsibility for their contents and to maintain their presence in future releases of firmware and even future cameras.
  • To open a format means to trigger an unnecessary public discussion. Let's say users can access the information that registers the focusing distance. The immediate reaction of certain users will be to get a ruler to check the focusing accuracy to motivate his claims that the camera doesn t focus correctly. In the recent case over the out-of-focus problems with 1D MkIII moral damage and financial losses of Canon could be much worse if only the respective field in RAW data would be officially documented. And of course such claims would not be limited to just Canon.
  • Some camera manufacturers are trying to get additional money out of RAW converters. Not long ago native RAW converters had no competition at all to the extent of monopolizing the market. Of course they prefer to maintain a competition at minimum. It is not unusual to hear from camera manufacturers that only their converters do the justice to their cameras while third party converters only compromise their cameras decreasing image quality, distorting colours and sacrificing resolution and on top of that adding noise.

Manufacturers claim sometimes that encoding of data fields is done in the best interest of the users, that only such encoding allows to ensure data integrity and also to prove authenticity and authorship of the original shot. Those of our readers who are familiar with the modern state of cryptography will surely smile here. It looks like the manufacturers are not overly concerned with providing us with convincing and satisfactory arguments.

All other parties except the manufacturers are genially interested in open formats as well as in reducing the current manifold of formats.

  • Photographers want to process the images taken with different cameras through only one or two standard processing workflows (as it was with films). Photographers benefit from the competition between the developers of different image processing programs. With the competition there is a hope for better processing quality and lower price for image processing programs. Another important consideration for photographers is an interaction between different programs to allow using best features of each of those.
  • Photo labs today can t accept RAW files for batch processing and printing at all. In the absence of the standard processing routings printing from RAW can be done only manually with absolute minimum of automation.
  • Program developers are also suffer from multiple formats and lack of documentation. They spend an awful lot of time studying alien formats and decoding the meanings of the fields. Such a waste of time and labour could be easily avoided with a little help and good will from the side of manufacturers. In the absence of such support from the camera makers their new cameras remain unsupported by the third party products for many months.
  • Archivists, in the broad sense of this word, and that includes photo banks, advertising agencies and individual photographers, can t sleep well. The pandemonium of formats gives them nightmares. They need to store not only the RAW images but also the programs that can open those files, manuals for those programs, their own notes on using the programs and the sequences of user actions needed to set processing parameters to render the necessary result. Today we have at least one case when the compatibility between versions was lost. Processing parameters, if set in an older version of the program, are ignored by a newer version. Sometime setting those parameters in an older version can cause a crash of a newer version. This is because the RAW format was changed between those two versions. We mean Nikon Capture here. Maybe this is not the only case. We were not running especial investigation of this problem because even one case is more than enough.

The speed of development of digital cameras doesn t allow creating such a universal data format that would last forever. However the chaos that exists today is not inevitable. One of the reasons for the current situation is that distinct and agreeable attempts of introducing standards are absent. The photo industry hasmodern photography always enjoyed diversity, sometimes even too much of it. New film formats emerged and vanished (828, 110, 126, APS, disk film); different recipes of film processing felt into decay or went into oblivion (Polacolor, C-22, K-14, E-4). Many are not aware of the reasons of such excessive diversity (and by the way, such diversity was caused not only by economical or technological factors, but to a certain extent raison d tre was just an attempt to hook the consumer up and to get some additional revenue from selling the rights to use the format or the process to other manufacturers). But pretty much everybody knows the results: the archives of images taken with a use of ill-fated formats can be maintained only by professional archivists but not by the individuals or small companies. It would be just horrible if the current archives of digital photos will follow the pattern. It is even more so if we are taking into account that during the last 10 years the amount of photos captured nearly equals the amount captured during previous 30 years.

Data, metadata and meanings

One of the most important achievements of modern photography is storing not only the image data but metadata as well.

  • Data is the image, captured by the camera. That is, it contains information about the level of brightness for each sensel. If the image is recorded in RAW format, very little is done to this data (usually only normalization and some noise reduction are applied). If the image is recorded as JPEG data is processed through multiple color and tone corrections, and also sharpened.
  • Metadata is the information about the image. It contains exposure parameters, time the shot was taken, possibly geographical location of the shot, information on lighting conditions (white balance), make, model, and serial number of the camera, information about the lens, and so on. The description of data format (parametadata) should include; details of the method used to store image data (bit width, compression scheme, etc.) but also define metadata in full detail.

Parametadata (that is data format of a RAW-file) is exactly that, which brings meaning to stored bits sequences ( this field contains focal distance expressed in one tenth portions of inch). Since manufacturers do not document format, the quest for bits meaning is to be fulfilled by hackers (in a good sense of this word), who, using different methods, make their own definitions of formats (we will talk about it hereinafter).

With a view of a further discussion lets divide data and metadata into the following groups:

  1. Those which are necessary to get a good quality image from RAW-file: the manufacturer, the camera s model, light sensitivity while taking the shot, image size, white balance data, the use of flash while shooting, as well as some other critical parameters, and of course, the image data that is the brightness map registered by sensor.
  2. Those, which might be used while processing a RAW-file: camera s presets (contrast-saturation-tone curve-sharpening-colour space), optics, and focusing parameters.
  3. Those, which are not necessary for processing but useful for demonstrating, cataloguing and search: like date and time, GRS coordinates, author, description of photo etc.

One cannot say that no standard for metadata exists, there is an EXIF standard and the most of cameras manufacturers follow it. But EXIF, which in a first place was established to support ready, viewable images, describes fields prerequisite for cataloguing (the third group in our classification) and provides no help to the RAW-processing software developers.

It is also not true to say that data and metadata are completely not documented. They are, but in accordance with the grievous joke affirming that 'FreeBSD kernel is very well documented, unfortunately it is all on C '.

  • Data and some part of metadata are documented in well-known program dcraw by Dave Coffin, the program, which currently supports (that is able to unpack) formats of 312 of digital cameras.
  • Metadata are documented in the ExifTool program by Phil Harvey. This program deals with a much broader spectrum of information than just the EXIF. The program also recognizes and deciphers a number of utility fields, among them those which some converters include in RAW file if the file is not just out of camera, but modified and saved in the raw converter.

It is interesting to mention that the size of program code of ExifTool exceeds the size of program code of dcraw by nearly a whole order of magnitude (75 thousands of lines against 8 thousands). This proportion quite adequately reflects the ratio of laboriousness of data deciphering to metadata deciphering: metadata are much more diverse.

Of course this documentation is not enough. Despite of all hackers efforts, mistakes happen and completeness of description is far from perfect. Sometimes the correct deciphering of a data field of some camera becomes possible only when this camera has already been discontinued. As a result even developers of RAW processing software can t declare with any degree of certainty that they do everything in a correct way. Funny consequence is that, any attempts to compare the quality of RAW processors based on 1 or 2 examples, are completely meaningless.

Revolutionary situation in digital photography

According to herein-above in the digital photography industry a revolutionary situation is emerging in compliance with its definition given by nobody else but Vladimir Lenin himself:

Photographers (and the whole industry, which uses the results of their work) cannot live as of old: the variety of non-documented formats suits nobody, especially taking into account that the quantity of new modifications of formats is increasing nearly exponentially.

Cameras manufacturers cannot rule over as of old: despite of all their efforts, including (and mainly) their attempts to conceal information, converters produced by independent developers prevail by users number and sometimes even deliver higher quality results compared to native converters.

It is known that development of a revolution situation into a revolution depends on the existence of party, which is ready and capable of taking a lead of a struggle.

And here the reasonable question emerges: how is anything at all works in such a chaos?

Developers mainly use 2 approaches, decreasing the level of entropy a little:

  1. Some programs support only a very limited number of data formats, therefore dramatically simplifying the problem.
  2. If the author of a program declares that his program provides support for the majority of data formats, it means that most probably he is using dcraw source texts either as the ready solution or as a documentation. Among others who uses this approach is such a major player as Adobe. It is nothing less but amazing that such a huge industry largely depends on just one person and 8 thousand lines of code written by him.

It is easy to see that the both methods are having no prospects, especially from strategic point of view.

Adobe DNG

The DNG format was presented by Adobe in September 2004 as a universal format of a digital negative , intended for the eternal archive data storage. The specification DNG 1.0 was poorly thought-out, and in a half a year Adobe presented the specification DNG 1.1. Together with the description of its format DNG SDK was released. Unfortunately this DNG SDK cannot be considered as anything but a run-around : easy-to-read documentation, useful examples as well as program templates are virtually ascent.

Before we move on, let's check on Adobe s statements: the archive properties and the universality.

Is it really archival?

Let s make a very simple experiment: we will try to imitate the situation which could take place 3 years ago. To make this experiment we will take a source RAW-file from the old enough camera (Canon Powershot G6) and convert it into DNG with an old version of the Adobe converter. To check archival properties we ll convert both files source RAW and its derivative DNG with the same presets into bitmapped RGB-format using current version of Adobe Camera Raw v.5. Let s have a look at the difference between the results (photo 1). Visual difference between conversions of both files is small and it is highly possible that it won t be noticeable when printed in a magazine. But straightforward subtraction shows that the difference exists.

It is rather difficult to consider a format to be an archival while it does not provide the identity with the source given that the archival file and the source have been processed equally.

Is it universal?

To check universality let s perform the inverse operation: we ll take a shot with a current camera (Canon 1D Mark III), convert it into DNG using the modern version of a DNG-converter, and try to feed it to an old version of Adobe RAW converter (ACR), which does not know this camera. This experiment is quite topical, and here is why: the support of new cameras in Adobe Photoshop CS2 has been discontinued, but not everybody is ready to pay for upgrade to Adobe Photoshop CS3 or CS4 given that these versions do not provide any distinct advantages to a particular user.

It has been found that the version 2.4 of Camera Raw does not open the file at all, while versions 3.x open it but results of conversions of RAW and derived DNG into RGB (photo 2) differ even more than it was in the previous experiment.

Shortcomings and evolution of DNG

The reason for the both above-mentioned reciprocity failure is that not enough metadata is specified in DNG format specifications. At every stage of the DNG progress Adobe unpack and standardize only the metadata which is necessary to support their current conversion methods. All other metadata even if stored is still present in the initial (non-documented) form. The level of metadata standardization in DNG is not enough for the goal (universal archival format). Gradually Adobe are modifying DNG, discovering information content of metadata and accordingly adding to the specifications, but that metadata have been ignored in previous versions of converters. If the file had been converted by one of these previous versions, some data might have been lost forever (as it was shown above).

The specification DNG 1.2, which was released several months ago, contains some additional fields of metadata colour data, but since they are intended mainly to support Adobe products, they have been added in a form as they are used by Camera Raw and Lightroom. This data has no relation to source RAW formats and hence it is artificial. Thus, DNG more and more becomes the internal format of the company which have developed it.

The DNG format does not help developers to support non-standard sensors (such as Foveon, Fuji Super CCD SR having 2 different images in one shot etc.). Of course, it is not difficult to invent the way to store non-standard data, but consequently such data requires non-standard algorithms. Unfortunately, those are not accounted for by DNG.

At the same time some manufacturers (Panasonic, Leica, Samsung) have started to use the DNG format as the output format of their cameras, though it does not prevent them from recording non-documented metadata, since there is a special place assigned for such undocumented tags in DNG specifications.

One can easily see DNG as one more RAW format. In this sense DNG is a little bit better than all others since some fields are somehow documented. But it is absolutely impossible to use DNG as the universal archival format , and one can see it from the simple experiments we offered here. Moreover, the acceptance of DNG in its current state as the standard leads to the situation when the method of conversion used by Adobe is also imposed, though implicitly.

OpenRaw

In year 2005 the OpenRaw initiative emerged. In fact it boiled down to the call to cameras manufacturers to publish specifications of their respective RAW formats. This call was ignored altogether, despite the fact that well-respected people were holding polite and slow negotiations in accordance with all the rules of the Japanese etiquette with very influential managers of the leading manufacturers of digital photo equipment.

However, suppose that these manufacturers turned to be kind enough to publish all their internal raw cuisine, even if in the scope that was already known. Would it be of any help for raw converters developers? Our opinion is not too much; due to (traditionally) a couple of reasons:

  • To program the processing of all data formats is a lot of work. Dave Coffin has been involved in it for more than 10 years, Phil Harvey about 5 years. Of course, given you have the descriptions it is no need to hack any more, which would have reduced the amount of work, but even reduced volume is still exorbitant high.
  • In fact one needs not a description of all and every bit of a format but only descriptions of fields plus instructions of what to do with all that jazz. Unfortunately the founders of OpenRaw did not even think about asking for such instructions.

There would be no reason to mention OpenRaw initiative if not for the intense PR campaign held 3 years ago. The campaign was successful and now many think (unfortunately, mistakenly) that there is such a format as OpenRaw.

Who is to be blamed for and what is to be done?

All the problems of the photo industry mentioned can be attributed, among other reasons, to the fact that the list of requirements for RAW data has never been seriously and openly discussed. As a result in every particular case a subset of metadata interpreted by a converter reflects an opinion of developers on what is the correct way to process the data. There is no need to go far away to catch an example of how it happens one of the authors of this article is guilty of the fact that even knowing how to decipher and handle Nikon s camera tone curve recorded in the metadata, he still thinks that those curves are not very useful and consequently ignores them in the converter. The DNG format also lines up with the same tendency: the data tags added in the version 1.2 are intended to be used by Adobe programs in the first place.

The information industry have faced these problems many times already and have solved them by adopting the standards on data formats. Before adoption those standards were widely discussed by all parties, and in case of necessity they are revised; but again in accordance with a standard procedure. With RAW-formats adoption of the standard on metadata is a reasonable solution for the present situation. The standard should describe required and recommended metadata. Moreover, the encryption of required fields, that is those fields that are critical for baseline raw conversion ought to be explicitly prohibited. Yes, we need to agree upon what is that baseline , too.

We can discuss what tags should be included into metadata, as well as the structure of the fields (that is where the standardizing committee comes useful), though some of them are pretty obvious. One of these obvious fields, absence of which considerably slows down the work and leads to significant waste of time and resources, is the description of spectral characteristics of camera sensor. These characteristics are well-known to manufacturers and sometimes even have been published in the sensor spec sheets, - unfortunately in the form of curves of a rather general type instead of tables with exact values.

The absence of open information on colour characteristics of a camera results in the situation when to implement camera support one needs either to perform a number of tiresome, expensive and sometimes very approximate tests, or to choose the color visually, or to resort to data extracted from DNG. It takes time because with new cameras released initially the only converter available is the one offered by camera s manufacturer (sometime as a separate purchase). This converter not always (to put it mildly) suits users, sometimes because it does not provide the necessary quality and sometimes because it does not fit into the workflow adopted by a user or a big corporation.

For example, many design bureaus, photographic studios and pre-press bureaus use only Adobe suits. The support of new cameras with Adobe software might be delayed for months. As it was mentioned above, the push to DNG to a considerable extent is an attempt to impose the industry with Adobe s standard as the only one and thus to remove the problem of delayed support of new cameras, the problem which extremely irritates Adobe users.

Meanwhile DNG is not hopeless and is quite capable to become the basis for the standard. To achieve this it is quite sufficient to expand the list of metadata fields, to make non-prerequisite some of the fields required for Adobe conversion software to work, and to prohibit data encryption. The problem is that camera and software developers are extremely well-aware of the history of TIFF format and consequently are very cautious about Adobe initiatives. But it is still possible that authorizing some committee on standards to bring DNG to agreeable status might solve this problem.

As for the both approaches existing today insisting on opening of all the tags of tens (or even hundreds) of data formats without the description of the meaning of tags (like it is suggested by OpenRaw); or imposing the scanty single format (DNG) they are not serving the purpose of resolving industry problems. Both ways are in fact the paths to nowhere.

Comments

I see a few fundamentals

I see a few fundamentals issues with the article.

1. The most important is that the author does not seem to have looked at the DNG spec 1.2 very closely. There were several important changes incorporated in that document that make the format considerably more friendly to third parties. These include:
a. Provisions for any software to attach its color decoding profiles into the DNG, even if they are entirely different than those used by Adobe.
b. A tag was added to indicate which application built the embedded rendering. This was added specifically because users will have DNG files created by different applications, and need to send them for output and get predictable results.
c. The licensing terms for Adobe's color matrix was updated to address concerns of third-party software, and licensing terms were added to enable third-party software to protect or distribute its own rendering profiles, if they so choose.

2. The article makes two claims about the rendering differences produced by DNG files processed by Adobe software. In the first, the author uses "an old version of the DNG converter." It would be helpful to know which one. Is it the first implementation, or a later one? There were acknowledged problems with the very first version that have been rectified in later versions. And what does "amplified" mean, exactly?

In the second claim, the author states that the renderings produced by different versions of Camera Raw are, in fact, somewhat different. This is pretty unsurprising, since the current software has had further engineering, and refinements to the rendering engine. For the comparison to be of real value, the author should document how he has created and applied the parametric rendering instructions for the comparison of the files. It's quite likely that the first image is taking advantage of improvements in sharpening and shadow rendering that were built into the later version but not available to version 3.x (he does not document which versions he used to test with). The 5.x version is also highly likely to be using a different profile to render, although he could choose to use the old one if he wishes.

Either the author tested in the manner he did to purposely skew the results, or he does not understand rendering and forward compatibility very well. For instance, it would be much more relevant to know if a DNG made in ACR 3.6 looked the same when rendered out of 5.2, and what rendering instructions were present. The method he chose will clearly employ the improvements in the current version when rendering the file, including sharpening, vibrance, new profiles and other rendering tools. These will not be available in all past versions, just as some TIFF layer adjustments from Photoshop CS3 can't be understood by Photoshop CS, and so a different result will be produced. At minimum, this needs to be understood.

Most importantly, the author's criticism's here seems to be confused by the capabilities of the specification, and Adobe's implementation of the spec. He also does not seem to understand that the universality of DNG is that of a container, not a particular rendering. The *only* way to get a universal static rendering of the file it to embed a non engine-specific one: this has also been greatly enhanced by the tools in the 1.2 spec, which the author should take some time to try and understand. This kind of unclear thinking, along with sloppy research from the primary documents, does not move the discussion forward.

I would be interested in hearing specific suggestions regarding these issues, if they are informed by clear thinking about what is the job of the specification, and what is the job of the software, and how to handle forward compatibility in a parametric image editing environment. While there are surely a number of important issues to address - I can think of at least a half a dozen - very little in this article approaches that.

Peter Krogh
Author, The DAM Book, Digital Asset Management for Photographers

Dear Peter, Please, in your

Dear Peter,

Please, in your own words, take some time to try and understand what the article is about. In doing so you will find the answers to many of your questions right in the article, including specific suggestions and the archival value of allowing more proprietary tags like colour decoding profiles of un-specified structure.

By the way, in your opinion, shall we consider that the history of DNG starts with v. 1.2 and ACR v. 4.5? In that case, it is not a bad start. But what to do with the users invested into DNG at v.1.0 stage? Were they misled?

We would appreciate if instead of simply ruling out the results of our experiment as purposely skewed you would do those yourself and post the results proving your point. As one of the methods of verification archival properties, we suggest you make a DNG out of a RAW file, and then delete from a copy of that DNG all vendor-specific EXIF fields to evaluate the difference in processing results between the "pure" DNG stripped of those vendor-specific EXIF fields and more complete data.

We would also love to see your priorities being sorted out. You begin your second paragraph with "most important", and continue with "most importantly" in the paragraph next to your last one. Such phrasing indicates that you were rather sloppy, again, to use a word of yours, while writing your comment.

It is our feeling that the readers of your comment would benefit from a disclosure more detailed than just The DAM Book author; like mentioning your special relations with Adobe in the status of Adobe Alpha Tester.

--
Iliah Borg

>Please, in your own words,

>Please, in your own words, take some time to try and understand what the article is about.

I'm pretty sure I understand where you're going with it. I just think the fundamental underlying assumptions are flawed. On top of that, you don't clearly outline what is an issue with the format, and what is an implementation issue.

>In doing so you will find the answers to many of your questions right in the article, including specific suggestions and the archival value of allowing more proprietary tags like colour decoding profiles of un-specified structure.

Well, the DNG spec does currently provide for anyone to embed profiles that describe the color any way they wish. If a manufacturer wanted to create his own tags and embed, or make profiles available, it could do that, as I understand it.

I think the far more important and challenging issue is how to deal with tags for further decoding and output of the files. As imaging technology moves from simply trying to render the files "correctly" to making more creative interpretations from the captured data, the challenge becomes attaching the custom profiles that can control that output, and keeping them available as the file gets passed from user to user. In this regard , the new 2.5-D profiles that the DNG allows is a really ingenious solution. I suggest checking them out.

Charting the spectral response of the camera's chip is far less important than enabling the user to create a portable parametric rendering. I

Once again, all parametric renderings will be engine-specific, and are likely to change somewhat over time, as applications are improved. (And those are implementation decisions made by the application's creator, and not an issue of format.) In fact, the rendering does not even *exist* until the raw data is loaded into the imaging pipeline along with the instructions. That's why the ability to attach a fixed rendering is so important. It is the *only* possible solution to create a totally predictable rendering, unless we stop software from developing any further.

Likewise, *backwards* compatibility of parametric rendering engines simply cannot work the way you outline unless:
1. The engines are not allowed to change and improve. or
2. A software manufacturer is obligated to update all old versions with new capability whenever it's developed.
Neither one of these will happen.

>By the way, in your opinion, shall we consider that the history of DNG starts with v. 1.2 and ACR v. 4.5?

I'd say 1.1, rather than 1.2. That was when all image data,a swell as all private makernotes was first transferred. Prior to that, the format definitely left some stuff behind.

>In that case, it is not a bad start. But what to do with the users invested into DNG at v.1.0 stage? Were they misled?

Sure, if that's the way you want to look at it. The first implementation had problems. It would be great if all 1.0 implementations were problem-free.

>We would appreciate if instead of simply ruling out the results of our experiment as purposely skewed you would do those yourself and post the results proving your point.

Did you understand my criticisms? The premise of what you are asking for is impossible and/or undesirable.

>As one of the methods of verification archival properties, we suggest you make a DNG out of a RAW file, and then delete from a copy of that DNG all vendor-specific EXIF fields to evaluate the difference in processing results between the "pure" DNG stripped of those vendor-specific EXIF fields and more complete data.

I have no idea what you're driving at here. Are you saying that some DNG-aware applications can make use of EXIF?

When you talk about "archival" what are you really talking about? Are you talking about access to the metadata, access to a particular rendering, or the survivability of the format.

The first of these is really the province of the camera maker, who encode the data, and could make it more discoverable. The second one can *only* be addressed by an embedded rendering. And the third is an area that you glossed over entirely in your discussion of the spec. The data validation tags in DNG 1.2 are a huge leap forward in archive stability, as they offer a way to determine the file integrity automatically. I suggest you check out this part of the spec.

>We would also love to see your priorities being sorted out. You begin your second paragraph with "most important", and continue with "most importantly" in the paragraph next to your last one. Such phrasing indicates that you were rather sloppy, again, to use a word of yours, while writing your comment.

Okay, fine, it was sloppy. I should have said that all of those things are important. It's important to read and understand something before you send out a purportedly comprehensive discussion of it. And it's also important to understand what is the problem with the spec, and what is the problem with any specific implementation. And it's important to understand what's possible, and what's not.

>It is our feeling that the readers of your comment would benefit from a disclosure more detailed than just The DAM Book author; like mentioning your special relations with Adobe in the status of Adobe Alpha Tester.

Okay, yes, I am an unpaid alpha tester for Adobe, which gives me access to the development teams. And I helped Adobe improve the DNG spec, with particular emphasis on addressing the issues raised by other software vendors, such as Bibble (ask Eric - he brought specific problems to light, and they were addressed in 1.2). I also focused on the ability to facilitate predictable rendering, which can only be achieved by use of embedded rendered data. In addition, Adobe occasionally pays me (poorly) to speak. Adobe also paid me to write a white paper on this subject.

http://www.adobe.com/digitalimag/ps_pro_primers.html

I have also licensed photographs to Adobe in my capacity as a professional photographer.

Peter Krogh

I have no idea what you're

> I have no idea what you're driving at here. Are you saying that some DNG-aware
> applications can make use of EXIF

Sure!

I've done simple experiment in two minutes
- image from Canon 50D converted to DNG with Adobe 5.2.0.65 converter
Results
* Black Level tag value: 1023 1023 1023 1023
* Black level repeat dim: 2 2
* White level tag: 13200 (i.e. data is unscaled, 14-bit preserved as is)

But the real Black Level for this image is 511, not 1023.

Right RAW processor should:
- think: "this is Canon! we should calculate black level by left side of black mask"
- ignore black level tag from DNG
- calculate black level by averaging black (masked pixels)

If you strip all vendor-specific metadata from this file you will get unacceptable nonlinearity in shadows.

-- Alex Tutubalin

I'm still not sure what

I'm still not sure what you're driving at.

In the first experiment, you say to alter the EXIF data, and then seem surprised that it changes the output. What is unexpected here?

In the second example, are you saying that Adobe changes an EXIF value or improperly calculates the Black Level Tag? If so, that sounds like an implementation issue, not one of the spec. (Unless you see something in the spec that indicates that the right data is being tossed out by design.

Mostly what I think you're saying is that if you pull stuff out form DNG's as Adobe makes them, they don't render predictably. Again, that does not seem to be a huge surprise.

What is the problem one would encounter if he were not trying to intentionally damage the file?
Peter

> I'm still not sure what

> I'm still not sure what you're driving at.

A converter can't rely on just the information as contained in DNG proper, like the black level in the example above. Interpretation of the DNG data in a raw converter often depends on the specific make/model and some other EXIF tags; and worse, converters that can render quality images from DNG need to ignore certain DNG data substituting it with hard-wired data of their own, derived from hacking the original raw format.

>> We would appreciate if instead of simply ruling out the results of our experiment
>> as purposely skewed you would do those yourself and post the results
>> proving your point.

> The premise of what you are asking for is impossible and/or undesirable.

Please explain why.

--
Iliah Borg

When you say "the DNG proper"

When you say "the DNG proper" are you referring to the XMP space? Or to the Adobe-generated tags? Isn't the EXIF data available to the third-party raw converter from the Adobe-created DNG? How specifically would you suggest modifying the spec, and/or modifying the behavior of the DNG converter?

Is there anything else you can put your finger on that is concrete? I hope we can agree that it is impossible for a live parametric rendering to be identical from program to program, and that the DNG presents the best current possibility for attaching universally accessible renderings to raw image data. I hope we can agree that the 1.2 extension made the format considerably more friendly to third parties. I also hope we can agree that full backwards compatibility of rendering engines is not going to happen. I hope we can agree that forward compatibility is the most important goal, and that there are some really imaginative solutions included in the DNG that are not found anywhere else.

I also hope we can agree that most of the other points you've identified are ones that are coming from the camera manufacturer side, such as the difficulty of parsing some encoded or encrypted data, and that camera manufacturers don't publish the spectral response of their cameras.

I'd say that if we are on the same page, a more appropriate title for the article, regarding DNG, would be "A few small suggestions for the next DNG spec" rather than "Path to Nowhere." I'll spend a lot of time undoing misconceptions that are spread with analysis like this one.

Peter

If we need to override data

If we need to override data supplied in one of the DNG fields using hard-wired data or specific algorithms, all derived from original raw, - what is the place for DNG than? Each DNG interpreter will do the interpretation in its own way, just as it happens with original raw.

--
Iliah Borg

It's unavoidable that

It's unavoidable that different applications will use different tools to render from raw. No matter how the information is saved, there will be tags attached to images that other applications can't make use of. Get used to it. The most important thing is that data is preserved and accessible.

As to what is the use of DNG. I'd say the following:

1. An openly structured raw data container that can be used to hold the image data, as well as other stuff described below. Apple can currently make use of this to render raw filetypes it has never seen before.
2. A way to attach all the rendering instructions and color profiles needed to replicate a user's intention for image rendering. This is particularly important as users will begin to render images with custom profiles to create distinctive looks. If the profiles don't travel with the files, the rendering cannot be replicated, even on software of the same version. This is something you need to get a better understanding of.
3. A way to attach one or more fixed renderings to a raw file container that is not application-specific in any way, and lets the user employ multiple raw converters in an archive, and bring the renderings along. It also allows the use of third-party asset management software that can correctly render the files. Lastly, it enables the user to leave the program entirely and take the renderings with him.
4. A raw data container that metadata can be written to in a safe and documented manner, unlike proprietary raw files.
5,. A format to save rendered filetypes in when they are edited parametrically. This is starting to be a large problem, and is going to grow larger fast.
6. A way to make image files self-validating in a way that can survive subsequent changes to the metadata or image settings. No other graphic file format offers anything like this.

I consider these to be extremely important facets of image archiving.

It is not, and never has been, intended as a way to standardize the raw rendering output.

Again, any input on what specifically needs changing in the spec, or in the applications that create DNG is most welcome. If there is specific information you need from a DNG that you can't get, that would be helpful to know. It would also be good if you corrected the inaccuracies in your article before it spreads too far around the internets. Like I say, battling these misconceptions makes a lot of needless work for me.

;-)

Peter

> It's unavoidable that

> It's unavoidable that different applications will use different tools
> to render from raw.

There is a huge difference between "will" and "must".

Currently, one can't store just DNG. At the very least the original RAW should be embedded into the DNG, and thus the amount of storage and hence reliability of the archives in diminished.

At the bare minimum, a converter renders from the original RAW the same as from derived DNG; but in certain cases the rendition from the original RAW is better (we will discuss why in a due time). Some converters, important ones from camera manufactures being the prime concern to the end-users, do not acknowledge DNG files; and no roadmap to include those is announced. Those converters that can't render from the original RAW but can process derived DNG - I do not know of any worth the effort, the results from such converters are sub-par. This all means the DNG is very far from being accepted and needs a lot of work. The only way we see to improve DNG to the point of being usable and commonly accepted by the industry is through joint effort and wide discussion, as outlined in the article. Currently I do not see any blog or other Internet resource, not even a mailing list that is dedicated to such a purpose. All I see is special interests.

As far as "inaccuracies" that cause you needless work - you have yet to show any. It takes real work, experimentation and effort to understand RAW conversion. Such work can't be considered needless. Slogans are not going to work, same as printing press is not going to cure credit crisis.

--
Iliah Borg

The rendering tools will

The rendering tools will inevitably use differing sub-components. The only way to stop that s to forbid progress in rendering tools.

The DNG spec has been submitted to the ISO for consideration, so there is some discussion happening there, I assume.

Other people can have input, but I can tell you that Adobe is most interested in speaking to people who are at least willing to acknowledge the good work they have already done, and who seem to have a good grasp on the issues.

When you lead with a discussion of backward compatibility that, if implemented, could only serve to cripple development, that is an issue. Your discussion also seems to be unaware or inexplicably dismissive of the modifications done to the DNG spec that were made in the interest of third parties.

Camera Manufacturers unwillingness to support DNG is a political issue, not a technical one, as far as anyone has ever been able to show me. It would be nice if Adobe could order them to support DNG, but that's not possible.

As to usable, DNGs can be read by most Adobe products, Capture 1, Bibble 5, Lightzone, Apple on system level, and Windows on a system level, and nearly all DAM products. Calling it "unusable" is probably not a good way to start a discussion, and is, I would suggest, not accurate.

Keep in mind, that I do not speak for Adobe in any way. They may be happy to talk to you, but in general, they like people who have done their homework first.

I am curious as to who exactly "we" is.

Peter

> The rendering tools will

> The rendering tools will inevitably use differing sub-components.
> The only way to stop that s to forbid progress in rendering tools.

Once again you are trying to pass on the issue at hand. Problem is not what we want to do, but what DNG forces us to do - to ignore the contained information to assure normal processing.

> The DNG spec has been submitted to the ISO for consideration,
> so there is some discussion happening there, I assume.

Thank you to be so opened about the nature of that discussion. It sounds especially unusual given it comes from a person who founded the Digital Standards and Practices Committee with the American Society of Media Photographers.

> I can tell you that Adobe is most interested in speaking to people who are
> at least willing to acknowledge the good work they have already done

You seem not to realize that Adobe are not the only party here to decide. Somehow I doubt that you gave here an accurate profile to Adobe, too; and that you really understand that it is not just about Adobe doing good job with their applications, but about the future of our archives, and the quality of our conversions being on par to the money invested into the lenses and cameras and studios etc, all other things being equal. If only Adobe ACR/LR would be the best converter in the world, DNG might look much more attractive. But it is not.

> Camera Manufacturers unwillingness to support DNG is a political issue,
> not a technical one

Please try to support your statement with some evidence.

> As to usable, DNGs can be read by most Adobe products, Capture 1, Bibble 5,
> Lightzone, Apple on system level, and Windows on a system level,
> and nearly all DAM products.

To read is not everything. There are at least two questions here, firstly, is the rendition in third-party raw converters based solely on DNG data fields; and secondly, how different that rendition is when compared to the rendition from the original raw data.

> I am curious as to who exactly "we" is.

Once again in your own words - we like people who have done their homework first.

I remember well your writing from the past, September 2005: "While DNG is not guaranteed to be around forever it has a better chance than any particular individual camera format currently available". You continue than: "As more photographers see its benefits, the number of DNG files in existence will dwarf any other single format." That is, you were acknowledging that DNG can go to oblivion yet suggesting to use it as widely as possible. Sound logic and good advice.

--
Iliah Borg

To whoever you are, Since you

To whoever you are,
Since you are not responding to any points I'm bringing up, other than to pick them apart, this will be my last post here.

> The rendering tools will inevitably use differing sub-components.
> The only way to stop that s to forbid progress in rendering tools.
Once again you are trying to pass on the issue at hand. Problem is not what we want to do, but what DNG forces us to do - to ignore the contained information to assure normal processing.

And I asked, can you not find that information if the DNG? Is it not in the EXIF space for you to find? THis is an actual question.

> The DNG spec has been submitted to the ISO for consideration,
> so there is some discussion happening there, I assume.
Thank you to be so opened about the nature of that discussion. It sounds especially unusual given it comes from a person who founded the Digital Standards and Practices Committee with the American Society of Media Photographers.

I don't understand your point here. All I happen to know about this is that the DNG spec has been submitted to the black hole that is the ISO organization. That's public knowledge. If you're interested, why don't you ask the people that do know?

> I can tell you that Adobe is most interested in speaking to people who are
> at least willing to acknowledge the good work they have already done
You seem not to realize that Adobe are not the only party here to decide. Somehow I doubt that you gave here an accurate profile to Adobe, too; and that you really understand that it is not just about Adobe doing good job with their applications, but about the future of our archives, and the quality of our conversions being on par to the money invested into the lenses and cameras and studios etc, all other things being equal.

I'm acutely aware of that. It's just that Adobe is the only company that has bothered to try and create a universal raw container. I assume that you would never consider saving a file as a PSD, which is far more proprietary and undocmented than DNG.

>If only Adobe ACR/LR would be the best converter in the world, DNG might look much more attractive. But it is not.

No arguement there. "Best" is a matter of personal taste, to a large extent.

> Camera Manufacturers unwillingness to support DNG is a political issue,
> not a technical one
Please try to support your statement with some evidence.

Have you spoken to any camera manufacturers about it? I have. And I've never heard any technical barrier. If there is one, please feel free to present it. Saying they have not done it,is not the same as saying that there is a technical limitation.

> As to usable, DNGs can be read by most Adobe products, Capture 1, Bibble 5,
> Lightzone, Apple on system level, and Windows on a system level,
> and nearly all DAM products.
To read is not everything. There are at least two questions here, firstly, is the rendition in third-party raw converters based solely on DNG data fields;and secondly, how different that rendition is when compared to the rendition from the original raw data.

The only thing that I think you're saying is that you need to bypass the DNG's Adobe renedering tags for those in the EXIF that is also contained in the file. If it's something different, please correct me.

> I am curious as to who exactly "we" is.
Once again in your own words - we like people who have done their homework first.
I remember well your writing from the past, September 2005: "While DNG is not guaranteed to be around forever it has a better chance than any particular individual camera format currently available". You continue than: "As more photographers see its benefits, the number of DNG files in existence will dwarf any other single format." That is, you were acknowledging that DNG can go to oblivion yet suggesting to use it as widely as possible. Sound logic and good advice.

Hmm, I would think that a format person would understand that "going into oblivion" is pretty unlikely for an openly documented format. Adobe could go under tomorrow, and DNGs could still be opened by third parties. And I do believe that there are probably more DNGs in existence than any other single raw format.

Since you won't identify yourself, or your own financial interest, I'm out.
Peter

DNG submitted to ISO???

The ISO keeps track on everything, submitted, rejected, retired, accepted, and lists the status of everything. I keep hearing from unofficial voices, bloggers, Adobe evangelists etc that DNG was submitted to the ISO, yet I could not find ANY trace of it at www.iso.org, which has a comprehensive search engine. In fact the ISO is a champion of traceability and exactly the opposite of a black hole. So, Peter, where is the DNG submission? Please be thorough and post the submission number and date.

Axel, A quick use of the

Axel,
A quick use of the Google shows that Dietmar Wueller, a member of TC42 WG18, claimed to receive the offer from Adobe in March of 2007. I was not at the Archiving conference this year where I would have probably talked to Herr Wueller in person. Here is the email referenced on the DNG Wikipedia page.

http://mail.kde.org/pipermail/digikam-devel/2007-April/012129.html

Perhaps you could write to him or to the secretariat of the working group to find out what the current thinking of the group is. Here is the Secretariat's info from the ISO website.
Secretariat direct:

Tel: +1 646-460-7897
Fax: +1 212 398 00 23

E-mail: jknopes@ansi.org

Peter

Just the ISO submission number and date, please!

Peter, I just ask you for the ISO submission number and date.

Not links to comments on blogs, posts on usenet, obscure bulletin boards or suggestion to use Google. I want to see a trace of it on http://www.iso.org, something OFFICIAL.

As far as I can tell, DNG was never submitted for standardization.

The post you mention is just from a single individual, and it does not say anything about DNG being submitted for standardization. TIFF/EP being revised with input from DNG ("[having] Adobe's permission to incorporate modifications and developments made for DNG") does not say the Adobe DNG 1.3 specification will become an ISO standard. In fact the post in question was a call for input from anyone, saying the group in charge of TIFF/EP was open to suggestions (which is hardly surprizing).

There is a fine line between incorporating elements of DNG into an existing standard, and making DNG a standard in itself even if DNG was indeed derived from the TIFF/EP. Unofficial Adobe voices (like yours) are riding that line and interpreting it in a way that makes people believe that DNG is a standard, or close to become one.

In fact, unlike TIFF and TIFF/EP, DNG is proprietary and I'm under the impression that Adobe never really had any real intention to standardize it. Proof is the recent additions of the undocumented lossy and fast-load options that creates "DNG" files that only Adobe can read back safely.

DNG just became a closed, opaque format as those additions (some of them turned on by default in recent Adobe software e.g. Lightroom 4.0 and 4.1) are breaking and the lack of documentation makes it very difficult at best for anyone else to read those files.

How do you correlate this recent behavior with a standard, documented and stable format? I call it bait-and-switch from a company working on securing its future more than anything else.

DNG as a container

Peter,

my experiments show two interesting things
1) Some DNG implementations use camera specific processing paths. I.e. one way for black calculation for Canon cameras and another way for 'xxx' one. So, these DNG engines are not camera independent.

2) Most common DNG converter calculates wrong Black Level tag. This is not a specification problem, but implementation one. Anyway, from practical point of view, DNG specs are a theory, but Adobe DNG Converter is the reality.

So, in current reality it is not possible to build good RAW processor for DNG files which uses only Adobe-documented DNG tags. At least for DNGs produced from Canon .CR2 good RAW engine should ignore DNG-specific Black level tags and calculate black level by itself. If application relies on DNG tag, it will result in very bad rendering in Zone V and below (Also, dynamic range will drop from 8-11 stops to 5-6).

Of course, DNG is more documented than usual (undocumented by vendor) RAW format. This is definitely a step in the right direction. But this step is not big enough: DNG processors must use vendor-specific EXIF fields for good rendering, vendor-independent DNG-fields are not enough.

Some DNG problems are related not to the specs itself, but to particular Adobe DNG implementation. On the other side, there are not many DNG converters on market, so Adobe DNG converter problems and DNG format problems are not different in mass mind.

-- Alex Tutubalin

Very simple experiment

- take any .CR2 from Canon camera
- convert it to .DNG by Adobe (i.e. 'right') converter
- replace Make/Model fields by Exiftool:
exiftool -Make=xxx -Model=yyy canon_eos_50d_01-02.dng

- try to render this file with some Non-Adobe DNG-enabled applications. I've used ACDSee Pro 2.5 and shadows are _very_ different. I'm sure, that LightZone (example of another non-Adobe DNG-enabled app) will show the same problem.
Updated: ACDSee and Lightzone show *right* rendering for Make/model preserved and incorrect one if Make/Model changed in Exif.

-- Alex Tutubalin

If you have two PNG

If you have two PNG converters, that take a BMP and turn it into PNG form and found that the output from these converters were different, what would you do? Note that PNG is simply using lossless compression on a BMP any differences means someone is at fault!

Why does this change for RAW converters? Aren't RAW converters "suppose" to be "lossless" or are they horrifically lossy programs that have been touted about as needed by people who have a vested interest in keeping the status quo?

OpenRAW makes far more sense. Proprietary formats don't matter so much as holding a knife to Adobe's throat and saying "we're documenting our RAWs now, so update your code for all platforms because you only exist because of our customers." Companies working together for the betterment of the consumer rather than whatever they think they're doing, should be the real goal here.

>In fact, unlike TIFF and

>In fact, unlike TIFF and TIFF/EP, DNG is proprietary and I'm under the impression that Adobe never really had any real intention to standardize it. Proof is the recent additions of the undocumented lossy and fast-load options that creates "DNG" files that only Adobe can read back safely.

I don't think this constitutes proof of nefarious intentions. If no DNG 1.4 specification is published, or if the license for the 1.4 functionality changes, I'd be inclined to agree with you. I don't think advancement is proof of anything other than forward thinking and the need for new functionality.

As to "only Adobe can read back safely", I'm not sure exactly what you are talking about. There are two basic issues. If you are speaking about the issue with Fast Load previews becoming unreadable, here is my understanding.

There is a flaw in the existing dcraw code that improperly treats an alternate IFD as the main IFD. (I think this has been corrected in newer dcraw code.) Since a lot of software uses dcraw, it makes this a widespread issue. The existing specification outlines how DNG readers should understand IFDs. Here's the relevant part of the 1.2 spec as I understand it.

"DNG 1.2.0.0 allows a new value for NewSubFileType equal to 10001.H. This value, used for alternative or non-primary rendered previews, allows for multiple renderings (not just multiple sizes of a single rendering) to be stored in a DNG file. DNG reading software that displays a preview for a DNG file should, by default, display a preview from an IFD with NewSubFileType equal to 1. Alternative renderings should only be displayed if requested by the user."

I believe that dcraw was grabbing the wrong IFD because it was not paying proper attention to the tag.

If your objection is based on the new lossy compressed options, then I'd simply say to wait a little while and see if Adobe continues to do what they have always done so far - fully document the changes and release free SDK code for other applications to make use of (some of) the new functionality.

As to the addition of lossy source image data, it's my understanding that this allows DNG to be a more suitable container for JPEG originals. There is a significant file size economy, which will be useful at times. I'm personally very happy about this addition, since it allows JPEG shooters to store their parametrically edited images in a more appropriate format. (I personally believe that stuffing the EDL back into the JPEG header is a bad hack.) There are other really good and interesting ways for lossy compression to be of use that anyone here should be able to imagine with a little thought.

You are certainly free to believe that any advancement in the specification is prima facie evidence that Adobe is up to no good. It is my hope that they will document and license the new DNG capabilities as they have done for the last few revisions, and that these capabilities will be used by many different software and hardware vendors in valuable ways. i think they have a good track record in this regard.

As to the ISO stuff, if you are really curious, then I suggest you ask them what's up.

Peter

Add new comment