Sorry, missed these notices because of *very long* reply:
>> Also properties such as gimbal pitch, yaw, roll among others are removed.
>>Should I give you a copy of this image to take a look at the fields that are removed?
LibRaw::write_tiff_ppm() used in dcraw_emu creates very minimal set of EXIF tags.
There is no plans to extend this.
not too convenient to look at tags as a flat list.
In verbose output I see PrimaryPlatform set to four zeroes in ICC profile section.
I don’t think this is a big problem (unless some real software will complain about it).
Thank you for the quick response; I downloaded compiled and ran the latest github version.
I had another question; the tiff version adds a property, this one in particular looks like the malformed gps info.
[+Primary Platform : Unknown ()]
Is that a normal property value
Also properties such as gimbal pitch, yaw, roll among others are removed.
--- RAW01 2020-07-04 16:29:18.567144467 +0800
+++ TIFF01 2020-07-04 16:29:26.662419543 +0800
@@ -1,138 +1,80 @@
ExifTool Version Number : 11.88
-File Name : IMG_0999.DNG
+File Name : IMG_0999.DNG.tiff
Directory : .
-File Size : 40 MB
-File Modification Date/Time : 2020:07:04 13:48:55+08:00
-File Access Date/Time : 2020:07:04 13:48:55+08:00
-File Inode Change Date/Time : 2020:07:04 13:48:55+08:00
-File Permissions : rwxr-xr-x
-File Type : DNG
-File Type Extension : dng
-MIME Type : image/x-adobe-dng
+File Size : 57 MB
+File Modification Date/Time : 2020:07:04 16:26:51+08:00
+File Access Date/Time : 2020:07:04 16:26:51+08:00
+File Inode Change Date/Time : 2020:07:04 16:26:51+08:00
+File Permissions : rw-rw-r--
+File Type : TIFF
+File Type Extension : tif
+MIME Type : image/tiff
Exif Byte Order : Little-endian (Intel, II)
-Make : Hasselblad
-Camera Model Name : L1D-20c
-Orientation : Horizontal (normal)
-Software : 10.00.11.04
-Modify Date : 2020:06:30 16:02:39
-Image Width : 5568
+Subfile Type : Full-resolution image
+Image Width : 5472
Image Height : 3648
-Bits Per Sample : 16
+Bits Per Sample : 8 8 8
Compression : Uncompressed
-Photometric Interpretation : Color Filter Array
-Strip Offsets : 1041083
-Samples Per Pixel : 1
+Photometric Interpretation : RGB
+Image Description :
+Make : Hasselblad
+Camera Model Name : L1D-20c
+Strip Offsets : 1872
+Samples Per Pixel : 3
Rows Per Strip : 3648
-Strip Byte Counts : 40624128
+Strip Byte Counts : 59885568
+X Resolution : 300
+Y Resolution : 300
Planar Configuration : Chunky
-CFA Repeat Pattern Dim : 2 2
-CFA Pattern 2 : 0 1 1 2
-CFA Plane Color : Red,Green,Blue
-CFA Layout : Rectangular
-Black Level Repeat Dim : 2 2
-Black Level : 4096 4092 4091 4096
-White Level : 65535
-Default Scale : 1 1
-Default Crop Origin : 4 4
-Default Crop Size : 5464 3640
-Bayer Green Split : 0
-Anti Alias Strength : 1
-Best Quality Scale : 1
-Active Area : 0 96 3648 5568
-Opcode List 3 : GainMap, WarpRectilinear
-Default User Crop : 0 0 1 1
-Subfile Type : Reduced-resolution image
-Preview Image Start : 75970
-Preview Image Length : 965113
-Y Cb Cr Coefficients : 0.299 0.587 0.114
-Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2)
-Y Cb Cr Positioning : Co-sited
-Reference Black White : 0 255 128 255 128 255
-About : DJI Meta Data
-Format : image/dng
-Absolute Altitude : +254.35
-Relative Altitude : +120.20
-Gimbal Roll Degree : +0.00
-Gimbal Yaw Degree : +12.20
-Gimbal Pitch Degree : -90.00
-Flight Roll Degree : -4.90
-Flight Yaw Degree : +8.50
-Flight Pitch Degree : +4.50
-Cam Reverse : 0
-Gimbal Reverse : 0
-Self Data : DJI Self data
-Version : 7.0
-Has Settings : False
-Has Crop : False
-Already Applied : False
+Resolution Unit : inches
+Software : dcraw v9.26
+Modify Date : 2020:06:30 16:02:39
+Artist :
Exposure Time : 1/120
F Number : 4.0
-Exposure Program : Program AE
ISO : 100
-Exif Version : 0230
-Date/Time Original : 2020:06:30 16:02:39
-Create Date : 2020:06:30 16:02:39
-Exposure Compensation : 0
-Max Aperture Value : 2.8
-Metering Mode : Center-weighted average
-Light Source : Fluorescent
-Flash : No Flash
Focal Length : 10.3 mm
-File Source : Digital Camera
-Scene Type : Directly photographed
-Exposure Mode : Auto
-White Balance : Auto
-Digital Zoom Ratio : 1
-Focal Length In 35mm Format : 28 mm
-Scene Capture Type : Standard
-Gain Control : None
-Contrast : Normal
-Saturation : Normal
-Sharpness : Normal
-Serial Number : 0K8TFAA0020268
-Lens Info : 28mm f/2.8-11
-GPS Version ID : 2.3.0.0
+Profile CMM Type :
+Profile Version : 2.1.0
+Profile Class : Display Device Profile
+Color Space Data : RGB
+Profile Connection Space : XYZ
+Profile Date Time : 0000:00:00 00:00:00
+Profile File Signature : acsp
+Primary Platform : Unknown ()
+CMM Flags : Not Embedded, Independent
+Device Manufacturer : none
+Device Model :
+Device Attributes : Reflective, Glossy, Positive, Color
+Rendering Intent : Perceptual
+Connection Space Illuminant : 0.9642 1 0.82491
+Profile Creator :
+Profile ID : 0
+Profile Copyright : auto-generated by dcraw
+Profile Description : sRGB gamma 2.222 toe slope 4.5
+Media White Point : 0.95045 1 1.08905
+Media Black Point : 0 0 0
+Red Tone Reproduction Curve : (Binary data 14 bytes, use -b option to extract)
+Green Tone Reproduction Curve : (Binary data 14 bytes, use -b option to extract)
+Blue Tone Reproduction Curve : (Binary data 14 bytes, use -b option to extract)
+Red Matrix Column : 0.43608 0.2225 0.01393
+Green Matrix Column : 0.38509 0.71689 0.09709
+Blue Matrix Column : 0.14305 0.06061 0.71402
+GPS Version ID : 2.2.0.0
GPS Latitude Ref : North
GPS Longitude Ref : East
GPS Altitude Ref : Above Sea Level
-DNG Version : 1.4.0.0
-DNG Backward Version : 1.3.0.0
-Unique Camera Model : Hasselblad L1D-20c
-Color Matrix 1 : 1.2385 -0.7159 -0.0657 -0.0982 1.0352 0.0721 0.0457 0.0241 0.7437
-Color Matrix 2 : 0.731 -0.2746 -0.0646 -0.2991 1.0847 0.2469 0.0163 0.0585 0.6324
-Analog Balance : 1 1 1
-As Shot Neutral : 0.356545961 1 0.5400843882
-Baseline Exposure : 0
-Baseline Noise : 1
-Baseline Sharpness : 1
-Linear Response Limit : 1
-Camera Serial Number : 0K8TFAA0020268
-Shadow Scale : 1
-DNG Private Data : (Binary data 19829 bytes, use -b option to extract)
-Calibration Illuminant 1 : Standard Light A
-Calibration Illuminant 2 : D65
-Profile Name : Embedded
-Profile Hue Sat Map Dims : 18 6 1
-Profile Hue Sat Map Data 1 : (Binary data 3408 bytes, use -b option to extract)
-Profile Hue Sat Map Data 2 : (Binary data 3439 bytes, use -b option to extract)
-Profile Embed Policy : Allow Copying
-Noise Profile : 8.108e-05 6e-08
-Original Default Final Size : 0 0
-Original Best Quality Size : 0 0
-Original Default Crop Size : 0 0
+GPS Time Stamp : 00:00:00
+GPS Map Datum :
+GPS Date Stamp :
Aperture : 4.0
-CFA Pattern : [Red,Green][Green,Blue]
-Image Size : 5568x3648
-Megapixels : 20.3
-Preview Image : (Binary data 965113 bytes, use -b option to extract)
-Scale Factor To 35 mm Equivalent: 2.7
+Image Size : 5472x3648
+Megapixels : 20.0
Shutter Speed : 1/120
GPS Altitude : 254 m Above Sea Level
+GPS Date/Time : 00:00:00Z
GPS Latitude : xxx
GPS Longitude : xxx
-Circle Of Confusion : 0.011 mm
-Field Of View : 65.5 deg
-Focal Length : 10.3 mm (35 mm equivalent: 28.0 mm)
+Focal Length : 10.3 mm
GPS Position : xxx
-Hyperfocal Distance : 2.39 m
Light Value : 10.9
Should I give you a copy of this image to take a look at the fields that are removed?
Hi Alex,
Is the API call is_fuji_rotated() the one to use to identify Super CCD sensors?
Also it looks like number of pixels in imgdata.rawdata.raw_image and number of pixels in imgdata.image seem to be consistent with raw_height/raw_width and height/width.
For Super-CCD sensors, there is a 45 degree rotation applied to the image. So the height/width fields in sizes report the bounding box that is needed to store this rotated image i.e. if you apply a 45 degree rotation on an image with dimensions 2177 rows and 2894 cols, u will need a buffer of size 3587 x 3588 pixels to store it. It will have a bunch of black pixels around the image. So, image contains height*width*4*2 bytes.
A few follow up questions I have are:
1. Does COLOR() function work for these types of images? Because for this specific image, cdesc is "RGBG" and COLOR(0, 0), COLOR(0, 1), COLOR(1, 0) and COLOR(1, 1) are 0, 2, 1, 3 respectively which results in a "RBGG" Bayer Pattern but I don't think that is correct because these are not really Bayer images?
2. Which fields/props in imgdata do I use to detect these kinds of files such that i can update my copying code suitably? The call to raw2image is not needed for typical Bayer or X-Trans sensors. You can copy from raw_image or color3_image etc.
So raw2image() works the same way for these files i.e. only one of the 4 channels will have a valid value and this will correspond to COLOR(r, c).
Is the size of the buffer (n bytes) allocated after raw2image() equal iheight*iwidth*2? This info is necessary for me to allocate the suitable output buffer in my code.
Alex,
This API call appears to only impact the output width/height. Do we treat this to be the visible dimensions (height/width) to just extract the visible portion of the RAW data?
Does this mean the COLOR(r, c) does not work for these formats?
To reiterate, what I am trying to do is extract the Visible Portion of the RAW Image and implement the pipeline myself? Will calling raw2image() on this image only give me the visible portion of the CFA? If so, what dimensions should I assume the resulting buffer has
So, is there a way to identify if the file is from a Super-CCD sensor such that I can branch my code to handle it separately.
Alex,
Thanks for that. I will read up on Fuji Super CCD images and go over the dcraw code.
Does that mean that some of the properties in the sizes field such as width, height are incorrect/invalid for these files? I was just trying to just read out the visible part of the raw image for me to apply my own camera pipeline.
Also, how do I identify if a file has Super-CCD sensor?
This is the way how Fuji Super-CCD files are processed (this method borrowed from dcraw.c):
- raw data is not 2:3 but stretched in one dimension (different for different cameras)
- color order is not rg/gb, but green columns and r/b columns
On processing:
- entire image is rotated 45 deg to get normal bayer pattern with diagonal greens (it is also unstretched to get 2:3 aspect ratio)
- standard debayering is used
- output rotated back and cropped
Matrix conversion is not gamut limited (unless you cut negative values in some intermediate steps)
ccm is 'camera matrix parsed from metadata. period', it is preserved as is (w/ possible value normalization). Usually it 'just some color data with not known camera/vendor specific meaning'.
That makes perfect sense. So, the matrix to convert from camera space to XYZ becomes xyz_rgb*rgb_cam. Does that impact the gamut of colors that can be represented as Camera Space -> sRGB is a narrowing conversion?
But, what exactly is ccm? I know in your earlier comment you mentioned
"ccm is also 'camera color matrix' retireved from RAW in 'as is' (excl. normalizing) form."
Alex,
A quick follow up. So based on your descriptions, cmatrix is the transformation from Camera Space to sRGB.
Is ccm similar to cam_xyz i.e. transformation between Camera Space and XYZ?
I have a DNG file shot from a Google Pixel 3 that has all-zeros for cam_xyz and ccm. The rgb_cam and cmatrix have the same values. The profile is also not available.
For such files, how do I convert from input space/camera space to XYZ?
There is no such program in LibRaw 'sample code' programs.
One could implement such program using LibRaw or Adobe DNG sdk for raw (dng) decode and something like libtiff for tiff write.
Apologies for the long response.
Do you have a program that can just convert the dng file to tiff without changing any of the dng's exif metadata?
Sorry, missed these notices because of *very long* reply:
>> Also properties such as gimbal pitch, yaw, roll among others are removed.
>>Should I give you a copy of this image to take a look at the fields that are removed?
LibRaw::write_tiff_ppm() used in dcraw_emu creates very minimal set of EXIF tags.
There is no plans to extend this.
I just downloaded the sample image and the exif file; I ran the test and still get the same errors when I try to load the .tiff file.
I ran a exiftool -v and there were some warnings
[Warning = ValueConv GPSTimeStamp: Argument "undef" isn't numeric in addition (+)]
There's no such errors in the raw file
Here are tiff file produced by dcraw_emu -T (from github/LibRaw/master branch) and result of exiftool -v3 tiff-file: https://www.dropbox.com/sh/6q9d3hd10z8x06m/AAA7QgHY9yf_KFK38JmZ5WF9a?dl=0
There are no tags 0xa40.. in this file.
apologies about the output, I sent a sample DNG file through the contact link.
I am still getting errors when trying to open the .tiff files
I am using colmap; which uses freeimage to import supported files. DNG files are unsupported but TIFF's are.
I think those unknown() values trips up freeimage.
not too convenient to look at tags as a flat list.
In verbose output I see PrimaryPlatform set to four zeroes in ICC profile section.
I don’t think this is a big problem (unless some real software will complain about it).
Thank you for the quick response; I downloaded compiled and ran the latest github version.
I had another question; the tiff version adds a property, this one in particular looks like the malformed gps info.
[+Primary Platform : Unknown ()]
Is that a normal property value
Also properties such as gimbal pitch, yaw, roll among others are removed.
Should I give you a copy of this image to take a look at the fields that are removed?
Followup:
was able to reproduce, fixed by this patch: https://github.com/LibRaw/LibRaw/commit/5a5b227a715c914e8fbd3ee7f97c0d76...
Please provide sample RAW file(s) to play with.
Alex,
Thanks for your help. FC(r, c) worked as expected. I believe I am all set.
Dinesh
yes, is_fuji_rotated() could be used to identify 'rotated processing' used in LibRaw/dcraw.c
COLOR(r,c) is correct in raw-coordinates (and FC(r,c) is correct in imgdata.image[] coordinates)
Hi Alex,
Is the API call is_fuji_rotated() the one to use to identify Super CCD sensors?
Also it looks like number of pixels in imgdata.rawdata.raw_image and number of pixels in imgdata.image seem to be consistent with raw_height/raw_width and height/width.
Dinesh
Alex,
I think I figured it out.
For Super-CCD sensors, there is a 45 degree rotation applied to the image. So the height/width fields in sizes report the bounding box that is needed to store this rotated image i.e. if you apply a 45 degree rotation on an image with dimensions 2177 rows and 2894 cols, u will need a buffer of size 3587 x 3588 pixels to store it. It will have a bunch of black pixels around the image. So, image contains height*width*4*2 bytes.
A few follow up questions I have are:
1. Does COLOR() function work for these types of images? Because for this specific image, cdesc is "RGBG" and COLOR(0, 0), COLOR(0, 1), COLOR(1, 0) and COLOR(1, 1) are 0, 2, 1, 3 respectively which results in a "RBGG" Bayer Pattern but I don't think that is correct because these are not really Bayer images?
2. Which fields/props in imgdata do I use to detect these kinds of files such that i can update my copying code suitably? The call to raw2image is not needed for typical Bayer or X-Trans sensors. You can copy from raw_image or color3_image etc.
Appreciate your help.
imgdata.image buffer is NOT equal to iheight*iwidth*2
So raw2image() works the same way for these files i.e. only one of the 4 channels will have a valid value and this will correspond to COLOR(r, c).
Is the size of the buffer (n bytes) allocated after raw2image() equal iheight*iwidth*2? This info is necessary for me to allocate the suitable output buffer in my code.
I think it will be faster and more productive to try raw2image() than discuss this here.
Alex,
This API call appears to only impact the output width/height. Do we treat this to be the visible dimensions (height/width) to just extract the visible portion of the RAW data?
Does this mean the COLOR(r, c) does not work for these formats?
To reiterate, what I am trying to do is extract the Visible Portion of the RAW Image and implement the pipeline myself? Will calling raw2image() on this image only give me the visible portion of the CFA? If so, what dimensions should I assume the resulting buffer has
So, is there a way to identify if the file is from a Super-CCD sensor such that I can branch my code to handle it separately.
Dinesh
https://www.libraw.org/docs/API-CXX.html#adjust_sizes_info_only
Alex,
Thanks for that. I will read up on Fuji Super CCD images and go over the dcraw code.
Does that mean that some of the properties in the sizes field such as width, height are incorrect/invalid for these files? I was just trying to just read out the visible part of the raw image for me to apply my own camera pipeline.
Also, how do I identify if a file has Super-CCD sensor?
This is the way how Fuji Super-CCD files are processed (this method borrowed from dcraw.c):
- raw data is not 2:3 but stretched in one dimension (different for different cameras)
- color order is not rg/gb, but green columns and r/b columns
On processing:
- entire image is rotated 45 deg to get normal bayer pattern with diagonal greens (it is also unstretched to get 2:3 aspect ratio)
- standard debayering is used
- output rotated back and cropped
Matrix conversion is not gamut limited (unless you cut negative values in some intermediate steps)
ccm is 'camera matrix parsed from metadata. period', it is preserved as is (w/ possible value normalization). Usually it 'just some color data with not known camera/vendor specific meaning'.
That makes perfect sense. So, the matrix to convert from camera space to XYZ becomes xyz_rgb*rgb_cam. Does that impact the gamut of colors that can be represented as Camera Space -> sRGB is a narrowing conversion?
But, what exactly is ccm? I know in your earlier comment you mentioned
"ccm is also 'camera color matrix' retireved from RAW in 'as is' (excl. normalizing) form."
but could you provide more details?
Thanks,
Dinesh
ccm is not similar to cmatrix.
To convert to XYZ use cmatrix or rgb_cam, multiplied by srgb2xyz conversion matrix, see convert_to_rgb() source for details.
Alex,
A quick follow up. So based on your descriptions, cmatrix is the transformation from Camera Space to sRGB.
Is ccm similar to cam_xyz i.e. transformation between Camera Space and XYZ?
I have a DNG file shot from a Google Pixel 3 that has all-zeros for cam_xyz and ccm. The rgb_cam and cmatrix have the same values. The profile is also not available.
For such files, how do I convert from input space/camera space to XYZ?
Pages