Jump to content


Photo

Raw vs Log Filesize


  • Please log in to reply
7 replies to this topic

#1 Dennis Schaller

Dennis Schaller
  • Basic Members
  • PipPip
  • 10 posts

Posted 01 June 2013 - 08:06 AM

Hey guys, 

I'm writing my bachelorthesis to finish my studies here in germany. My topic is "production of professional videomaterial to compare different cameras". 

I read an article by Art Adams over at provideocoalition about the differences between Raw and Log Files. 

I've got a few questions going a bit deeper into the topic though:

 

Art wrote the following in his article:

16,384: totally saturated sensor (maximum white)

8,192-16,383: First stop down from maximum white

4,096-8,191: Second stop down from maximum white

2,048-4,095: Third stop down from maximum white

1,024-2,047: Fourth stop down from maximum white

512-1,023: Fifth stop down from maximum white

256-511: Sixth stop down from maximum white

128-255: Seventh stop down from maximum white

64-127: Eighth stop down from maximum white

32-63: Ninth stop down from maximum white

16-31: Tenth stop down from maximum white

8-15: Eleventh stop down from maximum white

4-7: Twelfth stop down from maximum white

2-6: Thirteenth stop down from maximum white

1-2: Fourteenth stop down from maximum white

 

So he's talking about the digital values and the brightness stops they represent using Raw files. 

Then in log mode the data is compressed using logarithmic scales. 

This is how I imagine it's working:

For every pixel in the image data using Raw is stored like this:

Red: 10111011100000 (representing 12000 in binary values) 

Blue: 10011100010000 (representing 10000 in binary values) 

Green: 1111101000000 (representing 8000 in binary values) 

I dont exactly know what color this would be but just to have some example values. 

So in Raw the file is getting quite big because those values

for every pixel is quite much data (at least in 4:4:4 for every pixel). 

 

First question: is this roughly how data is stored (there will be some more meta data in the file, but i guess this is how pixel color data is stored?) 

Second question: how are those values compressed in log? 

Everyone is always talking about brightness being compressed in log, but i guess what is actually meant are the brightness values for each color channel seperately (as I did in my example above). 

Since in log the data has to be stored in binary values too, how are the log values written exactly? 

 

Hope someone is able to help me. I may be asking some more in-detail questions in this thread over the next few weeks while im writing. 

 

Thanks in advance. 

 


  • 0

#2 David Mullen ASC

David Mullen ASC
  • Sustaining Members
  • 19761 posts
  • Cinematographer
  • Los Angeles

Posted 01 June 2013 - 10:56 AM

Trouble is that raw, by definition, doesn't store RGB values -- it's a monochrome image that was photographed through a mosaic color filter pattern (Bayer filter) -- it has to be de-mosaiced (debayered) in post (or in camera for viewing a color image, or for recording an RGB format).  At this point it triples in size because you have converted, let's say, 4K raw to 4K RGB, meaning 4K red, 4K green, and 4K blue.  

 

To record log, the raw signal has to be debayered in camera to RGB with a log gamma curve applied.  Often the AD processor is working at something like 14-bit raw and converting it to 10-bit log, though the Alexa can record 12-bit log 4444.

 

Since a raw image cannot really be viewed as a color image without conversion to a video format with some sort of gamma curve applied, whether log or Rec.709, I'm not really sure it matters what the raw luminance values are, but I guess it is linear but usually in 14-bit depending on the camera.


  • 0

#3 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11939 posts
  • Other

Posted 01 June 2013 - 11:10 AM

The big question is linear with regard to what. Sensor data isn't linear to anything other than the response of silicon to light, so while it's often referred to as linear, it isn't very mathematically useful until it's had a curve put on it that the camera manufacturer thinks is reasonable, which is a matter of opinion. This is one reason for all the complexity around this subject; nothing is really linear, and everything is subject to the opinion of the imaging engineer at the camera company. The need to publish impressive specifications has in the past tempted some companies to excessively manipulate their raw data, compromising the aesthetic appeal to make numbers bigger, particularly in terms of trading noise floor for dynamic range.

 

P


  • 0

#4 Dennis Schaller

Dennis Schaller
  • Basic Members
  • PipPip
  • 10 posts

Posted 01 June 2013 - 12:15 PM

Ok David you're right, I messed up quite a lot in my post.
 
I was wrong about raw saving RGB-Data for every pixel of course. So what it really does is just saving a value for the lights intensity for every pixel. Those pixels are either red, green or blue. Got it till there.
Then you're saying that raw files are smaller then a debayered image. A question regarding this point: Is there a format that outputs raw debayered video without a log/rec709 curve? No right? You can either output raw or log/rec709 compressed material which will be smaller in filesize than raw. (Just for me to clarify)
 

To record log, the raw signal has to be debayered in camera to RGB with a log gamma curve applied.

And that is the point where I need some more information. What exactly happens in the "apply a log gamma curve" step?
I guess the curve compresses the raw values right? So when you still have only one value (representing the light intensity) for every pixel you compress those values logarithmically.

My big question is "Why are raw files bigger than log files? How is the data stored in raw files different from the data stored in log/rec709 format files?".

 

To get back to my example:

Data in raw files is stored like

Pixel coordinates - light intensity that hit the pixel (<- for every pixel there is)

Since its digital values, it all has to be 1/0. In a 2K Alexa image (measuring 2880 x 1620 pixels) you'd have to have at least 12bit to store the X coordinates of the pixel, 11 bit for the Y coordinates plus the 14bit or whatever it is that youre recording the brightness values in.

Example (12bit X coords, 11bit Y coords, 14bit brightness):

 

101101000000 11001010100 11111111111111

X: 2880 - - - - - -Y: 1620 - - - - -B: 16383

 

So those 1/0 combination will tell the pixel on the bottom right corner to be maximum white.

 

Now how comes log files are smaller in filesize? What is done to those ones and zeros to compress them?

 

Oh and Phil: Yeah I know the topic is quite confusing because some cameras even compress raw files with a log curve etc. I think I understand it enough to get the basics and to know when to use what or why it looks like it does, but I'm really interested in those geeky things maybe noone other than me thinks about. If I understand the real basics it makes everything much clearer for me so I try to get to the bottom of everything in my thesis as well.


Edited by Dennis Schaller, 01 June 2013 - 12:19 PM.

  • 0

#5 David Mullen ASC

David Mullen ASC
  • Sustaining Members
  • 19761 posts
  • Cinematographer
  • Los Angeles

Posted 01 June 2013 - 12:49 PM

I don't know if you can store an Alexa recording as 2.88K log-c, I think the choices are 1080P or 2K log-c (10-bit or 12-bit) or 2.88K raw (12-bit).

 

But I believe it is possible to debayer and color-correct in "linear light" gamma instead of log-c or Rec.709.  The ACES workflow for example involves converting footage to 16-bit linear light gamma for color-correction.  It's just that if you are going to work in 10-bit, log is a more efficient way of storing luminance values than linear.  But once you start color-correcting in 16-bit I don't think it matters, you might as well work with the original linear light gamma.

 

Raw files are not necessarily bigger than log files.  If you are using the original Alexa and recorded 2880 x 1620 pixels in raw versus, let's say, recording uncompressed 1080P RGB, that would be a 4.66MP frame in raw versus three 2.07MP frames of HD, which totals 6.21MP. So uncompressed HD out of the Alexa takes up more data in theory than 2.88K raw out of the Alexa (assuming both are 12-bit).  It's just that most people are using some sort of compressed format like ProRes 4444 when shooting in log-c on the Alexa, and some are recording log in 10-bit (or even 8-bit!)  And cameras like the Red One or Epic use selectable rates of compression for their raw recording.

 

You're probably reading the same article, but Art Adams here explains why log is an efficient way to store exposure values:

http://provideocoali...gamma_curves/P2


  • 0

#6 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11939 posts
  • Other

Posted 01 June 2013 - 12:57 PM

Are we confusing data rate compression with dynamic range compression?


  • 0

#7 Dennis Schaller

Dennis Schaller
  • Basic Members
  • PipPip
  • 10 posts

Posted 02 June 2013 - 07:48 AM

Ok that's good to know David. I thought raw files were so much bigger by default. That it wouldn't matter what settings you're recording in, raw would be bigger than log. But what you're saying makes more sence. Good to know that raw files aren't necessarily bigger with the same bitdepth.

 

But that actually sounds a bit like compressing data in log wouldn't really pay off, because raw files will be smaller? That souldn't be like that should it?

 

Phil no I wouldn't say that. I know those are different things and that raw doesn't mean big filesize and log doesn't mean small filesize. But as far as I see things, one of the benefits of recording log should be that you're getting smaller files (and that you can view them and that it maintains nearly the same dynamic range that raw files have).

 

 

David I read the page of Arts article again but it actually looks like he's having an error in there somewhere too? (or I don't get it right)

 

His example with the candles: In raw you coun't every lit candle in the room like 1 2 3 4 5 6 7 ... and in log you only count 1 2 4 8 16. While it makes sence that counting only the powers of 2 will result in less data to be saved, I don't think that's all you need to make your picture. You need those small steps in brightness difference and saving only the powers of 2 is like having full stops of brightness difference between each step.


  • 0

#8 Dennis Schaller

Dennis Schaller
  • Basic Members
  • PipPip
  • 10 posts

Posted 05 June 2013 - 12:52 PM

Just to let you know:

I guess the answer I was expecting was "Log data isn't stored otherwise, the bits are just used differently because the LUTs assign them differently"

I never heard of LUTs, so it was very unclear to me what Log actually does to manipulate the data.

 

Thanks anyways, you two gave me some good points too :)


  • 0


Metropolis Post

Glidecam

Broadcast Solutions Inc

Aerial Filmworks

rebotnix Technologies

The Slider

Tai Audio

Paralinx LLC

Rig Wheels Passport

Opal

Ritter Battery

Media Blackout - Custom Cables and AKS

FJS International, LLC

Gamma Ray Digital Inc

Technodolly

Abel Cine

CineTape

Willys Widgets

Visual Products

Wooden Camera

CineLab

Gamma Ray Digital Inc

Media Blackout - Custom Cables and AKS

Technodolly

Visual Products

CineTape

Glidecam

Aerial Filmworks

Paralinx LLC

Metropolis Post

Opal

The Slider

Ritter Battery

Wooden Camera

Willys Widgets

Abel Cine

CineLab

Broadcast Solutions Inc

rebotnix Technologies

FJS International, LLC

Rig Wheels Passport

Tai Audio