Jump to content


Photo

Red Workflow


  • Please log in to reply
39 replies to this topic

#1 blain murphey

blain murphey
  • Basic Members
  • PipPip
  • 14 posts
  • 1st Assistant Camera

Posted 23 February 2009 - 04:03 PM

Hi, I recently got a job for a Red Feature and I just found out im responsible for the workflow also along with my 1st AC duties. I have never done offloading Red Footage. Can anyone tell me what all needs to be done. We will be shooting with the Red Drive and CF Cards
  • 0

#2 Oliver Christoph Kochs

Oliver Christoph Kochs
  • Sustaining Members
  • 323 posts
  • Film Loader
  • Germany

Posted 23 February 2009 - 04:32 PM

I have no idea about RED or anything, but wouldn't this be something like:
Boot a laptop, plug in a 10$ CF Card Reader, insert CF card, copy data from CF card (with RED footage) on 2 seperate harddrives.
Make sure that the seperate harddrives are stored and transported seperately.
Pray and then press the "format CF card button".
But, hey, please get a person that KNOWS everything about RED raw format. It'll help.
  • 0

#3 Mike Thorn

Mike Thorn
  • Basic Members
  • PipPip
  • 64 posts
  • Grip
  • Cleveland, OH

Posted 23 February 2009 - 06:51 PM

Oliver is right, with one caveat: cheap CF card readers have been known to cause data corruption. Make absolutely sure your card reader is UDMA-approved.

A couple other tips: use Firewire 800 if possible (Macbook Pros have a FW800 port built in). This will significantly increase your offload time. Likewise, use FW800 drives as well. Advice on drive brand will vary wildly. LaCie has had some issues with their larger drives in the past; I know people that won't touch Western Digital drives with a sanctified touchin' pole.

Double backups are a must. On some shoots we've even gone to double duplicate (backups of backups). If you back up to two drives on set, make sure to copy from the card to the drives both times, instead of once to one drive and then from that drive to the other, just in case something twibbled.

Make sure to clearly mark which cards are full and which are empty. I use two separate and different colour CF card cases for this.

Using FW800 and reasonably quick drives, you can expect to spend about five minutes per card copy.
  • 0

#4 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 06 March 2009 - 03:05 PM

Hi, I recently got a job for a Red Feature and I just found out im responsible for the workflow also along with my 1st AC duties. I have never done offloading Red Footage. Can anyone tell me what all needs to be done. We will be shooting with the Red Drive and CF Cards


This sounds like a bad situation. Offloading and safeguarding the finished work is traditionally the job of the Second AC, not the First. We're finding that the data wrangling workload is too big for just a single Second AC. It takes a Second Second AC to sit at a computer workstation and watch the picture as the CF's are offloaded, just to be sure that nothing really horrible is wrong. The original Second slates, keeps camera reports, and shuttles CF's back and forth. Avoid the mechanical disk drive, they have reliability issues.

Here's a good place to get the stuff you need for the QC/Offload station:

www.melrosemac.com

You'll need a couple CF readers, a Mac, Raid array, and shuttle drives of some sort. You can set this up on the camera truck in the film changing darkroom. Get a UPS, too, if you'll be running off the generator - or if there's a chance somebody might trip over the AC cable and kick it out.




-- J.S.
  • 0

#5 Chris Fernando

Chris Fernando
  • Basic Members
  • PipPipPip
  • 148 posts
  • 1st Assistant Camera

Posted 06 March 2009 - 03:47 PM

Advice on drive brand will vary wildly. LaCie has had some issues with their larger drives in the past; I know people that won't touch Western Digital drives with a sanctified touchin' pole.



I think Western Digital coined the phrase, 'It's not a matter of if it will crash, but when.'

Save yourself much stress, hearthache and frustration; stay away from WD drives.

Edited by Chris Fernando, 06 March 2009 - 03:48 PM.

  • 0

#6 Adrian Sierkowski

Adrian Sierkowski
  • Sustaining Members
  • 7117 posts
  • Cinematographer
  • Los Angeles, Ca

Posted 06 March 2009 - 04:27 PM

I like Seagate drives, personally, for my storage. Definitely back up as much as you can. On larger productions and for posterity, they'll often also lay out to LTO tape.
  • 0

#7 Chris Durham

Chris Durham
  • Basic Members
  • PipPipPipPip
  • 290 posts
  • 1st Assistant Camera
  • New York, NY

Posted 06 March 2009 - 04:56 PM

Data wrangling is a must and shouldn't be a 1st AC thing. You need someone to dump the cards, watch the footage, and dupe the footage before handing the card back for reformatting. Nothing exists until it exists in two places. And don't put off dumping and watching for another day, even if you have a dozen cards or hard drives. I've seen cards corrupt. Know as soon as possible if you have to reshoot something and get it same day if you can.
  • 0

#8 Jonathan Bowerbank

Jonathan Bowerbank
  • Basic Members
  • PipPipPipPip
  • 2815 posts
  • 1st Assistant Camera
  • San Francisco, CA

Posted 06 March 2009 - 05:19 PM

If you know someone who is computer savvy, perhaps not an AC and would be willing to Camera PA at a lower rate. You could probably get that approved by production. There really should be someone fully dedicated to making sure the footage is getting "in the can". While you're working on set, you can't keep running back to the laptop to see how the downloading is going, or to check footage.
  • 0

#9 Brian Drysdale

Brian Drysdale
  • Basic Members
  • PipPipPipPip
  • 5070 posts
  • Cinematographer

Posted 07 March 2009 - 05:12 AM

I don't think a 1st AC would have time to do much data wrangling. It needs someone to supervise the process and make the back up copies. Is the production going to come to a halt while this is happening? You don't want to be under any time pressures while doing the downloading and saving money on this process is short sighted.

If the 2nd AC isn't trained or an experienced data wrangler available, perhaps an assistant editor could be employed. Although, the latter may still need training; someone with an editing background told me that he thought it would require a day's training just to learn the downloading process. However, I suspect that may depend on the trainee and how much detail you want to go into during the training. After the training, they should also know what to do when things start going wrong - that usually takes more time to learn.

The process has to be pretty bullet proof, so you really can't take short cuts.
  • 0

#10 Chris Pickle

Chris Pickle
  • Basic Members
  • PipPip
  • 12 posts
  • Cinematographer

Posted 07 March 2009 - 09:15 AM

I know this might sound sacrilegious, but data wrangling is not the big deal people say it is. Actually, on anything other then a very big budget you don't even need a DIT. Everybody, (Especially Red owners) like to perpetuate the idea that it's some big scary difficult issue--data and DIT.

As far as the camera goes, well, someone on the crew should know the camera menus,etc. But that's no different then any camera. The data is as easy as pulling the drive or card and copying it to two other drives. No one sits there and watches the footage. That would take forever. Even the data transfer takes forever on big shoot, so once it's started, all you can do is maybe count the ceiling tiles.

You double check the files sizes on all 3 drives, watch a bunch of clips, maybe a few seconds from each, then carry on. It's ridiculously easy. On a small indie crew, the 1st could do it, or at least get it started over lunch if that's how you want to do it. Hell, the director or DP could do it.

On our last two features we simply went back to the office and transfered footage at the end of the day. Someone on the crew took care of it, but we all communicated to make sure no one wiped drives until all double checks were made. We shot on a drive so we didn't stop mid-day to transfer.

Chris
  • 0

#11 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 09 March 2009 - 02:06 PM

I know this might sound sacrilegious, but data wrangling is not the big deal people say it is. Actually, on anything other then a very big budget you don't even need a DIT.


True, this is driven mostly by budget, and the associated insurance/bonding requirements.

On a network TV budget, we use CF's instead of the hard drives. Drive reliability and having too many eggs in that fragile basket....

We do want to have somebody watch as they transfer the CF's -- and yell "Oh, s--t!" if there's something major wrong. The quicker the UPM finds out, the cheaper it is to fix/re-shoot.

The DIT is really a sort of full time on-set dailies colorist. Basically they perform the same function that the dailies telecine colorist does on a film show. But they're on the clock 18 hours a day instead of 4 - 6, which is where the expense is. That's necessary on a tape shoot, but not for Red.





-- J.S.
  • 0

#12 Stuart Brereton

Stuart Brereton
  • Basic Members
  • PipPipPipPip
  • 3065 posts
  • Cinematographer
  • Los Angeles

Posted 09 March 2009 - 02:40 PM

You don't necessarily need a DIT, but you should definitely have someone dedicated to the task. Also, in addition to the drag and drop process to two drives you should have some sort of verification process. On a simple level the TOAST software can do a simple checksum analysis. For more in depth support there are numerous other software packages. Conrad Huntzinger, who posts here sometimes has authored some software which is well regarded.
  • 0

#13 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 09 March 2009 - 03:22 PM

On a simple level the TOAST software can do a simple checksum analysis.


Once you've had a person watch to check that there's a good picture, a checksum will ensure that things continue in good order down the line. But if it's NG on the CF, a checksum will merely verify that the same corruption gets duplicated.





-- J.S.
  • 0

#14 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11938 posts
  • Other

Posted 09 March 2009 - 03:27 PM

I'm absolutely terrified by what's being proposed in some of these situations.

Material should come off the camera, be backed up to duplicates, one of which should go to the salt mines for long term storage. Second duplicate goes to post and is verified as being what was expected, complete, and bit for bit verifiable. Only then do you nuke the mag and start again. This means you need more cards. They're cheap. They can deal with it.

You absolutely need a paperwork system, if only to protect yourself.

Problems I have seen with this sort of thing include (not limited to Red):

- Material is verified, but it's the wrong material. Specify, retrieve and compare timecode ranges and roll numbers to prevent this. Ask the question: It's a reasonable-looking image, but is it the right reasonable-looking image.

- All the backing up of hard disks and mag tapes is often done by one person in one room on one computer. Programme interchange problems still exist (disk/tape/card only readable on the computer that wrote it, etc). Redundant backups should be redundantly verified - send the primary to post and have them verify it, verify the secondary yourself, on a different computer, and have the production office compare the numbers then sign a form to let you delete the mag. Don't do it all yourself - humans have an unpleasant tendency to write down what seems right, rather than taking a step back and reevaluating the correctness of what they're seeing. In any case, making the decision to delete camera original material is way above the pay grade of most of the people who end up doing it.

- Common problem: Both sets of a redundant backup are sent to post, while both sets of another redundant backup are sent to the salt mines. One backup, usually representing a whole reel, appears to go missing completely, causing panic, whereas another has a dangerous hole in its data security. Ask the question: who is handling my media, how am I labelling and controlling where it goes, and how do I keep it and its paperwork together.

- Secondary media are often not subjected to any form of testing, with the result that, if there's a problem with the primary, you won't find out until you really need it. Both sets of backups from the first few days of shooting should go to post for checking. This particular problem is endemic in all forms of data backup and archiving in all industries. Ask the question: do I know if my secondary backups are actually good, or do they only get tested when I find I need one, which is already a failure situation?

- Image is thought to be OK, but includes horrible problems such as swapped R/G/B channels (dual-link SDI cables misconnected), bad LUTs, or some other issue. Material should be watched and compared to frame grabs, photos, or written notes describing the image content. Ask the question: it's a DPX file, it's readable, it's verifiably identical to the one I backed up, but is it reasonable.

Whatever you do, get it in writing from production that they are happy with it, that the completion and insurance people are happy with it, and ensure that you personally are fully absolved of any repercussions.

In my experience, Red shoots appear to be unaware of any of this and tend to handle it all very, very badly.

P
  • 0

#15 Keith Walters

Keith Walters
  • Sustaining Members
  • 2219 posts
  • Other
  • Sydney Australia

Posted 09 March 2009 - 05:20 PM

In my experience, Red shoots appear to be unaware of any of this and tend to handle it all very, very badly.


MR RHODES!!!
"I-
"Oh never mind...."
  • 0

#16 Brian Drysdale

Brian Drysdale
  • Basic Members
  • PipPipPipPip
  • 5070 posts
  • Cinematographer

Posted 09 March 2009 - 07:24 PM

On our last two features we simply went back to the office and transfered footage at the end of the day. Someone on the crew took care of it, but we all communicated to make sure no one wiped drives until all double checks were made. We shot on a drive so we didn't stop mid-day to transfer.

Chris


Sounds like a pretty simple shoot if you're only using the drives, using CF there's continuous changing of the cards and downloading CFs in batches.

Contractual demands can be a lot more demanding than just going back to the office and transferring footage. What may be OK on a low budget indie feature or a short won't pass on a larger production. Also, it mightn't be a good idea to have the 1st AC doing all this after the nearly standard 12 hour shooting day when they may be wanting to check the gear and prep things for the next day's filming.
  • 0

#17 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 09 March 2009 - 07:51 PM

I'm absolutely terrified by what's being proposed in some of these situations.

.... It's a reasonable-looking image, but is it the right reasonable-looking image.


Good point given the reports of CF cards not erasing, leaving last time's stuff in place.

Here's how we propose to work on a TV series, for your comments:

On set, the second AC slates, keeps camera reports, and carries CF cards back and forth to the second second AC/data wrangler.

The data wrangler works in the old film changing darkroom on the camera truck. His workstation consists of a UPS, a Mac computer with two CF readers and a 10.5 TB Raid array. The movable media are six 500 Gig jump drives and enough CF's for three days' work.

When a CF comes off the camera, the data wrangler transfers it to the Raid array, watching to see that the picture is OK, and that the slates correspond to the camera reports. If there's a problem, he alerts the UPM or first AD immediately. Then the CF card goes into the holding box in the darkroom, and he transfers the new material from the Raid array to two of the six jump drives. This is always "print all", not circled takes. At the end of the day, one jump drive goes into the holding box, the other goes with the sound to the video facility.

The video facility transfers the jump drive to their SAN, and from the SAN to two LTO-4's. Again, print all, not circled takes. Only then do they do a circle takes and syncing pass for the editor and viewing DVD's, with one light correction. When the LTO's are checked and OK, they return the jump drive to the data wrangler, with the written OK to move the CF's and the other jump from the holding box to the available box. The data wrangler re-uses stuff from the available box, erasing the oldest first.

Everything stays on the Raid array for the duration of the episode, so the director and DP have it available for reference.

Do you see any big holes in this?


Thanks --





-- J.S.
  • 0

#18 John Brawley

John Brawley
  • Sustaining Members
  • 834 posts
  • Cinematographer
  • Atlanta Georgia

Posted 10 March 2009 - 02:32 AM

I know this might sound sacrilegious, but data wrangling is not the big deal people say it is. Actually, on anything other then a very big budget you don't even need a DIT. Everybody, (Especially Red owners) like to perpetuate the idea that it's some big scary difficult issue--data and DIT.



Well only if you're 1st is meant to be changing lenses and getting everything in focus.

I think the point being that you shouldn't have your 1st doing it.

It's really a separate job to make sure it's done well and in a timely manner. Otherwise its the end of the day or in breaks and then the 1st is going to do ? Just a few extra hours of overtime ??

If I was you I'd NOT be taking on that responsibility. It's like being asked to load and 1st at the same time. Except to download you red files takes a fair bit longer and requires more phaffing (depending on if you're just copying or doing other business)

It a large workload to take on, and requires zero mistakes. Do you really want that headache ??

jb
  • 0

#19 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11938 posts
  • Other

Posted 10 March 2009 - 08:46 AM

Do you see any big holes in this?


That's better than most people seem to be doing. I like the fact that you immediately make three copies of it - two jump drives, the RAID, and CF are all available shortly after shooting, so four copies of it exist including the CF. I'd be fractionally, though probably paranoically, concerned that there isn't much bit for bit verification going on; the LTO 4s are five copy operations (CF -> RAID -> Jump -> SAN -> LTO) away from camera original, and there is no way of making sure that nothing's been even fractionally broken in that time. This could be fixed by putting MD5 checksums on things.

The thing is, this only really works well with very heavily compressed originals as Red produces. On a shoot with a real camera...

P
  • 0

#20 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 10 March 2009 - 11:42 AM

This could be fixed by putting MD5 checksums on things.


Thanks, Phil. I've googled it, and found some freeware such as CheckSum+ for the Mac. I'm going to recommend that we create MD5 checksums of the Raid copies using the on-location mac.





-- J.S.
  • 0


Paralinx LLC

Wooden Camera

Media Blackout - Custom Cables and AKS

Gamma Ray Digital Inc

Ritter Battery

Opal

The Slider

Willys Widgets

Metropolis Post

CineLab

Tai Audio

Rig Wheels Passport

Technodolly

FJS International, LLC

Visual Products

CineTape

Aerial Filmworks

Abel Cine

rebotnix Technologies

Broadcast Solutions Inc

Glidecam

Media Blackout - Custom Cables and AKS

Broadcast Solutions Inc

FJS International, LLC

Opal

Technodolly

Rig Wheels Passport

Visual Products

CineTape

Aerial Filmworks

The Slider

rebotnix Technologies

Glidecam

Tai Audio

CineLab

Willys Widgets

Paralinx LLC

Metropolis Post

Gamma Ray Digital Inc

Abel Cine

Wooden Camera

Ritter Battery