Jump to content


Photo

Digital "Loading"


  • Please log in to reply
42 replies to this topic

#1 Chris Keth

Chris Keth
  • Sustaining Members
  • 4427 posts
  • 1st Assistant Camera
  • Los Angeles

Posted 25 March 2010 - 04:13 AM

I was asked by a guy who I have loaded film for what I know about digital loading and I had to tell him "not much." I did it for a RED job where I transferred the drives to one external drive, randomly spot checked them, and then made copies to two other drives.

Can anyone share the finer points? Is there a "standard" way to do all of this stuff yet? I'd like to be able to tell the guy that I've brushed up significantly enough that he doesn't go elsewhere for his digital work.
  • 0

#2 Gus Sacks

Gus Sacks
  • Sustaining Members
  • 287 posts
  • Cinematographer
  • Los Angeles, CA

Posted 25 March 2010 - 08:31 AM

Hey Chris. I've done enough of this to fortunately or not be able to call myself a 'Digital Loader,' which is a silly title, but sometimes you have to differentiate yourself from a DIT, unless Production lumps both jobs into one title, but I'd say there is a difference.

When I do "Loading" for a RED job, I manage the media - by means of doing the same as you'd do with Mags, Letter them and Build a numbering system - then doing file management backups into Folders, checking the clips, then I keep a detailed spreadsheet I combine with the 2nd AC's camera reports (which they don't always keep) to give Post/Editorial a clearly defined method of matching the Raw clips with the Scene/Take numbers, and if there were any issues or discrepancies with the footage. Then I generate TIFF files, similarly, for post and color correction. I reformat the Mags on my computer and send them back out relabeled. Sometimes productions ask for other things, but that's typically the base package, so to speak. Some productions don't want that all, so sometimes I don't do it. This workflow is for one/two day commercial jobs or low budget features. Otherwise, it'd be great to send the footage to be backed up a third time before formatting drives and cards.

Hope that helps,
Gus
  • 0

#3 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11941 posts
  • Other

Posted 25 March 2010 - 09:54 AM

The critical thing about this is getting the backups verified. It's terrifyingly easy to go through a whole workflow assuming you have backups, but only find that there was a problem with them once you need them.

I've harped on about this endlessly on this and other fora, but while making lots of backups helps, it's really just reducing the chances of failure - to verify things, you need to actually recover the backups and check their validity, and this ideally needs to be done by a third party. Deleting orginal media is potential armageddon and should be done with the same sort of trepidation with which you'd burn the o-neg of a 35mm shoot.

So:

- Record
- Sanity check viewing - use this as an opportunity to make your paperwork notes
- Copy off the card once (A dub)
- Copy off the card again (B dub)
- Optionally, bit for bit compare A and B dubs (will save time if there's a problem at this point, but on aggregate may save time to find it later, even if it makes one incident take longer to fix)
- Make 2 sets of long-term backups from A dub (#1 set and #2 set)
- Recover long term backups and compare to B-dub
- Get in writing from production that they believe and accept that what came off the card is identical to what went to long term storage
- #1 set of long term backups goes to offsite storage
- #2 set gets used for post.

Critically I would never delete anything from original media without a signoff from production.

P
  • 0

#4 Chris Keth

Chris Keth
  • Sustaining Members
  • 4427 posts
  • 1st Assistant Camera
  • Los Angeles

Posted 25 March 2010 - 12:03 PM

Critically I would never delete anything from original media without a signoff from production.

P


Is that really ever going to happen, though?

What's the best way to compare (bit for bit, as you say) the copies and the copies from copies? Is it just going to be looking at file sizes and trusting or is there a more comprehensive method or program to do it?
  • 0

#5 Saul Rodgar

Saul Rodgar
  • Basic Members
  • PipPipPipPip
  • 1682 posts
  • Cinematographer

Posted 25 March 2010 - 12:12 PM

Is that really ever going to happen, though?

What's the best way to compare (bit for bit, as you say) the copies and the copies from copies? Is it just going to be looking at file sizes and trusting or is there a more comprehensive method or program to do it?


As Phil says, the key to success is verifying the transfer. Which in the case of oft chaotic on-set circumstances is easier said than done, unfortunately.

A piece of software called Shotput Pro allows to back up to several hard drives at once. The software allows one to choose formats, like SxS, P2, Red, Canon 5D, etc. It is $90 to buy online, and it gives you plenty of control and a log of your transfer at the end of the card back-up process. However, it is not fail proof.

Recently, we were transferring a bunch of P2 cards to hard drive with it, but one of them failed without the software warning us about it. It was caught before the card was reformatted, so we were safe, but we weren't on-set. But it is certainly something to be aware of.

On Macs, I like the Disk Utility app for creating .dmg's, it has never failed me.

Edited by Saul Rodgar, 25 March 2010 - 12:17 PM.

  • 0

#6 Chris Keth

Chris Keth
  • Sustaining Members
  • 4427 posts
  • 1st Assistant Camera
  • Los Angeles

Posted 25 March 2010 - 12:20 PM

As Phil says, the key to success is verifying the transfer. Which in the case of oft chaotic on-set circumstances is easier said than done, unfortunately.

A piece of software called Shotput Pro allows to back up to several hard drives at once. The software allows one to choose formats, like SxS, P2, Red, Canon 5D, etc. It is $90 to buy online, and it gives you plenty of control and a log of your transfer at the end of the card back-up process. However, it is not fail proof.

Recently, we were transferring a bunch of P2 cards to hard drive with it, but one of them failed without the software warning us about it. It was caught before the card was reformatted, so we were safe, but we weren't on-set. But it is certainly something to be aware of.

On Macs, I like the Disk Utility app for creating .dmg's, it has never failed me.


Shotput seems like a pretty good way to log all of those copies and to have a first check. How exactly did it miss that copy? Was it in "compare file size" or "compare file contents" mode at the time? How did you guys catch the mistake? I hate to say it but it seems like a lot of luck is involved with the average red production.
  • 0

#7 Saul Rodgar

Saul Rodgar
  • Basic Members
  • PipPipPipPip
  • 1682 posts
  • Cinematographer

Posted 25 March 2010 - 12:35 PM

Shotput seems like a pretty good way to log all of those copies and to have a first check. How exactly did it miss that copy? Was it in "compare file size" or "compare file contents" mode at the time? How did you guys catch the mistake? I hate to say it but it seems like a lot of luck is involved with the average red production.


I'm pretty sure it was in the compare file size mode. It was also set for fastest verification. It did not warn of the problem. the Log text file was fine. I caught it because, since we weren't on set, I tried transferring the files to FCP, where it gave me a "Invalid Format" warning. So, we were able to look for the card (which its always id'd by serial number on the backup), and go from there.

The reason it was on fastest verification mode is because it takes long enough as is, if it took any longer I would rather use Disk Utility --which does take longer--but like I said, it has never failed me.

Edited by Saul Rodgar, 25 March 2010 - 12:38 PM.

  • 0

#8 Chris Keth

Chris Keth
  • Sustaining Members
  • 4427 posts
  • 1st Assistant Camera
  • Los Angeles

Posted 25 March 2010 - 12:41 PM

I'm pretty sure it was in the compare file size mode. It was also set for fastest verification. It did not warn of the problem. the Log text file was fine. I caught it because, since we weren't on set, I tried transferring the files to FCP, where it gave me a "Invalid Format" warning. So, we were able to look for the card (which its always id'd by serial number on the backup), and go from there.

The reason it was on fastest verification mode is because it takes long enough as is, if it took any longer I would rather use Disk Utility --which does take longer--but like I said, it has never failed me.


So you would use disk utility to make a disk image rather than just copy the contents of the card/drive to a folder? Why?
  • 0

#9 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 25 March 2010 - 01:37 PM

.... unless Production lumps both jobs into one title, but I'd say there is a difference.


I'd say there's a big difference. The loader's job is to get everything you shoot turned over to post exactly as shot. The DIT is your full time on set colorist.

Given that we can't afford a whole day's worth of CF cards, what we've done is to put a computer and RAID array in the darkroom on the camera truck. The loader transfers the cards to the RAID, and to a pair of removable solid state jump drives. At breakoff, one jump goes to the post facility, where they dump everything to their SAN and make LTO backups.

CF's get re-used as needed after they've been transferred to the RAID and jumps. Jumps get re-used after the LTO's have been confirmed.

The RAID is big enough for a whole episode's worth of dailies, which makes them available for review during production.





-- J.S.
  • 0

#10 Saul Rodgar

Saul Rodgar
  • Basic Members
  • PipPipPipPip
  • 1682 posts
  • Cinematographer

Posted 25 March 2010 - 02:08 PM

So you would use disk utility to make a disk image rather than just copy the contents of the card/drive to a folder? Why?


What I like about Disk utility is that it is a dedicated one-task-at-the-time app, unlike the Finder which does many things at once. It was specifically created by Apple to handle hard drives, form repair to creating mirror disk images. So far it has not let me down, the Finder has. If the Finder gets over whelmed by other tasks, it is likelier to botch up a file transfer than Disk Utility.


From wikipedia: A disk image is a single file or storage device containing the complete contents and structure representing a data storage medium or device, such as a hard drive, tape drives, floppy disk, CD/DVD/BD and key drive, although an image of an optical disc may be referred to as an optical disc image. A disk image is usually created by creating a complete sector-by-sector copy of the source medium and thereby perfectly replicating the structure and contents of a storage device.

Edited by Saul Rodgar, 25 March 2010 - 02:12 PM.

  • 0

#11 Saul Rodgar

Saul Rodgar
  • Basic Members
  • PipPipPipPip
  • 1682 posts
  • Cinematographer

Posted 25 March 2010 - 02:16 PM

For the record: Disk Utility handles somewhat simple hard drive repairs. For more complex repairs, Alsoft's Disk Warrior rules unchallenged.
  • 0

#12 John Brawley

John Brawley
  • Sustaining Members
  • 834 posts
  • Cinematographer
  • Atlanta Georgia

Posted 25 March 2010 - 04:15 PM

As Phil says, the key to success is verifying the transfer. Which in the case of oft chaotic on-set circumstances is easier said than done, unfortunately.



Oh yeah. That's why I hate backing up on set. I'd much rather treat it like film and send the rushes offsite to be backed up in more controlled conditions.

I've always been sceptical of on set backups. The chance of interrupted power, accidental hard drive knocks, constant moving of the set, constant moving of the DIT trolly. It just screams painful. I'm even more sceptical of onset colour correction, a la DIT.

On a recent TV series, we simply shot to RED DRIVES instead of card. Every couple of hours of set time we swapped out the drive and sent them off to the lab (in this case Deluxe) with a runner at lunch and the end of the day.

Deluxe would do visual and tech checks of every print take, two sets of backups to editorial drives, backed it up to their networked storage and this was then also backed up on their LTO tape system.

The main chance of failure with this approach is then the camera drive itself. I feel like it's minimised by swapping it out religiously every couple of hours. Even if the drive fails, there's often a chance of recovery of most of the drive if a particular sector / take goes down.

The most you loose is a couple of hours of set time if the drive totally fails. More than likely you'd also know on set if that happens to the drive because the camera will report a fault as you shoot. Not a bad trade off. And i can have one less person on set. Just a loader instead of a DIT and loader.

jb
  • 0

#13 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 25 March 2010 - 05:00 PM

Backing up not on set, but within walking distance, is sort of like what we always had with film -- getting it out of the camera mags and into metal cans. The backup station should always be running on a UPS. As cards and RAM drives become more affordable, sending them to post starts to make sense. Running mechanical drives around is a bit more scary. They're just not as rugged as solid state.

I, too, have seen on-set color correction not work worth a damn over and over again. Trying to make a look that way is a fool's errand. But going to tape, having a DIT to make sure what you get is within range for final timing is somewhere between a real good idea and a necessity. It's one of those temporary things that should be going away now, as the technology improves -- like the Technicolor consultants of old.




-- J.S.
  • 0

#14 John Brawley

John Brawley
  • Sustaining Members
  • 834 posts
  • Cinematographer
  • Atlanta Georgia

Posted 25 March 2010 - 05:35 PM

Backing up not on set, but within walking distance, is sort of like what we always had with film -- getting it out of the camera mags and into metal cans.


Not really backing up though is it ? You've just moving it to another container for transport. A step that does have a small degree of risk. But you still only have one instance of your content until it hits the lab and is processed to a more stable and safe state. Even then though there's still only the one instance of the content, but at least it's in a safe, stable location where conditions are more controlled.


I, too, have seen on-set color correction not work worth a damn over and over again. Trying to make a look that way is a fool's errand. But going to tape, having a DIT to make sure what you get is within range for final timing is somewhere between a real good idea and a necessity. It's one of those temporary things that should be going away now, as the technology improves -- like the Technicolor consultants of old.



I agree. I've had the best results simply taking a still and using lightroom. I'll email some notes to the colourist. Nothing else is worth the effort and seldom gives a better result.
  • 0

#15 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 25 March 2010 - 06:04 PM

With film, you're just putting it in another container. With the Raid array and jump drives, we have two portable copies, and one non-portable but more reliable copy. That seems to be working so far.

Communication between the DP and colorist is the key here. We want always to have tests in pre-production, and the DP and colorist sit together in the timing bay and establish the look for dailies using those tests. Thereafter, notes by e-mail, voice mail, or whatever is all it takes. The colorist works for you, just like your gaffer and operators do.




-- J.S.
  • 0

#16 Chris Keth

Chris Keth
  • Sustaining Members
  • 4427 posts
  • 1st Assistant Camera
  • Los Angeles

Posted 25 March 2010 - 07:53 PM

With film, you're just putting it in another container. With the Raid array and jump drives, we have two portable copies, and one non-portable but more reliable copy. That seems to be working so far.


So 3 copies from source material made on set and verified seems to pretty much be the emerging standard?

I'm trying to get a bead on what is "usual" and some of the best practices because I'm starting to be asked about my prowess in loading for digital systems like the RED.


Another related topic: Anyone have a favorite backup power supply?
  • 0

#17 Chris Keth

Chris Keth
  • Sustaining Members
  • 4427 posts
  • 1st Assistant Camera
  • Los Angeles

Posted 25 March 2010 - 08:14 PM

Any experience with R3D Data Manager? It seems like a well thought out program for this type of thing.

Do any of you keep paperwork of all these file transfers, like a post-camera log so to speak?
  • 0

#18 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 25 March 2010 - 10:59 PM

So 3 copies from source material made on set and verified seems to pretty much be the emerging standard?
Another related topic: Anyone have a favorite backup power supply?


It's what we've done. I can't say for other companies. As the comfort level goes up with time, the backup level may go down.

The UPS I have is an APC:

http://www.apcc.com/





-- J.S.
  • 0

#19 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11941 posts
  • Other

Posted 26 March 2010 - 06:58 AM

I tend to agree that in an ideal world this would not be done on set, but it certainly can be done on set and, given that, it's probably a tough sell to ask someone to pay for it to happen elsewhere. I'm happy with either scenario.

And yes you can get signoff from production. You just need a form for each roll which has two halves: one to be filled in by you, one to be filled in by the people who read back your LTOs and verify they're good. Then all the production office has to do is be the arbiter, be the person who checks that all the numbers match up, and sign the "permission to delete" box.

I have used this system on the basis that I was not being paid enough to make those kinds of decisions about the camera original - I'm truly in awe of what loaders take on, for so little money.

P
  • 0

#20 John Waterman

John Waterman
  • Basic Members
  • PipPip
  • 49 posts
  • 1st Assistant Camera
  • Chicago

Posted 27 March 2010 - 10:17 PM

Any experience with R3D Data Manager? It seems like a well thought out program for this type of thing.

Do any of you keep paperwork of all these file transfers, like a post-camera log so to speak?


I'd like to chime in here. The emerging standard for all of these issues has been established by the ICG Data Handling workshop. I highly recommend this training for all L600 members.

For Red, R3D data manager is the standard. Make sure you use check sums when copying data from place to place to verify that it was copyed correctly. R3D data manager or shot put pro both create logs of every file transfer you make. Make sure to keep this 'reciept' if there ever is a question where/when data was copied.

The more copies the better. Make sure they are descrete seperate drives and live in different phyical locations. I recently day played on a Red indie feature where the digital loader had the second dive mirrored to the first drive, and the first drive got acedentally formated.. so it formated the second drive too, and production lost severals days of work. LTO tape back-ups are the best for ultimate archive.

Defitely go do the ICG training - two days of intensive study with the best people in the biz doing this kind of work.
  • 0


Opal

CineLab

The Slider

Technodolly

Ritter Battery

Broadcast Solutions Inc

rebotnix Technologies

Aerial Filmworks

Metropolis Post

Paralinx LLC

Willys Widgets

Wooden Camera

FJS International, LLC

Rig Wheels Passport

Media Blackout - Custom Cables and AKS

Tai Audio

Glidecam

CineTape

Abel Cine

Gamma Ray Digital Inc

Visual Products

CineTape

Tai Audio

Abel Cine

Wooden Camera

Gamma Ray Digital Inc

Ritter Battery

The Slider

Opal

Aerial Filmworks

Metropolis Post

Glidecam

Visual Products

Broadcast Solutions Inc

Rig Wheels Passport

Paralinx LLC

CineLab

Willys Widgets

Media Blackout - Custom Cables and AKS

rebotnix Technologies

Technodolly

FJS International, LLC