Jump to content


Photo

Proper Data Managing on Set


  • Please log in to reply
11 replies to this topic

#1 Tyler Leisher

Tyler Leisher
  • Basic Members
  • PipPip
  • 91 posts
  • Director
  • El Segundo, CA

Posted 27 August 2010 - 10:50 PM

Hey all,

I'm working on learning to be a Digital Loader/2nd AC, and for my current job I ingest a lot of HD footage from P2 Cards and 5D/7Ds.

Up until now, I've just been checking the directory file size and number of files copied but I dont think thats the most efficient (or real) way of checking.

So, I'm curious if there are any standard industry proven ways of checking these to make sure:
- The video from the P2 was transfered to the hard drive without problems (Checksums dont match of .mxf vs .mov when ingested.)
- FIles have been copied from one drive to another

I know that MD5 Checksumes are normally the standard, but I'm not sure how to go about getting a checksum of an entire folder (including subfolders) on a mac.

Should I do a checksum of each file and then compare to the same file? Is there a faster way of generating multiple checksums?

I'm also curious what you guys use to backup in the field in terms of an array, are you using a Drobo or G-Raid type RAID system, or do you have a custom built machine to handle that?

Any other tips or advice you have for me would be great as well.
  • 0

#2 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11936 posts
  • Other

Posted 28 August 2010 - 06:41 AM

There is no short cut to producing an MD5 sum. Its purpose is to check for single bit errors, so the computer is intrinsically required to go over every bit in the source file. It isn't quick. That's life. The only way to speed it up is to throw more hardware at it.

I'm not so convinced about these USB- or firewire-attached RAIDs. I'd rather have two physically separate devices, so that you can also satisfy offsite-backup requirements - or to put it another way, if building A burns to the ground, you still have backups in building B.

Should I do a checksum of each file and then compare to the same file?


Copy the file off the card twice, produce two checksums, and if they're identical you're good. If not, delete both files and retrieve them from the card again, twice. Repeat until you've got a match or until it's failed a reasonable number of times.

Here's what slightly perturbs me about this. I'm sure I'd be sneered at, as being entirely underqualified to do this kind of work personally, and yet I constantly get phone calls and emails about it from "real" film ACs asking questions exactly like this. Is nobody running some sort of course?

Anyway, I wrote an article on it on my website: http://philrhodes.co...e_wrangle.xhtml

P
  • 0

#3 Oliver Christoph Kochs

Oliver Christoph Kochs
  • Sustaining Members
  • 323 posts
  • Film Loader
  • Germany

Posted 28 August 2010 - 07:02 AM

Maybe software from Imagine Products can help. I know that my company bought the software but I haven't tried it myself yet.
  • 0

#4 Tyler Leisher

Tyler Leisher
  • Basic Members
  • PipPip
  • 91 posts
  • Director
  • El Segundo, CA

Posted 28 August 2010 - 01:50 PM

Do you do an md5 of each video file or an md5 of the entire folder?
  • 0

#5 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11936 posts
  • Other

Posted 28 August 2010 - 02:25 PM

Most MD5 tools work on files, but you really want one per file anyway - so you can then put that hash into the filename, and anyone can check the integrity of the file again at any time in the future, by redoing the hash and comparing it with the one in the filename.

P
  • 0

#6 Jamie Metzger

Jamie Metzger
  • Basic Members
  • PipPipPipPip
  • 773 posts
  • Cinematographer
  • San Francisco

Posted 28 August 2010 - 07:17 PM

Shotput Pro for data transfer and Clone Carbon Copier for....well, you know.
  • 0

#7 Elhanan Matos

Elhanan Matos
  • Sustaining Members
  • 432 posts
  • Digital Image Technician
  • Santa Monica, CA

Posted 19 November 2010 - 12:31 PM

If you're using a mac I would highly recommend you learn how to use the terminal; Using just a plain old mac with osx (or PC with linux) you can copy safely and quickly using Rsync then create an MD5 checksum and compare it using diff. Sounds a little daunting, but if you're serious about data managing, then you better learn it.
  • 0

#8 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11936 posts
  • Other

Posted 19 November 2010 - 01:06 PM

All this is quite doable on windows too; I usually write JScript to do it.

P
  • 0

#9 Tony Brown

Tony Brown
  • Sustaining Members
  • 689 posts
  • Cinematographer
  • London

Posted 20 November 2010 - 02:57 PM

Hey all,

I'm working on learning to be a Digital Loader/2nd AC,


Am I being a Luddite thinking they are completely separate grades? I'm sick to death of seeing 1st AC's trying to organise a shoot single handedly whilst the 'Digital Loader' stares at his Mac Book Pro. Either you wrangle data or you're a 2nd AC.....no?
  • 0

#10 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11936 posts
  • Other

Posted 20 November 2010 - 03:04 PM

I don't think that being a data wrangler should be any more difficult than being a conventional loader - really it should involve less of your time and more machine time, which can happen unattended. When I've written software designed to work in these scenarios, anyway, that's the assumption I've worked on. Copying a few files around, which is what we're really talking about here, should not be a full time job, so my feeling is that if you would not have had a separate loader and second, you should really be able to do without a separate wrangler and second.

That said the practical reality is that some current cameras and their support software can be much more attention-seeking than perhaps they should be. There's also the factor that a lot of film-oriented ACs want to sell themselves as computer people when perhaps that's a bit ambitious/

Phil's Two Cents: Play by ear. Depends on the camera system.

P
  • 0

#11 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 20 November 2010 - 04:08 PM

I'm also curious what you guys use to backup in the field ....


We have one show that's taking a radical approach, not backing up in the field at all. They just put the cards from the camera in a pellican case with the camera reports, and at the end of the day, the whole thing goes to the post house. Just like exposed film, only easier.... So far, it's working.





-- J.S.
  • 0

#12 Jamie Metzger

Jamie Metzger
  • Basic Members
  • PipPipPipPip
  • 773 posts
  • Cinematographer
  • San Francisco

Posted 20 November 2010 - 09:36 PM

We have one show that's taking a radical approach, not backing up in the field at all. They just put the cards from the camera in a pellican case with the camera reports, and at the end of the day, the whole thing goes to the post house. Just like exposed film, only easier.... So far, it's working.





-- J.S.



I remember reading that Fincher did that on "the social network" I can't see that as being a very viable option these days, as I normally blow through 20+ CF cards /day. I recently did a a 3 Camera RED shoot where we shot 500GB/day, plus I was transcoding on set. This was a very "full time" job.
  • 0


FJS International, LLC

Wooden Camera

CineLab

Willys Widgets

Tai Audio

Media Blackout - Custom Cables and AKS

Rig Wheels Passport

The Slider

Opal

Technodolly

Metropolis Post

Abel Cine

Aerial Filmworks

Glidecam

Ritter Battery

Broadcast Solutions Inc

rebotnix Technologies

Gamma Ray Digital Inc

Visual Products

Paralinx LLC

CineTape

The Slider

FJS International, LLC

Wooden Camera

CineTape

CineLab

rebotnix Technologies

Media Blackout - Custom Cables and AKS

Visual Products

Gamma Ray Digital Inc

Abel Cine

Technodolly

Willys Widgets

Paralinx LLC

Broadcast Solutions Inc

Tai Audio

Metropolis Post

Aerial Filmworks

Opal

Ritter Battery

Rig Wheels Passport

Glidecam