Jump to content


Photo

Tell me about your workflow for backup


  • Please log in to reply
1 reply to this topic

#1 Brian Rose

Brian Rose
  • Basic Members
  • PipPipPipPip
  • 896 posts
  • Student
  • Kansas City area

Posted 16 January 2012 - 03:20 PM

All,

At the production company where I find myself employed, one of my duties is data management and archiving. The company I work for has grown slowly, but lately begun to accelerate in expansion, which is great, though we now find ourselves having to revise and make more efficient our methods of data backup.

The method used to be fairly simple, with files on an external drive, and the original tapes (HDV, DVCPRO etc) serving as the backup. Now we're doing file based stuff necessitating two copies. Prior to my hiring on, the method of backup was to burn the files to blu-rays. But now we're generating so much footage that this isn't really practical any more, and blu-rays are hardly archivally sound.

I'm pushing for LTO5 backup, which everyone is on board for. But what complicates things is for projects often we'll have three or even four people editing the same thing, from a central SAN to which we are all linked via fibre.

So what I'm struggling with is to create a good workflow, that ensures everything is getting backed up, so we can be safe to delete from the SAN to free up space; yet we don't want to be wasting LTO space by backing up two and three times. There's also the issue of spanning projects across multiple tapes, or if one backs up several times during the course of a project, soon a project is scattered across multiple tapes with other things.

I think we just need to start from the ground up, devise a total workflow start to finish. But I could use your help. Tell me about your workflows for working with and backing up footage. What suggestions would you have? What backup programs would you recommend to aid us in our work?

Many Thanks!
BR
  • 0

#2 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11937 posts
  • Other

Posted 16 January 2012 - 07:42 PM

This stuff is so very dependent on exactly how you work.

Are your editors making local copies of things? Are they producing significant amounts of rendered output (if they're doing graphics, or whatever)?

Usually if it's normal longform edit, there'll be a bunch of media files which won't change much as things progress, so they only need backing up once, and the project files and ancillary things like graphics and audio snippets which may change quickly. On one occasion we had one folder for stuff that was static or anticipated to be static, and one folder for stuff that was expected to change. The changeable folder was backed up often, and the static one wasn't. It was crude but effective, but only as long as we were all reasonably technical people who knew what we were doing.

Of course this relies on people following the rules, and the biggest problem is not designing the system, it's making people follow it. There are commercial products which claim to automate all this stuff, but there are problems: they may still force you to work in particular ways and require you to follow rules, and frankly getting checksums of huge media files to see if they've changed is a lengthy process in itself. Mostly I see people running some sort of custom-written stuff, whether that's just a nightly backup script or a more fully-featured thing with a web interface and a way to book files in and out (but that still requires people to take action to ensure the backup reflects changed files).

The short answer is that while some people do write collaborative edit software that at least claims to handle these sorts of situations, there is no perfect universal solution that's used by everyone. Most of the situations I've seen, even at some quite big post places, were pretty hacked-together.

One way of making it faster to find out what's altered is to rely on the filesystem's date-last-changed fields, or the filesize, but sometimes these may not be reliable ways of identifying a file's contents.

P
  • 0


Aerial Filmworks

Opal

The Slider

Technodolly

CineLab

Broadcast Solutions Inc

Tai Audio

Glidecam

Rig Wheels Passport

Willys Widgets

Ritter Battery

Wooden Camera

Visual Products

CineTape

Gamma Ray Digital Inc

Media Blackout - Custom Cables and AKS

Metropolis Post

FJS International, LLC

rebotnix Technologies

Abel Cine

Paralinx LLC

Abel Cine

The Slider

Visual Products

FJS International, LLC

Willys Widgets

Metropolis Post

Tai Audio

Broadcast Solutions Inc

Technodolly

rebotnix Technologies

CineLab

Rig Wheels Passport

CineTape

Ritter Battery

Paralinx LLC

Aerial Filmworks

Media Blackout - Custom Cables and AKS

Gamma Ray Digital Inc

Wooden Camera

Opal

Glidecam