Jump to content


Photo

Managing Massive Amounts of Footage


  • Please log in to reply
4 replies to this topic

#1 Brian Rose

Brian Rose
  • Basic Members
  • PipPipPipPip
  • 896 posts
  • Student
  • Kansas City area

Posted 13 October 2011 - 10:57 AM

I'm about to embark on a massive, and I mean massive footage archiving project for my (new) employer. They possess a substantial collection of historical reenactment footage, which they license as stock footage, and want it all digitized to take their operations to the next level.

The footage spans several decades, and every imaginable format: D2,Beta SP, digibeta, DVCAM, miniDV, HDCAM, HDV, file types of every kind, super 8mm, 16mm, 35mm...

Digitizing the footage will be no problem, as we have all the inhouse facilities for tape transfers, and the film will just be a matter of telecine to pro-res.

What I'm wrestling with is just how to managed so much footage. Rough estimates are about 1000 hours of material. Given DV is around 12 gigs an hour, and you're looking at a minimum of 12 tbs, and really there will be much more than that, since a substantial chunk of that footage is HD material.

Spanning across a bunch of 2tb hard drives seems impractical, and dangerous, as each hard drive I add, the law of large numbers says one will fail, which will SUCK. Not to mention those drives are not exactly idea for long term storage. LTO struck me as another option, though that seems more suited as a backup/storage solution, whereas this material would need to be in a state that is accessible, since the point is to have this material available for license.

We also have a SAN and a fibre network, though at this point it is more of an active terminal, in that it is for live projects, and it's divided into smaller hard drives of 2-4 TB which are each dedicated to a particular editor. Total size is 22 or 24 tbs. Is expansion of the SAN an option, perhaps to establish a dedicated stock drive? Or do I look at constructing a RAID in tandem with our server?

Man, so many, many options, and in many ways I am a novice, and learning as I go. I'd sure love some input from you all, whom I'm sure have worked with large amounts of footage. How do you manage it in a way both efficient in terms of accessibility, while ensuring redundancy and safety of storage?

Best,

BR
  • 0

#2 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11939 posts
  • Other

Posted 13 October 2011 - 12:13 PM

If this is the basis of a business, it's valuable data, so I think you should put it onto LTO (twice) in any case.

Otherwise yes keep it on a SAN or other network-accessible drive, that would be the normal approach. 12TB of storage is no longer the hundreds of thousands it used to be. If you also have it on LTO if you do decide to do something else you can just pull it back off the LTO.

Why ProRes?

Are you going to create low-res thumbnails for people to view? Is that going to go on a website?

P
  • 0

#3 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 13 October 2011 - 02:31 PM

It's best to get a complete solution from one vendor, so there's only one place to point the finger when something needs to be fixed. I haven't used them, but I've heard good things about Isilon:

http://www.isilon.com/




-- J.S.
  • 0

#4 Brian Rose

Brian Rose
  • Basic Members
  • PipPipPipPip
  • 896 posts
  • Student
  • Kansas City area

Posted 13 October 2011 - 03:03 PM

Phil,

the pro res is merely the native format (such as our converted 5D material). For online/web purposes we'd be generating motion jpeg proxies with BITC for comping.

I've been reading all day, and consulting with another techie in the office, and I think I'm leaning toward a three level approach:

Level 1: Proxies. This would be available always for use for inhouse comping, as well as for those needing stock footage

Level 2: Online originals. Factoring on cost as well as existing infrastructure, we're probably going to go with spanning the stock footage across 2TB hard drives which would be accessed when it comes time to relink clips, or provide the client with full res versions

Level 3: LTO backup. But considering hard drives DO fail, and as we add hard drives, the law of large numbers suggests a failure is increasingly likely, all the footage would be backed up to LTO 5, whose tapes roughly equal the hard drives we use (1.5 TB). We've been needing a more effective archiving solution for some time, as our current workflow has been to back up raw footage to blu-ray. Of course, we've filled two massive binders already and we're burning our way through a third. I think the environment is right that I'm going to compile a cost comparison report, to show the savings of using LTO for our archival storage.

So thanks all for your help, and if you have anything else to add, I'd be eager to read it!

Best,

BR
  • 0

#5 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11939 posts
  • Other

Posted 13 October 2011 - 03:46 PM

Sounds reasonable, although the acid test is if your insurers are willing to underwrite it. They are likely to require that you make two sets of LTOs and hold one of them at an offsite location, which isn't difficult to do. At that point I think you're doing more than you need to.

What sort of disk arrays are you using? I'm not so sure about RAID-5, especially in big arrays. It saps performance, often ends up costing more than the extra drives required to maintain a given space or rate in RAID-10, takes forever to rebuild, and loses performance when broken. In many situations the rebuild time for a big RAID-5 is so large that the likelihood of a second drive failure during the rebuild (which will destroy the array) becomes uncomfortably high.

P
  • 0


Metropolis Post

rebotnix Technologies

Opal

Broadcast Solutions Inc

CineTape

FJS International, LLC

CineLab

Media Blackout - Custom Cables and AKS

Tai Audio

Glidecam

Paralinx LLC

Technodolly

Gamma Ray Digital Inc

Wooden Camera

Willys Widgets

Abel Cine

Rig Wheels Passport

The Slider

Ritter Battery

Visual Products

Aerial Filmworks

Tai Audio

Paralinx LLC

Visual Products

rebotnix Technologies

Glidecam

Gamma Ray Digital Inc

FJS International, LLC

Media Blackout - Custom Cables and AKS

Broadcast Solutions Inc

Willys Widgets

Opal

Abel Cine

CineLab

The Slider

CineTape

Technodolly

Wooden Camera

Aerial Filmworks

Metropolis Post

Ritter Battery

Rig Wheels Passport