Jump to content


Photo

How do you handle storage/backup of your footage/projects?

RAID Storage Backup

  • Please log in to reply
17 replies to this topic

#1 Sean Emer

Sean Emer
  • Basic Members
  • PipPip
  • 44 posts
  • Cinematographer
  • Los Angeles

Posted 06 February 2019 - 03:01 PM

Hi guys! As my career progresses I've found myself without an adequate solution for storing my footage and old projects. I try to cut a new reel every year or so, and right now to get to work I need to have as many as 8 separate HDDs plugged in. These are mostly personal drives or backup drives from the projects themselves. I'd love to build myself one giant tower of hard drives that I could have everything on and which would be RAID-0 for safety, but I've never built such a thing before. I've got around 10TB of footage to store right now, but would like to be future proofed, so some kind of enclosure in which I can slide drives in/out depending on my needs would be great. Do any of you have good recommendations for external enclosures, brands to prefer/avoid, that sort of thing? Thanks for any feedback you can provide!


  • 0

#2 Sean Emer

Sean Emer
  • Basic Members
  • PipPip
  • 44 posts
  • Cinematographer
  • Los Angeles

Posted 06 February 2019 - 03:37 PM

I should clarify that I meant to type RAID-1 above, not RAID-0


  • 0

#3 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 12405 posts
  • Other

Posted 06 February 2019 - 06:00 PM

If you want a RAID, the drives will generally remain part of it indefinitely. I have an 8-disk RAID-10 which gives me 12TB of fast online storage. More is easy; it's just more money. If you like, I can go into the details on how it was made; it's not difficult to do if you can handle a screwdriver and afford the parts. What sort of system do you want to connect it to?

 

I use half a dozen 4TB USB-3 external drives for shuttling data around.

 

The real solution to this is something like an LTO tape drive, for deeper archiving. Unfortunately this is not a service anyone's ever willing to pay for - they don't think it's important until suddenly there's a problem, at which point it magically becomes very important. LTO drives are expensive, but it's more or less the only realistic archiving solution out there. You can often get LTO drives that are a generation old on ebay for reasonable money.

 

P


  • 0

#4 Sean Emer

Sean Emer
  • Basic Members
  • PipPip
  • 44 posts
  • Cinematographer
  • Los Angeles

Posted 06 February 2019 - 06:14 PM

Thanks for the reply Phil! I'd love to hear how you did yours. Right now I'm looking at an external enclosure with USB 3 connection. LAN connections don't seem particularly worth it for my applications, and my PC doesn't have eSATA or anything like that. I custom built my PC, so I am familiar with the screwdriver :)

 

I have 6 8TB drives right now, but can buy more to fill in the gaps if needed.


  • 0

#5 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 12405 posts
  • Other

Posted 06 February 2019 - 06:40 PM

I simply built my workstation into a rack case with eight drive bays in the front and put a Highpoint RAID controller in it. At this point you're free to buy any drives you like at commodity prices. I had to buy multilane SAS to SATA cables and there are the usual issue of fit and finish when cabling everything up. A bit of fiddling around in the controller BIOS. Usual stuff. The result is a 12TB storage bin that sustains 550MB/second when half full and somewhat fragmented. If you want more you could put 4TB drives in it. If you want a lot more, get a 16-bay case and a 16-way controller. It is the cheapest, simplest way to create a big, fast bit bucket.

 

A few things I found: 

 

- People will say "use SSDs." So far whenever I've done the mathematics, spinning rust is still better on a size basis. Putting SSDs on a SATA RAID controller will often saturate the four-lane PCI bus connection anyway. If you absolutely need to do several streams at once, when seeking will destroy the performance of a disk-based setup, you may need SSDs.

 

- Don't use RAID-5, or RAID-6, use RAID-10 no matter what anyone tells you. You will need a very expensive controller to make the more complex RAID levels fast enough, and just adding more disks to offset the space-inefficiency of RAID-10 is often a cheaper and faster approach.

 

- People will say "use LSI, not highpoint. Highpoint is cheap." I tried an LSI controller and returned it; it wouldn't do big block sizes, it wouldn't do RAID-10 (I had to do RAID-1 on the LSI controller, and RAID-0 the resulting pairs in Windows disk management.) Yes, the Highpoint is cheaper. It also seems to work better. Your mileage may vary.

 

- Ponder cooling. The Highpoint controller gets alarmingly hot if there isn't adequate airflow; I had to add an extra cooling fan for peace of mind. More to the point, it's common to put a 120mm fan behind each stack of four 3.5" disks; I used Noctua fans, which are quiet, and found it was possible to run them at reduced power for even lower noise if you have to be in the room with it, while maintaining reasonable temperatures.

 

- Ponder power. A lot of PCs people build have either very inadequate power supplies becuase they got cheap, or very overspecified ones because of a bad experience with the inadequate option. Hard disks aren't that hungry, but it's worth checking. If your current PSU doesn't have enough SATA power connectors to run all the drives, that's a warning sign.

 

- USB3 is nice if you want to move it around. If you don't, just get a rack case with drive bays and put it all in there. Internal will be faster, given the direct PCIe connection, and probably cheaper and more reliable.

 

- Rack cases go easily into off-the-shelf flight cases for portability. Woohoo!

 

Oh, and edit: replace the drives before they fail. If you leave the drives running 24/7, they'll tend to do two or three years before starting to hit problems. The controller may be able to give you some warning by reporting the SMART data, and of course when a drive fails you have the RAID protection. When failures start to occur, it's naturally likely that since all the drives are the same age they'll start to cascade-fail. When that happens, bite the bullet and replace them all. Then you just have to figure out what to do with the data while you do that. RAID is not backup of course, and you still need LTO.

 

P


  • 0

#6 Sean Emer

Sean Emer
  • Basic Members
  • PipPip
  • 44 posts
  • Cinematographer
  • Los Angeles

Posted 06 February 2019 - 07:06 PM

Thanks for the great writeup Phil, I appreciate your help!

 

Are you sure that a RAID 10 setup is cheaper than a 5-enabled controller? I could get 70TB out of 8x 10TB drives w/ RAID5. I'd need 14 drives to get that from RAID 10! Raid 5 just sounds like a good deal, especially when the basic pre-built enclosures don't seem to be more expensive to add RAID 5 to.


  • 0

#7 Daniel D. Teoli Jr.

Daniel D. Teoli Jr.
  • Basic Members
  • PipPipPipPip
  • 300 posts
  • Other
  • Born in L.A....NYC is 2nd home...Rustbelt is home base.

Posted 09 February 2019 - 10:20 AM

How do the big boys store and back up their feature films? That must be something!

 

If I hit the lotto I will buy one of these...or a few of them.

 

https://www.bhphotov...Top Nav-Search=


Edited by Daniel D. Teoli Jr., 09 February 2019 - 10:21 AM.

  • 0

#8 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 12405 posts
  • Other

Posted 09 February 2019 - 11:37 AM

RAID is not backup. Long term storage on big movies is either RGB separations on 35mm or, more commonly, LTO tape, which is designed to be archival. Hard disks are actually a pretty poor archive medium. Sometimes stuff will sit on RAIDs that are running and being maintained if it's anticipated to be needed.

 

Sean, it depends to a small degree on what you're trying to achieve. If you're buying 10TB drives, those will be expensive, and a RAID-5 controller may make more sense. I'd still bear the following in mind:

 

- A controller capable of RAID-5 that's as fast as a RAID-10 of the same capacity will be (much) more expensive. Calculate whether it's worth it.

- RAID-5 often gets very slow when degraded, to the point where it is unusable. The data may be recoverable, but you still can't work.

- Rebuilding a degraded RAID-5 can be very, very slow. Days, even. The likelihood of a further (and therefore catastrophic) failure during the recovery process is nonzero.

- A degraded RAID-5 is much less failure-tolerant than a RAID-10 of the same size. Failure of any further disk will destroy the RAID-5, whereas any disk except one can still fail on the RAID-10 without loss of data.

 

You're probably right that at very large drive sizes the economics may change but I'd look at what you can achieve with a larger number of smaller drives. 4TB disks are (based on a quick poll) about 0.28 times the price of 10TB disks, but 0.4 times the capacity.

 

P


  • 1

#9 aapo lettinen

aapo lettinen
  • Basic Members
  • PipPipPipPip
  • 1216 posts
  • Other
  • Finland

Posted 09 February 2019 - 12:37 PM

if you don't have huge read/write speed requirements you can buy a raid box with for example 6 bays but keep the drives independent (not raiding them at all so that the box is basically acting as a large hdd dock and every drive is showing independently to the computer) and then it is possible to change drives if needed and keep the most used ones attached. 

 

all hard drives will inevitably fail at some point and you may never know when that happens. I recommend keeping a separate archive copy of the material on LTO whatever the primary backup plan would be


  • 0

#10 aapo lettinen

aapo lettinen
  • Basic Members
  • PipPipPipPip
  • 1216 posts
  • Other
  • Finland

Posted 09 February 2019 - 12:41 PM

if you don't have huge read/write speed requirements you can buy a raid box with for example 6 bays but keep the drives independent (not raiding them at all so that the box is basically acting as a large hdd dock and every drive is showing independently to the computer) and then it is possible to change drives if needed and keep the most used ones attached. 

this is to keep the setup more neat than having multiple SATA drives and separate docks around. 

Bare sata drives are cheaper than external hard drives and if you don't need to change them every day it may be worth it to use them this way. I personally use lots of SATA drives for backups just because working with large amounts of material at a time and the sata drives give the best price/performance ratio plus they are easier to store if you have dozens of drives per project


  • 0

#11 Sean Emer

Sean Emer
  • Basic Members
  • PipPip
  • 44 posts
  • Cinematographer
  • Los Angeles

Posted 09 February 2019 - 02:27 PM

My plan for now is to use a raid tower and run 8x 10TB NAS drives in Raid 50. That way I have a fault tolerance of 2 drives and still get some speed benefits without halving my available space. Almost all of the media is backed up on other HDDs as well, and my hope is that in my career the footage that degrades and is lost on old hard drives will be irrelevant to me once they fail. Thank you for the tips on LTO though! Some day in the future when I'm rich perhaps I can do a true backup of my favorite works.


  • 0

#12 aapo lettinen

aapo lettinen
  • Basic Members
  • PipPipPipPip
  • 1216 posts
  • Other
  • Finland

Posted 09 February 2019 - 06:13 PM

My plan for now is to use a raid tower and run 8x 10TB NAS drives in Raid 50. That way I have a fault tolerance of 2 drives and still get some speed benefits without halving my available space. Almost all of the media is backed up on other HDDs as well, and my hope is that in my career the footage that degrades and is lost on old hard drives will be irrelevant to me once they fail. Thank you for the tips on LTO though! Some day in the future when I'm rich perhaps I can do a true backup of my favorite works.

 

if you know someone with an LTO7 or 8 setup you can ask them to back it up for you from the RAID when you get it up and running. it is generally not practical to own a expensive LTO system if using it for only couple of projects per year... if shooting something like 100TB per year or more or having extensive archive of projects on LTO then it would make sense. For example archiving a single movie project per year with perhaps 30TB of raw material and some smaller projects yearly it would not be very cost effective to own the LTO by yourself I think but it would probably be economical to pay a little for someone else to make the LTO backups for you, especially if running a newer system using LTO7 or 8 where tape capacities are large. 

 

As a comparison, I have backed up many projects raw materials around 30-120TB a piece on LTO-5, it is actually pretty practical if you can finance the tape costs because the one tape would be about one shooting day worth of material, sometimes two days even ( one LTO-5 tape would store around 1.38 - 1.47 TB of video material depending on the hdd's feeding the drive, the general system configuration and how fragmented the data is on the hard drives. LTO works by writing data and at the same time immediately reading it back to ensure it was written correctly and if not, it will write it again until it went correctly on the tape. No possibility to rewind and rewrite over the bad data, it must be written again to keep the tape speed in limit. This means that any errors in the backup process (for example caused by slow HDD's feeding the LTO station) will result in more write errors and will end up consuming more tape and will thus lower end capacity of the tape because more of the space was wasted due to the write errors. So the capacity varies from tape to tape depending on a lot of things. video data does not benefit from the LTO compression so the max capacities for raw video are normally the uncompressed capacity of the tape or lower) 

 

for your needs a LTO6 capable system would probably be more than enough so a LTO7 drive could be the practical choice if you really want your own system (you can write and read LTO7 and LTO6 tapes on an LTO7 drive and read LTO5 tapes and you could save some money on backing up on LTO6 if LTO7 tape is not needed for the job due to capacity. If you decide to update the system to LTO8 someday later you can still read the 6 tapes with it) . if using someone else's LTO system I would do the backups on LTO7 or LTO8 tapes to be able to save on the manual work with the tapes and get the material in smaller physical space. larger tape capacity would be beneficial because the material would be in larger chunks due to being already archived on raid and the fast raid setup would lower tape usage by lowering the amount of write errors. Thus the newer generation drive would save a lot I think in the case you would use someone else's LTO drive and if the material is already archived so that you could write the large tapes full )


  • 0

#13 Daniel D. Teoli Jr.

Daniel D. Teoli Jr.
  • Basic Members
  • PipPipPipPip
  • 300 posts
  • Other
  • Born in L.A....NYC is 2nd home...Rustbelt is home base.

Posted 10 February 2019 - 03:18 PM

RAID is not backup. Long term storage on big movies is either RGB separations on 35mm or, more commonly, LTO tape, which is designed to be archival. Hard disks are actually a pretty poor archive medium. Sometimes stuff will sit on RAIDs that are running and being maintained if it's anticipated to be needed.

 

Sean, it depends to a small degree on what you're trying to achieve. If you're buying 10TB drives, those will be expensive, and a RAID-5 controller may make more sense. I'd still bear the following in mind:

 

- A controller capable of RAID-5 that's as fast as a RAID-10 of the same capacity will be (much) more expensive. Calculate whether it's worth it.

- RAID-5 often gets very slow when degraded, to the point where it is unusable. The data may be recoverable, but you still can't work.

- Rebuilding a degraded RAID-5 can be very, very slow. Days, even. The likelihood of a further (and therefore catastrophic) failure during the recovery process is nonzero.

- A degraded RAID-5 is much less failure-tolerant than a RAID-10 of the same size. Failure of any further disk will destroy the RAID-5, whereas any disk except one can still fail on the RAID-10 without loss of data.

 

You're probably right that at very large drive sizes the economics may change but I'd look at what you can achieve with a larger number of smaller drives. 4TB disks are (based on a quick poll) about 0.28 times the price of 10TB disks, but 0.4 times the capacity. Plans are not set in concrete, still trying to figure it all out, as in best fit for a small budget.

 

P

 

 

Is this what you are talking about?

 

https://www.quantum....to-7/index.aspx

 

I never heard of them. Are they practical for the needs of this thread? I could not find a price. Usually when they hide the price it is very $$.

 

For me all I have is hard or flash drives. I thought about buying the 'M' disk for archival work of still photos, but they seem to not be that mainstream. I think the M disk only goes up to 100 GB as well. 

 

I have close to half a million feet of film that needs scanning and archiving. So I wondering what are the best options to store it. It wont be all MP4's. I hope to have it frame by frame TIFF then converted to HR jpeg for storage after the grading and then DVD. Converting TIFF to jpeg save maybe 80% of the space as a rough estimate. 


Edited by Daniel D. Teoli Jr., 10 February 2019 - 03:25 PM.

  • 0

#14 Daniel D. Teoli Jr.

Daniel D. Teoli Jr.
  • Basic Members
  • PipPipPipPip
  • 300 posts
  • Other
  • Born in L.A....NYC is 2nd home...Rustbelt is home base.

Posted 10 February 2019 - 03:31 PM

I guess the lto's can be a nightmare if things don't work out...

 

Terrible support & software, Rarely works on Mac, don't buy
By Matt
 
I'm a big believe in LTO for data storage and archiving. It's cost effective and secure, easy to safely store lots of data for many years. This is no solution though. I'm running it to a Mac pro over thunderbolt 2 via a Highpoint Rocketstor I/O box. I have 12 or so tapes of data archived. A recent Mac OS update seems to have killed the drives ability to mount tapes (again). I have spent lots of time with the friendly HP folks in India, have had two replacements overnighted (which took almost two weeks). There is an obscure program called HPE Library and Tape Tools which is the only way to check or update firmware from a Mac. With a little digging through HP's bafflingly awful, sprawling mess of a Web presence, it can be found, downloaded, and installed, but then it does not work on Mac. There is no way to update firmware, no way to decipher the many bizarre error codes and malfunctions it has, barely any way to convince HP's tech support that they even sell LTO drives (just google this drive and see what actual info and support you can find). Terrible. definitely go with Tolis, mLogic, anyone else.

 

 

https://www.bhphotov...#customerReview


Edited by Daniel D. Teoli Jr., 10 February 2019 - 03:33 PM.

  • 0

#15 Tyler Purcell

Tyler Purcell
  • Sustaining Members
  • 4463 posts
  • Other
  • Los Angeles

Posted 10 February 2019 - 04:32 PM

There are three types of storage; Online, Nearline and offline.

Online would be your directly connected in-use raid storage. This needs to be fast and large, not necessarily redundant. Raid 5 and Raid 6 overhead is massive, it slows down ANY array quite a bit. For at-home use, especially in the sub 20Tb category, you can get away with a Raid 0 solution that's simply backed up so if it fails, you'd be ok. If you're editing proxy media, you can run a Raid 5 or 6 solution, but again this is not a good backup or redundancy. A bad drive in a Raid 5 enclosure, could take day's to reconstitute the data, where a simple nearline backup of your media would be faster to get you back online again.

Nearline storage is that USB drive you have in your closet or safe. This is what you would use for short-term backup's of original media AND project files. These would normally be stored in a secure location so if your raid went down, you'd have some sort of backup. Many people use nearline storage for full-time use and then take the drive and store it afterwards. I personally prefer to keep the drives not spinning and leave nearline as only a backup resource. I use nearline storage for all of my backup's and simply buy new drives every 5 years to duplicate the media onto.

Offline storage is usually long-term. These are archived projects that have been delivered and you're ready to delete off your nearline storage drives. This is so incase a client needs that media sometime up the road, you will be able to access it. Many people use hard drives for this, but LTO is the best solution because it's pretty foolproof and if the data is not encrypted, it will be able to be read back 10 years from now without an issue.

In terms of a raid array, the more drives the better. Fewer, higher capacity drives are not a smart solution because they have a lower MTBF and the more platters you have, the faster the raid. I highly recommend over 6 drive raids, the 8 - 12 drive raids are the best "alternative" to a 24 or 32 bay enclosure which are the industry standards. Truthfully, the problem lies with what kind of connection you're using as well. Thunderbolt and USB C are great, but finding a computer that's not brand new with native support is hard and you can't just add it. So a lot of people resort to SAS, but it's not very fast either. One solution is buying a case that can take the drives internally. This is easy to do with 2.5" SSD's and with the cost of SSD's decreasing rapidly, in the next year or so, we should see 4TB SSD's in the price bracket of similar spinning disk's today. Imagine a 8X4tb SSD's as your "raid" array. The one we have at the office is around 1600Mbps and it's just 8 drives running off a sata PCI card. Really simple.

When talking nearline storage, I prefer the 2.5" portable drives, over the 3.5". The 2.5" drives seem to last a lot longer because they're designed for laptops and have a much higher MTBF. They do appear to sit better over long periods of time as well. I've had around 3 3.5" drives fail on me and not a single 2.5" yet. :fingers crossed: Also, the 2.5" drives are cheap these days. You can get 4tb for $95 bux on Amazon! Where a standard 4tb 3.5" USB drive is around $150 - $199. So cheaper, save space, no need for a power supply and last longer.

My process is very simple. I get the project on my raid first and start editing. Once I'm in it for a bit of time and the project folders are all organized properly. I will back those project folders up onto the 2.5" nearline storage. I will back the project files up on dropbox. Final outputs go onto dropbox and TWO nearline storage drives, one 2.5" and one 3.5", both stored in different locations. This way if anything goes down, I have the projects off-site backed up. I have the media on nearline storage backed up. I have the finals on the cloud AND two different spinning disk's backed up. Then I also give the client a drive with all the media AND the final's as well. So there are 4 copies of the finals and 2 copies of the media, before I delete anything off my raid. I figure in the length of time it takes for the drives to go bad, the client will either not care about the project anymore OR they'll have backed it up. I also don't ever erase my nearline storage like most people do. I keep it forever and as technology changes, I back it all up to other drives. I have drives dating back to 2001 that are still working and are backed up to other new drives as the connections have changed over the years.
  • 0

#16 aapo lettinen

aapo lettinen
  • Basic Members
  • PipPipPipPip
  • 1216 posts
  • Other
  • Finland

Posted 11 February 2019 - 04:10 AM

I guess the lto's can be a nightmare if things don't work out...

 

Terrible support & software, Rarely works on Mac, don't buy
By Matt
 
I'm a big believe in LTO for data storage and archiving. It's cost effective and secure, easy to safely store lots of data for many years. This is no solution though. I'm running it to a Mac pro over thunderbolt 2 via a Highpoint Rocketstor I/O box. I have 12 or so tapes of data archived. A recent Mac OS update seems to have killed the drives ability to mount tapes (again). I have spent lots of time with the friendly HP folks in India, have had two replacements overnighted (which took almost two weeks). There is an obscure program called HPE Library and Tape Tools which is the only way to check or update firmware from a Mac. With a little digging through HP's bafflingly awful, sprawling mess of a Web presence, it can be found, downloaded, and installed, but then it does not work on Mac. There is no way to update firmware, no way to decipher the many bizarre error codes and malfunctions it has, barely any way to convince HP's tech support that they even sell LTO drives (just google this drive and see what actual info and support you can find). Terrible. definitely go with Tolis, mLogic, anyone else.

 

 

https://www.bhphotov...#customerReview

 

LTO systems tend to be very picky about software versions, operating system version, driver and hardware compatibility so generally it is not a good idea to just shop for a LTO drive and random SAS card online and blindly purchase the first option. the incompatibilities may start with different SAS connectors on the card and the drive (just purchase another card then...) and the drive may be incompatible with the SAS card or the SAS card may be incompatible with your computer or your archiving software etc. 

 

I for example have the LTO5 set up on imac with the SAS card on external pci-e thunderbolt box (sonnet echo express se2) and I had to literally check EVERYTHING before dared to purchase that box in the first place. took about a week or more of hard work to make sure everything SHOULD work correctly before could purchase anything. Even if the drive is compatible with the SAS card and the SAS card is compatible with the Thunderbolt box, the drive and software may not support the data through VIA the box in LTO use  :blink:  there was other type of box which DID NOT support LTO use with that Tandberg drive even when it was compatible with the card and computer and software and everything else. 

 

SO, you need to be very careful when setting up the LTO system for the first time and especially if it is setup on a computer you need for other work like an edit computer. That's because you usually need to update your edit every now and then (for example update the operating system because FCPX or Adobe programs cannot be otherwise updated further) and basically ANY system or software update may disturb the LTO drivers or software, even rendering them unusable unless you purchase new software versions. 

 

Setting up a LTO system is relatively easy if everything goes well but these setups tend to have crazy compatibility problems with everything so it is best to have a dedicated computer just for LTO use and not for anything else and no one is allowed to update anything in it without permission. I personally don't even connect the dedicated LTO computer to the internet EVER so that I can be sure it does not auto update anything and I don't need to purchase new software versions for everything which would easily cost 1000 USD or more :o

 

with Thunderbolt compatible LTO drives it is slightly easier to setup them because you don't have to worry about the SAS card and possible external box compatibility so you have less things to worry about though the drive may still be incompatible with the computer and software/operating system and there is quite few of the Thunderbolt compatible drives available in the first place so the options are quite limited. But as said you really should dedicate a separate computer for LTO work and not try to use your edit for it. You also don't want to have the edit reserved by the LTO station multiple hours at a time if you have to back up something or restore something from the tapes... using the computer for something else at the same time tends to disturb the LTO transfers so you generally don't want to do anything with the computer when it backs up or restores and therefore it is not practical to use your valuable edit for the LTO tasks by my opinion


  • 0

#17 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 12405 posts
  • Other

Posted 11 February 2019 - 04:57 AM

I've reviewed several LTO drives under Windows and not had any problems. The LTO drive itself connects to the computer via SAS. You will require the drive, a SAS card and appropriate cable. The drivers you need will be those for the SAS card, and those to turn that raw block device into something you can actually store files on. The LTO group has developed LTFS, the Linear Tape Filesystem, which makes it possible to drag and drop files onto the LTO tape in exactly the same way as we would with any other storage device. Early versions of LTFS could be a bit finickity, but in my view that's long since been worked-out. I just plugged it into my normal workstation (at which I sit right now) and it was pretty straightforward.

 

I think by involving a Thunderbolt external enclosure you're inviting problems. At some point, get a workstation. Not everything has to be an iMac. I have no idea what software you're having to purchase; LTFS is free.

 

There is not really any such thing as a thunderbolt or USB3 LTO drive; they're all SAS natively. Such things are sold, but you're really just buying a box containing a conventional LTO drive in a box with the relevant converter electronics.

 

In terms of support, you are buying into something that's not a consumer product and the web presence supporting it is not aimed at consumers. These things go into banking and medical records archiving in big datacentres. It's not a CD burner. The upside is that you can probably get an email address which will be answered by a person - you're paying enough for it - so I've not found support to be an issue.

 

To Daniel's question:

 

I have close to half a million feet of film that needs scanning and archiving. So I wondering what are the best options to store it. It wont be all MP4's. I hope to have it frame by frame TIFF then converted to HR jpeg for storage after the grading and then DVD. Converting TIFF to jpeg save maybe 80% of the space as a rough estimate. 

 

I'd have said that was a pretty good LTO application, yes. The thing is, it's pretty much the only game in town; there is nothing else that gives you the archival permanence, write speed, or the cost per megabyte, at least after you write a certain number of tapes and amortise the cost of the drive. LTO is not a particularly cheap format, though in context it's a lot cheaper than even a used SRW-5800 HDCAM-SR deck. If you want to go even cheaper, you can buy a used LTO deck a couple of generations old from eBay. There's a degree of backward compatibility in new drives.

 

P


  • 0

#18 aapo lettinen

aapo lettinen
  • Basic Members
  • PipPipPipPip
  • 1216 posts
  • Other
  • Finland

Posted 11 February 2019 - 05:36 AM

I've reviewed several LTO drives under Windows and not had any problems. The LTO drive itself connects to the computer via SAS. You will require the drive, a SAS card and appropriate cable. The drivers you need will be those for the SAS card, and those to turn that raw block device into something you can actually store files on. The LTO group has developed LTFS, the Linear Tape Filesystem, which makes it possible to drag and drop files onto the LTO tape in exactly the same way as we would with any other storage device. Early versions of LTFS could be a bit finickity, but in my view that's long since been worked-out. I just plugged it into my normal workstation (at which I sit right now) and it was pretty straightforward.

 

I think by involving a Thunderbolt external enclosure you're inviting problems. At some point, get a workstation. Not everything has to be an iMac. I have no idea what software you're having to purchase; LTFS is free.

 

There is not really any such thing as a thunderbolt or USB3 LTO drive; they're all SAS natively. Such things are sold, but you're really just buying a box containing a conventional LTO drive in a box with the relevant converter electronics.

 

In terms of support, you are buying into something that's not a consumer product and the web presence supporting it is not aimed at consumers. These things go into banking and medical records archiving in big datacentres. It's not a CD burner. The upside is that you can probably get an email address which will be answered by a person - you're paying enough for it - so I've not found support to be an issue.

 

To Daniel's question:

 

 

I'd have said that was a pretty good LTO application, yes. The thing is, it's pretty much the only game in town; there is nothing else that gives you the archival permanence, write speed, or the cost per megabyte, at least after you write a certain number of tapes and amortise the cost of the drive. LTO is not a particularly cheap format, though in context it's a lot cheaper than even a used SRW-5800 HDCAM-SR deck. If you want to go even cheaper, you can buy a used LTO deck a couple of generations old from eBay. There's a degree of backward compatibility in new drives.

 

P

 

yep I meant the integrated thunderbolt conversion LTO drives. there is some of them available. 

 

All SAS cards are not compatible with all the LTO drives and it may be pretty challenging to try to find out which is compatible and which is not and what one can do about it. and driver/operating system/software compatibility issues complicate things more because there may be 5 or more different possible failure points in the chain from the drive to the backup software and thus it is complicated to try to find out what should work.... OR one can just randomly purchase stuff and hope that everything works correctly (most likely will not) .

 

I use BRU PE for lto backups, for me it has been more useful than LTFS for many reasons and more future proof. there is other options as well. 

 

 

Like you said the LTO has benefit of being pretty future proof because the tape drives are more common than pro video decks used in the past. And every post house is likely to have a compatible LTO system up and running so even if your own system would burn up you can always get the tapes restored somewhere near you. The backwards compatibility is also a great feature so the drive does not need to be even same generation than the tapes, just max. 2 generations newer and you can still restore the tapes with it.


Edited by aapo lettinen, 11 February 2019 - 05:38 AM.

  • 0



Paralinx LLC

Rig Wheels Passport

Visual Products

Glidecam

Media Blackout - Custom Cables and AKS

Serious Gear

Gamma Ray Digital Inc

Ritter Battery

Tai Audio

Broadcast Solutions Inc

Wooden Camera

New Pro Video - New and Used Equipment

CineLab

Abel Cine

Metropolis Post

Wooden Camera

Visual Products

Glidecam

Tai Audio

Serious Gear

Broadcast Solutions Inc

CineLab

Metropolis Post

New Pro Video - New and Used Equipment

Paralinx LLC

Ritter Battery

Media Blackout - Custom Cables and AKS

Gamma Ray Digital Inc

Rig Wheels Passport

Abel Cine