Jump to content


Photo

Data Wrangling with a MacBook Pro... eSATA via Thunderbolt?

HDD DIT WRANGLER macbook firewire thunderbolt eSATA G-DRIVE

  • Please log in to reply
43 replies to this topic

#1 Luke Wheldon

Luke Wheldon
  • Basic Members
  • PipPip
  • 21 posts
  • Editor
  • Leeds, UK

Posted 09 February 2014 - 05:06 PM

Hi,

 

I'm primarily an Editor dabbling in motion graphics. I'm wanting more on-set work and have been offered the opportunity to do some AC and data wrangling on a local short in Leeds.

 

I'll be using a MacBook Pro 15" (late 2011) 2.5GHz Intel Core i7, 16 GB RAM.

Shooting RED scarlet and backing up on two G-DRIVE's - Yet to have confrimation whether or not the drives have eSATA.

 

As my mac has one FireWire 800 port and one Thunderbolt, I'll be running into some problems on the way in working out how fast I can transfer data on the budget I have. 

 

I've heard and read of the problems of trying to daisy chain two FW800 drives and a RED station, all through one little FW port. I've also read a lot of positive things about the Sonnet Echo Pro Thunderbolt adapter. http://www.sonnettec...hunderbolt.html

 

I'm hoping to get the eSATA extension, but if the G-DRIVE's are not compatible the FW800 will do for now.

 

For £250 I'm wondering if this is the best option I have for transferring data as quickly as possible, while linking two G-DRIVE's through one eSATA and using the second eSATA for the RED station?

 

 

Thank you.

 

 

 

Luke Wheldon.

 

 


  • 0

#2 Luke Wheldon

Luke Wheldon
  • Basic Members
  • PipPip
  • 21 posts
  • Editor
  • Leeds, UK

Posted 09 February 2014 - 06:30 PM

I've just read that eSATA cannot be daisy chained. Duh.

 

How well would the set up work if the eSATA drives each had they're own port and then I used the FW800 for the RED station? Or one eSATA for one G-DRIVE and the other for the RED station... then the FW800 for the second G-DRIVE?

 

Thanks.


  • 0

#3 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11937 posts
  • Other

Posted 09 February 2014 - 06:47 PM

I've tried to use Red cameras twice and on both occasions had problems getting the red card reader to work via anything other than USB. Which is part of why we avoid red. But I digress.

My approach is to try and keep things on private buses as much as possible, and I'd prefer to have symmetrical performance from the hard disks. You will need some sort of software to do a double copy and checksum compare for real reliability.

Ideally you want to own an LTO drive if you're going to be doing any serious amount of this. Hard disks fail way, way too easily. And a macbook is not really the ideal tool. Laptop processors are comparatively rather underpowered, and by the time you've got this stuff all patched up it can become a reliability nightmare with all the cables and connectors. I like to have everything racked up in a server case and ready to go.

P
  • 0

#4 Luke Wheldon

Luke Wheldon
  • Basic Members
  • PipPip
  • 21 posts
  • Editor
  • Leeds, UK

Posted 10 February 2014 - 05:43 AM

Hi Phil,

 

Thanks for your post. For double copy and checksum I have been advised to use ShotPut Pro for now. At some point in the future the idea is to be doing a lot more of these jobs and to expand into DIT work. As I am very new to the role I have to work with what budget I have and the drives I'm provided with. The job I mentioned is unpaid, which is fine for now as I am still learning and would like to build up some relationships - I can't go around giving myself a new job title until I'm a lot more confident and experienced under the role.

 

I've been doing a lot of reading on the role and it seems that many people have different ways of working, mainly due to budget and production. Just out of interest what sort of set up do you have for data wrangling?


  • 0

#5 Freya Black

Freya Black
  • Basic Members
  • PipPipPipPip
  • 4161 posts
  • Other
  • Went over the edge... Central Europe

Posted 10 February 2014 - 05:43 AM

Okay, first up be careful with the firewire stuff. NEVER hotplug firewire, it's easy to burn out your firewire ports that way as they aren't well designed in that respect.

 

Secondly I agree with Phil that it's best to keep things on separate buses as much as possible. So if for instance you can only get the red card reader to work from USB then just put it on the USB bus and make sure nothing else is on that USB bus.

 

If you can get eSATA working then it would be great to have the RED station on eSATA so that you can make the most of the really fast pipeline. The Red SSD's should be able to pump out data very quickly. If the hard drives have the capability to work with eSATA then that might be good but there is something to be said with having them on the firewire 800 bus anyway as they are slower devices  and they will be on a separate bus that way.

 

You might want to experiment a bit and see what works fastest based on those principles.

 

I assume your laptop doesn't have a cardbus slot? Just thinking that might make getting eSATA a bit cheaper.

 

Freya


Edited by Freya Black, 10 February 2014 - 05:45 AM.

  • 0

#6 Luke Wheldon

Luke Wheldon
  • Basic Members
  • PipPip
  • 21 posts
  • Editor
  • Leeds, UK

Posted 10 February 2014 - 04:05 PM

Great! Thanks, Freya. Unfortunately I don't have the expresscard slot on my 15" so I've just ordered the Sonnet with eSATA.

 

This all may seem very basic to most but it's essential info for me right now.

I'm really appreciating the time and advice from everyone. 

 

Luke.


  • 0

#7 John Miguel King

John Miguel King
  • Basic Members
  • PipPipPip
  • 153 posts
  • Digital Image Technician
  • London, UK

Posted 14 February 2014 - 04:19 AM

You need to assess the weakest link in the data flow. The G-Drives and FW are the definite ones in this case.

 

G-Drives are safe for storage but horribly slow for transfer, they peak at 100 MB/s on a good day. FW peaks at 80 MB/s. Add to this the checksum, which doubles your time, and you're looking at 40 MB/s total in the very best of all possible scenarios.

It looks like the wrong setup, to be honest. You need to ditch the G-Drives for G-Raids, and go either USB3 and/or eSata. Those drives are so slow they won't be able to actually play the footage.

It will all, however, work as you have planned it, but it'll be dreaduly slow. Make sure you get plenty of redmags to keep the camera rolling and arm yourself with patience.

How many USB3 does your laptop provide? Belkin has a nifty little thunderbolt hub that provides 3 USB3 and another thunderbolt. It's the cheapest option around, and with TBolt being a direct connection to the PCI Bus you won't have the slightest problem, bottleneck or conflict with it. Patch an esata through the belkin via tbolt and sonnet's expresscard reader. This way the weakest link shall be your drives, with esata2 (what the redmag reader has) being the second slowest at 3 gigabits, and usb3 the third slowest at 5 gigabits.

It's what happens when producers buy the drives!


  • 1

#8 Dan Collins

Dan Collins
  • Basic Members
  • PipPip
  • 33 posts
  • Cinematographer

Posted 16 February 2014 - 02:10 AM

Lacie has an eSata hub. 

 

There are several companies in addition to Sonnet who you mentioned, that are coming out with thunderbolt docks to offer many more ports of varing types. 


  • 1

#9 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11937 posts
  • Other

Posted 16 February 2014 - 04:28 AM

I'd avoid hubs and splitters if you can. They can have a detrimental effect on performance.

 

P


  • 1

#10 Chris Burke

Chris Burke
  • Basic Members
  • PipPipPipPip
  • 1675 posts
  • Boston, MA

Posted 16 February 2014 - 11:42 PM

use ShotPut Pro for back up and check-sum verification. I have managed over 8000 hours of footage for two seasons of a TV show and the software rocked. We used G-Speed Qs in RAID 5 hooked up to Macbook Pro 15" via an eSATA to USB3 converter. USB 3 is much cheaper and there are more devices made with it rather than Thunderbolt. Remember that the drives to which you are backing up to are the bottleneck. You can have a 10,000 RPM drive hooked up to a computer, both by Thunderbolt and then again by USB 3 and you won't see a difference in speed. Not until you use SSD as your storage medium will you see a speed boost. FW800 is only going to be marginally slower than either USB 3 or thunderbolt with spinning drives which are not striped.

 

With the set up I used and using a San Disk Imagemate USB 3 card reader (they are the fastest, faster than Lexar) I could back up a 32gig CF card with verification in 8 minutes. If I were you I would look into a RAID 5 or RAID 10 solution, going to two non striped drives is a little scary.


  • 1

#11 John Miguel King

John Miguel King
  • Basic Members
  • PipPipPip
  • 153 posts
  • Digital Image Technician
  • London, UK

Posted 17 February 2014 - 02:48 AM

 If I were you I would look into a RAID 5 or RAID 10 solution, going to two non striped drives is a little scary.

 

Two cloned G-Raids is essentially RAID 10. x


  • 1

#12 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11937 posts
  • Other

Posted 17 February 2014 - 06:01 AM

I've long felt that Raid5 is a poor choice for... well, a lot of things, with the only real exception being where you want the absolute minimum cost per megabyte and don't care too much about speed, which is not your situation. Raid10 is cheaper if you want it to be even slightly fast, less likely to fail, recovers more quickly if it does fail, and remains more usable while degraded.

 

I wrote a lengthy article on why I think this is the case: http://www.redsharkn...ou-need-to-know


  • 0

#13 Luke Wheldon

Luke Wheldon
  • Basic Members
  • PipPip
  • 21 posts
  • Editor
  • Leeds, UK

Posted 17 February 2014 - 03:32 PM

Thank you for the advice, everyone. Much appreciated! 

 

I'll write up a short report next week of how the shoot went with regards to transfer speeds etc


  • 0

#14 Luke Wheldon

Luke Wheldon
  • Basic Members
  • PipPip
  • 21 posts
  • Editor
  • Leeds, UK

Posted 17 February 2014 - 03:33 PM

I've long felt that Raid5 is a poor choice for... well, a lot of things, with the only real exception being where you want the absolute minimum cost per megabyte and don't care too much about speed, which is not your situation. Raid10 is cheaper if you want it to be even slightly fast, less likely to fail, recovers more quickly if it does fail, and remains more usable while degraded.

 

I wrote a lengthy article on why I think this is the case: http://www.redsharkn...ou-need-to-know

 

Thanks Phil. Just going to get stuck into your article now!


  • 0

#15 Luke Wheldon

Luke Wheldon
  • Basic Members
  • PipPip
  • 21 posts
  • Editor
  • Leeds, UK

Posted 17 February 2014 - 03:46 PM

Phil, I think I've had too much of a long day to take all of that article in.

 

I should really read it tomorrow when I'm feeling fresh...


  • 0

#16 Chris Burke

Chris Burke
  • Basic Members
  • PipPipPipPip
  • 1675 posts
  • Boston, MA

Posted 18 February 2014 - 07:56 AM

 Your only real solution is a Thunderbolt PCI expansion chassis, if speed is crucial. From there you can hook up many different kinds of pci expansion cards; USB 3, eSATA, mini-SAS.........


  • 0

#17 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11937 posts
  • Other

Posted 18 February 2014 - 08:04 AM

 Your only real solution is a Thunderbolt PCI expansion chassis

 

Or, you know, a real computer.

 

It is a significant source of frustration to me that a large proportion of people trying to do real, serious data handling work leap instantly and inexplicably to the conclusion that a laptop is the way to go. And that laptop has to be made by Apple.

 

An Apple laptop is just about the single most expensive way to buy computer capability. I have no bone to pick with Apple in particular, but they're not particularly cost-effective, and they no longer really have a properly expandable solution for heavy lifting. Laptops are expensive no matter who you get them from, as well as difficult to expand (as this thread demonstrates). Compound these factors and you're making a genuinely crazy decision to base a DIT cart around a macbook.

 

Get a 4U rack case with a lot of drive bays and load it up. It doesn't have to be expensive to begin with and you can update it as your needs grow. You can drop it in an off the shelf flight case for transport, too.

 

P


  • 0

#18 John Miguel King

John Miguel King
  • Basic Members
  • PipPipPip
  • 153 posts
  • Digital Image Technician
  • London, UK

Posted 19 February 2014 - 04:05 AM

 

Or, you know, a real computer.

 

It is a significant source of frustration to me that a large proportion of people trying to do real, serious data handling work leap instantly and inexplicably to the conclusion that a laptop is the way to go. And that laptop has to be made by Apple.

 

An Apple laptop is just about the single most expensive way to buy computer capability. I have no bone to pick with Apple in particular, but they're not particularly cost-effective, and they no longer really have a properly expandable solution for heavy lifting. Laptops are expensive no matter who you get them from, as well as difficult to expand (as this thread demonstrates). Compound these factors and you're making a genuinely crazy decision to base a DIT cart around a macbook.

 

Get a 4U rack case with a lot of drive bays and load it up. It doesn't have to be expensive to begin with and you can update it as your needs grow. You can drop it in an off the shelf flight case for transport, too.

 

P

 

Phil, I do this for a living, and trust me, TBOLT expansion chassis works perfectly fine. I use it on commercials and long form. Even for RED transcodes as long as you've the latest Retina and a Redrocket. The Belkin Hub is spreading like wildfire at the moment. Jigsaw 24 simply doesn't have enough stock to keep up with the demand.

The only situations in which I'd recommend bringing the big one are these:

- Multicamera transcodes. In 99% cases best go big one, however, I've gone retina MBP with up to three Alexas at prores 2k 4444 and it was absolutely fine. The intermediate drive was an 8 bay RAID 6, and I had my assistant operating it whilst I attended the three camera crews. The name of the game was "feed the monster". We managed to ingest, verify and transcode with a one light grade close to 1 tb of 2K prores 4444 per day. Staying only 30 minutes after wrap. We were real proud of this feat.

- Anything RAW, with the exception of a single Red camera, in which case red rocket on expansion chassis goes just fine.

What I've discovered is that the number of cores does matter greatly in the retina. The moment you have 4 verified copy processes in the background plus one transcode the GPU stops maxing its FPS due to the processor not being able to feed the GPU.

The retina is a real beast. And the moment apple finally (as in they'll never do it) decides to release thunderbolt GPU drivers, even single camera RAW will be doable with an external chassis.

But for data wrangling... even an i5 Macbook Air with the right expansion chassis and cards, or hub, is overkill.

And yes, I'd rather have a Linux or Windows going on in a bespoke computer. But ShotPutPro, the only pro verified copy appication in Windows is slow slow slow indeed. Silverstack beats it to a pulp every single time. And the list of OSX only DIT apps is larger, way way larger then the alternatives. Not what I personally like, but it's the status quo. :(

My humble opinions.


Edited by John Miguel King, 19 February 2014 - 04:09 AM.

  • 0

#19 John Miguel King

John Miguel King
  • Basic Members
  • PipPipPip
  • 153 posts
  • Digital Image Technician
  • London, UK

Posted 19 February 2014 - 04:11 AM

And Luke, don't buy anything. It can all be hired for very cheap indeed.  ;)


  • 0

#20 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11937 posts
  • Other

Posted 19 February 2014 - 05:28 AM

John, I shall defer to your more recent experience - last time I did this for money, the 5D mark 2 was new and we were barely on LTO4, although I've done a lot of related stuff in the meantime and always feel underdressed without properly racked up gear. Laptops become a hydra of flimsy cabling, in my view. Anyway.

 

I'm a bit concerned about ShotPut. Back when I started doing this stuff - more or less around the time anyone started doing this stuff - I just wrote my own scripts to do double copies and MD5 sums, and the like. I had thought of writing something more formalised to do copies and transcodes (via ffmpeg) but assumed ShotPut had the market sewn up. I've never been an Apple guy. Might this be worth revisiting if it's flawed?

 

P


  • 0



Also tagged with one or more of these keywords: HDD, DIT, WRANGLER, macbook, firewire, thunderbolt, eSATA, G-DRIVE

Technodolly

Abel Cine

Metropolis Post

rebotnix Technologies

Gamma Ray Digital Inc

CineTape

Broadcast Solutions Inc

Aerial Filmworks

Willys Widgets

Rig Wheels Passport

Visual Products

Paralinx LLC

Opal

Media Blackout - Custom Cables and AKS

The Slider

CineLab

Ritter Battery

FJS International, LLC

Glidecam

Wooden Camera

Tai Audio

Gamma Ray Digital Inc

Rig Wheels Passport

Ritter Battery

The Slider

rebotnix Technologies

Media Blackout - Custom Cables and AKS

Opal

Aerial Filmworks

Wooden Camera

FJS International, LLC

Metropolis Post

Glidecam

Broadcast Solutions Inc

Visual Products

CineTape

Willys Widgets

Abel Cine

Paralinx LLC

CineLab

Tai Audio

Technodolly