Jump to content


Photo

D-21 Work flow?


  • Please log in to reply
17 replies to this topic

#1 Saba Mazloum

Saba Mazloum
  • Basic Members
  • PipPipPip
  • 110 posts
  • Cinematographer
  • China

Posted 09 August 2008 - 10:17 PM

Hey friends,

If anyone has experience with the post work flow , could you share with us? Thanks

Does anyone know what feature films or commercials were shot on the d-21?

Thanks
  • 0

#2 David Mullen ASC

David Mullen ASC
  • Sustaining Members
  • 19759 posts
  • Cinematographer
  • Los Angeles

Posted 09 August 2008 - 11:18 PM

So far, the majority of D20 projects have been for television and have recorded 1080P to HDCAM-SR using an SRW1 deck and used an HD tape workflow.

The D21 is brand new but not that different, and I suspect the majority of users will continue to record to an SRW1 deck and use an HDCAM-SR workflow rather than record uncompressed 2.8K RAW to data recorders.
  • 0

#3 Keith Mottram

Keith Mottram
  • Sustaining Members
  • 824 posts
  • Other

Posted 15 August 2008 - 09:00 AM

Hi

I'm prepping post for a D21 feature currently. The shoot is going to be 4:4:4 LOG 1920x1080 to SR1, with single link 4:2:2 1920x1080 from the SR1 to macbooks via AJA IO HD for editorial in ProRes- these editing rushes will have baked in filmout Truelight LUTs. The film will then be conformed at 10bit uncompressed and graded on a Baselight. Steadicam will be shot to Flashmags and then dumped to SR1. I am hoping to do a data test, but am not sure whether this is going to be possible. Very early on it was decided that any benefits of shooting data would be negated by extra back up hassle, untested workflow etc- especially as the film is in 1.85 aspect. I am going to be looking at filmouts next week, so will let you know what my opinion is and what differences I can notice between images from the D20 and D21.

Keith
  • 0

#4 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11936 posts
  • Other

Posted 15 August 2008 - 10:55 AM

> Very early on it was decided that any benefits of shooting data would be negated
> by extra back up hassle, untested workflow etc

I'm really dismayed to keep hearing this. It can be done quite easily, it's just a case of finding the right people - that is, people who aren't hopelessly habituated to film/tape workflows and are willing to do data as it's been done for decades by other branches of computing.

It's not untested, it's just untested by the people who are currently trying to do it.

P
  • 0

#5 Dan Goulder

Dan Goulder
  • Sustaining Members
  • 1259 posts
  • Cinematographer

Posted 15 August 2008 - 12:18 PM

Does anyone know what feature films or commercials were shot on the d-21?

Thanks

The D21 is a recent update of the D20. "The Bank Job" was a wide release feature shot on the D20.
  • 0

#6 Keith Mottram

Keith Mottram
  • Sustaining Members
  • 824 posts
  • Other

Posted 16 August 2008 - 11:20 AM

> Very early on it was decided that any benefits of shooting data would be negated
> by extra back up hassle, untested workflow etc

I'm really dismayed to keep hearing this. It can be done quite easily, it's just a case of finding the right people - that is, people who aren't hopelessly habituated to film/tape workflows and are willing to do data as it's been done for decades by other branches of computing.

It's not untested, it's just untested by the people who are currently trying to do it.

P



Hi Phil,

Urm, am I one of the " people who are currently trying to do it."? do you think I am habituated to a film/tape workflow? Because I am quite capable of handling an all data work flow if I wanted to. I just don't. It's not necessary and although I was simplistic in my workflow explanation, my workflow knowledge itself is not simplistic. On this production we have one of the best DITs about and a DOP who is very enthusiastic about digital production. We also have a supportive producer and director. So it is my choice. And it is a legitimate choice. I have weighed up the pros and cons and guess what SR was the best way for this production to go. We are not shooting film and we are shooting data. We are just archiving it to tape. Which we would have if we were shooting to codex or s2. The difference being that we are archiving to SR on set in realtime, rather than running tape backups after the shoot every night. As I also said we are shooting direct to ProRes HD at the same time. I really don't understand why you feel this is a bad decision. Yes there is compression when going to SR but nothing that is unnaceptable in my opinion and nothing that I feel is detremental.

best

Keith
  • 0

#7 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11936 posts
  • Other

Posted 16 August 2008 - 01:13 PM

You're misinterpreting me. I'm dismayed to hear people whining about "untested workflows". More or less every workflow done on data/HD/DI work at the moment is untested in its applied form because almost every production, or at least every house, is doing things slightly differently - and it works fine because computers are flexible. I'd love to see a globally standardised way of doing data post. There isn't one and won't be a for a while. This is normal and accepted, it doesn't make it "untested".

The point of all this, the entire purpose of doing data, is to avoid the Sony tax and get your material into a format where it can be handled on mundane computer hardware. Workflow - in terms of "we'll get it from place X to place Y somehow" - has been tested to death over the last three decades of modernish computer equipment, and can huge quantities of money can be saved.

I think it is misguided to overlook this sort of thing on the basis that it's "untested". It isn't. There is a dangerous undercurrent of thinking here, I suspect, which says on the one hand "you can't save any money doing data if you do it properly," but which defines "properly" as "not doing data." It's self-defeating.

P
  • 0

#8 David Mullen ASC

David Mullen ASC
  • Sustaining Members
  • 19759 posts
  • Cinematographer
  • Los Angeles

Posted 16 August 2008 - 02:59 PM

Just in the three months I've been shooting on the RED camera, the workflow for handling the data has been modified constantly. Software updates, build updates, etc. It's rather time-consuming and we are still losing a day at the post house between shooting and when editorial gets Pro-Res HD files, which is fine... but it makes it no faster than shooting film, if not slightly slower, to get it into the editing phase. The post house tells me that this would all be going faster if we were transferring to HDCAM-SR and going into AVID's rather than doing a file-based workflow into FCP. But they are certainly trying hard now for three months straight to get this workflow to, well, work.

I believe this is the future, but I can also understand why some people would rather just fall into a standardized tape workflow for post that doesn't need testing to set-up.


It's also interesting that on location, whereas if I were shooting film, the loader would just get a darkroom in the back of the camera truck, on this low-budget movie, we need a portable air-conditioned truck to handle all of the data and back-ups. On "Manure", we were based on stage and just set-up an office to do this work, but now that we move around every day, sometimes twice a day, our data management set-up has to follow us around everywhere so we can get these drives unloaded and backed-up, shuttle drives loaded and sent to post, shuttle drives returning, plus the RAID drives holding our two back-ups. And we need a cool place to do all of this because our locations don't have any places for it. We are often just outside in the streets with our gear.
  • 0

#9 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11936 posts
  • Other

Posted 16 August 2008 - 09:56 PM

> we need a portable air-conditioned truck to handle all of the data and back-ups

Yeah, that's a fairly serious problem. There are two ways to round it:

- Use backup devices of sufficient robustness that they don't need this. I think this is already the case; an LTO3 deck is certainly not more and possibly quite a bit less fussy environmentally than an HDCAM-SR recorder. Because you can put the SRW1 on the back of the camera, you do, and nobody thinks anything of it, but the considerations are exactly the same as for every camera out there that has a built-in tape deck. You could throw it on a cart and forget about it if they'd let you.

- Build flash-based recorders and send it off to have this done, exactly as you would with SR tapes or 35mm. In reality world the physical robustness of inactive hard disks is at least as good as an HDCAM tape and quite a bit better than 35, but I fully understand that the insurance people have their underwear misconfigured on this point.

So in both cases the problem with these solutions to extant problems is not what it will do, it's what you're allowed to do with it. I could wax frustrated about how these decisions are being made on by people who are uninformed, but at the end of the day as an engineer I can only specify a product that works; if people choose to ignore the information, there's not an awful lot I can do about that.

The biggest concern I have with data backup is the desirability of running it overnight while LTO drives are significantly slower than mag speeds. The solution to this is to have more or less one backup station (that's two decks) per mag; aggregate bandwidth is a near-universal cureall.

P
  • 0

#10 David Mullen ASC

David Mullen ASC
  • Sustaining Members
  • 19759 posts
  • Cinematographer
  • Los Angeles

Posted 16 August 2008 - 11:04 PM

So you're recommending data be downloaded and backed-up with each camera drive change to on-set LTO tape recordings instead of RAID's?

We already archive to LTO's, but at the post house, not on location.
  • 0

#11 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11936 posts
  • Other

Posted 17 August 2008 - 09:07 AM

> We already archive to LTO's, but at the post house, not on location.

That's a choice I would think is determined by a number of factors:

- Mobility of the production. Sitting on a soundstage, you can put the backup gear on site. Running around three locations a day, not so much.

- Distance to post house - self-explanatory in terms of turnaround time

- Editors' requirements versus what the recorder will give you. A lot of post work is being done on these shows to turn LTO backups into something Final Cut will read.

And finally the biggest thing:

- Budget. By backing up onsite, you can limit your offsite work to just LTO verification, at the cost of more on-set tasks.

The last one I think is the big issue; making it simpler to do the backup on set will make it at least possible for more people to do it this way, whether they end up doing so or not.

And apropos of nothing I'm not hoplelessly wedded to LTO, it just seems to be gaining a degree of ubiquity. You could do it on AIT or DLT as well, other than that they're helical scan and there's more foundation-garment entanglement from insurance companies on this despite the fact that every HD tape format is also helical scan, and treated much more roughly.

P
  • 0

#12 Keith Mottram

Keith Mottram
  • Sustaining Members
  • 824 posts
  • Other

Posted 19 August 2008 - 11:08 AM

It's rather time-consuming and we are still losing a day at the post house between shooting and when editorial gets Pro-Res HD files, which is fine... but it makes it no faster than shooting film, if not slightly slower, to get it into the editing phase. The post house tells me that this would all be going faster if we were transferring to HDCAM-SR and going into AVID's rather than doing a file-based workflow into FCP.


Hi David,

I cannot fathom this explanation. Both Avid and FCP are NLE's which can ingest HD from SR, neither of which can digitise from SR faster than realtime. So why on earth would they tell you that it would be faster to use Avid. I can only think because they can find a way of charging you more and by saying "Avid" they think that a producer would accept a premium. The SR workflow I would assume would be to play out the Red files with Scratch onto SR (or any other TC encodable tape) for editorial/ rushes. I would seriously worry about a post house that is happy to talk such trousers.

best,

Keith
  • 0

#13 David Mullen ASC

David Mullen ASC
  • Sustaining Members
  • 19759 posts
  • Cinematographer
  • Los Angeles

Posted 20 August 2008 - 03:40 AM

Hi David,

I cannot fathom this explanation. Both Avid and FCP are NLE's which can ingest HD from SR, neither of which can digitise from SR faster than realtime. So why on earth would they tell you that it would be faster to use Avid. I can only think because they can find a way of charging you more and by saying "Avid" they think that a producer would accept a premium. The SR workflow I would assume would be to play out the Red files with Scratch onto SR (or any other TC encodable tape) for editorial/ rushes. I would seriously worry about a post house that is happy to talk such trousers.

best,

Keith


Not faster to digitize into the NLE. The problem is that we need to convert our RED footage to ProRes HD files and add letterboxing and windowburns, and for some reason, not doing this step in an SR tape environment is slower. So I assume the problem is basically the difference between creating Apple Quicktime ProResHD files with letterboxing and windowburns versus handing editorial an HD tape. The issue isn't the digitizing step into the NLE system.

Look, they aren't lying -- it really is taking them 18 to 27 hours daily to process our RED footage into ProRes HD files because of the letterboxing and windowburn steps.

Don't know the particular steps, I think they are using RED's Log & Transfer software, but I don't know if they are working from the R3D files or the QT proxies that the camera generates. Doesn't make them happy -- I'm sure they rather the process went quicker. I was there when they were trying to explain the conversion problem to a RED representative and the RED expert couldn't think up a faster solution either for them.

They did the dailies for Soderberg's RED movies and didn't have these problems because it was all done using SR tape.

You can talk to the folks at PlasterCity yourself to get the details as to why making these ProRes files is such a slow process.
  • 0

#14 Phil Rhodes

Phil Rhodes
  • Sustaining Members
  • 11936 posts
  • Other

Posted 20 August 2008 - 08:48 AM

I suspect what's going on here - and this is necessarily a speculation - is that this is being done as a renderable, non-realtime process on computer hardware.

The saving grace of doing it this way is that instead of it taking 27 hours on a $100,000 HDCAM-SR deck, it takes 27 hours on a $2,000 desktop PC. If it's taking too long, throw another $2,000 desktop PC at it, bash up a script of some kind to schedule it, and halve the time involved. This is where this sort of thing starts to save money. Yes I know producers don't believe it can possibly be any good if it isn't done on six figure gear, but the fact remains that it's doable.

Hmm, but.

On the other hand, most post houses, in my experience, seem very reluctant to do that sort of thing, and would much rather involve their very expensive Sony investments if they possibly can, so having thought about it - god knows how they're doing it.

P
  • 0

#15 Keith Mottram

Keith Mottram
  • Sustaining Members
  • 824 posts
  • Other

Posted 20 August 2008 - 12:09 PM

Not faster to digitize into the NLE. The problem is that we need to convert our RED footage to ProRes HD files and add letterboxing and windowburns, and for some reason, not doing this step in an SR tape environment is slower. So I assume the problem is basically the difference between creating Apple Quicktime ProResHD files with letterboxing and windowburns versus handing editorial an HD tape. The issue isn't the digitizing step into the NLE system.

Look, they aren't lying -- it really is taking them 18 to 27 hours daily to process our RED footage into ProRes HD files because of the letterboxing and windowburn steps.

Don't know the particular steps, I think they are using RED's Log & Transfer software, but I don't know if they are working from the R3D files or the QT proxies that the camera generates. Doesn't make them happy -- I'm sure they rather the process went quicker. I was there when they were trying to explain the conversion problem to a RED representative and the RED expert couldn't think up a faster solution either for them.

They did the dailies for Soderberg's RED movies and didn't have these problems because it was all done using SR tape.

You can talk to the folks at PlasterCity yourself to get the details as to why making these ProRes files is such a slow process.



Not saying they're lying about the time it takes to do the log and transfer in FCP, just saying that with regards to SR workflow it is irrelevant whether the NLE is Avid or FCP. Whatever process they use to create SR versions for editorial would be the same whether the edit is done on Avid, FCP or any other SDI capable NLE. With regards to working in ProRes from SR you can digitise straight to this codec (this is what will be happening with D21 footage or Genesis footage into FCP), with Avid it is the same- you would digitise straight to Avid DNX or whatever.

To get the rushes in, the choice currently is log and transfer to FCP or use Scratch to play the files out onto tape (and then digitise into whatever NLE editorial want to use)- so why Avid is brought up is either ignorance or nonsense. That is the point. Its not like post houses don't fib on a day by day basis, whether they are fibbing for ease of explanation (a white lie) or intentionally changes from fib to fib/ lie to lie. I've never worked with (or for) a post house that doesn't work on a need to know basis and with that in mind it is possible that they saw the Avid alternative as the way to give you the simplest explanation.

best,

Keith
  • 0

#16 David Mullen ASC

David Mullen ASC
  • Sustaining Members
  • 19759 posts
  • Cinematographer
  • Los Angeles

Posted 20 August 2008 - 12:27 PM

Here is their explanation:


?The conversion process is slow for ProRes because the only way to generate ProRes is with an Apple compression engine (be it Compressor, Final Cut Pro, Quicktime Player, etc.). Even with a render farm, it is difficult to create ProRes files in realtime. Creating DPX can be done in real time using multiple ways, but Quicktime is not so robust.

?The RedCode codec also changed (improved) between Build 15 and 16, which is the result for the 9x real time render times for ProRes 1080 or 2K files. (used to be 3x real time)
At PlasterCITY, we thread a dozen or so machines together in order to aggregate the 9x real time render process across more cores, which brings the render time significantly down, but it still a much slower process than in previous builds of the camera.

?There is one solution that works great, however, which is the realtime solution everyone wishes they had. It is the the system Scratch. Scratch is able to debayer the Red 4K files in real time and play them out in 2K or HD 1920x1080sF in real time. This allows us to process the red files in shooting order to tape, just like telecine.
We also have a Tangent CP200 (DaVinci control surface) that allows us to do grades in real time on all the footage as well. Scratch can also read the meta-data of the R3Ds accurately (unlike Quicktime) so it means we can exactly match your monitors on set. Quicktime just doesn't have these options...

?After the tape is made, we wrote an application that allows us to add in the camera audio automatically in synch and then output an AVID ALE (Avid Log Exchange) which acts as a flex file so the SR tape can match back to the R3Ds after an Avid EDL is produced after editing the film. This allows for easy match-back to R3Ds for a fast and accurate 2K or 4K conform.

***The main crux here is that the Scratch system works great for tape, and the ALE output works great for AVID. On the contrary, Final Cut Pro's ability to interpret Batch Lists like ALEs is very poor. So we can't even do the tape worklfow and digitize into Final Cut. -If we could, we would...because it's fast and accurate. With Final Cut, we can get the files into them via Scratch, but we loose all R3D source name and Timecode metadata headers.

We've had some major discussions with Apple on this front and they hate hearing that the AVID out performs their product. We are hoping they can get ALE support / Batch List support in FCP so we can load in flex files from Red telecine, but for now, the only way to use FCP and match back is via slow Quicktime outputs.
  • 0

#17 Keith Mottram

Keith Mottram
  • Sustaining Members
  • 824 posts
  • Other

Posted 20 August 2008 - 01:26 PM

Here is their explanation:


?The conversion process is slow for ProRes because the only way to generate ProRes is with an Apple compression engine (be it Compressor, Final Cut Pro, Quicktime Player, etc.). Even with a render farm, it is difficult to create ProRes files in realtime. Creating DPX can be done in real time using multiple ways, but Quicktime is not so robust.

?The RedCode codec also changed (improved) between Build 15 and 16, which is the result for the 9x real time render times for ProRes 1080 or 2K files. (used to be 3x real time)
At PlasterCITY, we thread a dozen or so machines together in order to aggregate the 9x real time render process across more cores, which brings the render time significantly down, but it still a much slower process than in previous builds of the camera.

?There is one solution that works great, however, which is the realtime solution everyone wishes they had. It is the the system Scratch. Scratch is able to debayer the Red 4K files in real time and play them out in 2K or HD 1920x1080sF in real time. This allows us to process the red files in shooting order to tape, just like telecine.
We also have a Tangent CP200 (DaVinci control surface) that allows us to do grades in real time on all the footage as well. Scratch can also read the meta-data of the R3Ds accurately (unlike Quicktime) so it means we can exactly match your monitors on set. Quicktime just doesn't have these options...

?After the tape is made, we wrote an application that allows us to add in the camera audio automatically in synch and then output an AVID ALE (Avid Log Exchange) which acts as a flex file so the SR tape can match back to the R3Ds after an Avid EDL is produced after editing the film. This allows for easy match-back to R3Ds for a fast and accurate 2K or 4K conform.

***The main crux here is that the Scratch system works great for tape, and the ALE output works great for AVID. On the contrary, Final Cut Pro's ability to interpret Batch Lists like ALEs is very poor. So we can't even do the tape worklfow and digitize into Final Cut. -If we could, we would...because it's fast and accurate. With Final Cut, we can get the files into them via Scratch, but we loose all R3D source name and Timecode metadata headers.

We've had some major discussions with Apple on this front and they hate hearing that the AVID out performs their product. We are hoping they can get ALE support / Batch List support in FCP so we can load in flex files from Red telecine, but for now, the only way to use FCP and match back is via slow Quicktime outputs.


David,

thank you so much for this detailed explanation and being so generous with your time. It makes interesting reading and the ALE section is of course very relevent and answers why log and transfer makes sense as far as meta data is concerned with FCP. Of course ALE's are simple files and if we're honest a bit crap as far as there limitations compared to say XML, but simplicity is so often the best way (especially in light of the complications of using a red in the first place). I am also a little confused as to what they mean by Avid EDL. The problem with reconforming with SR and FCP is interesting, as the TC would be on the SR tape after popping out of the scratch, as would the audio with whatever route that is used. So if everything is logged correctly then I am unsure why FCP would be a problem with this route. A guess is that there is problem reconforming if the TC is not unique on the clips, but this can happen with traditional film transfers as well. Finally it is simple to use DPX files in FCP it is just uses up a lot of storage. Either way I am sure glad that I'm not involved with any red post at the moment. Thanks again and when I get a chance I'm going to investigate this further.

take care,

Keith
  • 0

#18 John Sprung

John Sprung
  • Sustaining Members
  • 4635 posts
  • Other

Posted 20 August 2008 - 01:29 PM

On the other hand, most post houses, in my experience, seem very reluctant to do that sort of thing, and would much rather involve their very expensive Sony investments if they possibly can, ....

In the case of Plaster City, last time I was there they only had one SR machine. They may have more now, but certainly not the familiar rows of racks full we see in the old tape houses. They started doing file based post from scratch, without a legacy infrastructure. They were very much of the four digits per box rather than six digits per box price mindset.




-- J.S.
  • 0


Gamma Ray Digital Inc

Opal

Willys Widgets

Broadcast Solutions Inc

Glidecam

Visual Products

Ritter Battery

Paralinx LLC

Technodolly

Tai Audio

The Slider

CineLab

FJS International, LLC

CineTape

Rig Wheels Passport

rebotnix Technologies

Abel Cine

Aerial Filmworks

Media Blackout - Custom Cables and AKS

Wooden Camera

Metropolis Post

CineLab

Ritter Battery

Wooden Camera

Aerial Filmworks

Visual Products

Technodolly

Gamma Ray Digital Inc

Abel Cine

FJS International, LLC

rebotnix Technologies

Opal

Willys Widgets

CineTape

Rig Wheels Passport

Metropolis Post

Tai Audio

Broadcast Solutions Inc

Media Blackout - Custom Cables and AKS

Glidecam

Paralinx LLC

The Slider