Customers have long been asking for the ultimate in remote workflow flexibility. Whether that means hiring video editors from distant geographies or simply letting employees and contractors work from home, production houses no longer want to be tethered to their machines. COVID-19 has only accelerated this request. As a part of EFSv, we recently announced the capability to achieve true seamless proxy editing. This patent-pending feature enables the first cost-effective cloud editing infrastructure, overcoming one of the primary objections of migrating media workflows to the cloud.
Cloud editing economics – block storage vs object storage
Until today, the questionable economics of editing in the cloud has been one of the biggest objections to adoption. With EFSv seamless proxy editing, we have dramatically reduced the costs, making cloud editing economically viable for anyone. When we say dramatically better, we mean reductions in the region of 50-75% compared to current cloud editing infrastructure implementations.
When it comes to editing video in the cloud, assuming you plan to use standard applications such as Adobe® Premiere® Pro, Davinci Resolve, Media Composer®, ProTools® or Adobe® After Effects®, you need your shared cloud storage to appear as though it is just another drive attached to your workstation. That is to say, the storage needs to have a filesystem that supports all normal operations such as read, write, seek, modify, The filesystem also needs to use standard mechanisms to tell an application if your username or group has permission to read, write, and lock a file for exclusive use. To protect your media you also need file system auditing to see who is accessing files within your storage. And, of course, you need storage that’s fast enough to play back your media in real time. These are all features that can be realized easily in the cloud using block storage.
Unfortunately, block storage in the cloud is expensive. However, there is a lot of value that cloud providers are bringing on top of standard disk storage. Behind the scenes you are getting file protection and robust security for your data with high performance guarantees.
Being able to utilize object storage in the cloud (like AWS S3) would be much less expensive. However, out of the box, it doesn’t meet the requirements of media creation applications because it lacks the features needed to create a real filesystem and instead focuses on simplicity, scalability, and reliability. By limiting the features of object storage, cloud providers can offer significant cost savings. You only pay for the exact number of bytes you store, rather than reserving a whole block of storage that ends up only being partially used.
But what if you could make object storage appear as a standard filesystem so that you could enjoy the cost benefits and also get it to work with editing, mixing and visual effects applications? In fact, that’s part of what we have done with EFSv — making object storage look exactly like a normal file system, regardless of whether it’s connected to Windows, MacOS or Linux. But there’s still one more challenge to overcome that is specifically problematic for video and audio editing – latency.
Object storage is fine for throughput, and when you play a video from a streaming platform such as Netflix, Amazon Prime Video, or Disney+, you are almost certainly streaming the video from an object storage solution. These streaming platforms deal with latency (the time it takes to read the first byte of video) by having you wait a small amount of time to watch the first segment of video (you know, that little bit of buffering you see before the video plays) and then, because the videos are generally played linearly, it is always possible to keep ahead of the user, disguising the latency.
While this works for streaming services, such a solution is not acceptable for video and audio editing – by its very nature these are non-linear activities. When you edit video even a small amount of latency will impact your workflow as you attempt to read multiple video files simultaneously and jump quickly back and forth between different files on your timeline.
Current object storage trades latency for resilience and throughput. Indeed most object storage solutions not only have higher latency than is acceptable for video editing, they also have inconsistent latency. So the challenge is to find a way to blend object and block storage into a single filesystem in a manner that lets you have your cake and eat it too — in other words, get the cost savings of object storage but the low latency of block storage that’s needed for editing.
Fixing the headaches of proxy editing
EditShare isn’t the first company to come up with the idea of blending object and block storage. Since people first started editing in the cloud, a common approach to mixing the two types of storage and saving money has been to put high resolution original media into object storage, and then transcode everything to smaller, just good enough low-resolution proxy files for rough editing and store just those proxy files on the more expensive block storage. Because proxies are small, they don’t use up a lot of block storage — and so you save money. Then, at the end of a project, you restore from object storage back to block storage just the files that made it into the final cut and you “conform” your sequence back to high-quality originals.
Unfortunately, this kind of proxy workflow is fraught with problems.
Just ask anyone who has ever had to proxy edit a project. First, there are inevitably shots that don’t look great in the proxy version, and it’s really frustrating to have to wait for the conform to find out a shot isn’t any good. Second, it’s impossible to create accurate effects that match colors when you are dealing with proxy files. Third, it’s no fun to have rough cut and fine cut screenings based on your proxies because these screenings will never give the best impression. Fourth, at the end of an edit, it takes time to copy high-resolution files back to block storage and if your project is coming down to the wire, every minute counts. And finally, there’s nothing worse than conforming an edit and getting a “media offline” or “nothing relinked” message when you attempt to relink back to the original.
It’s probably safe to say, no one loves the offline and conform workflow. While it is not a ‘broken’ system, it’s far from ideal. And it doesn’t even save as much money as it could, because at the end of the day you still have to copy at least some content back to expensive block storage.
A seamless proxy workflow designed for everyone
The EFSv seamless proxy editing feature solves all these problems. It is open — we have yet to find a NLE that it doesn’t work with — and fundamentally, it changes the economics of editing in the cloud.
EFSv lets you put your high-resolution files into cost-effective object storage but makes those files appear as if they are on a normal mounted block storage file system. At the same time, EFSv provides the scalable high performance block storage needed for low-latency access to proxy files and renders. Proxies can be generated by our FLOW media management system, or by your own tools. Both high-resolution and proxy versions are accessible at all times to your media creation application. And when you import clips into NLEs such as Premiere Pro using our new FLOW panel, the panel “teaches” the application about the existence and location of both high-resolution and proxy versions, so you can toggle back and forth between versions any time you want. While you won’t be able to play back your whole timeline from the object storage — mostly because of the latency — you can nonetheless view the original high-resolution clips at any time (including for color correction), and when it comes time to rendering your timeline for a screening or for the final deliverables, you just toggle to (or link back to) the originals and hit the render or export button. There’s never anything to copy or restore from object storage. Instead, you render directly from object storage — a process that is not slow, by the way. The momentary latency you get when first accessing a file and that can frustrate timeline playback won’t make any noticeable difference for rendering.
Our new system not only works with cloud workstations you connect to by advanced remote desktop software such as Teradici, but it also works spectacularly over a VPN — so if you want to keep working from home with the editing application running on your own workstation or laptop, this is also an option. You can get the same proxy/high-resolution toggling — and this even works with macOS-based workstations (something you cannot run in any cloud today). And if you need to download proxies instead of working with them “in place” from central storage, that’s supported too.
There’s a reason we refer to this as “true” seamless proxy editing — because it is truly seamless.
For creatives, the new system eliminates any compromises on quality or workflow, and for the people who look after the finances, the news is equally good. Better. Faster. Cheaper.
Cost savings – it’s BIG.
The savings you get with EFSv seamless proxy editing are tremendous. For example, you may have a media library of about 100 terabytes for a project — consisting of a little over 2,000 hours of original media assuming you’re using a 100 Mb/s codec. With traditional block storage , storing this content for editing would cost about $4,500/month. On top of this, you’ll require fairly powerful storage and workstation instances to handle the bandwidth of this high-resolution content. Suddenly, cloud-based editing may stretch outside of your budget. In fact, in many cases, your high-resolution codec will be of a higher bit-rate; for example many ProRes codecs exceed 500 Mb/s.
Seamless proxy editing resolves these issues. By taking down the content to a lower bit rate for editing we can reduce your block storage needs down to the 2-4 terabyte range, a factor of 25x or more. In this scenario, your total storage costs (object and block) will come down to something closer to $1400/month. And since we will be editing proxy files, less expensive compute instances can be used for storage management and NLE workstations. Your high-resolution files will continue to be stored in cheaper storage tiers, where they can be accessed instantly by the NLE with no additional workflow effort required. An added bonus – with your high-resolution files residing in object storage, you can easily take advantage of cloud-based AI tools such as automatic object recognition and audio transcription, and also share your files with partners who may be doing other work on the project.
EFSv with seamless proxy editing not only makes cloud editing affordable for the first time, it also creates the opportunity for new cloud-centric workflows to make your media projects more efficient. Learn more.
