For this next step in our journey we review cloud video storage as it can be one of the biggest sources of confusion and costs when it comes to post-production storage. All major cloud providers offer a range of storage solutions, all optimized for different uses and price points.
In this article we’ll review some of the primary cloud storage options for editing in the cloud as well as the tradeoffs and associated costs.
A Look into Cloud Based Storage
Much like the traditional storage models, cloud-based storage options fall into one of five major categories:
- Local Storage: Analogous to a local computer configuration, this is an HDD or SSD physically attached to the cloud compute instance.
- Block Storage: This is network attached storage that must be provisioned against a compute instance during deployment time. Similar to local storage, it relies on the compute instance operating system to make it useful to applications. For those familiar with SAN’s (Storage Area Networks), block storage is similar in concept.
- File Storage: These are storage systems that act similar to a NAS (Network Attached Storage), and provide higher-level file-based access to a shared network of computers, typically using NFS.
- Object Storage: In this tier, files are stored as data objects referenced using an ID. The benefit is that there is no structural scale inhibiting storage growth. HTTP is the protocol for access.
- Archive Storage: A type of object-based storage, this is a lower-cost tier that is typically less accessible and requires a commitment of storage duration for several months or longer.
With the exception of local storage, these storage tiers can be highly scalable and extend dynamically during run-time or via deployment scripts. All of the storage tiers have trade-offs between costs, performance, scalability, and availability. We explored a bit of this in our prior Journey to the Cloud blog, where we focused on backup and archival storage.
For the purposes of video editing, one must focus on the usefulness of these storage types to the needs of the application as well as the costs. This requires a storage layer with low-latency, that can ideally scale to multi-TBs, and can be used by the editing application in a cost effective manner. A general rule of thumb is the further down in the above list you go the cheaper the storage, but that is not always the case. Offerings of SSD, NVMe, HDD, highly-performant file systems, and other permutations offered in each of these tiers make price/performance reviews a very complex analysis.
While costs are generally based on storage sizes and performance, other factors typically apply. As an example, depending on the vendor and storage tier, transferring content out of a tier may incur a fee and so will the actual I/O calls themselves, such as an HTTP GET and PUT, which read and write the data.
Video Editing Storage Choices
These considerations leave us with three choices for editing: Local Storage, Block Storage, and File Storage. Let’s review the usefulness of these choices.
Local storage has long been the tried and true means for editing workstations. These local disks come in a variety of sizes and speeds, and while they may be useful for a single, stand-alone editor, there are a few drawbacks. First, they can’t be resized dynamically so as a project grows you could find yourself short on space. Perhaps more importantly in this virtualized cloud world, once you shutdown the system you lose all your data, applications, and settings. This is not a factor in the on-premise world, but in a virtual environment the cost of running virtual workstations full-time is significant and you will want to shut-down these instances when not in use. Unless you can finish your project in one sitting or you have money to spare, this may not be the best option.
Block-based storage is a far more effective solution for video editing environments. In a virtual environment, you can provision the exact amount of low-latency storage you require and attach it to your editing environment when you spin up your system. Done correctly, you can even expand it as your needs grow. In addition, all of the public cloud vendors provide various types of block storage optimized for different performance and cost points so you can dial in exactly what you need for your editing applications.
From a cost point of view, block storage can have a permanence distinct from the compute instances. When you shutdown your cloud compute instances, whether that’s a virtual editing workstation or an asset manager, you can continue to maintain your storage so you don’t lose anything. Yes it’s not free, but it is far more economical than having to keep your entire environment running all of the time.
File storage is another possible option for video editing environments. Much like your NAS in the office, these systems provide a shared storage pool to multiple clients. They typically rely on NFS as the protocol. These file-based storage systems are sometimes offered with different performance characteristics, some optimized for higher throughput, but at a cost. These NAS-like systems, however, have limited storage capacity and performance. They are fine for IT applications, but video production environments can quickly run into issues.
Finally, object storage is a popular and cost-competitive storage tier, and you only pay for what you use, however for video editing it has its drawbacks due to latency and native application support. And while there are solutions for translating object storage to file-system calls, these gateways will continue to be hampered by the latency issues and often require caching workarounds. This is not ideal for professional or collaborative editing purposes.
The Best of all Worlds
For many teams exploring cloud production for the first time, the choices are overwhelming. Between the major cloud vendors, there are hundreds of choices of cloud storage tiers, performance choices, and tradeoffs. Too many choices can result in analysis-paralysis and leave organizations behind in the technology curve.
EditShare developed FLEX Storage as a software-defined-storage layer to abstract these complexities and provide high-performance, cost-optimized, video production storage. FLEX uses a mix of both block storage and object storage to provide that balance between storage performance and costs, and this mix can be altered to meet any workflow or budget. We’ve also benchmarked and tested the various storage performance types within these tiers – such as SSD, HDD, etc – to find the right balance for different application mixes.
As many of our customers require editing libraries of 10’s or 100’s of TBs of media, often captured at high bit-rates, we found a need to be more clever about storage cost optimizations. In 2020 we began offering FLEX Seamless Proxy Editing, which provides a cost optimized workflow that leverages proxy editing, but in a unique simple workflow. For those interested in learning more, you can read more here.
EditShare offers FLEX as both a standalone node or a cluster of nodes. Both configurations can be scaled out or scaled up to meet different performance requirements. The important thing to note, is FLEX is not a one size fits all solution. At EditShare we focus on openness and choice, giving our customers the ability to deploy what’s right for them, as well as the ability to adapt their environments as their needs change.
Get in Touch
Explore easy ways to get started in the cloud
Schedule time with an EditShare solutions specialist
