Choosing the right type of post production storage for media production and post production is critical for enabling creative teams to collaborate effectively and deliver work on time. The needs of media workflows are quite different from other applications and also from traditional IT-oriented considerations. Instead of focusing on the number of operations (IOPS) available from the storage layer, the critical metric for working with high resolution video is throughput: how many (typically) large media files can be read from (and written to) shared storage simultaneously by multiple editors and other users of the system.
The simplest form of storage used in media production is DAS: direct attached storage. That is, a disk drive (or maybe a RAID array of disk drives) attached directly to a workstation, typically via USB or Thunderbolt or other external protocols such as SAS. It is fast and, by definition, there can be no contention because only one person can use it at a time. If more than one person needs access to the media on the drive, they will have to unplug it from one workstation and plug it into another. The alternative is for each person to have their own DAS, which quickly adds up in terms of raw drive costs and means a lot of copying files around, with the inevitable challenges of managing versions and keeping changes in sync.
To avoid all these limitations we need to move to an architecture which provides shared storage.
One flavor of shared storage is SAN (Storage Area Network) storage. This is a large server environment to which multiple clients can attach and access a shared pool of storage available from across many servers and drives. When a client connects to a SAN it appears the same as a local drive, essentially like DAS, and applications will talk to the storage at the block level.
While the SAN provides a pool of storage, meaning that each workstation attached to it can have access to a large amount of storage when required, a given segment of the raw storage can still only be attached to a single workstation at a time and so has many of the similar limitations for media workflows as DAS.
Also, to get its promised performance, the SAN typically uses fibre channel connectivity (today, more usually FCoE: fibre channel over ethernet), a protocol which delivers fast data rates, but which really only works on local area networks, so is not suitable for remote collaboration.
There may be times when you need blisteringly fast connectivity, like dealing in uncompressed 4K or beyond with data rates in excess of 2GB/sec for digital cinematography, film restoration, and high end finishing. In these cases, SAN could be the solution, however these workflows can also be achieved with other shared storage options that also allow for easier collaboration, further optimising workflows and reducing overall data storage capacity requirements.
The alternative storage option is NAS, network attached storage. EditShare’s software defined shared storage is a high performance scale out NAS, whether deployed on-premises or in the cloud. The EditShare File System (EFS) is renowned for its excellent performance in production and post-production. Whilst you might initially think of NAS as a low performance storage designed for smaller applications, with a clustered architecture and other optimizations NAS can deliver high data rates and all the flexible and cost effective advantages of NAS without the need to build out a SAN.
A NAS, as its name suggests, is storage which sits on your core TCP/IP network, not a branch off to a separate, dedicated network. Any other device on the network can access any part of the NAS, so collaborative workflows become completely simple and transparent.
Unlike a SAN, though, NAS is not accessed at the block level: file operations are sent over the network. This file-level locking has the huge benefit of making it simple to integrate with creative software and intuitive in use.
The question, then, is how to get the performance we need for premium post-production with shared storage on a NAS?
Typically, workstations and NAS servers communicate using standard protocols like SMB or NFS. These are designed to be suitable for whatever the computer needs, however can have limitations when pushed to the limits demanded by high bit rate editing and post.
So EditShare developed its own, media-specific network protocol, accessible using EFS Native Client file system drivers that are available on MacOS, Windows and Linux.
Media applications rely heavily on multi-threaded operations to present multiple video, audio and other data to be streamed to users in high resolution. The EFS Native Client takes advantage of this pattern to balance the throughput load across multiple storage nodes and determine the fastest way to deliver files back to the application. The result is much higher performance than with conventional network protocols.
EditShare EFS storage systems are also inherently scalable: when you need more space, you add additional nodes and hard drives to grow the overall capacity of the system. A major advantage of this approach is that you can get a two-for-one deal: providing additional protection by spreading data across servers but also gain throughput benefits when reading data from multiple nodes to avoid hotspot of contention on particular hard drives or server nodes.
The advantages of software defined NAS are many. You can freely mix SSD and spinning disks (although in the real world you may well find that you do not need SSD as much as you think you do). You do not need to define the number and allocation of clients. Multiple clients can access the same files: you would have to manually define copy tasks on a SAN. If you need to re-size volumes from project to project, you do it with a couple of mouse clicks, not the wholesale restructure you need for SAN. In addition, the commodity TCP/IP network connectivity infrastructure you already have can be utilized, instead of having to deploy costly and inflexible parallel SAN infrastructure or bespoke dual personality Fibre Channel/NIC cards.
Importantly, for solutions such as EditShare EFS, there is a focus on the aggregate performance of the storage across multiple workstations simultaneously accessing a large pool of media. Of course, individual client performance is critical and EditShare continuously optimises both client and server to align to the sweet spot for real media applications. But beyond the individual it is just as important to consider the overall capabilities of the storage layer to support many users collaborating together across a range of applications that put different demands on the storage throughout the production lifecycle. It is this focus on performance of the applications using the storage that is important: can multiple users edit multiple streams without latency, jitter or dropped frames? The answer with EditShare EFS is, they can.
