Updates to EFS Collaborative Storage and FLOW Media Management Solutions
Our Director of Business Development, Jeff Barnes, sat down with SVG’s Chief Editor, Jason Dachman, to provided an update on our EFS collaborative storage and FLOW media management solutions and to highlight some of the key partnerships that are strengthening our presence in the sports industry.
In this journey stage, we will explore the concept of ‘editing in the cloud’. Here editors connect to cloud virtual workstations to perform post-production work with their video content management systems.
What is a Virtual Workstation?
Conceptually, a virtual workstation is simple.
The editing suite and assets don’t run on a high-powered local computer. Instead, the same software runs on a virtual machine in a cloud data center.
This virtual machine provides cost savings since you only pay for the time you use it.
You also don’t have to:
Maintain the server
Apply security patches
Deal with broken hardware
Have all of the other headaches associated with that workstation in your office
Virtual workstations in the world
Similar to workstations in the physical world, virtual workstations come in all configurations suitable for different workloads.
One can pick from hundreds of combinations of CPUs, RAM, storage, network interfaces and other features.
For video editing, workstations configured with GPUs are generally required. This allows the software configuration to run optimally for this video-processing-heavy software stack.
Virtual workstations and non-linear editing
Virtual workstations are considered headless systems. Like all cloud servers, they don’t have a monitor, keyboard or mouse.
Running a database is easy, but running a graphics-heavy program like a non-linear editing suite is more challenging.
To make this workstation useful, it is necessary to be able to attach a monitor. To do this, we rely on a remote desktop client.
Remote desktop software such as Remote Desktop for Windows or VNC Connect permits a remote user to log into a workstation. They allow the desktop window, keyboard and mouse controls to become remotely accessible, much like they are being accessed locally.
However, many of these general-purpose remote software desktop packages don’t cut it for video editing. They were not designed for the high-resolution and high-refresh rates required to project an editing interface remotely.
Fortunately, there are solutions to meet those challenges.
Virtual workstations for media
Teradici Cloud Access Plus and Nice DCV are software packages that address these video-intensive needs.
These applications are installed on virtual workstations. The workstation’s GPU encodes and sends video, keyboard, mouse, and USB data to a remote client. It’s as though you are running a very long HDMI and USB cable from the cloud data center to your local computer.
Additionally, these solutions can support two monitors for many environments, giving you a larger work area.
While these high-performance applications make it feel like a remote computer sits on your desktop, there are practical limits. The primary limitations are physics, the internet, and WiFi.
Even over the best network, the further the physical distance between the virtual workstation and your local computer, the more perceptible the lag will be.
For that reason, when deploying a cloud editing solution, it’s essential to choose a cloud data center physically close to your local office or home.
The internet routing itself will have an impact. Aside from distance, ensure IP connectivity, typically measured using ping times and throughput, is considered.
Lastly, it’s a well-known fact that the WiFi network most computers connect to is the biggest culprit.
For that reason, hardwiring your local computer to a router can help eliminate many sources of headaches. In addition, EditShare recommends a broadband download speed of at least 10-15 Mb/s per workstation monitor for best performance.
Workstation Instances and Software Images
Virtual workstations comprise server hardware and software, so let’s look closely.
Compute Instances
All major cloud vendors offer a family of GPU-enabled compute instances suitable for video editing. Like buying a PC or Mac, you can choose how powerful a system you want and how much you’re willing to pay.
You’ll want to choose the appropriate compute instance depending on your typical workload:
What codec bit rates are being edited?
How many concurrent streams are being accessed and edited?
Adobe has done a great job of publishing guidelines for selecting the appropriate workstations for different personas of video editors.
Here’s a table summarizing their choices for AWS, GCP and Azure:
The beauty of the cloud is your choice is never final.
If you choose the right cloud production solution, you can easily migrate from a low-end workstation to a high-end workstation and back again. Moreover, you can do this while preserving your editing process, environment and projects.
Software Images
In a virtualized environment, software images can easily be deployed to a machine. They can operate for a while and then spin back down when not in use.
Software images are pre-baked into a golden image to make this process efficient. This means they are pre-configured and ready to go when they are deployed to a virtual machine.
AWS calls these Amazon Machine Instances, GCP refers to them as an Image, and Azure names them Virtual Disk Image. The concept is the same. An image is created with the operating system, applications and optimizations. This image can then be quickly deployed to a system, and no further software installations are necessary.
In our example, this software image will include the remote desktop software, editing applications, and other supporting software. It provides a fully-function environment to start working right away.
Once deployed, users can then make changes to their specific environment – customizing their environment, updating software, and adding plug-ins.
Then, using persistent storage such as block storage, each individual workstation can be shut down while preserving that software image. This frees up the expensive compute instance. It also saves the image at a reasonable cost, ready to be used by the user at a later point on-demand.
As virtual workstations are equipped with GPUs, the editing applications often require the use of Microsoft Windows. Therefore, the cost of running a virtual workstation 24×7 can get pricey.
Saving the software environment to persistent storage allows users to only use workstations when they are active. Since a 40 hour work week is about 24% of the total hours in a week, the savings can be fairly substantial.
Cloud-based video editing
For the most part, the major non-linear application vendors are embracing cloud-based video editors.
Adobe Premiere Pro can operate in virtual environments. Adobe has done a great job documenting its best practices for Virtual Desktop Infrastructures on the prominent three cloud vendors.
DaVinci Resolve can operate in virtual environments. You can read about DaVinci on AWS here.
With the other non-linear editor vendors, we expect to see a progression towards cloud-based video editing tools as well.
Final Cut Pro is limited by MacOS not being available in cloud environments for editing use cases. (Macs have recently been supported in AWS specifically for the development of software applications and at this moment, don’t support FCP).
Avid Media Composer is also offered in the cloud via an add-on VM license. There are restrictions Avid applies to the use of their software.
Who can help me create virtual workstations for cloud video editing?
The EditShare team can help you create your virtual workstation for cloud video editing.
The ability to support multiple workstations and scale editing bandwidth to meet the challenge of production teams is dramatically more flexible in FLEX. Servers and storage can be scaled up and down to accommodate the needs of any budget or project.
Multiple editing stations, with any mix of horsepower, non-linear editing applications, and workflow requirements, can be instantly provisioned to meet a workgroup’s demands.
** This webinar has passed. Please fill out the form to access the recording.
May 12 Session 2: 3:00 PM USA Eastern Time (duration 1 hour)
Join EditShare’s VP of Business Development, Tom Rosenstein, and Lead Pre-sales Engineer, Nathan Rausch, as they outline the pragmatic steps toward a complete end-to-end production workflow in the cloud. Each stop along the way brings maximum value for an incremental investment. We’ll share sample diagrams, demos, and cost comparisons for cloud production workflows to help begin your Journey to the Cloud.
Save Your Seat
Webinar Speakers
Dan Schaffer, Director of Partner Strategy and Enablement, EditShare
Tom Rosenstein, VP of Business Development, EditShare
** This webinar has passed. Please fill out the form to access the recording.
May 12 Session 1: 9:00 AM USA Eastern Time (duration 1 hour)
Join EditShare’s VP of Business Development, Tom Rosenstein, and Lead Pre-sales Engineer, Nathan Rausch, as they outline the pragmatic steps toward a complete end-to-end production workflow in the cloud. Each stop along the way brings maximum value for an incremental investment. We’ll share sample diagrams, demos, and cost comparisons for cloud production workflows to help begin your Journey to the Cloud.
Access the Recording
Webinar Speakers
Dan Schaffer, Director of Partner Strategy and Enablement, EditShare
Tom Rosenstein, VP of Business Development, EditShare
In our previous article in the Journey to the Cloud series, we looked at migrating backup and archives of media content to the cloud. The primary intent being to move off of legacy on-premise archive solutions and provide a higher degree of resiliency and flexibility in the storage and access of media libraries. Once those high-resolution media files are stored in a public cloud, they are much more accessible for a variety of use cases.
In this article we’ll cover cloud-based, remote proxy editing which extends the usefulness of those cloud media backups to make them accessible for video editing. We’ll also cover one other benefit within this stage of the Journey, and that is a proxy-based browsable content library accessible to non-editors.
Video Proxies in the Cloud This stage in our journey builds upon the high resolution content stored in the cloud from on-premise backups. While this content is accessible for restoring to on-premise solutions, it’s not necessarily useful for editing from these cloud-based libraries. Large file size, object based stores, and latencies all make remote editing of these files impractical at this point in time. To address these hurdles we can turn to video proxy files.
Video proxy files are lower bit-rate and lower quality versions of the original high resolution source. Instead of working with source content which can be 100’s of Mb/s, we can generate proxies of these files which can be on the order of 1 Mb/s. This much smaller proxy file can then be used for editing. Upon playback that file may not be as high-quality and pristine as the original, but it can be good enough to edit against. The important thing to note is that the proxy file is frame aligned to the high resolution, so that whatever you see in the proxy at a particular timecode will match the high resolution at that timecode. This means you can create a sequence using mixes of several different video and audio tracks, edit them in a tool such as Adobe Premiere Pro, and then take the resulting EDL (Edit Decision List) back to the original high-resolution content where it can be conformed against the original high-resolution content. Most editing tools have the ability to easily re-link the proxy files to the high resolution (using rules to match files) and some also have the ability to link the high resolution and proxy together so you can toggle at any time.
Generating Proxies in the Cloud Now that we know proxy files are lighter weight facsimiles of high-resolution content we can focus on how they can be created. Several tools exist that can generate proxy-based content from original sources. EditShare FLOW is a media asset management solution that includes built in proxy generation capabilities. FLOW can be set up to automatically scan an object store and generate proxies of all this stored material. In addition, as a media asset manager, it takes care of managing the links between the two files, correlating those files for conforming and other workflows. In addition to FLOW, other 3rd party transcoders, such as AWS Elemental, can be configured to perform the same function.
The beauty of proxy files is the size. While the proxy bit-rates can be configured to suit one’s needs, the important thing is that they are significantly smaller in size than high resolution. For example, one may have high resolution content stored in DNX145 codecs (145 Mb/s) but the corresponding proxy can be 1 Mb/s – in other words, less than 1% of the original file. The result is that proxies are almost inconsequential in terms of overall storage costs.
Editing Proxies from the Cloud We have our high-resolution content in the cloud and we have created their corresponding proxies, now what? As with everything in video workflows, there are choices.
Remote Proxy Editing For many users, the attractiveness of using a local computer for editing is compelling. Home or office based computers can be set up to work against cloud video storage quite easily. Depending on the tool, proxies can be either downloaded or streamed down to the local computer for easy editing. This process of working with proxy material is referred to as off-line editing. Originally this workflow was useful because most computers could not meet the processing or storage requirements necessary for high-resolution editing. While this may still be the case with the demands of some codec’s, this well established workflow is perfectly suitable for remote-editing proxy editing.
Setting up access to your cloud environment is quite straightforward and will allow editors to quickly become productive. In establishing connectivity to your cloud environment, it is worth noting, access to proxies should be controlled through a secure protocol such as HTTPS or via a VPN to ensure security of your environment. The FLEX solution comes bundled with standard security features enabled by a range of cloud services. When it comes to rendering/conforming a sequence, high-resolution files can be accessed via similar secure protocols, however it may be faster and more cost effective to remotely render projects on a workstation within the cloud environment.
Leveraging the FLOW system can further allow video editors to become more productive through the use of the FLOW Panel for Adobe Premiere Pro and the FLOW Window for DaVinci Resolve. These ‘plug-ins’ provide the ability to map both the proxies and high-resolution content so editors can have the ability to edit against the proxies while providing the ability to view specific scenes by toggling over to the original content directly within the NLE tool set. Note that this high-resolution playback may be practical for viewing a specific frame or a media clip, if bandwidth is sufficient. However, this is not the best method for reviewing a sequence of high-resolution clips as the user experience will likely run into playback issues. Still, it serves as a very convenient spot-check for editors.
The Journey to Cloud Based Proxy Editing Once you’ve made the decision to backup your content to a public cloud provider, the steps to cloud based proxy editing are rather straightforward. This is simply a matter of creating the content proxies, and with the assistance of a media management solution such as EditShare FLOW, one can simply point their editing tool at the proxy and get started.
For many customers, there’s an additional benefit to building out proxy environments in the cloud, and that is content browsing and access. EditShare is seeing an increased interest in using a cloud environment to serve as the universal content library for organizations looking to unlock their media libraries from private vaults.
Once content is loaded into a cloud, FLOW can surface a browsable repository that’s managed, controlled and viewable by not just editors but other creative individuals who may be interested in looking at finished and raw footage previously stored in proprietary vaults. In doing so FLOW provides a user-friendly method for content re-use within organizations, and enable workflows that can reduce infrastructure spend.
Your Cloud Journey EditShare continues to provide its customers with solutions that make the most sense for them. If that’s on-premise, EditShare has you covered. And if you recognize that a migration to cloud services will offer benefits to your organization over time, EditShare can provide you with that pathway that makes sense for your business. If that’s simple backup and archives, we can help. If it’s extending that investment to allow for remote proxy editing or perhaps a browsable content library we can help there too. And if you are looking at perhaps moving your entire video production environment to a cloud environment we can help with that as well. In upcoming articles in the series, we’ll dive into additional aspects of the Journey to the Cloud.
As organizations embrace cloud technologies for video production, often the first area of focus is the use of a public cloud for media backup. The economics and practicality make sense now more than ever, different price points and capabilities can support a variety of needs. For many teams, it’s now a matter of if, how, and when.
Deciding if it’s time to move media from on-premise storage to the cloud begins with a review of costs and benefits. Traditionally, archives have meant tape – it’s density and costs are quite compelling but they have limitations such as accessibility, obsolescence, and missing tapes. More recently, object-based on-premise storage has become an attractive option. Solutions can use low cost, commodity hardware managed by software (referred to as Software-Defined Storage), and they offer performance and scalability across servers as needed.
Tape based systems, such as LT0, and on-premise object storage can be attractive to many organizations looking to build scalable content libraries. Increasingly though, the benefits of cloud based storage have emerged, and along with it the use of this methodology. To begin, cloud video storage offers data durability levels far beyond any on-premise storage, often keeping multiple data copies across multiple geographic zones. And of course, the pay-as-you-grow economics, lack of IT overhead, zero footprint and robust security features can all factor into one’s decision making process.
Navigating the Strata of Cloud Offerings Much like traditional storage systems, many public cloud providers offer a hierarchy of options when it comes to cloud storage. For media storage archives, object-based storage provides the scale that’s appropriate. Generally these tiers vary in cost, durability, availability, accessibility and other metrics. Below is a quick snapshot of AWS and GCP based tiers:
This chart is not inclusive of all features/costs but provides a quick overview of two public cloud vendors’ object storage options.
The obvious attraction to cloud object storage is the ability to select a tier based on expected access needs and archive duration – the options are plentiful. One point of note, users are generally charged for access to these storage tiers, whether egress fees for pulling the content off of the cloud or access fees for retrieving files stored in deeper tiers. These fees should be considered in any archival economic analysis. If archival is your single cloud consideration there are also cloud providers such as Wasabi that provide competitive storage pricing without any additional access & egress charges.
Okay, you’re pulling the trigger on cloud archival – now what? Migrating a media catalog of a substantial size will take some time to transfer. Due to this fact of physics, you may want to consider the brute force method. An AWS Snowball, GCP Transfer Appliance and other similar solutions provide a hardware device that allows you to copy your data locally and ship the device to the cloud provider for upload. For catalogs of TB or PB’s of media, this is often the most practical option.
Alternatively, one can always simply copy content up to an object store. This may be practical for smaller catalogs or incremental updates where upload catalog sizes are reasonable and bandwidth is suitable. Beware though, a standard HTTP or FTP transfer will be slow for even the largest of pipes – TCP/IP was designed for reliability, and not for raw speed. In many cases it is recommended that a user implements a file transfer acceleration solution from such companies as Data Expedition or Signiant. These applications allow for data to be moved up to 10x faster than a traditional protocol will allow.
Managing your Archive Library Intelligently managing your content library pays off in storage cost savings and reduces headaches due to aggravation caused by – “where’s that file”, “did I delete it”, and other similar maladies. A proper media asset management platform will integrate seamlessly into existing workflows and allow for content producers to focus on storytelling. The best solutions give administrators the ability to easily and quickly set up configurations and infrastructure to automate the backup and archival of raw content, finished sequences, and final productions to the proper locations with minimal overhead.
EditShare’s FLOW solution was designed as a skinny MAM to accommodate these workgroup demands, in conjunction with FLEX storage. FLOW and FLEX make it easy to set up these media spaces for raw content and projects and design and apply robust automations. This enables seamless workflows that backup content to near-line archives, cloud object stores, or deeper cloud archives with no tax on production teams activities.
Disaster Recovery Historically fires and other catastrophes drove companies to build out their disaster recovery plans – last year we added pandemics to the list. The increasing need to ensure business continuity in the face of future office closures is forcing many video production teams to look to cloud backups as an anchor point for DR plans.
As backup content libraries get stored to cloud platforms for resiliency purposes, cloud production solutions can also offer an on-demand DR solution in times of crisis. EditShare offers the ability to instantly use these backup cloud libraries as a source of editable content. In the event on-premise systems can’t be accessed, EditShare’s FLEX and FLOW systems can be instantly spun up, allowing virtual workstations to access and edit content with no downtime. By backing up the FLOW database to the cloud, we can also ensure project continuity between on-premise and cloud workflows. Editor’s may also choose to use their home workstations to connect into FLEX to allow for proxy editing remotely.
This ability to re-purpose media archives instantly and provide a video production environment allows organizations to manage their risks and costs while being prepared for future uncertainties. A compelling insurance investment for most organizations.
The EditShare Advantage EditShare continues to be singularly focused on helping video production teams increase their efficiencies and manage their costs. Media production teams come to us for help assessing the scale and scope of their backup, archival, and disaster recovery needs because we can tailor a cost effective solution that meets their requirements. With FLOW and FLEX, we make it easy to get media into and out of the cloud.
Your Cloud Journey Our ability to offer on-premise, cloud, and hybrid solutions enables the transition to cloud for existing environments and balanced approaches that optimize return on investment. One of the first ways organizations leverage the cloud is for media backup, and they find the most success today focusing on object based storage and carefully considering their needs and the fees of a variety of solutions and providers. After you choose a service, the next hurdle is selecting how to get their content into the cloud. Solutions are optimized to the various transfer sizes and speeds required. Once media is in the cloud, asset management plays a key role, aiding in the organization of media assets, as well as automation of changes and retrieval, integrating seamlessly with production workflows on-prem, in the cloud, or in between. Finally, implementing a robust disaster recovery plan for media back up in the cloud adds risk mitigation for minimal investment.
This approach to cloud adoption is a journey, undertaken by teams looking for flexibility and capability without significant investment. Key to their success is finding the most valuable ways to leverage the cloud, and adopting them gradually as their needs require.
