Text-based video editing: Lumberjack Builder, Premiere & Resolve
Text-based editing powered by AI is taking the post-production world by storm. Adobe and Blackmagic Design announced that their respective editing apps would showcase it as a built-in feature. Text-based editing uses artificial intelligence to produce a transcript of your videos that provides a way to edit the video by selecting text. Time savings is the major advantage of text-based editing, especially for documentaries and interviews. The Lumberjack system introduced AI-powered text-based editing scenes several years ago for Final Cut Pro, so the big news is that DaVinci Resolve and Premiere Pro now have it built in. But we’ll look at each system’s strengths and weaknesses so that you can decide which tool is right for your workflow.
The case for text-based editing
The script lays out the story for narrative films and TV shows. AVID’s Media Composer features ScriptSync to help editors match their edits to the shooting script. Their PhraseFind feature then allows you to search your audio clips and find just what you are looking for. But non-scripted shows like reality TV and documentaries operate in reverse. The final “script,” as it were, is really the byproduct of the editing process. It’s even been recognized that documentary editors are writers. Documentaries often cull together an enormous amount of interviews. Those interviews often overlap on common subjects. Those subjects form the building blocks of the film’s story. So there was a real need for more powerful and more cost-effective solutions for text-based editing.
It should be noted that there’s a huge amount of buzz regarding “Text-to-video” tools like Runway. That tool uses AI to create video clips from text prompts. “Text-based editing” uses AI to create a transcript from a video, like an interview. Then you use that transcript to edit your video together by pulling together important clips. These clips might come from a single interview or multiple interviews.
A “paper edit” is the product of using transcripts of interviews printed out on paper to craft an edit. This is the analog method of “text-based” editing. You can actually cut up the portions of the interviews and lay them out, and then group them by topic. This sounds archaic, but it can really help you to see the full story. Another version of the paper edit is to print out a list of markers from the interviews summarizing each point and interviewee discussed. Here’s an example of a paper edit from the 2017 documentary Fragments of Truth.
Paper edit from the Fragments of Truth documentary (2017, Reuben Evans)
In this example, each of the markers was typed out after a portion of an interview had been watched. Then the markers were printed out, and cards were made that listed the common subjects. Those became the building blocks for the film.
Paper edit from the Fragments of Truth documentary (2017, Reuben Evans)
As you can see, this process could benefit greatly from some technological improvements. This is one area where artificial intelligence can shave days, if not weeks, off the time it takes to log and organize your footage.
Those improvements took center stage at NAB 2023 when Adobe and Blackmagic Design announced that text-based editing would ship with their NLEs. You just have to love Adobe’s marketing tagline for Text-based editing, “No more paper cuts.”
DaVinci Resolve
Blackmagic Design included text-based editing in the DaVinci Resolve 18.5 beta. It brings the basics of text-based editing to DaVinci Resolve Studio ($295). Blackmagic calls it “Speech to Text.”
DaVinci Resolve Speech to Text (Blackmagic Design, 2023)
Resolve can automatically create transcripts for you using AI. It will identify silent portions of your clips as well. Simply select a clip in your media bin and click the “Transcribe Audio” button. Resolve will transcribe the text and note silent portions with ellipses. When you highlight the text of your transcription, Resolve will highlight that portion of your clip in the timeline. Resolve can use that transcription to create captions for your video as well. The YouTube channel “Creative Video Tips” has a great tutorial on Speech to Text editing in DaVinci Resolve.
You can see in the video that Resolve only addresses a couple of aspects of text-based editing. The reviewer is having to implement a “hack” where he is using one timeline to organize clips and another to do his edit. That other timeline functions like the “organization cards” in a paper edit. That makes DaVinci Resolve’s implementation pretty good for a single interview or a short video. But it falls a bit short of indexing and organizing the contents of an entire film because it doesn’t incorporate some key metadata. Identifying who is saying what in a documentary interview is highly beneficial. For instance, you may have a host appear in multiple locations or several speakers in a single interview.
Premiere Pro
Just a few days before Blackmagic Design announced Speech to Text, Adobe announced, “Premiere Pro is the only professional editing software to incorporate Text-Based Editing.” While that claim didn’t last long, Adobe’s implementation did go further than Blackmagic’s feature. Premiere Pro automatically transcribes clips and produces captions. Importantly, it allows you to identify the speakers.
It would be nice to see some more advanced tools when it comes to identifying speakers. Currently, the editor has to go through each phrase and identify the speaker. Adobe has shown off the ability for it to identify speakers, but they haven’t shipped that feature to beta yet.
Premiere’s text-based workflow adds a couple of other important features as well. Editors can import a transcript that has been created through a service like Rev.com, and you can associate that transcript with the clip. This is handy if your audio has technical words or foreign languages.
Adobe Premiere Pro Text-based editing, 2023 Adobe
Editors can export the transcripts that Premiere provides as well. This adds value to the transcripts because those transcripts can be uploaded to social media sites along with the video for increased SEO performance.
Both Premiere Pro and DaVinci Resolve allow you to insert clips from the transcription window. You can identify silent sections in your clips in both NLEs.
Adobe also provides a workspace for text-based editing in Premiere, making the feature feel more refined than Blackmagic’s implementation. It feels like Adobe has laid the foundation for more functionality in this workspace in the future. But currently, it is still limited in its ability to function as an organizational tool for a film with common topics across multiple speakers, as is the case with most non-scripted work. So Premiere appears to have the upper hand when it comes to built-in integration.
Lumberjack Builder
In 2018, Philip Hodgets from Intelligent Assistance presented Lumberjack Builder. When you organize footage, it is known as “logging” footage, hence the name Lumberjack.
Lumberjack then grew into a whole suite of logging and editing tools, culminating in the release of their new Lumberjack Builder NLE. Originally released for FCP, the Lumberjack system also works with Premiere Pro. It was the first system to connect AI for transcription with an editing interface and the first text-based editing tool.
Lumberjack combines transcription with keywords and other metadata that allow you to organize an entire project’s worth of footage and cull it together into an actual text-based edit. This comes from a deep understanding of the purpose of a paper edit. It is designed to work with keywords across clips the way an editor uses cards for organizing when doing a paper edit.
The key difference here is that Resolve and Premiere use the text as a “source,” but the “destination” is still the timeline. You read the words in the “source,” but you have to listen in the “destination.” Whereas, Lumberjack features the same interface when you are working through your source interviews or the timeline that you are assembling. The editor is working with blocks of text. This makes it a powerful tool for documentary filmmakers.
For films in languages other than English, Lumberjack offers 16 languages for free. And it integrates with a third-party transcription service for another 50 languages at 25 cents a minute.
Finally, Lumberjack offers real-time logging for interviews with their iOS app. The app enables metadata tagging by people, locations, or other key topics right on set. When combined with AI transcription and text-based editing, it’s a powerful solution. When the editor has finished their “paper edit” in Lumberjack, just send it over to FCP or Premiere and start the process of trimming.
Descript
The AI-powered online video editing app, Descript, uses a text-based editing approach as well. It’s designed to be easily accessible for anyone who needs to make simple videos like presentations. Descript also features an audio mode that is designed for podcasters. One of the big features of Descript is that it will help to identify and eliminate “verbal clutter.” Those are the umms and ahhs that we say when we don’t quite know what to say next.
Descript offers “Scenes” as an easy way to insert your b-roll. The editor inserts a slash into the transcript to identify the beginning and end of a scene. And then you just drag a clip or graphic onto that spot.
Descript now has the backing of OpenAI, so it will be really interesting to see what they come up with in the future.
Conclusion
Text-based editing is nothing new in the sense that Intelligent Assistance has been offering it for years. At the same time, it feels totally new, because far more people have accessed it through Resolve and Premiere Pro in the past few weeks than in the past few years. It is a tool that has proven its worth, whether through the old-school paper edit or the latest AI tech. So many AI-powered features will be coming to post-production professionals that it will be hard to keep up. Some will be of dubious usefulness, while others will transform job descriptions overnight. But the best tools will be the ones that empower storytellers to efficiently work their craft so that we can all do more of what we love.
MediaSilo allows for easy management of your media files, seamless collaboration for critical feedback, and out-of-the-box synchronization with your timeline for efficient changes. See how MediaSilo is powering modern post-production workflows with a 14-day free trial.
IBC launch of major new unified interface
Boston, MA – July 20, 2023 – EditShare®, the technology leader that enables storytellers to create and manage collaborative media workflows, is set to introduce EditShare One as its unifying user experience at IBC2023 (Stand 7.A35, 15-18 September). EditShare One gives a single, streamlined experience across all of the company’s high performance production asset management and media storage solutions.
EditShare One boosts productivity by simplifying interactions, with an intuitive user experience at every stage of the creative process. Initial applications include Producer View, for assigning tasks and delivering comments and feedback to the production team, making collaboration simple, even across multiple locations.
EditShare One Transcription View
The AI-integrated Transcription View is a powerful new application that speeds up identifying the key points in large media captures. Users will find Transcription View and other productivity improvements visible in FLOW directly as well as in the FLOW panel in the Adobe® user interface, and Resolve thanks to the seamless integration.
EditShare is already transforming post-production through Universal Projects, which allows projects to be set up, linked to bins, and synchronized with whichever editing software package is chosen. The new EditShare One user interface and its innovative debut applications will deliver even greater productivity and creative control.
“Our users around the world talk to us about the challenge of creating exceptional content within tight time constraints,” said Sunil Mudholkar, VP Product Management at EditShare. “That is why we have put the focus on strong workflows, using automation where it is practical. Now, with EditShare One, we have user experiences which are thoroughly intuitive and consistent, so you are able to sit at any workstation and understand the complete content flow and the state of any project.”
IBC2023 will also see the continuing extension of EditShare core technologies to support the modern media production industry. It is now common for content to be stored at multiple locations, in the cloud, and in remote workstations such as editors working on site or even at home. A single instance of FLOW maps all the content in every location and ensures it is ready. Automated proxy creation and integrated file acceleration ensure that the right media is always to hand.
“IBC brings the media industry’s leading creative people together“, Mudholkar added. “We are excited to show how we’re continuing to develop our platform to meet their real day-to-day challenges. More new introductions to the EditShare line up will be announced next month, so watch this space.”
Demonstrations of EditShare’s proven storage solutions and workflow software can be seen at IBC2023, on stand 7.A35.
EditShare is a technology leader that enables collaborative media workflows on-premise, in the cloud, or in a hybrid configuration. With customer and partner success at the heart of EditShare’s core values, our open software solutions and robust APIs improve workflow collaboration and third-party integrations across the entire production chain, ensuring a world-class experience that is second to none. The high-performance software lineup includes media optimized shared storage management, archiving and backup, and media management, all supported with open APIs for extensible integration.
EditShare’s cloud-enabled remote editing and project management technology was recently recognized by the National Academy of Television Arts and Sciences (NATAS) with a prestigious 2021 Emmy® Award for Technology and Engineering.
You turn in your cut and wait for the inevitable notes. “But my cut was amazing; they won’t have any notes…” and then, the email arrives. There are notes. Pages of notes. Some of them in all caps, some of them in bold. Page three even has some pictures. But it’s okay because, as any experienced editor will tell you, the review process is critical to all forms of filmmaking. And most especially so for television.
Television has come a long way. Traditional broadcast television still exists but is far less prominent than it was a decade ago. Streamers have taken over, and while the concept of television has changed slightly, the majority of shows available to audiences are created in a similar manner.
As an editor working on a television or streaming show, you are likely to work on a small team of post-production professionals who intimately understand the show, even when different directors bounce in and out across various episodes.
Being a good communicator
First and foremost, it is crucial to learn to communicate with your team. Not just your assistants, but your Post-Production Supervisor, other editors working on the show, the VFX and sound mix teams, the executives, showrunners and directors.
Editors working in television tend to be in a unique position. They are the shepherds of the show, supporting the vision of not just the director but the overall show itself, from showrunner notes to executive notes. That’s why TV editors, like editors across all mediums, need to be great communicators.
Here is some advice on how you can effectively communicate with your team and guide the process most effectively.
Set your ego aside
Remember that you are there to support the goals of the show. You may have a vastly different personal style than that of the show or the particular director working on an episode with you, but if you always come from a place of supporting the show, you’ll be able to more effectively pitch ideas that resonate with the team.
Be willing to accept any ideas that come your way, not just the ones that you like the most. Being open to hearing the ideas of others and implementing them into your cuts will help your colleagues and collaborators see the vision of the show and push it forward in the review process.
Internal notes vs. network notes
The process of editing a TV show typically looks like this, although every show can be slightly different. First, the editor will deliver a cut. Then the director will have somewhere from two to four days to spend with the editor to deliver a cut that fits their vision. Notes are then given on the cut, and the editor works directly with the show’s producers and showrunner to prepare a cut to share with the network.
Then the network begins sharing their notes. The network notes can vary greatly depending on the show you are on, the relationship that show has with the network, and how much that show fits the network’s brand and goals. You may be collaborating with your showrunner on how to achieve some network notes while pushing back on others. During this process, you’ll be attending tone meetings, which help align all of the creative goals on the production under the guidance of the showrunner.
Remember that during this process, the show is being given notes as a whole. The showrunner is likely feeling stress from the network notes and needs to decide how to achieve them, whether they are going to push the show too far in one direction or another, and whether or not to fight the notes. As an editor, one of the only people who has seen all of the footage forwards and backward, you can help guide this part of the process by being supportive of any internal workflows your showrunner puts in place to help try out ideas. Being open to receiving notes at this time will help you share ideas and concepts with your team that will help them push the show forward to the network at the next screening.
Showing options
If you have done any sort of client work in the past, you might be familiar with the idea of never saying, “No, we can’t do that.” Instead, if you find an idea is asked of you that you don’t agree with or know won’t work, don’t give a firm “no.” Rather, find a way to achieve something similar but different. Present an option and explain why the original note would have been too complex to accomplish. Presenting an option that is exciting and does work is a great way to showcase your talents as an editor but also diplomatically say, “I’m very supportive of this idea, but the idea that you wanted to do would have been too difficult to achieve.”
Nothing can stop a review process in its tracks like simply saying “no.” Saying “no” is an easy way to get told “just do it” and then forcing everyone down a path of resistance that ultimately will not work. If you spin your wheels on an idea for a long time that ultimately you knew wouldn’t work, then all of that time is lost and could have been better spent punching up other parts of the show. Finding a collaborative way to help directors and executives feel that their idea was still heard helps them push the show in a forward direction.
Working with technology
We live in a world full of technology. Whether your editorial team communicates on Slack, Discord or just a lengthy text thread, here are some great technologies that can be utilized to aid in reviewing episodes.
Messaging tools
Your team will likely need to speak to each other often. In today’s world of remote work, it’s entirely possible that many members of your team are in distant places.
Find a communication tool that works for you. Slack and Discord are great options. If your entire show runs on Microsoft products, you might consider Microsoft Teams. For Google Teams, you might consider Google Meet and Spaces. Whatever you choose, make sure that the communication product you use is easy to set up and that everyone can access it both on their computers and mobile devices.
Meeting tools
Just like communication tools, it’s important to get everyone on the same page about how you will be talking to each other. It can be helpful to guide your team to one singular communication tool for having live conversations to cut down on the difficulty of setting up calls. For instance, if your team is used to using Zoom, then use Zoom for everything.
Alternatively, if your team is entirely a Google team, then consider skipping Zoom and relying solely on Google Meet.
The goal is that you can easily say to anyone on the team, “Let’s meet on Zoom,” and they know exactly what you mean. Sometimes you may just have a question or an idea that can be figured out in less than 5 minutes on a call. This can cut down on lengthy back-and-forth emails.
Reviewing
For reviewing scenes or episodes, it’s important to decide on a tool that will help your team all work together. That’s where a platform like MediaSilo comes in. It’s important for your team to always be looking at the same thing and sharing their ideas in a way that is easy for the editor to understand and accomplish.
Tools like MediaSilo can help consolidate feedback, remove the complexity of writing out timecode notes, and make it easier for everyone involved to know when an idea was shared and have a conversation about that idea. Having all of your feedback and approvals in one place helps expedite the review and approval process by keeping everyone on the team aware of any changes that are being made.
Wrapping up
As an editor, you are the only person who has seen every frame of film that was shot. You know the show forwards, backward, and in some cases upside down (depending on if you flipped some shots). It’s up to you to help guide your team to success. Your role goes beyond simply pushing buttons in the Avid; you are there to help guide your team, discuss ideas, manage schedules and ultimately be the one responsible for crafting the flow of the show.
It’s not an easy job. But with good communication skills, you can be highly effective.
MediaSilo allows for easy management of your media files, seamless collaboration for critical feedback, and out-of-the-box synchronization with your timeline for efficient changes. See how MediaSilo is powering modern post-production workflows with a 14-day free trial.
Three tier storage provides rapid access and security
Boston, MA, July 19, 2023 – EditShare®, the technology leader that enables storytellers to create and manage collaborative media workflows, has a continuing relationship with PROGRESS.film, one of the biggest theatrical distributors in Europe and owner of vast archives of historical film. As well as supporting in-house production, the EditShare storage network is closely integrated with AI technology from Veritone to build a platform to monetize the archive.
PROGRESS was founded in East Berlin in 1950 and, until the fall of the Wall, was the only film distributor in East Germany. Today it holds the complete film heritage of East Germany, along with exclusive collections from East and West Europe, Vietnam, Ukraine, the US and more. The archive runs to more than 26,000 films.
Part of PROGRESS is a production company, LOOKS.film, specializing in historical documentaries. Original footage shot by LOOKS is also added to the archive, becoming a resource for film-makers of the future. LOOKS selected EditShare as the best platform to support its production activities, and parent company PROGRESS recognized that the flexibility, security and power of the EditShare solution was ideal as the foundation of its archive projects.
The goal was to create workflows which would enable the team to bring online vast amounts of film at multiple locations across Europe, and link it with excellent metadata in part generated by AI software. By integrating the EditShare storage network with a highly automated commerce platform from Veritone using APIs to build a seamless solution, PROGRESS provides archive access to film-makers everywhere, with a cost-effective license and delivery workflow.
Gunnar Dedio, CEO at PROGRESS
“We acknowledge that our archive represents a unique library of unmatched historical and cultural significance,” said Gunnar Dedio, CEO at PROGRESS. “We wanted to make this as widely available as possible, by giving film-makers simple online access to search through our archive, select the footage they need, and to create a license and download their content. “After extensive technical trials, we identified the two best players in the field, EditShare and Veritone,” he continued. “They worked together on the APIs which enabled us to create the powerful, seamless system we have today.”
At the heart of this system is the three-tier storage network from EditShare. At the top layer is an EFS server with a total of 120TB of storage, providing fast and agile access for online users including eight editing suites. Supporting that is another EditShare disk structure providing 320TB of nearline storage, making all content in current use readily accessible, and providing a buffer layer to move content in and out of archive.
The third tier is an EditShare ARK LTO8 tape library. Currently this has around a petabyte of storage but can continue to grow, and provides secure backup as well as highly resilient long-term archiving. EditShare FLOW asset management manages all levels of storage and generates proxies as required, and runs on its own servers.
“This scope and significance of the archive project at PROGRESS is huge,” said Said Bacho, chief revenue officer at EditShare. “But it is achieved with standard building blocks from EditShare: server nodes, nearline storage and tape archives, brought together with FLOW software. We are very proud of this project, and it shows clearly what can be achieved with technology based on open standards and simple APIs.”
EditShare is a technology leader that enables collaborative media workflows on-premise, in the cloud, or in a hybrid configuration. With customer and partner success at the heart of EditShare’s core values, our open software solutions and robust APIs improve workflow collaboration and third-party integrations across the entire production chain, ensuring a world-class experience that is second to none. The high-performance software lineup includes media optimized shared storage management, archiving and backup, and media management, all supported with open APIs for extensible integration.
EditShare’s cloud-enabled remote editing and project management technology was recently recognized by the National Academy of Television Arts and Sciences (NATAS) with a prestigious 2021 Emmy® Award for Technology and Engineering.
Avid Media Composer is one of the leading edit software packages in use today, and has some unique features that indexes media so that it can be referenced in projects without knowing the exact path to the media. This can be extremely helpful on large projects but does require a specific media ingest process.
The way that Avid handles content is that each file is given a unique ID reference (known as a Media Object or MOB ID). Ingested media is wrapped into the Material eXchange Format (MXF), and the ID is saved as part of the MXF metadata. The whole of the familiar Avid workflow depends upon this.
On a simple project, the editor sorts material as the content is ingested, and it is automatically formatted and ready to go. But in a busy post environment, this is an inefficient way of working. Most obviously, it ties up an expensive editing workstation and related facilities simply to ingest content. It is also a manual process, so an edit assistant is needed to control the ingest.
EditShare’s integrated production asset management software layer, called FLOW, provides a number of automated tools to simplify this process. The idea is to help users streamline their workflows so they become more productive.
For Avid users, an important tool in FLOW is the ability to create Avid format files, complete with unique ID burnt into the MXF data. This is done not in the Avid workstation but on FLOW servers As soon as the editor sits down to work, all the files are ready in the right format.
Not only is this not taking up edit workstation time, but it can also be completely automated. Set up a watch folder, and new content will be automatically prepared as it arrives on the storage.
More than that, FLOW and its Universal Projects software tool allow you to organize material into universal bin structures that can be synchronized into Media Composer and other editing tools. An editor or edit assistant can structure the bins to suit the specific requirements of the project. All this happens in a web browser that can be accessed by anyone wherever they are located without tying up the suite.
This is a real boost to productivity because it separates organization from creativity. Much of the content preparation and Avid file format conversion happens automatically. Bin structures, markers, subclips and sequences can all be managed from a browser (or even automated). Then all of this appears on the Media Composer screen so the editor can start working right away.
For any busy facility, this is a perfect application of technology. Everything that can be automated is: no one needs to manage file ingest transcoding and rewrapping. Processes that need to be prepared in advance are: content is selected; bins are created and populated; the timeline is set up.
And there is nothing in the way of the creative process. No preparation, no waiting for file conversion or transfers. The editor simply focuses on making the content as good as it can be.
The attention of creatives, lawmakers, and technologists is fixated on generative AI. The ability of a computer to turn a line of text into an image, sound, or video is simultaneously exciting and scary. Adobe has introduced its Firefly (beta) engine in an attempt to empower rather than replace creative professionals. The Firefly page reads, “Adobe is committed to developing creative generative AI responsibly, with creators at the center. Our mission is to give creators every advantage — not just creatively, but practically. As Firefly evolves, we will continue to work closely with the creative community to build technology that supports and improves the creative process.”
Not for commercial use
Adobe is just getting started with generative AI tools. The images produced by the Firefly beta are only for non-commercial use, according to the FAQ page. In this article, we’ve used images produced by Firefly when commenting on them (under Fair Use), but not for the header image (just to avoid any problems). One of the goals of Firefly is for creatives to be able to include imagery created with the help of AI while eliminating this kind of second-guessing.
AI tools for still images
Firefly for still images works on the web and in Photoshop. We’re going to focus on the web version. Adobe allows you to upload your own images to the site or use some of their sample images. Adobe claims that all of the images that Adobe uses to train its AI have been appropriately licensed.
In-painting
Here’s a sample image that Adobe provided. The woman is wearing an orange jacket and standing in a restaurant. Her portrait has been taken with a shallow depth of field, that’s why the background is blurry, and she is sharp.
You can use the Insert tool to highlight her clothes and describe a new look. The line “A black cocktail dress” is entered into the search box.
Almost instantly, Firefly puts her in a dress appropriate for the evening. Several options are provided. The first one wasn’t great, but this option with the necklace should work.
Changing out the background is just as easy. Click the “Background” button and enter a prompt, like “a cocktail bar.”
And Firefly delivers an appropriate background, with a bit of an awkward attempt at a hand holding a glass with a clutch purse attached.
Is this image going to go up on a billboard anytime soon? Probably not. Could a professional Photoshop artist do it better? Or course. But there will be plenty of uses for this level of imagery. And as time goes on, the AI will keep improving.
Text to Image
The next tool that Adobe offers is called “Text to Image.” You can describe a scene and see what comes up. Just for fun, let’s go across the room and see what Adobe gives us for “a man holding a drink wearing a black suit in a cocktail bar.”
And a dashing selection of well-dressed gentlemen appear. And their hands don’t look too bad, just a little off. Maybe one of them would be a good match for our lady above.
Text effects
Firefly’s next tool lets you experiment with some crazy text effects. In this example, The word “Yum” is filled with a 3D pattern of “Mediterranean cuisine.” The sidebar shows you a bunch of different options like Snake, Ballon or Bread Toast. You can change the background color and then copy and paste the image for use elsewhere.
There are also ways to produce variations on the text effects. For instance, you can change the “fit” from medium to loose. Now you can see how the design spills outside of the letters.
Generative recolor
Adobe has made one more tool available in the Firefly web demo. It’s called “Generative recolor.” You upload an SVG (Scaleable Vector Graphic). And then, you can choose from several tools that allow you to rework the color pallet.
You can choose from the suggested themes or use text to create your own. Additionally, you can select the “harmony” of the color palette, like complementary or triad.
Firefly for video
Firefly is still in beta and focused on still images, but here’s a look at what they have coming for video. There’s an exciting set of features on the way for video editors and motion graphics artists.
Depth
Adobe demonstrated the ability to add depth to an image with a prompt, “A sunlit living room with modern furniture and a large window.”
Firefly then scans the image and appears to understand the dimensional aspects of the space.
This allows for the image to be shown with multiple styling options. I could see this kind of tool being used by production designers to create looks for shoots.
3D to image
Adobe demonstrates how Firefly will go beyond understanding 3D depth in 2D images. Firefly will actually be able to make 3D objects to place in your scenes.
In this example, they show how a 3D object of a castle can be composited into a generated scene with a prompt.
And then that same model could be restyled into a “castle dessert.” Firefly changes the appearance of both the model and the scene. It understands the context for desserts might be a plate or picnic table.
Conversational editing
Most of the time, the initial image that you get from AI won’t be exactly what you need. But if you refresh the search, you end up back at square one. Conversational editing allows you to keep tweaking the image until you get what you want by “texting” your image. You can become the ultimate annoying client, and your AI designer will never grumble.
The sequence starts with the image of a dog.
The first prompt is to dress the dog in a Santa suit.
And that’s followed by a request to put him in front of a gingerbread house.
Unlimited iterations
Unlimited, instant iterations of artwork have the potential for absolute chaos when it comes to creative deliverables. The mind boggles at the revision requests graphic artists will endure. And then, once they have shipped their work, clients will go to work texting the image to change it further.
VFX artists on “Spider-verse” talked about the cycle of revisions and the long hours that went with that. The executive responded to their concerns: “I guess; welcome to making a movie.” On one side, artists fear losing their jobs to AI. But there is a disconcerting intermediate step. It will be so easy to change art that one’s artistic intent may not be reflected in the “final” project. This may have a chilling effect on people’s desire to enter the arts in a professional capacity. Nobody knows the future, but we know that it will look different than it does today.
Audio production
Adobe’s video showcased many advancements in video production. And they aren’t limited to images. Custom music and soundtracks will be incorporated into Adobe’s tools.
Sound effects based on items in the images will automatically be created. Firefly will understand the elements in your images and suggest appropriate sound effects.
The “effect” this will have on the stock music and stock sound effects industries will be monumental. We’ve already seen many AI tools that can help with voice isolation and noise reduction. Currently, editors subscribe to music and sound effects websites. If Adobe builds AI tools into their video editing apps that automatically suggest sound effects and music from libraries, those sites will have a major uphill climb getting people to purchase music and sound effects files that don’t adapt to the duration of their timelines.
Video Editing
The Firefly engine looks like it will take a significant amount of grunt work out of video editing. Adobe is building workflows to automatically insert B-roll based on the script or voice-over. It would only make sense that those b-roll clips could be generated by AI rather than limited to what you shot that day.
Adobe showed off automated storyboards and previs based on the script. Color grading and relighting based on text prompts. Captions and animated 3D text are just a prompt away.
The high end of the editing world may again coalesce around masters of the craft. However, the medium and low ends of the video editing world will undoubtedly shift, as creators will be able to craft films with nothing more than a keyboard.
Adobe’s goals
Rather than Firefly being a standalone product, Adobe wants to integrate it into its existing tools. We’ll see more generative AI tools throughout their products as they gradually eliminate formerly time-consuming tasks. Adobe has committed itself to an approach to AI that doesn’t steal work and avoids biases. They are putting in tools to help imagery avoid being scanned by AI bots. At the same time, governments like the UK are considering laws to label AI images, and AI sequences like Marvel’s Secret Invasion intro are seeing some backlash. Adobe knows its customers, so hopefully, it can walk that fine line of empowering creatives without displacing them.
Conclusion
Adobe’s vision for the Firefly engine is technically ambitious and thrilling for anyone who wants to create. The desire to turn our words into worlds is as old as time itself. The result may be an explosion of creativity, or it may end up being a mountain of uncanny images. But one thing is for certain; this is just the spark of the AI revolution.
MediaSilo allows for easy management of your media files, seamless collaboration for critical feedback, and out-of-the-box synchronization with your timeline for efficient changes. See how MediaSilo is powering modern post-production workflows with a 14-day free trial.
CLIOs. Tellys. EMMY’S. DEFINITION 6 knows what award-winning work looks like because they’ve produced it. Chris Reinhart, the SVP of Post-Production for DEFINITION 6’s Entertainment Business Unit, has helped lead their team to win multiple Daytime Emmy Awards for his work on Sesame Street, in addition to a Sports Emmy for editing ABC’s coverage of the 90th Anniversary of the Indianapolis 500. Behind many of these awards, Chris and his team continually rely on MediaSilo to craft compelling stories and bring their client’s work to the finish line.
Before MediaSilo, Chris and his team constructed their own in-house approval system, in which they would manually digitize their individual video assets and upload them to their site. This process was burdensome and not sustainable. The digital asset management world was evolving, and Chris knew they needed to find a solution that allowed them to seamlessly work on multiple projects at a time and expedite the completion rate of their work to their client’s satisfaction. Having worked with MediaSilo in the past, Chris knew it was a tool that could improve their post-production workflows. Chris led his team to make the switch to begin using MediaSilo for their entire Audio/Visual needs.
“DEFINITION 6 navigates hundreds of versions of assets across dozens of clients with MediaSilo, and it’s incredibly straightforward.”
DEFINITION 6’s work ranges from short promotional ads to documentaries. The lifecycle of their projects varies from a couple weeks to several months. Regardless of project length, the workflows are roughly the same. As content is being shot on location or in a studio, cuts are uploaded to MediaSilo and securely shared with external stakeholders and clients for their actionable feedback and approval in MediaSilo Review Links. During the review and approval process, the editors at DEFINITION 6 will make the necessary revisions to the cuts and then send those back to the customers to take a look at. Additionally, inside the customer Review Links, Chris is able to “stack” multiple cuts on top of each other inside the original link for the client to easily swap between the original asset and the new, edited versions to see if the correct changes were applied. Without this feature, client work can get lost in the shuffle and force clients or execs to go digging in a sea of links and emails in order to tell if their feedback was addressed properly. MediaSilo ensures clients can view all their content and feedback in a single, easy-to-navigate location.
With MediaSilo, Chris and his team have the ability and confidence to work on dozens of client projects simultaneously while keeping all their work organized and moving in the right direction. For Chris personally, he loves to be able to seamlessly switch between his projects at DEFINITION 6 and into his other customer’s MediaSilo workspaces and projects without needing to log out and back in under a different user name. The lack of workspace switching capabilities could take up valuable time better spent on their work.
In addition, MediaSilo’s ease of use for their customers and clients keeps Chris and his team relying on it project after project. Onboarding new users and employees added to their workspace takes very little training time, which makes transitioning from project to project effortless. And MediaSilo’s powerful mobile app allows users to put their work in their pocket and take it on the road while providing peace of mind that their assets are secure.
Chris emphasized DEFINITION 6 works with some of the most globally well-known clients and influential brands in the Media & Entertainment world today. If a piece of media, big or small, is leaked or put in the wrong hands, it could have drastic consequences for all parties involved. Time and time again, their trust is placed in MediaSilo to securely store and share their assets with only the intended users. MediaSilo’s SOC 2 Compliance not only gave Chris and his team peace of mind regarding their client’s work but also gave DEFINITION 6 the confidence to broaden its user base and implement the MediaSilo platform into other departments in their organization, such as Production, Casting and Sales. This grew MediaSilo as not only a place to collaborate on their work-in-progress projects but also a platform to serve as a library to organize and store their finished work. Furthermore, DEFINITION 6’s MediaSilo users are spread out across different project bases, from Entertainment to Public Relations Projects. MediaSilo allows administrators on the DEFINITION 6 workspace, such as Chris, the ability to strictly govern which users have access to which files to make sure all their work is secure and in the right hands.
“You want the thing you’re gonna do over and over again to be reliable and as simple as possible.”
At the end of the day, Chris emphasized that cool new features can only go so far with any platform, and the most important aspect that continues to bring him and his team back to MediaSilo project after project is its reliability. According to their Chief Engineer Luis Albritton, DEFINITION 6 uploaded over10,000 assets, sent nearly 7,000 review links and hosted almost 24,000 viewers of their content in MediaSilo during 2022 alone.
Additionally, Chris pointed out that they send roughly 50-60 MediaSilo review links per day to one of their top clients. With numbers like this, it is imperative for MediaSilo to consistently be a platform that is both reliable and secure. DEFINITION 6 has put their trust in MediaSilo for over half a decade to reach deadlines and keep their clients coming back for all their Media and Entertainment needs. MediaSilo continues to be a tool that checks all the boxes for DEFINITION 6, and we hope to continue this partnership for many more projects and awards to come.
MediaSilo allows for easy management of your media files, seamless collaboration for critical feedback, and out-of-the-box synchronization with your timeline for efficient changes. See how MediaSilo is powering modern post-production workflows with a 14-day free trial.
Remote and high throughput post for busy broadcast facility
Boston, MA, June 27, 2023 – EditShare®, the technology leader that enables storytellers to create and manage collaborative media workflows, continues to work with Red Pepper Pictures, now adding a powerful EFS 450 storage system at the company’s Johannesburg headquarters. The EditShare media storage and management systems are also accessible from Red Pepper’s other facility in Cape Town, South Africa.
Red Pepper is a production and post business serving broadcasters, advertisers and corporate clients, in Africa and globally. Focused on creativity, across all their activities the company generates more than 4500 minutes of content each month.
This level of business calls for highly tailored workflows as well as carefully controlled storage and media management. This is particularly true in the case of high profile, fast turnaround reality television like the “Housewives” reality franchise, “Who Wants To Be A Millionaire”, “Big Brother” and a popular Afrikaans music reality show “Die Kontrak”. Red Pepper initially invested in EditShare storage in 2013, and has scaled its facilities since then, most recently adding the powerful EFS 450 system, designed for demanding bandwidth applications.
EFS is a software-defined storage ecosystem built on commodity hardware and designed for practical workflows, with the ability to scale from a single storage node to an enterprise cluster, with a single namespace for simple management. The EFS 450 incorporates fault tolerance and very high bandwidth, with intelligent data distribution to ensure the timely support of large numbers of users.
It incorporates EditShare’s FLOW media management software with dedicated, redundant metadata servers. The workflow toolset is particularly important when multiple editors need access to content simultaneously, in applications like fast turnaround editing of programs with a very large number of sources.
“We turned to EditShare because we needed something that could manage high throughput, fast turnaround edits,” said Christian Nenkov, Head of Post Production Operations at Red Pepper. “We found that it could do the job far better than anything else on the market. Our staff are all comfortable with the EditShare toolkit, and it makes sense to continue to invest in it as our business grows.”
Protea is EditShare’s partner in South Africa, and provides the sales, installation and support for Red Pepper. “They are a really busy company, with some huge productions on their slate,” said James Macpherson of Protea. “FLOW lets them set up customised workflows for complex reality shows, and it is simple to add remote access to and from Lagos and Cape Town. We are pleased to continue this strong relationship with Red Pepper.”
“We’ve seen substantial growth in the region over the past two years thanks to Protea. Their ability to highlight the key end user workflows and requirements so that we can provide the right solution has been critical in the successful deployment of this and many other systems,” commented Said Bacho, CRO at EditShare.
EditShare is a technology leader that enables collaborative media workflows on-premise, in the cloud, or in a hybrid configuration. With customer and partner success at the heart of EditShare’s core values, our open software solutions and robust APIs improve workflow collaboration and third-party integrations across the entire production chain, ensuring a world-class experience that is second to none. The high-performance software lineup includes media optimized shared storage management, archiving and backup, and media management, all supported with open APIs for extensible integration.
EditShare’s cloud-enabled remote editing and project management technology was recently recognized by the National Academy of Television Arts and Sciences (NATAS) with a prestigious 2021 Emmy® Award for Technology and Engineering.
This summer, three of the major Hollywood unions are negotiating new contracts. The WGA went on strike last month. SAG-AFTRA is currently in negotiations, and speculations are they will join in solidarity with WGA when their current contract expires June 30 (although negotiations may extend past this date). On Friday, June 23rd, the DGA voted (by 87%) to ratify a three-year contract with the studios.
One of the key issues in contention with all three major guilds is the use of AI technologies and how they affect the relative aspects of the film industry. It’s more clear how AI affects writers and actors. For the former, it can be used to complement or even completely supplement aspects of the writing process. For actors, the advancements in AI to create lookalikes and soundalikes are both fascinating and frightening.
What’s less clear is how AI impacts members of the Director’s Guild.
According to The Hollywood Reporter, the contract specifies “that generative artificial intelligence (Gen AI) is not a person and that work performed by DGA members must be assigned to a person. Moreover, “Employers may not use Gen AI in connection with creative elements without consultation with the Director or other DGA-covered employees,” and top entertainment companies and the union must meet twice annually to “discuss and negotiate over AI.” There’s a lot in that little paragraph to unpack. So let’s dig in.
What is “generative” AI?
You’ve heard of ChatGPT. The “G” stands for generative, “generative pre-trained transformer.” Chat refers to the way you interact with it. It can generate original content based on your requests. DALL-E (As in the artist Salvador Dali and the Pixar character Wall-E) generates original art. McKinsey has a great rundown on the basics of AI. And Runway is focused on using generative AI technologies for storytelling. So there’s a lot of concern about the loss of jobs in the creative industries when you have technologies that are designed to emulate the original output of people. Everyone would like technology to make their jobs easier, but we’re wary of its ability to threaten our ability to “generate” an income.
Different kinds of AI
If you ask ChatGPT, it will tell you that there are at least eight commonly recognized forms of AI. Narrow, General, Superintelligent, Machine, Deep, Reinforcement Learning, Natural Language Processing and Computer Vision.
Narrow AI
Apple’s Siri and Amazon’s Alexa fall into this category. They focus on a specific task, and their intelligence allows them to excel at that one thing. These systems can be used for recommendation engines but don’t possess a “general intelligence. Narrow, or weak AI, can provide plenty of value in post-production. It can do things like automatically match the color of two shots or duck the volume of the music. If NAB 2023 was any indication, post-production pros will see new Narrow AI-powered tools coming on everyday.
General AI
Also known as AGI (for artificial general intelligence), is a theoretical AI that is the opposite of Narrow AI. It would be smart across many domains. It could have many skills. Wired reports, “Microsoft Research, with help from OpenAI, released a paper on GPT-4 that claims the algorithm is a nascent example of artificial general intelligence (AGI).” In theory, AGI would have the ability to think on a human level. And the problem with that is how do we align it with our own interests.
Superintelligent AI
Also known as ASI, this is again a theoretical AI that surpasses human intelligence rather than just matching it. This Artificial Super Intelligence would have thinking skills of its own. We don’t know if this is possible to create, and if it is possible, we don’t know if we can control it.
Machine Learning (ML)
Machine learning comes in various forms, including supervised, unsupervised or semi-supervised learning. A machine is trained on a data set, and it may or may not really understand the “right” answer. But machines can begin to spot patterns that might not be apparent to us. Eventually, the system can learn to spot patterns, like identifying a dog in a picture. Generative AI uses machine learning algorithms. It is the foundation for the technologies that we are seeing today.
Deep Learning
This is a subset of Machine Learning. Google’s new generative AI search results say, “Deep Learning structures algorithms in layers to create an “artificial neural network” that can learn and make intelligent decisions on its own.” Statements like this might send a chill down your spine. Those “decisions” have more to do with recognizing and identifying patterns than Terminator-style decisions about who lives and who dies. But how the Deep Learning machine comes to its conclusions can be a bit of a mystery. We know that Deep Learning requires a large set of data. So It is easy to see how a platform like Netflix can use Deep Learning. By analyzing the viewing habits of its members, it can use that analysis to feed a recommendation engine.
Reinforcement Learning
When you put an AI into an unpredictable environment, how does it make decisions? That’s what Reinforcement Learning teaches the machine. It makes decisions and observes the consequences. If rewarded or punished, it learns to do more or less of that. This kind of AI will make a huge impact on online marketing. Multiple iterations of ads will proliferate, and AI will be able to adapt, deploy and adapt again based on whatever is most profitable.
Natural Language Processing (NLP)
This aspect of AI speaks to its ability to understand and output language like a person. By understanding linguistics, the computer learns how to sound like us. This capability is great for things like spell check, transcription and translation. This tech is already taking post-production by storm.
But this capability feels like an existential threat rather than a helping hand for those who make their living writing. This explains why the WGA has focused on the need to refrain from supplanting writers with machines. Writers can use AI to assist their process, but they don’t want the studios to replace them with machines.
Computer Vision
“If AI enables computers to think, Computer Vision enables them to see, observe and understand,” says IBM. Netflix is using Computer Vision to create match-cut tools. This technology provides the input necessary for a computer to “watch” a film. Then Deep Learning begins to understand patterns in those films. VFX tech like rotoscoping and relighting are already being simplified with Computer Vision.
Notice that “generative” is not on that list. This omission is because generative AI combines these different algorithms. The “pre-trained transformer” part of ChatGPT refers to its ability to use statistics to establish relationships between words in a sentence (or a larger body of script) based on an enormous pile of texts written by people. It is trying to copy the way people write by studying their writing. You can see how Netflix uses Machine Learning and Computer Vision in their video about how it is transforming the entire industry from four years ago.
The controversy around “generative” AI in the DGA contract
The specificity of “generative” AI has caused concern among those who feel it is too specific because there are various kinds of AI. Others argue that it is sufficient. The argument is that if AMPTP is using such precise language for “generative AI,” that may open the door to other AI derivative technology that goes by a different name. Others have argued that there are provisions in the contract to prevent that kind of loophole from being exploited. Ultimately, the lawyers will have to battle that out.
In the meantime, generative AI keeps improving. The internet is full of examples of images created by AI. But what about the aspects of lighting, historical era, costumes, focal lengths, capture medium and lens choices? Here are three examples of prompts of DALL-E and Stable Diffusion that incorporated that kind of language:
Two men playing chess photo at golden hour in New York City with a bounce fill used on a 19mm leica lens at maximum aperture shot on Kodak Kodachrome film.
A portrait of a woman on a beach with a bounce fill used shot on a Leica 50mm lens at f.95
Two women martial arts fighters shot with a telephoto lens from the 1990s.
“Without consultation”
The examples above feel simultaneously laughable and threatening. If AI can reference the characteristics of specific lenses and lighting techniques to generate images, you better believe it will be deployed in all corners of post-production. This is why the agreement stipulates that AI can’t be used “without consultation” with the director. Of course, the concern is that this consultation may not be in good faith. Will directors simply be “notified” (rather than genuinely consulted) when AI is brought in to change or generate imagery? The role of video collaboration/review and approval tools will become exponentially more significant to enable directors to navigate these waters.
Impact on assistant directors and UPMs
The DGA agreement focuses on AI “in connection with creative elements.” This qualifier raised concerns regarding the “non-creative” aspects of the director’s department. Script breakdowns, scheduling and more are a part of the responsibilities of assistant directors and unit production managers. The concern was that much of the work of these “below-the-line” positions may be automated by AI.
Impact on other union negotiations
Forbes points out that it may be easier for the DGA to get concessions regarding the use of AI for their jobs as opposed to other unions. SAG-AFTRA and the WGA will be watching closely because they may feel they are more vulnerable to having their work replaced by AI than directors. The WGA desires assurances that they won’t be replaced by a chat interface. SAG-AFTRA has the concerns of actors, stunt coordinators and background actors to represent. The members of SAG-AFTRA have already authorized a strike if necessary.
Conclusion
The impact of AI will be felt deeply across the entire industry. It will change both the size and shape of the industry as a whole. New roles will be created, and current ones will have their responsibilities reshaped. At the same time, it is imperative that we recognize that it is the human resonance in a work of art that makes it unique. It’s the capturing of creative energy that makes it valuable, and a machine can’t replace that.
MediaSilo allows for easy management of your media files, seamless collaboration for critical feedback, and out-of-the-box synchronization with your timeline for efficient changes. See how MediaSilo is powering modern post-production workflows with a 14-day free trial.
