Respeecher’s CEO on working on Star Wars, Book of Boba Fett Luke, & more

If you’ve ever watched a Star Wars show on Disney Plus, there’s a good chance you’re familiar with the work of tech startup Respeecher, whether you realize it or not. The Ukrainian company’s AI-powered voice cloning platform provided Mark Hamill’s de-aged vocal performance in both Mandalorian And Boba Fett: The Book of Boba FettBoth for a yet-unidentified person in Obi Wan Kenobi. Lucasfilm has asked Respeecher to keep the name of that character a secret for now — and with so many franchise veterans returning for the series, there’s certainly no shortage of potential candidates.

Polygon spoke with Respeecher CEO Alex Serdiuk to better understand a process that to many fans no doubt borders on sacrilege: using technology to create entirely bespoke performances for one (or possibly even two) of the Star Wars saga’s most iconic characters. Serdiuk insists on the humanity of the platform from the beginning. “With our technology and our services, we can create a digital copy of a particular voice and enable another person to speak in this voice,” he explains. “And therefore we enable [studios] to scale voices, to age voices, and even resurrect voices for some projects.”

So, far from the mental image conjured up by terms like “artificial intelligence” and “voice cloning” — that of a sound engineer running lines of dialogue through a computer algorithm that then spits out audio files. Respeecher’s work on Star Wars is surprisingly performance driven. While Darth Vader himself may be more machine now than man, if the Ukrainian startup is supplying his voice (and remember, we said “if”) the essence of the character’s voice is still very much flesh and blood.

Lucasfilm

“There is no AI yet, and I don’t believe it would exist, that would let us use it just on a turnkey basis to create the performance we wish to create. […]Human voices are needed elsewhere [to provide] input because that human voice gives all the inflections, the accent, the style of speech and pace that AI is not good at creating,” Serdiuk insists. He adds: “Our system requires a performance on input, so it can be made by the same person that is being de-aged for example, or someone else. […] It takes all the performance, all the acting from what we call a ‘source voice,’ and then we do the conversion.”

What’s more, Respeecher’s pipeline allows actors like Hamill to record different takes just like they would on an actual set, which the company’s experts can later adjust at their end based on notes from showrunners like Jon Favreau or directors like Deborah Chow.

“With studio projects and films they might record thousands of takes for each line, and that means that we would need to convert all those [into the younger voice], send them back, and maybe send different versions because we used to train different models with different setups,” he says. “And sometimes we also need to meet creative expectations so they can just direct us, Do you want to learn how? [a line reading]Sound a bit like that? and we would work to make it sound a bit more like they ask.”

So, is the ultimate aim to recreate a traditional performance through nontraditional means — in the case of Mandalorian Oder Boba Fett: The Book of Boba Fett, as though Hamill’s lines were somehow transmitted directly from the set of 1983’s Star Wars: Return of the Jedi? “The goal is to make it sound like it was recorded yesterday in the studio by the target voice themselves,” Serdiuk agrees.

Of course, there’s still the risk with voice cloning technology like Respeecher’s that the synthesized performance it produces will sound artificial, even if viewers aren’t entirely sure why. Serdiuk even admits that Hamill’s de-aged voice sounds “way, way better” in Boba Fett: The Book of Boba Fett It did so in a better way than it did back then Mandalorian, thanks to small yet significant improvements in how Respeecher’s AI model was “trained” to emulate the actor’s vocals. The CEO also pointed out that many of his fans had complained about the use of visual effects to depict Luke Skywalker, 20. Mandalorian finale, few realized the Jedi Master’s voice was also synthetic until Lucasfilm spilled the beans in a Disney Gallery “making of” special several months later.

Serdiuk mentions that Hamill’s convincing de-aged vocals in Mandalorian And Boba Fett: The Book of Boba Fett They were even more impressive when you consider the outstanding legacy assets Respeecher received form Lucasfilm. “That [data] was quite old, so we had something from tapes, we have some old ADR recording, something from a video game,” Serdiuk recalls. “And the thing is you need to get this data trained in your model to enable it to produce the output quality that would fit into a modern production. There are many projects that involve either aging or resurrecting. [performer’s voices], this might be the main challenge because lack of data and quality of data introduce additional blockers in terms of making it sound good.”

Respeecher CEO says that the data-related challenges have been overcome and that Lucasfilm, an industry titan, has accepted his work. “[We]The idea to build a synthetic voice was the first thing that came up. [platform]at a level that would pass Hollywood studios’ sound engineers. It could also land in large productions. So, when they accept our sound, when they say something good about the sound we were able to produce — and that’s a very complicated and heavy technical challenge, to make synthesized speech on the level where it would be indistinguishable from a real recording — in such cases, it really encourages us and helps us grow,” he says.

Serdiuk suggested that due to the acceptance of voice-cloning technology, studios might not need talented soundalikes for actors who have died. For example, wouldn’t Star Wars Story: Rogue One — in which Guy Henry imitated the voice of the late Peter Cushing as Grand Moff Tarkin — be the kind of project that would automatically land on Respeecher’s desk now? Not necessarily, according to Serdiuk, who sees Respeecher’s voice cloning technology as one of several viable options at filmmakers’ disposal.

Image: Lucasfilm/Disney

“There are always different visions of how things should happen in the industry and the fans have different thoughts about how they should happen. I wouldn’t say that [Respeecher]is well-suited to serve as a judge or in charge. [which approach is best],” he says. Serdiuk also made it clear that should Lucasfilm ever call upon Respeecher to re-create the voice of a dead actor, the company would only do so with the approval of that actor’s estate. And while Serdiuk’s response will set voice actors at ease for now, the fact that Lucasfilm is a repeat Respeecher customer suggests the startup has made serious in-roads for voice cloning technology.

Indeed, Serdiuk already has a vision for Respeecher’s future that extends beyond de-aging actors’ voices, although he remains adamant that what the company has planned will expand filmmakers’ creative horizons, not shrink them. The technology should be accessible to smaller videogame developers and studios for film and television. He also speaks enthusiastically about the groundbreaking health care applications of Respeecher’s platform — even citing one instance where the company is collaborating with a voice actor who has lost their voice to allow them to perform again.

Yet looking to the future doesn’t mean Serdiuk has lost sight of what it means for Respeecher to be part of a certain space opera set a long time ago in a galaxy far, far away. “It’s something special. I mean, you’re part of this story. Star Wars is a movie that has made a significant impact on Hollywood. They’ve been disrupting the industry, from a technical standpoint, from the very beginning, and the way they do their movies is exceptional. So, it’s a big honor to be able to work with those people and learn from them.”