Is that Tom Hanks Speaking in Japanese? No, it’s Just AI

Tom Hanks. (AP)
Tom Hanks. (AP)
TT
20

Is that Tom Hanks Speaking in Japanese? No, it’s Just AI

Tom Hanks. (AP)
Tom Hanks. (AP)

Bad lip-syncing in dubbing and subtitles can put off audiences and hurt box office takings of foreign films.

AI may be about to change all that.

Start-up Flawless AI, co-founded by film director Scott Mann, has a tool that it says can accurately recreate lip sync in dubbing without altering the performance of the actors.

The tool studies how actors move their mouths and swaps the movements out according to the dubbed words in different languages, making it seem like Tom Hanks can speak Japanese or Jack Nicholson is fluent in French.

Mann was inspired to come up with the tool when he saw how dubbing affected the narrative cohesion of his 2015 film "Heist", starring Robert De Niro.

"I hate dubbing as it stands," he told Reuters. "You have to change so many things to try and catch sync. You're changing words filmmakers and performers have thought about so deeply. They're thrown out to find a different word that fits, but it never really does."

Mann decided to do something about this conspicuous mismatch. After some research, he discovered a white paper by Christian Theobalt of the Max Planck Institute for Informatics, laying out a new approach that uses artificial intelligence to recreate photo-real human visual effects.

The tool worked to translate a De Niro scene in "Heist".

"It's able to essentially take an 'ooh' sound from De Niro 20 minutes earlier and place that in a different moment in the film," explained Mann. "It measures at the same time and blends it so that the performance is the same but it's a different mouth movement."

Flawless AI is now working with producers and studios to integrate the technology into post-production. The first films using this tool are around a year away.

Mann believes this technology will be a game-changer, helping to eradicate foreign-language categories at award shows and creating a more inclusive film industry with a real global stage.

"From a filmmaking point of view, you're going to see the rise of a much more diverse range of stars," he said.

There might be a dissonance and a focus on the AI technology in the beginning when it is introduced, but eventually Mann hopes its use will be seamless.

"When it's a common thing, we're not going to think about it anymore and we'll just enjoy the movies," he said.



Video Game Actors Are Voting on a New Contract. Here’s What It Means for AI in Gaming

A picketer holds a sign for the SAG-AFTRA video game strike at Warner Bros. Games headquarters on Aug. 1, 2024, in Burbank, Calif. (AP)
A picketer holds a sign for the SAG-AFTRA video game strike at Warner Bros. Games headquarters on Aug. 1, 2024, in Burbank, Calif. (AP)
TT
20

Video Game Actors Are Voting on a New Contract. Here’s What It Means for AI in Gaming

A picketer holds a sign for the SAG-AFTRA video game strike at Warner Bros. Games headquarters on Aug. 1, 2024, in Burbank, Calif. (AP)
A picketer holds a sign for the SAG-AFTRA video game strike at Warner Bros. Games headquarters on Aug. 1, 2024, in Burbank, Calif. (AP)

An 11-month strike by video game performers could formally end this week if members ratify a deal that delivers pay raises, control over their likenesses and artificial intelligence protections.

The agreement feels "like diamond amounts of pressure suddenly lifted," said Sarah Elmaleh, a voice actor and chair of the Screen Actors Guild-American Federation of Television and Radio Artists' interactive branch negotiating committee.

Union members have until Wednesday at 5 p.m. Pacific to vote on ratifying the tentative agreement.

Voice and body performers for video games raised concerns that unregulated use of AI could displace them and threaten their artistic autonomy.

"It’s obviously far from resolved," Elmaleh said. "But the idea that that we’re in a zone where we might have concluded this feels like a lightening and a relief."

AI concerns are especially dire in the video game industry, where human performers infuse characters with distinctive movements, shrieks, falls and plot-twisting dialogue.

"I hope and I believe that our members, when they look back on this, will say all of the sacrifices and difficulty we put ourselves through to achieve this agreement will ultimately be worth it because we do have the key elements that we need to feel confident and moving forward in this business," said Duncan Crabtree-Ireland, the SAG-AFTRA national executive director and chief negotiator.

Here’s a look at the contract currently up for vote, and what it means for the future of the video game industry.

How did the current strike play out? Video game performers went on strike last July following nearly two years of failed negotiations with major game studios, as both sides remained split over generative AI regulations.

More than 160 games signed interim agreements accepting AI provisions SAG-AFTRA was seeking, the union said, which allowed some work to continue.

The video game industry is a massive global industry, generating an estimated $187 billion in 2024, according to game market forecaster Newzoo.

"OD," and "Physint" were two games delayed due to the strike during the filming and casting stage, video game developer Hideo Kojima wrote in December. Riot Games, a video game developer, announced that same month that some new skins in "League of Legends" would have to use existing voice-overs, since new content couldn't be recorded by striking actors. Skins are cosmetic items that can change the visual appearance of a player and is sometimes equipped with new voice-overs and unique recorded lines.

The proposed contract "builds on three decades of successful partnership between the interactive entertainment industry and the union" to deliver "historic wage increases" and "industry-leading AI provisions," wrote Audrey Cooling, a spokesperson for the video game producers involved in the deal.

"We look forward to continuing to work with performers to create new and engaging entertainment experiences for billions of players throughout the world," Cooling wrote.

Video game performers had previously gone on strike in October 2016, with a tentative deal reached 11 months later. That strike helped secure a bonus compensation structure for voice actors and performance capture artists. The agreement was ratified with 90% support, with 10% of members voting.

The proposed contract secures an increase in performer compensation of just over 15% upon ratification and an additional 3% increase each year of the three-year contract.

How would AI use change in video games? AI concerns have taken center stage as industries across various sectors attempt to keep up with the fast-evolving technology. It’s a fight that Hollywood writers and actors undertook during the historic film and TV strikes that forced the industry to a stop in 2023.

"In the last few years, it’s become obvious that we are at an inflection point where rules of the road have to be set for AI, and if they aren’t, the consequences are potentially very serious," Crabtree-Ireland said. "I think that really made this negotiation extra important for all of us."

SAG-AFTRA leaders have billed the issues behind the labor dispute — and AI in particular — as an existential crisis for performers. Game voice actors and motion capture artists’ likenesses, they say, could be replicated by AI and used without their consent and without fair compensation.

The proposed contract delineates clear restrictions on when and how video game companies can create digital replicas, which use AI to generate new performances that weren't recorded by an actor.

Employers must obtain written permission from a performer to create a digital replica — consent which must be granted during the performer’s lifetime and is valid after death unless otherwise limited, the contract states. The time spent creating a digital replica will be compensated as the same amount of work time it would have required for a new performance.

The agreement also requires the employer to provide the performer with a usage report that details how the replica was used and calculates the expected compensation.

Elmaleh, who has been voice acting since 2010 and had to turn down projects throughout the strike, said securing these gains required voice actors bring vulnerability and openness to the bargaining table.

"We talked a lot about the personal, the way it affects our displacement as workers and just the sustainability of our careers," Elmaleh said. "Our work involves your inner child. It’s being very vulnerable, it’s being playful."

What’s next for the video game industry? The tentative agreement centers on consent, compensation and transparency, which union leaders say are key elements needed for the industry to keep progressing.

As the contract is considered by union members, Elmaleh and Crabtree-Ireland said further work needs to be done to ensure the provisions are as broad as necessary.

"Even though there’s a deal that’s been made now, and we’ve locked in a lot of really crucial protections and guardrails, the things that we haven’t been able to achieve yet, we’re going to be continuing to fight for them," Crabtree-Ireland said. "Every time these contracts expire is our chance to improve upon them."

Elmaleh said she hopes both the video game companies and performers can soon work collaboratively to develop guidelines on AI as the technology evolves — a process she said should start well the proposed contract would expire in October 2028.

Leading negotiations has felt like a full-time job for Elmaleh, who took on the role in a volunteer capacity. As the efforts die down, she said she anxiously anticipates returning to video game acting in a landscape that is safer for performers.

Voice acting "is core to who I am. It’s why I fought so hard for this. I wouldn’t do this if I didn’t love what I do so much. I think it’s so special and worthy of protection," she said.