YouTube is developing an artificial intelligence feature to generate lip-syncing for its auto-dubbed videos. The technology aims to enhance realism by modifying a speaker’s mouth movements to align with translated audio tracks, intended to increase viewer engagement.
According to Digital Trends, the system’s technical foundation, as detailed by YouTube’s product lead for Auto-dubbing, Buddhika Kottahachchi, relies on a custom-built AI. Kottahachchi explained that the technology executes intricate, pixel-level changes to a speaker’s on-screen mouth to create synchronization with dubbed audio. The AI model incorporates a three-dimensional perception of facial structures, allowing it to analyze the geometry of the lips and teeth. It is also designed to interpret and replicate facial expressions that accompany speech. This 3D modeling approach allows the system to more accurately simulate the physical movements required for speaking in a different language.
In its initial phase, the lip-sync feature will have specific technical and linguistic limitations. The AI processing is currently restricted to videos with a 1080p resolution and cannot be applied to 4K content. Language support at launch will be confined to English, French, German, Portuguese, and Spanish. Following this introductory period, YouTube plans to expand support to more than 20 languages. This expansion is designed to bring the lip-sync feature into alignment with the full range of languages currently offered by YouTube’s auto-dubbing service.
YouTube has not announced a firm release date for the feature. The company is expected to first introduce the technology through a pilot program with a small group of creators, a strategy that mirrors the rollout of the auto-dubbing function. That auto-dubbing service was expanded to a wider audience just last month, indicating the lip-sync addition may undergo a prolonged testing period. Creators will be provided with controls to manage its use, including the reported option to disable the feature for their entire channel or for individual videos, giving them final say over their content’s presentation.
The feature may come at an additional cost, though a specific price has not been finalized. It is undetermined if the creator or consumer will bear the fee, but reports suggest it will likely be the consumer. To address potential misuse, YouTube plans to implement safeguards. These include a descriptive disclosure to inform viewers of the AI alteration and an invisible, persistent fingerprint embedded into the video. This digital watermark is described as being similar in function to SynthID, a tool used to identify AI-generated content, providing a mechanism for tracking and authentication.
YouTube is not the only platform developing this technology. Meta has a comparable initiative for its Instagram platform, where it launched a pilot program last year to dub and lip-sync Reels. While details on the program’s success are limited, it was recently expanded to support four languages: English, Hindi, Portuguese, and Spanish.
