MATCHING MOUTH SHAPE AND MOVEMENT IN DIGITAL VIDEO TO ALTERNATIVE AUDIO

Register WIPO Patent
Application Number US2019033862
Publication Number 2019/226964
Status In Force
Filing Date 2019-05-23
Publication Date 2019-11-28
Owner WARNER BROS. ENTERTAINMENT INC. (USA)
Inventor
  • Stratton, Thomas
  • Lile, Shaun

Abstract

A method for matching mouth shape and movement in digital video to alternative audio includes deriving a sequence of facial poses including mouth shapes for an actor from a source digital video. Each pose in the sequence of facial poses corresponds to a middle position of each audio sample. The method further includes generating an animated face mesh based on the sequence of facial poses and the source digital video, transferring tracked expressions from the animated face mesh or the target video to the source video, and generating a rough output video that includes transfers of the tracked expressions. The method further includes generating a finished video at least in part by refining the rough video using a parametric autoencoder trained on mouth shapes in the animated face mesh or the target video. One or more computers may perform the operations of the method.

IPC Classes  ?

  • G06T 13/40 - 3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
  • G06K 9/46 - Extraction of features or characteristics of the image