AI-Based Deep Fake Avatars Using Gaussian Splatting

Abstract

This paper investigates the technical feasibility of creating an interactive conversational avatar using Gaussian Splatting technology in the Unreal Engine environment. The study focuses on verifying the possibility of implementing a talking avatar without using parametric face models. The result of the study is a prototype consisting of a local FastAPI backend and a client application in Unreal Engine 5.3.2, which integrates speech recognition, response generation by a local large language model, speech synthesis and data generation for lip synchronization. The avatar is rendered using the 3D Gaussian Splatting plugin from Luma AI. The paper includes the implementation of the application in the VR environment of Unreal Engine. The implementation section of the study also introduces comparative testing of four large language models and latency measurement of the entire conversational data pipeline. The study confirms the technical feasibility of the combination of Gaussian Splatting and a conversational AI application only partially, as the main open problem remains the implementation of the avatar's lip animation.