Proposal and Implementation of Conversation AI System for Avatars in Virtual Reality

Abstract

This article presents the design and implementation of a conversational AI system for avatars in virtual reality. The system enables spoken interaction with MetaHuman avatars in Slovak and English and is implemented in Unreal Engine 5. The work connects speech recognition, large language model response generation, speech synthesis, audio playback, avatar state feedback, and lip-sync animation. Existing local and cloud-based models are integrated into a modular STT-LLM-TTS pipeline so that different processing configurations can be compared. The evaluation focuses on response latency, hardware load, and the selection of a suitable configuration for VR use. Technical testing showed that the lowest average latency was achieved by the configuration using cloud STT, local LLM, and local TTS with an average response time of 1.94 s. For user testing, cloud STT, cloud LLM, and local TTS were selected as a better compromise between response quality and latency. The results show that LLM-based con-versational AI can be integrated with VR avatars in Unreal Engine while preserving flexibility for fur-ther model and performance evaluation.