AI Studio

This framework is a byproduct of a case study exploring a step-by-step integration of OpenAI's GPT-3.5 model into Unity 3D for the purpose of enhancing non-playing characters (NPCs) in video games and interactive applications. By leveraging GPT-3.5's natural language processing capabilities, we aimed to create NPCs capable of dynamic real-time interactions with players and each other! Our methodology involved integrating GPT-3.5, implementing speech-driven dialogues, and incorporating humanoid avatars with lip-syncing and emotive animations.

To run the sample scenes in AI Studio, several third-party licenses and packages must be acquired and installed by the user. These packages are not included in the AI Studio package and until installed the sample scenes will contain objects with missing component scripts. Once the required packages are installed, these missing component script errors will be resolved. Instructions for acquiring and installing these packages are included in the README and are also documented in a set of YouTube videos.

To run the “Simple” sample scenes, the following dependencies must be satisfied:

• An OpenAI API key.
• Integration for the OpenAI API in Unity. This is an open-source Unity package hosted in GitHub governed by the MIT license.
• A subscription key for the Microsoft Azure Speech Service.
• The Microsoft Cognitive Services Speech SDK for Unity package.

To run the “City” sample scenes except for “Avatar Wars”, these additional packages must be installed:

• The Unity Cinemachine package from the Unity Registry.
• The UMA 2 Unity Multipurpose Avatar package from the Unity Asset Store.
• The CITY package from the Unity Asset Store.
• The SALSA Lip Sync Suite from Unity Asset Store.
• The SALSA Lip Sync Suite OneClick UMA DCS package from the Crazy Minnow Studio website.
• A package of several Mixamo animations.

To run the “City” sample scene “Avatar Wars”, this additional dependency must be satisfied:

• The Ready Player Me Unity SDK package.