Description
Why is this important?
Large Language Models (LLMs) are redefining the possibilities for interactive game experiences, especially in how players engage with Non-Playable Characters (NPCs). Traditional NPCs rely on pre-scripted dialogue trees and behavior systems, which limit immersion and adaptability. By leveraging LLMs locally, we can move beyond static interactions and create dynamic, real-time NPC behaviors that respond intelligently to players. This not only enhances immersion but also demonstrates the feasibility of running advanced AI directly on-device without reliance on cloud infrastructure. Such innovation could shape the future of gaming, particularly in mobile platforms where performance and autonomy are critical.
Project Summary
This project challenges you to propose and implement novel methods of using LLMs to control or interact with NPCs during runtime. Beyond dialogue systems, the focus is on new, creative mechanisms that showcase the potential of LLM-driven NPCs in real-time gameplay. Your implementation should demonstrate a working prototype running locally at a stable 30–60 FPS, ensuring both responsiveness and playability. While achieving mobile deployment is the ultimate goal, a PC/laptop demo will also be accepted as proof of concept. The final submission should include source code, reproducible build instructions, supporting documentation, and a short demo video.
To qualify, your submission should include, where possible:
- Source code (with clear documentation and build instructions)
- A reproducible setup (e.g. scripts, datasets, or dependencies)
- A supporting document describing the project and design decisions
- High-quality images and a video (≤ 3 minutes) demonstrating the demo in action
Please ensure your contribution contains no confidential material and that you have rights to share it, especially if affiliated with an academic or commercial institution.
Prequisites
- Familiarity with a modern game engine (e.g., Unity, Unreal Engine, Godot)
- Experience with integrating machine learning models into real-time applications
- Knowledge of C++, Python, or a game scripting language
- Understanding of on-device ML optimization (e.g., quantization, pruning, mobile deployment)
- Access to hardware capable of running LLM inference locally (PC or mobile)
Resources from Arm and our partners
- Blog: The Future of Interaction with Mobile Game Characters
- Blog: The Future is Here: Verbal Interaction with NPCs on Mobile
- Blog: Google AI for game developers- Google Developers Blog
- Webinar: Generative AI in Game Development
- Paper: A Survey on Large Language Model-Based Game Agents
- Paper: Large Language Models and Games: A Survey and Roadmap
Support Level
This project is designed to be self-serve but comes with opportunity of some community support from Arm Ambassadors, who are part of the Arm Developer program. If you are not already part of our program, click here to join.
Benefits
Standout project contributions will result in preferential internal referrals to Arm Talent Acquisition (with digital badges for CV building). And we are currently discussing with national agencies the potential for funding streams for Arm Developer Labs projects, which would flow to you, not us.
To receive the benefits, you must show us your project through our online form. Please do not include any confidential information in your contribution. Additionally if you are affiliated with an academic institution, please ensure you have the right to share your material.