Microsoft has unveiled Muse, a groundbreaking generative AI model designed to revolutionize video game creation. This innovative tool, the first World and Human Action Model (WHAM), can generate both game visuals and controller actions, marking a significant leap forward in gameplay ideation. Muse was developed by Microsoft Research Game Intelligence and Teachable AI Experiences teams in collaboration with Xbox Games Studios’ Ninja Theory. The model was trained using data collected from Xbox gamers, potentially offering game studios a way to cut costs by automating the creation of game scenes that would normally require human programming and animation. Microsoft has open-sourced the weights and sample data, making the executable available for the WHAM Demonstrator on Azure AI Foundry, allowing other researchers to explore and build upon their work.

The development of Muse was driven by insights gathered from game developers who were looking for AI tools that could generate gameplay sequences adhering to game rules and physics, while also allowing for adjustments throughout the development process. To train Muse, the research team used seven years’ worth of gameplay data from “Bleeding Edge,” a multiplayer battle game from Xbox’s Ninja Theory studio, extracting video frames and controller inputs from 500,000 anonymized game sessions. This extensive dataset, involving up to 1.6 billion parameters, enabled the AI to develop a deep understanding of the game environment, its dynamics, and how it evolves in response to actions.

Muse’s capabilities extend beyond just generating visuals; it can also understand and model 3D worlds, enabling rapid iteration, remixing, and creation within video games. This allows developers to create immersive environments and fully unleash their creativity. According to Microsoft, Muse can develop a practical understanding of interactions in the world simply by observing human gameplay. This understanding allows the model to generate coherent and immersive video-game worlds that respond to player inputs. The integration of AI in video game development addresses the rising costs and the need for fresh ideas in complex worlds, where previously, generative AI tools had limited ability.

The potential applications of Muse extend beyond gaming, with possibilities for AI assistants that can visualize and help with tasks such as reconfiguring a kitchen, redesigning a retail space, or building a digital twin of a factory floor to test different scenarios. Microsoft is also reportedly developing another in-house AI language model, MAI-1, to compete with models from Google and OpenAI. Overseen by Mustafa Suleyman, MAI-1 will have approximately 500 billion parameters. While the precise purpose of MAI-1 remains undetermined, its development signifies Microsoft’s broader commitment to advancing AI capabilities across various sectors.