Roblox researchers released OpenGameEval in October, a benchmark that tests AI coding agents on real game-development work inside Roblox Studio. According to the paper, posted to arXiv, the best of 13 frontier models solved 51.7% of tasks on its first attempt.
The researchers ran each model as an agent on 84 human-curated tasks, with 16 attempts per task. Every run happens in a reproducible, stateful Studio session and is scored with executable checks, both on the edited scene and in a simulated play session.
That means an agent doesn't pass because its code looks plausible. It passes because the game actually behaves correctly.
The top model was Claude Fable 5. Its 51.7% single-attempt score fell to 39.4% when it had to solve a task 5 times out of 5. Six tasks were not solved by any model tested.
The leaders were close overall: the top 5 finished within about a point of each other, but they solved different tasks to get there. Split by the kind of work involved, the top 5 spread out by 5.0 points on script-writing tasks and 12.5 points on scene-change tasks.
The most useful finding for agent builders concerns exploration. OpenGameEval splits its 8-tool action space into "observation" tools and "editing" tools, which let the authors measure how much an agent looked around before it acted.
With task and model held fixed, runs that inspected every object the reference solution touches passed 13.4 points more often on scene-only tasks than runs that inspected none of them. On script-only tasks, the gap was 9.8 points.
Why it matters: game engines are stateful in a way most coding benchmarks don't capture. Before an agent changes a scene, it has to know what's already in it. The data supports a practical rule for studios wiring agents into their editors: give the agent ways to inspect the scene, and make sure it uses them before it edits.
Roblox has released the task suite, its place files, per-task annotations, a Studio plugin for running the tasks and an updated leaderboard on GitHub under the MIT license.
Caveat: the results are the authors' own. The live GitHub leaderboard now uses an expanded 87-task set, so its figures differ slightly from the paper's 84-task results.
Next, a shorter version of the paper is due at a NeurIPS 2026 workshop on evaluating interactive agents.
