StarSkirmish Bench

The StarSkirmish Bench evaluates how strong of a StarCraft bot can each LLM write given 1 hour of wall clock time.

An LLM’s score reflects its win rate against a roster of competitive human written bots and demo bots.

Scores are scaled so that Stardust, the top human written bot, scores 100 and Four Gate Dragoon, the weakest demo bot, scores 0.

GPT-6 Astra and Claude Opus 5.5 are functionally tied as the top 2 scoring StarSkirmish Bench LLMs. Along with GPT-6 Sol, these three LLMs are a clear step above every other tested LLM at writing StarCraft strategy code in C++.

GPT-6 Sol offers uniquely good value in its StarSkirmish Bench performance.

Model Breakdown

Objective

Each LLM writes a Protoss bot in C++ against BWAPI 4.4.0, played on OpenBW. Games are Protoss versus Protoss on three ladder maps: Heartbreak Ridge, Benzene and Destination. A game ends when one side's buildings are all destroyed. Games that reach the 60 minute cap (86,400 frames) are decided as on the BASIL ladder: the higher kills + razings score wins.

The LLM has three tools: one to compile, one to play a batch of practice games against a tier of opponents, and one to read a game transcript with build timings, fight summaries and economy recaps. Practice opponents are grouped into tiers: D (the three demo bots), C and B (three mid-ranked human written bots each), A (BananaBrain, Locutus) and S (Stardust).

Tournament Setup

Every entrant plays every other entrant 6 times, twice on each map. The field is 50 LLM bots (10 models x 5 runs), the 3 demo bots and 9 competitive human written bots, for 62 entrants. Ratings are Elo ratings fitted from all games at once. Games ran on Prime Intellect sandboxes.

We report each model's average rating over its five bots, not just its top ranked bot.

Harness Setup

Every model ran inside the same harness: Inspect's ReAct deepagent, with bash, a text editor, a memories tool and research subagents. Models were called through OpenRouter with high reasoning effort. Our OpenRouter settings required first party inference only, so that open weights models were guaranteed to get their full quality. The model is told to keep going until the hour is up. There is no submit tool, the harness automatically picks up the bot code when the time runs out.

Rating Calculation

Ratings are fitted from all tournament games at once. A model's Bench score comes from the average rating R of its five bots in two steps.

Benchmark Correlations

StarSkirmish isn't a novelty game bench, it captures meaningful long term reasoning and agentic coding abilities. It correlates strongly with 4 public coding benchmarks, and most strongly with Vals IOI, olympiad programming problems solved in C++, and WeirdML, an independent benchmark of unusual machine learning coding tasks thought to be less benchmaxed than most.

Future Work

SOTA reasoning models now benefit significantly from reasoning periods longer than 1 hour. Because of this, we plan on launching versions of StarSkirmish that can highlight model capabilities with substantially longer reasoning periods.