BrickBench asks coding agents to design buildable LEGO sets

What is BrickBench asking agents to do?

BrickBench gives a text prompt and requires a LEGO assembly that is semantically faithful, well designed, and physically buildable from real parts in the LDraw library. The three settings change the constraint: Model caps parts at 400, Set targets retail-set scale (400–4000 parts), and Alt-Build freezes the inventory to the 783 pieces of set 10698. Prompts span ten theme categories aligned with BrickLink Designer Program themes.

BrickAgent is the accompanying environment: part search, connector-based placement, subassemblies, rendering, and a validator that names faulty parts. The authors contrast this with prior LEGO generators that used specialized models on small objects. For how executable checks differ from vibe scoring, see evaluating AI agents and SWE-bench explained.

The HTML paper restates the construction loop in LDraw terms: agents must pick discrete parts, reason about stud connectivity, and keep the whole assembly physically coherent, not just a mesh that looks right in a render. Prompts are organized around ten BrickLink Designer Program–aligned themes so the same agent is judged across vehicles, creatures, interiors, and other set genres rather than a single toy category. Alt-Build is the inventory stress test: the agent cannot invent missing colors or rare pieces outside set 10698’s 783-part bag.

How do they score validity, alignment, and design?

Physical validity covers part limits, collisions, and stability under a rigid-body gravity simulation in PyBullet; the paper notes that stud holding force and axle play are not modeled. Semantic alignment uses VQA-style yes/no questions derived from each prompt. Design quality uses pairwise judgments turned into ELO ratings, with a human study used to validate the automatic design metric.

The project site restates the headline tension: agents can satisfy verifiable constraints while still losing a “LEGO Turing test” against human-designed models of similar part count. That framing is the authors’ and depends on their rater pool and comparison protocol.

The authors also evaluate eleven frontier agents plus two data-driven baselines. Their site’s overall table mixes Valid, VQA, Align ELO, Design ELO, and token list-price cost. One reported pattern is that GPT-6.1 Sol can approach Astra’s ELO band at a lower listed cost—an author plot, not an independent pricing study. Use those columns to compare systems inside BrickBench, not to crown a general coding model.

Stability checks treat connected components as rigid bodies of uniform density in PyBullet with high friction and no restitution, so unsupported pieces fail by falling rather than bouncing. The authors are explicit that stud clutch force and axle play are omitted—so “valid” here means their simulator’s connectivity/collision/gravity stack, not a claim that every assembly would survive a child’s table shake. Semantic alignment is built from yes/no visual questions derived from the prompt graph; a design that is physically solid can still fail alignment if the pelican has no bicycle or the cowboy has no lasso.

What did the authors find about validity versus design?

In the paper’s environment ablation, GPT-6 Astra remains valid on every prompt with BrickAgent and on 40% without it; GPT-5.6 Luna stays valid with BrickAgent and falls below 1% without it. Alignment and design scores do not collapse the same way—without buildability constraints, some agents produce assemblies that look better but cannot be built. Across eleven frontier agents, the authors say five deliver a valid assembly for every prompt, while Design ELO separates systems more sharply than VQA.

Human raters, in the authors’ study, identified human-designed assemblies in 323 of 360 comparisons and did not take any agent for human more than one time in five. Those are paper-reported figures. Cost columns on the site show list-price token spend per assembly; rank does not simply follow spend in the authors’ plots.

The project site’s agent table is useful as an apples-to-apples BrickBench leaderboard, not as a general coding ranking. Token list-price columns show that spend and Design ELO do not move in lockstep in the authors’ plots, which is the practical takeaway for teams budgeting agent evals: paying more for a frontier model does not automatically buy human-preferred part efficiency or silhouette. When you cite the five agents that hit 100% validity, keep the companion fact that the human preference study still separates those systems on craft.

What should builders take from BrickAgent?

If you evaluate coding agents on constructive tasks, an environment that returns named physical faults is doing different work than a final screenshot judge. BrickBench’s ablation is a reminder that “the model can write LDraw” is not the same claim as “the model can iterate against a validator.” Conversely, saturating validity does not mean the aesthetic or part-efficiency bar is cleared.

The public surfaces we opened are the arXiv abstract/PDF, https://brickben.ch, and https://github.com/BrickBench/BrickBench. Licensing of the LEGO brand and of any third-party part libraries is outside this summary; use the repositories’ own terms before shipping demos.

Practically, if you already have connector-level part metadata—as BrickNet-style annotations provide—an agent loop that searches, places, validates, and revises is closer to how software agents win on SWE-bench than a single-shot mesh dump. BrickBench’s contribution is making that loop scoreable at set scale, with design quality kept as a first-class axis instead of a footnote.

If you fork the environment, start by logging validator fault names—collision, unsupported component, illegal part—because those strings are the training signal that made Astra and Luna’s ablations diverge. A second practical habit is to keep design ELO or human pairwise preference as a separate gate after validity saturates; otherwise your leaderboard will celebrate buildable blobs. The Stanford / MPI / partners author list and the brickben.ch release are the citation objects; LEGO trademarks and third-party LDraw part libraries remain outside AiLookout’s reuse claims.

What did we not test?

We did not install BrickAgent, generate assemblies, run PyBullet validation, or repeat the human preference study. Model names and scores are quoted from the paper and project site as of our 10 October check.