AtelierBench finds no AI model reaches the professional design bar

What the benchmark found

Hyde, an enterprise AI start-up, and Silverside, an AI creative lab founded by advertising executive PJ Pereira, published AtelierBench on 20 October 2026, a benchmark that asks whether AI can produce design work a professional would actually send to a client. Their answer, in a research post on Hyde's blog, is no: none of the eight models and three harness setups they tested reached the professional bar.

The weakest area was precision. Text and logos scored worst of ten quality dimensions, and models also struggled to keep reference images unchanged. A misspelled tagline or a distorted logo makes an asset unusable however good the rest of it is, which is why the authors treat those failures as decisive.

The other headline finding is about harnesses, the scaffolding of tools and limits a model works inside. Giving weaker models tools to check their work helped a lot, but the gains flattened quickly while costs kept rising, and for the strongest models a heavier harness sometimes made the output worse. Results like these echo other recent agent benchmarks, such as Hillclimb's 48-hour research benchmark, where the same frontier models were compared on long, open-ended work.

How AtelierBench tests a model

AtelierBench differs from most image benchmarks in three ways. First, its tasks are built from real client briefs from Silverside's work, ranging from social posts to point-of-sale assets and high-resolution outdoor ads. The authors say there are hundreds of tasks, and that they are private and not in any public dataset, to avoid contamination.

Second, scoring follows how a designer judges work. Each brief is broken into specific claims about what the output must include, avoid or match, and each claim is tagged to a category such as required scene content, text and logo readability, or delivery format. Silverside designers verified every claim and weighted its importance, and scoring combined language-model judges with human designers.

Third, the model does not draw the image itself. Like a designer using node-based tools, it has to build a workflow in ComfyUI: plan the steps, load reference images, choose which image-generation models to call, run the pipeline and fix its own errors. That workflow can then be re-run and adjusted later.

The study tested eight models in three price tiers: premium (GPT-6 Astra and Fable 5.1), standard (Claude Opus 5, GPT-5.6 Sol and Gemini 3.8 Flash) and open-weights (Kimi K3, Qwen3.8 Max and DeepSeek V4.1 Flash). Each model attempted every task twice on each of three harnesses. On Base it writes the workflow in one go with only the brief and a catalogue of available models. On Extended it can use ComfyUI's command-line tools to look things up and validate the workflow before running it. Extended Plus is the same with double the tool-call limit; the post's notes cite runs of 15 and 30 tool calls.

Most models cannot run a design workflow first time

On the Base harness only Astra, Fable and Opus produced an output more than 90 percent of the time. The other five models failed to execute more than half the tasks; their execution rates ranged from 3 to 41 percent.

Tools changed that. On the Extended harness the same five models jumped to between 59 and 82 percent, which the authors read as evidence that most failures were workflow-construction errors the models could detect and fix once they were allowed to check. Doubling the tool budget on Extended Plus added at most six percentage points, so a fairly simple harness captured most of the benefit.

Quality: nobody reaches the bar

Execution is only the first hurdle. The authors calibrated a professional bar with Silverside's designers at a quality score above 4.5, the point where a designer's verdict switches from “redo” to “ship to client”. No model reached it on any harness.

To compare models fairly, the authors also looked at like-for-like tasks, those that ran under all three harnesses. On that measure, moving from Base to Extended Plus left Astra at 4.1 and Opus at 3.8, while Fable dropped from 4.1 to 3.8. Weaker models gained: DeepSeek rose from 2.5 to 3.4 and Sol from 3.4 to 3.9, while Kimi barely moved. Looking only at tasks each model completed, Sol gained 0.47 points, Gemini Flash 1.05 and DeepSeek 1.1.

Across all models, text and logos averaged only 2.7 on Base and 3.0 on Extended Plus, the lowest of the ten dimensions. Keeping references unchanged was also weak and improved by just 0.2 points with a better harness. Models did well at following explicit rules, such as excluding certain content and meeting delivery specs. Text rendering is a known weak spot that open models such as OpenMOSS's MOSS-Image have been built specifically to address.

When more help makes models worse

The most counter-intuitive result concerns the top three models, which completed almost every task on every harness. Astra held steady, but moving from Base to Extended, Fable's delivery-format score fell from 4.6 to 3.5 and its text-and-logos score from 3.4 to 2.7. Opus fell from 4.2 to 3.8 and from 3.2 to 2.7 on the same two dimensions.

The authors offer two early hypotheses. The harness's tools check that a workflow runs, not that the output matches the brief, so a model can fix the workflow at the brief's expense. And a longer loop gives more chances to lose track of exact requirements such as image size and wording. Neither has been tested yet.

What it costs

On Base, premium models cost about three times as much per attempt as standard models and seven times as much as open-weights ones. But a failed attempt still burns tokens, so once failures are counted the gap largely disappears. Kimi costs about a third as much as Opus per attempt but slightly more per successful output, US$0.41 against US$0.38, and at lower quality. In a single shot, the authors say Opus offered the best balance of cost and quality, and only DeepSeek stayed genuinely cheap.

Better harnesses raise the cost of every task, by about 2.2 times on Extended and 3 times on Extended Plus. Because far more attempts then succeed, cheaper models become the best value per successful output: Sol on Extended matched the quality of Opus on Base for about 15 percent less per success. Going to Extended Plus added about 40 percent to total spend for a small gain, and Astra on Extended Plus scored about the same as on Base while costing 55 percent more per success.

Across every combination, only DeepSeek, Sol, Opus and Astra were ever the best value at their price, and Fable never was. The cheaper models still failed about one task in four, so the authors say a top model on the simple harness remains the safer choice when an attempt has to work first time.

Caveats

AtelierBench is a vendor study. The task set is private, so outsiders cannot rerun it, and the example task shown in the post is illustrative rather than part of the benchmark. Only one execution environment, ComfyUI, was tested, and the quality bar was calibrated by the same designers who wrote the claims. Taste-based scoring is hard to standardise, a problem other judgment benchmarks such as TasteVal and BrickBench also wrestle with.

Both companies also have a commercial interest in the conclusion. Hyde sells private evaluations, fine-tuning and self-hosted models, and the post ends by arguing that enterprises need private evals and post-training on their own design decisions. Silverside sells AI creative production to brands. The numbers are still useful, but they come from a benchmark whose authors also sell the proposed fix.

The authors say they are working with design-led companies on next steps, including letting models check their own output and retry, putting a designer in the loop, and extending the benchmark to other media.