Patent-lawyer AI trial finds better drafts for all but skill gains only for seniors

What did the patent-lawyer experiment find?

The study is NBER Working Paper 35720, “Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting”, by MIT economist David Autor with Tanya Rodchenko, Josh Martin, Zanna Iscenko, Scott Strand, David Pearl and Melissa Ferere. Google Research summarised it in a blog post on 7 October 2026. The paper is dated September 2026 and was revised in October.

It asks a question many employers now face: when AI makes people better at their jobs, are they also getting better at the job itself? The trial separated the two by measuring how lawyers performed with an AI assistant and then how they performed on a related task without it.

The answer depended on experience. AI access improved everyone’s drafts. But three months of using the tool only improved the unaided skills of senior lawyers. Junior lawyers, who gained the most speed from the AI, showed no average gain in measured expertise.

How was the trial set up?

The trial was pre-registered in the American Economic Association’s RCT registry on 21 May 2025 and ran from May 2025 to February 2026. The researchers started from 24 intellectual-property firms that do regular, non-exclusive business with it, and eleven agreed to take part. Of 156 lawyers they put forward, 133 completed onboarding and were randomised. Participants were paid their normal hourly rate for time spent on study tasks.

Within each firm, about two-thirds of lawyers were assigned to the AI group and one-third to a control group, stratified by experience. In practice 90 were treated and 43 were controls. The analysis defines seniors as lawyers with seven or more years of experience. Controls received some general AI training but could not use the tool until the study ended.

The tool was a then-unreleased Google Labs patent-writing assistant. As a pre-release product it changed during the study, including a redlining feature added in July 2025. The paper says it was not released as a standalone product and its features went into other Google AI products; Google’s blog says it is now part of Gemini Notebook.

Lawyers did three main tasks. After 10 days and after 90 days, they drafted parts of a patent application, typically five to seven dependent claims and a detailed description, from simulated inventor materials and figures, with the AI group allowed to use the tool. At 90 days they also redlined a flawed, hypothetical patent draft without AI. Blinded patent attorneys at an independent firm scored every submission on enforceability, accuracy, strategic ambiguity, completeness and clarity, with at least two raters per task. 91 lawyers completed the full protocol.

How much did AI improve the drafts?

With the tool, drafting quality rose by 0.34 standard deviations at 10 days (p = 0.03) and by 0.38 SD at 90 days (p = 0.01). Google’s blog translates those into gains of about 10 and 11 percentile points against the control group. The improvements were consistent across all five quality measures. Juniors gained more than seniors: 0.58 and 0.60 SD at the two assessments, against 0.22 and 0.30 SD for seniors.

The gain came from fewer bad drafts, not more brilliant ones. Treated lawyers were much less likely to land in the bottom fifth of scores and more likely to be in the lower-middle and middle ranges, with no change in the share of excellent drafts. That echoes earlier studies in writing, customer support and software, where AI helped weaker performers most.

AI also saved time. Treated lawyers finished the 10-day drafting task about 10 minutes faster than controls, whose average was 110 minutes (p = 0.05). Point estimates of the saving were larger for juniors, though the difference with seniors was not statistically significant. On the 90-day draft, treated juniors averaged 0.36 on the standardised scale, above untreated seniors at 0.13, where the overall control mean is zero. A Harvard study of coding agents recently found a related pattern: more output does not automatically mean more finished work.

What happened when the AI was taken away?

The 90-day redlining task is the study’s test of skill. It asked lawyers to find and fix errors in a draft without AI, the kind of review work that catches what lawyers call “patent profanity”, wording that narrows or undermines a patent’s legal protection.

Overall, lawyers who had used the AI for three months beat controls by 0.32 SD (p = 0.04). The whole effect came from seniors, who improved by 0.45 SD (p = 0.02), about 13 percentile points in Google’s framing. Juniors’ effect was −0.03 SD (p = 0.93), effectively zero. Using the tool did not make redlining any faster.

The average hides a split among juniors. Treated juniors had fewer middling redlining scores, more poor scores and more good-but-not-excellent ones. The authors read this as AI speeding up skill-building for some juniors while replacing it for others, and say the experiment cannot explain which is which. In post-task surveys, treated lawyers reported large perceived speed gains, and treated juniors felt their drafting quality had improved, by 0.95 SD at 10 days.

Why did junior and senior lawyers diverge?

The paper does not establish the cause, but Google’s blog describes patterns from reading the redlines and follow-up interviews. Juniors in both groups worked top to bottom through the document, often spending their time copy-editing low-stakes introductory sections before reaching the claims, and swapping synonyms rather than fixing the scope of protection.

When juniors did spot a serious flaw, they often wrote a comment describing it instead of rewriting the text. The authors call this “diagnose-without-execute”. Because it was just as common among control juniors, they treat it as a baseline junior gap that three months of letting AI execute rewrites did not close.

Treated seniors instead rebuilt claims from scratch, removed wording that could narrow legal rights and tied many edits to legal doctrine. In interviews they described treating AI output as a “logic auditor” rather than a finished product, something that loosened their attachment to existing text and made them explain their structural edits. A COLM 2026 study found a related risk in a much shorter setting: brief AI help left people quicker to give up when working alone.

How solid are the results?

The authors run several checks. The drafting and redlining effects hold on all five scoring dimensions, and the senior-only redlining gain appears in every one. A permutation test, which does not rely on standard large-sample assumptions, gives p = 0.05 for all lawyers, 0.02 for seniors and 0.93 for juniors. The same pattern holds for seniority cut-offs between three and eleven years.

One worry was that some lawyers secretly used AI on the no-AI redlining task. Using LLM checks and manual audits, the authors flagged 15 of 91 redlining submissions as possibly AI-assisted, erring on the side of flagging. Excluding them pushes the full-sample effect below conventional significance, but the senior effect stays significant at the 10% level. Ratings by Gemini broadly agreed with the human raters on drafting gains, though less consistently on whether juniors or seniors gained more.

What does it mean for training junior staff?

The authors call it a trade-off between immediate productivity and lasting skill. Juniors capture the biggest short-term gains from AI, yet those gains did not become better unaided judgment. They suggest foundational expertise may be what lets AI-assisted work turn into learning, and they expect professional work that needs judgment exercised without AI to remain important in law.

For firms, the practical point is that productivity data alone may hide a training problem. Junior work that AI now speeds up, such as first drafts and routine review, is also how people traditionally learn. The same worry appears in Singapore’s new cyber workforce survey, where 86% of professionals said heavier reliance on AI could weaken entry-level staff’s ability to investigate independently. Deliberate practice without AI, close review of junior edits and teaching juniors to act on the flaws they find are reasonable responses. That is AI Lookout’s reading, not a recommendation from the paper.

What are the study’s limits?

The sample is small: 133 randomised lawyers, 91 completing every task, and junior and senior subgroups of a few dozen each. The firms all had business relationships with Google, Google ran and funded the study, and the tool was Google’s own pre-release product that changed during the trial. MIT’s human-subjects committee determined that MIT was not engaged in the research.

There was no baseline skill test, because firms resisted the extra time; the redlining comparison relies on randomisation to balance skills at the start. The tasks were simulated, scored on a rubric, and covered one specialised profession over three months. It is an NBER working paper, not yet peer reviewed, and AI Lookout is not aware of an independent replication.