A 2015 skin cancer CNN beat GPT-6 Astra and Claude Fable 5.1 in a dermatologist's test

What Magnus Lynch tested

On 25 October 2026 Dr Magnus Lynch, a consultant dermatologist and Mohs surgeon at Guy's Hospital in London and honorary reader at King's College London, published A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis. He gave the same 200 skin lesions to OpenAI's GPT-6 Astra, Anthropic's Claude Fable 5.1 and a fine-tuned copy of Inception v3, an image network Google first published in 2015. The old network named the right diagnosis for 125 lesions, or 62 percent. The two frontier models managed 84 and 85, about 42 percent each. The post reached the front page of Hacker News within hours.

The lesions come from MILK10k, a public data set released through the International Skin Imaging Collaboration in 2025, in which every lesion has two images: a clinical close-up and a dermoscopic image taken through the lit magnifier dermatologists use. Lynch drew 200 at random, all confirmed by histopathology. Half should never be discharged: 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas and 20 actinic keratoses. Half are safe to discharge: 50 benign moles and 50 seborrhoeic keratoses.

Every lesion was biopsied, including the harmless ones, which makes the set harder than everyday practice. As Lynch puts it, a mole somebody decided to cut out is one that worried a clinician.

How the three models were set up

The two LLMs got both images and nothing else, with the same prompt at default settings. They were asked, as a clinician might use them, for the most likely of six diagnoses or unsure, a confidence, a second choice, and whether the lesion could be safely discharged with no treatment, biopsy or follow-up. Neither model refused a case.

The CNN was deliberately not state of the art. Lynch started from Inception v3 pretrained on everyday photographs and fine-tuned all of its layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU, in about twenty minutes. The 200 test lesions, and every other photograph of the same lesion or patient, were removed from its training and validation data, and they were not used to pick the model or set its threshold. It scored both images, weighted towards the dermoscopic one, and chose from the same six diagnoses.

The network is small enough to run in a phone browser with WebGPU, and Lynch has put a demo online that uploads nothing. The cases, prompts, every LLM reply and the scoring code are in one download.

The headline numbers

The gap holds up beyond the headline figure. Allowing a second choice, the CNN got 173 of 200 right against 134 for GPT-6 Astra and 138 for Claude Fable 5.1. It named 13 of the 30 melanomas as its first choice, against 8 and 11. The biggest differences were on harmless lesions: the CNN recognised 37 of 50 moles and 26 of 50 seborrhoeic keratoses, against 16 and 13 for GPT-6 Astra and 33 and 6 for Claude Fable 5.1.

It did not win everywhere. GPT-6 Astra identified all 25 basal cell carcinomas, one more than the CNN, and both LLMs did better on actinic keratosis, where Claude Fable 5.1 got 12 of 20, GPT-6 Astra 10 and the CNN 9. On squamous cell carcinoma the CNN got 16 of 25, GPT-6 Astra 12 and Claude Fable 5.1 just 5.

Where the LLMs went wrong

Lynch found that each LLM had a favourite answer. GPT-6 Astra called 75 lesions basal cell carcinoma when only 25 were, including 14 of the 30 melanomas. That explains its perfect basal cell score: it said basal cell carcinoma so often that it caught every real one. Claude Fable 5.1 leaned the other way, calling 53 lesions actinic keratosis when 20 were.

The post opens with a melanoma that both LLMs called a seborrhoeic keratosis, a harmless warty growth, and said could be discharged; the CNN got it right from the dermoscopic view. In another example, a thin, barely pigmented invasive melanoma, GPT-6 Astra answered basal cell carcinoma with 95 percent confidence while Claude Fable 5.1 and the CNN named melanoma. The CNN's own melanoma errors went in a different direction: of the 17 it missed, it called 8 a mole and 6 a seborrhoeic keratosis.

The safety numbers cut both ways

Diagnosis is not the whole story, and Lynch's table shows why. On the question that matters most in a clinic, whether a lesion can be sent home, the LLMs were cautious. GPT-6 Astra kept 97 of the 100 lesions that should not be discharged and Claude Fable 5.1 kept 96, discharging 3 and 4 of the 30 melanomas. The price was that they discharged only 32 and 22 of the 100 safe lesions, so most harmless spots would still be referred.

At its preset threshold the CNN was far less cautious: it kept 77 of the unsafe lesions, discharged 76 safe ones and let 18 of the 30 melanomas go, which would be unacceptable in practice. When its threshold was matched to the LLMs, the table reports it kept 98 or 99 unsafe lesions and discharged 2 or 1 melanomas. In other words, the CNN's advantage depends on where the line is drawn, and the post does not report how many safe lesions it discharged at those matched settings.

Lynch also asked the LLMs as a patient would, with the clinical photograph alone and the question "do I need to see a doctor?" Both advised seeing a doctor in almost every case, harmless lesions included, and for 99 and 98 of a further 100 biopsy-proven melanomas. For a worried member of the public that is safe advice, but it is not a diagnosis.

The caveats Lynch lists himself

Lynch is open about the limits. Two hundred lesions show a pattern but cannot rank close results, and 30 melanomas leave wide uncertainty on the number that matters most. Each LLM case was run once, and answers vary between runs. Because every lesion was biopsied, the cases are hard: on ordinary archive images the same CNN separated lesions that should and should not be discharged far better, with an area under the ROC curve of 0.96 against 0.86 here.

MILK10k has been public since 2025, so the LLMs could have seen it in training, which if anything would have helped them; the CNN had certainly not. And Inception v3 is a baseline. Lynch notes that current skin cancer AI uses ensembles of image transformers trained on more data, and that regulated products built on them perform considerably better. He also discloses that the analysis and first drafts were produced with Claude Code, which runs on a Claude model, while scoring was done by script against the histology.

What it means for AI in medicine

The result fits a pattern we keep seeing in medical AI: narrow models trained and validated on one task still beat general chatbots at that task. Google's TB chest X-ray model, for example, became a certified device before it reached clinics, as covered in Google says its TB X-ray AI now screens patients in six African countries, and its blind-sweep ultrasound system was tested against expert sonographers rather than chat models, as covered in Google's blind-sweep ultrasound AI matched expert sonographers.

For people uploading photos of a mole to a chatbot, the practical message is narrower than the headline. The LLMs in this test were poor at naming the lesion but almost always told patients to see a doctor, which is the right answer. Where they fall short is the clinician's question: they flagged so many harmless lesions that they would add little to a triage clinic, and when they were wrong about a melanoma they were sometimes very confident. A twenty-minute fine-tune of an old network did better on most measures, on a small, hard test that its own author warns against over-reading.