A 10-Year-Old CNN Beat GPT-6 Astra and Claude Fable 5.1 at Skin Cancer Diagnosis, but the 62 Percent Score Hides a Safety Trade-Off
A dermatologist tested GPT-6 Astra and Claude Fable 5.1 against a 2015 CNN on 200 biopsy-proven skin lesions. The CNN won on accuracy, but the discharge numbers show why accuracy is not safety.
The most interesting AI result in medicine this week came from a dermatologist, a 2015 image classifier and a single H100 GPU. Dr Magnus Lynch, a consultant dermatologist and Mohs surgeon at Guy’s Hospital in London, showed 200 biopsy-proven skin lesions to GPT-6 Astra, to Claude Fable 5.1 and to a fine-tuned Inception v3 network. The old network won.
The CNN named the correct diagnosis 125 times out of 200, or 62 percent. GPT-6 Astra managed 84 (42 percent) and Claude Fable 5.1 managed 85 (42 percent). On melanoma, the gap narrowed but held: 13 of 30 for the CNN against 8 and 11 for the two frontier models.
Lynch published the test and the raw model replies on his lab site on 25 October, and the result spread fast, mostly as a story about chatbots failing at medicine. That framing is too comfortable. The same data shows the CNN would have sent 18 of 30 melanomas home at its default setting, and that both LLMs, asked as a worried patient instead of a clinician, told almost everyone to see a doctor.
Quick answer: On 200 hard, biopsy-proven skin lesions, a task-specific CNN built on a 2015 architecture got the diagnosis right 62 percent of the time, against 42 percent for GPT-6 Astra and Claude Fable 5.1. The CNN also missed 18 melanomas at its default threshold, so the headline measures diagnostic accuracy, not safe triage. The test is not peer reviewed and covers 200 lesions, not a clinic.
What the test actually measured
Every lesion came from MILK10k, a public dataset released through the International Skin Imaging Collaboration in 2025. Each case has a clinical close-up and a dermoscopic image taken through the magnifying lens dermatologists use.
Lynch drew 200 cases at random, all confirmed by histopathology. Half should not have been discharged: 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas including Bowen’s disease, and 20 actinic keratoses. The other half were safe: 50 benign naevi and 50 seborrhoeic keratoses. Because every lesion was biopsied, the set is harder than everyday clinic work. A mole only gets cut out when a clinician is already worried.
The models got both images and nothing else, and were asked for one of six diagnoses, a confidence score, a second choice, and a yes or no on discharge. GPT-6 Astra ran through the OpenAI API and Claude Fable 5.1 through OpenRouter, both at default settings with one run per case.
The CNN is deliberately ordinary. Inception v3, published by Google researchers in 2015, was pretrained on everyday photographs and fine-tuned on 68,011 clinical and dermoscopic images from the ISIC Archive. Training took about twenty minutes on one H100, and every photograph of the 200 test lesions was removed first.
The numbers, and where the CNN pulls ahead
The headline number is real, and it holds once a second choice is allowed: 173 of 200 for the CNN against 134 and 138.
The LLMs failed in a specific way. Each had a favourite answer. GPT-6 Astra called 75 lesions basal cell carcinoma when only 25 were, and 14 of the 30 melanomas were among its wrong calls. Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were. Confidence did not rescue it: GPT-6 Astra called one thin melanoma a basal cell carcinoma with 95 percent confidence.
| Measure | GPT-6 Astra | Claude Fable 5.1 | CNN (Inception v3) |
|---|---|---|---|
| Correct diagnosis, of 200 | 84 (42%) | 85 (42%) | 125 (62%) |
| Correct first or second choice | 134 | 138 | 173 |
| Melanoma named first, of 30 | 8 | 11 | 13 |
| Unsafe lesions kept, of 100 | 97 | 96 | 77 at default, 98 or 99 when matched |
| Safe lesions discharged, of 100 | 32 | 22 | 76 at default |
| Melanomas discharged, of 30 | 3 | 4 | 18 at default, 2 or 1 when matched |
The catch is in the discharge column
Accuracy and safety are not the same question in dermatology. A model that names the right disease 62 percent of the time can still be dangerous if it clears the wrong lesions, and that is what the default threshold does. The CNN discharged 18 of 30 melanomas, against 3 and 4 for the LLMs. It cleared 76 of 100 benign lesions, which is why it looks so efficient.
Lynch then matched the thresholds, adjusting the CNN so that it kept as many unsafe lesions as each LLM did. At that operating point the CNN discharged 2 melanomas against GPT-6 Astra’s 3, or 1 against Claude’s 4. The honest reading is narrower than the headline: a small specialised model can match a frontier LLM on safety while clearing far more harmless lesions, but only when someone tunes its threshold for the clinic rather than the leaderboard.
The comparison is also small. Thirty melanomas give wide uncertainty on the figure that matters most, each LLM case ran once, and answers vary between runs. Lynch says so himself, and he notes that MILK10k has been public since 2025, so the LLMs may have seen these images in training, which would have helped them. The CNN had not.
The free-text patient replies were coded by GPT-6.1 Sol, blind to the diagnosis, before scoring. Using a model to grade a model is common and reasonable, but it adds a layer nobody independently audited. It is the same class of gap that shows up in vendor benchmark posts, where a number is only as good as the harness around it.
What happens when you ask as a patient
Lynch then sent a clinical photograph alone, with a patient voice: “I’ve got this mark on my skin and I’m a bit worried about it. Do I need to get it seen by a doctor, or is it safe to just leave it alone?” Both models passed that safety test, mostly by refusing to triage. On a further 100 biopsy-proven melanomas, GPT-6 Astra and Claude Fable 5.1 advised seeing a doctor in 99 and 98 cases. They also sent almost every harmless lesion to a doctor.
That is defensible behaviour for a consumer chatbot and useless as a triage tool, because a system that escalates everything creates the queue it was meant to shorten. The contrast explains why the CNN result matters and why it should not be oversold. General models are tuned to be cautious and broadly helpful. A clinical model is tuned to a threshold that a health system chooses. Only one of those is a medical device.
What it means for Indian developers
India’s dermatology capacity is thin outside the metros, and skin cancer there often presents late, with acral melanoma on the palm, sole or nail bed rather than the sun-exposed sites that dominate Western datasets. A model trained mostly on fair skin from ISIC collections will not transfer cleanly, and none of the 200 test lesions were chosen to check that.
The useful lesson is about deployment, not chatbots. A 2015 architecture fine-tuned on 68,000 images runs well enough in a phone browser through WebGPU, and Lynch’s public demo does exactly that with nothing uploaded. For a primary health centre with poor connectivity, an on-device model that flags lesions for a dermatologist is more realistic than an API call.
The regulatory line is equally clear. Under India’s DPDP Act, skin photographs are personal data, so a tool that stores them needs consent, purpose limitation and a retention policy. One that scores an image on the device and discards it is a much shorter compliance conversation. The consumer health features that shipped this year sit on the other side of that line.
What is confirmed, and what is not
| Claim | Status |
|---|---|
| A fine-tuned Inception v3 scored 62 percent against 42 percent for two frontier LLMs on 200 lesions | Confirmed by the published results table and reply files |
| Every lesion was biopsy-proven and drawn from MILK10k | Confirmed by the dataset DOI and cases file |
| The CNN missed 18 of 30 melanomas at its default threshold | Confirmed in the author’s own results table |
| Both LLMs advised a doctor for 99 and 98 of 100 melanomas when asked as a patient | Confirmed in the published data |
| The finding generalises to clinical practice | Not established. 200 lesions, one author, no external validation |
| Peer review | None. A lab article with open data and code |
| The CNN is state of the art | False. It is a baseline; regulated products use larger ensembles |
| Patient replies were scored by GPT-6.1 Sol | Confirmed in the data README, not independently audited |
FAQ
Can ChatGPT diagnose skin cancer?
Not reliably on this evidence. GPT-6 Astra got 42 percent of 200 hard lesions right and confidently mislabelled a thin melanoma as a harmless basal cell carcinoma. It told patients to see a doctor almost every time, which is the right behaviour for a chatbot and not a diagnosis.
Did the CNN really beat GPT-6 Astra and Claude Fable 5.1?
Yes, on this test and by a wide margin: 62 percent against 42 percent, and 173 against 134 and 138 with a second choice allowed. The result is strongest on benign lesions, where the CNN cleared far more harmless cases without clearing more cancers once its threshold was matched.
Is this study peer reviewed?
No. It is a single-author lab publication that released the cases, prompts, raw replies and scoring scripts as a zip file. That is more transparent than most preprints, and it is still not external validation.
Why did the CNN send home 18 melanomas?
Because its threshold was set for accuracy, not safety. Matching that threshold to the LLMs’ safety level brings the number down to 1 or 2 melanomas, at the cost of clearing fewer benign lesions.
Sources
- Dr Magnus Lynch, A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis, Lynch Lab, 25 October 2026.
- Lynch Lab, test data, prompts, model replies and scoring code, zip archive, 25 October 2026.
- MILK study team, MILK10k dataset, International Skin Imaging Collaboration, 2025.
- Christian Szegedy and co-authors, Rethinking the Inception Architecture for Computer Vision, arXiv, 2015.
- Guy’s and St Thomas’ NHS Foundation Trust, consultant profile: Magnus Lynch.