GPT-6 Astra and Claude Fable 5.1 lose to a 2015 CNN in a dermatologist's skin cancer test

What the test found

Dr Magnus Lynch, a consultant dermatologist and Mohs surgeon at Guy's Hospital and honorary reader at King's College London, published the results on 25 October 2026. He gave the same 200 skin lesions to OpenAI's GPT-6 Astra, Anthropic's Claude Fable 5.1 and Inception v3, a convolutional neural network (CNN) Google published in 2015, which he fine-tuned on public dermatology images.

The CNN named the correct diagnosis for 125 lesions (62%). GPT-6 Astra got 84 and Claude Fable 5.1 got 85, about 42% each. Allowing a second choice, the CNN reached 173 against 134 and 138. It identified 13 of 30 melanomas, against 8 for GPT-6 Astra and 11 for Claude Fable 5.1. The largest gap was on harmless lesions: the CNN recognised 37 of 50 moles and 26 of 50 seborrhoeic keratoses.

How the test worked

The lesions come from MILK10k, a public International Skin Imaging Collaboration dataset in which each lesion has a clinical photo and a dermoscopic image. Lynch drew 200 at random, all confirmed by biopsy: 100 that should not be discharged, including 30 melanomas, and 100 benign moles and seborrhoeic keratoses. The LLMs got both images and the same prompt at default settings, and were asked for the most likely of six diagnoses, a confidence level, a second choice and whether the patient could be discharged.

The CNN was fine-tuned on 68,011 images from the ISIC Archive on a single H100 GPU, in about twenty minutes, with every image of the test lesions and patients removed first. The models also failed in different ways. GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of the 30 melanomas. Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were. The test cases, prompts, all LLM replies and the scoring code are available for download.

What it means, and its limits

Lynch's conclusion is that a small, old, specialised network recognises skin lesions far better than two general frontier models. He framed the prompts the way a clinician might use the assistants, and ran a second test as a patient would. Asked with only a clinical photo whether to see a doctor, both LLMs said yes in almost every case, harmless lesions included, and for 99 and 98 of a further 100 melanomas. So the weakness he found is in naming the diagnosis, while the patient-facing advice erred toward caution.

Lynch lists the caveats himself: 200 lesions and 30 melanomas give wide uncertainty, each LLM case ran once, and every lesion was biopsied, which makes them harder than routine cases. He notes the LLMs may have seen MILK10k in training. He also says regulated skin cancer products perform considerably better than his baseline. He disclosed that the analysis and first drafts were produced with Claude Code. The work is a blog report, not a peer-reviewed paper.

What to watch next

Repeat runs with different prompts, larger test sets and responses from OpenAI and Anthropic would show whether the gap holds. For a regulated clinical use of AI, compare the NHS evaluations under AI in Industry. More research coverage is under AI Research.

Questions people ask

Can ChatGPT or Claude diagnose skin cancer?

In Dr Magnus Lynch's test of 200 biopsy-proven lesions, GPT-6 Astra and Claude Fable 5.1 named the right diagnosis about 42% of the time, against 62% for a fine-tuned 2015 image network.

Is the skin cancer LLM study peer reviewed?

No. It is a report on Dr Lynch's lab website, with the test data, prompts, model replies and scoring code available to download.

Which model did best on melanoma?

The CNN named 13 of 30 melanomas as its first choice, against 11 for Claude Fable 5.1 and 8 for GPT-6 Astra.

Sources

1. A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis (Lynch Lab, 25 Oct 2026)