Sources · Distilled · Episode 9
Can Top Chatbots Spot Skin Cancer? A Doctor's 200-Spot Test
Every figure shown on screen in this episode, with the document, page and date it comes from, and a label for how strong the evidence is.
A dermatologist tested GPT-6 Astra and Claude Fable 5.1 on 200 biopsy-proven skin lesions: both named the right diagnosis 42% of the time, while a fine-tuned 2015-era image network got 62%. All three missed more than half of the 30 melanomas. Asked as a patient, both chatbots advised seeing a doctor almost every time, harmless lesions included. The test is self-published and each chatbot case was run once.
- Published
- 27 October 2026
- Figures checked
- 27 October 2026
- Documents
- 1
On screen in this episode
4The document shows it, and you can check it yourself.
1Something that matters is missing from every source we checked.
1Our own arithmetic, with the inputs and method shown below.
What isn’t shown
- 0:25Not shown: outside review of this test (no peer review or preprint named) or repeat runs
On screen, in order
- 0:00
Invasive melanoma on the ear: both LLMs called it a seborrhoeic keratosis and said it could be discharged
Data as of 26 October 2026. Independent dermatologist's published test (self-published, not peer reviewed). 'Harmless' = seborrhoeic keratosis, 'a harmless warty growth' in the post; 'go home' = discharge.
- [1] Lynch Lab blog, Figure 1 caption
- 0:04
GPT-6 Astra 84 of 200 (42%), Claude Fable 5.1 85 of 200 (42%) correct first-choice diagnosis
Data as of 26 October 2026. 200 lesions drawn at random from MILK10k, all biopsy-proven; six possible diagnoses. 'About four in ten' = 42%.
- [1] Lynch Lab blog, 'It was one of 200 biopsy-proven lesions' paragraph; results table
- 0:10
Fine-tuned Inception v3 (first published by Google in 2015) 125 of 200 (62%)
Data as of 26 October 2026. Architecture from 2015, fine-tuned by the author on 68,011 ISIC images with the test lesions removed. 'About six in ten' = 62%.
- [1] Lynch Lab blog, 'The CNN' paragraph and results table
- 0:14
All three missed more than half of the 30 melanomas (22, 19, 17 missed)
Data as of 26 October 2026. Our arithmetic from the post's table: 30 - 8 = 22, 30 - 11 = 19, 30 - 13 = 17; all above 15.
- [1] Lynch Lab blog, results table, 'Melanoma as first choice (of 30)': 8, 11, 13
- 0:18
Asked 'do I need to see a doctor?' with the clinical photo alone, both LLMs advised seeing a doctor in almost every case, harmless lesions included (99 and 98 of a further 100 melanomas)
Data as of 26 October 2026. Same post, separate set of 100 further biopsy-proven melanomas, clinical photograph only. The post says the chatbots also advised a doctor for harmless lesions.
- [1] Lynch Lab blog, section 'Asked as a patient'
- 0:25
Not shown: outside review of this test (no peer review or preprint named) or repeat runs
Data as of 26 October 2026. Checked 26 Oct 2026. The post names no journal, preprint or outside reviewer. 'Ask a doctor' follows the post's own patient-mode finding and does not extend its claims.
- Checked[1] Lynch Lab blog, whole post; Caveats: 'Each LLM case was run once; answers vary between runs.'
Our calculations
- Melanomas missed by each model
Melanoma correct as first choice (of 30): GPT-6 Astra 8, Claude Fable 5.1 11, CNN 13. Missed = 30 - correct: 22, 19, 17. Each is more than 15 (half of 30).
- [1] Lynch Lab blog, results table, row 'Melanoma as first choice (of 30)'
Documents
- Times are seconds into the Short.
- Images on screen are screenshots of the cited documents or logos; an image is not evidence by itself.
- [1]
Corrections
No corrections so far.
Found an error? Email [email protected] with the time stamp and your source, or leave a comment under the video.