AI and phone-based eye screening: how accurate is the evidence?

Software that reads an eye scan or runs a vision test on a phone can extend screening to people who would otherwise never see a specialist, and the best-studied examples work about as well as their makers claim — within tight limits. An autonomous artificial-intelligence system for diabetic retinopathy, tested in 900 primary-care patients, detected more-than-mild disease with a sensitivity of 87.2% and a specificity of 90.7% [s1]; a smartphone visual-acuity test validated in rural Kenya agreed with a gold-standard chart to within 0.07 logMAR [s2]. Both are screening and triage tools that flag people who need an eye doctor — not devices that diagnose or treat.

The diabetic-retinopathy system, and what its numbers mean

Diabetic retinopathy is a leading cause of blindness in working-age adults, and it is preventable if caught early — but it requires a dilated eye exam or expert reading of retinal photographs, which most primary-care visits do not include. The system tested here, marketed as IDx-DR, uses a fundus camera in a clinic and a cloud algorithm to decide, without a human grader, whether a patient has "more than mild" retinopathy or diabetic macular oedema and should be referred [s1]. Note the hardware: this is a dedicated retinal camera plus AI, not a phone snapshot.

In the pivotal trial, the system enrolled 900 people with diabetes and no known retinopathy at primary-care clinics, and compared its verdict against Early Treatment Diabetic Retinopathy Study grading by a certified reading centre using widefield stereoscopic photography and optical coherence tomography [s1]. Of the participants, 198 (23.8%) actually had more-than-mild disease [s1]. Against that reference standard the algorithm reached a sensitivity of 87.2% (95% CI, 81.8–91.2%) and a specificity of 90.7% (95% CI, 88.3–92.7%), and it returned a usable result — the "imageability" rate — in 96.1% of patients (95% CI, 94.6–97.3%) [s1]. Those figures cleared the trial's pre-specified thresholds, and on that basis the US Food and Drug Administration authorised it as the first autonomous AI diagnostic system in any field of medicine [s1].

What the sensitivity number means in practice: roughly one in eight patients with referable disease was missed on a single screening pass [s1]. That is a good result for a fully automated primary-care test, but it is not a substitute for an eye exam in anyone with symptoms, and screening programmes handle the residual miss rate by repeating the test over time rather than treating one negative as an all-clear. The trial also measured performance at the moment of FDA authorisation in a defined population; it does not establish how the system performs across every camera operator, ethnicity or comorbid eye disease it will meet in the field.

The smartphone acuity test

Visual acuity — reading progressively smaller symbols on a chart — is the most common eye measurement in the world, and the equipment it usually needs (a calibrated, retro-illuminated chart at a fixed distance) is exactly what community and home settings lack. Peek Acuity is a smartphone app that presents a rotating letter "E" and records which way the user says it points, designed to work without a wall chart or familiarity with the Latin alphabet [s2].

Its validation study, nested in the Nakuru Eye Disease Cohort in central Kenya, tested 300 adults aged 55 and older, comparing the app against both a Snellen chart and the ETDRS logMAR chart used as the research reference [s2]. The mean difference between the smartphone test and the ETDRS chart was 0.07 logMAR (95% CI, 0.05–0.09), and against Snellen 0.08 logMAR (95% CI, 0.06–0.10) — small enough that the authors concluded the phone measurements agreed well with both charts [s2]. Test–retest variability was within ±0.033 logMAR, and the app took 77 seconds on average versus 82 seconds for the Snellen test [s2]. In other words, a health worker with minimal training got acuity readings close to a research-grade chart, roughly as fast, on a phone.

Where these tools help, and where they mislead

The case for assistive screening technology is access. A retinopathy algorithm lets a diabetes clinic screen without an ophthalmologist on site; a phone acuity test lets a community worker find people who need glasses or referral where no clinic exists. Both address bottlenecks that keep treatable eye disease undetected — the same access argument that underpins other consumer eye and ear tools, from over-the-counter hearing aids to self-fitted AirPods amplification.

The failure mode is treating a screen as a diagnosis. Neither tool identifies the cause of a problem: the acuity app measures how well someone sees, not why; the retinopathy system flags referable disease but does not stage it or replace the specialist exam that follows a positive result [s1][s2]. A reassuring number from either can create false confidence, particularly in diabetes, where retinal damage can advance without symptoms and where the drug treatments that drive glucose down carry their own early retinopathy signal to watch. These are front doors to eye care, and only useful if the door leads somewhere — to a clinician who can confirm, stage and treat what the software could only flag.

Sources

• [s1] Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices — npj Digital Medicine, 2018-08-28. https://doi.org/10.1038/s41746-018-0040-6
• [s2] Development and Validation of a Smartphone-Based Visual Acuity Test (Peek Acuity) for Clinical Practice and Community-Based Fieldwork — JAMA Ophthalmology, 2015-08-01. https://doi.org/10.1001/jamaophthalmol.2015.1468