A Persian–English bilingual medical question-answering benchmark
@inproceedings{ranjbar2026persianmedqa,
title={PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark},
author={Ranjbar Kalahroodi, Mohammad Javad and Karimi, Sepehr and Sheikholselami, Amirhossein and Ranjbar Kalahroodi, Sepideh and Faili, Heshaam and Shakery, Azadeh},
booktitle={Proceedings of LREC 2026},
year={2026}
}LLMs score well on English medical benchmarks, but accuracy drops sharply in other languages — especially in medicine, where cultural and clinical context can be the difference between a right and a dangerous answer. Simple translate-then-test pipelines strip out terminology, regional protocols, and population-specific clinical cues.
Persian is spoken by 100M+ people and had no expert-validated medical QA benchmark. PersianMedQA fills that gap with questions drawn from 14 years of official Iranian national medical exams — sourced from non-crawlable PDF exams, with year-by-year analysis confirming minimal contamination in LLM training data, and translated to English (Gemini-2.5-Flash) with 3-level verification by Iran's National Medical Education Assessment Center plus a board-certified internal medicine specialist.
Most questions translate cleanly. But regional protocols don't — and when they fail, they fail in a way that would matter in an actual clinic. Here, translating a question about a dog bite into English silently erases the Iranian vaccination context the correct answer depends on:
نوجوان ۱۳ سالهای به دلیل گاز گرفتگی سگ — جهت پیشگیری از کزاز، کدام تجویز لازم است؟
✓ Model answer: TIG + Td vaccine — correct
"A 13-year-old due to a dog bite — which should be administered for tetanus prevention?"
✗ Model answer: Td only — incorrect
Why it flips: Iran's vaccination schedule gives the last routine DTP dose at age 6, so a 13-year-old is 7 years post-vaccination. The Iranian protocol calls for TIG + Td for tetanus-prone wounds after 5+ years; Western guidelines allow Td alone under 10 years. Translation erases the regional context the correct answer depends on.
Across 41 models in four categories — general closed, general open, medical-tuned, and Persian-tuned — accuracy on Persian trails a 75% human baseline for all but the strongest models.
| Model | Persian | English | Avg |
|---|---|---|---|
| GPT-4.1 | 83.1% | 80.7% | 81.9% |
| Gemini-2.5-Flash-Preview | 82.4% | 79.1% | 80.7% |
| Claude-3.7-Sonnet | 75.2% | 77.4% | 76.3% |
| LLaMA-3.1-405B-Instruct | 67.0% | 73.5% | 70.3% |
| Meditron3-8B (medical) | 38.7% | 50.0% | 44.3% |
| Dorna2-LLaMA-3.1-8B (Persian) | 34.9% | 51.2% | 43.1% |
| Human baseline | 75.0% | — | — |
GPT-4.1 and Gemini-2.5-Flash score higher on Persian than on the English translation — machine translation introduces semantic drift that disadvantages models on the translated version.