The advent of GPT-3 and successor models spurred extensive research into medical applications of LLMs ( , ), leading to the development of many benchmarks to evaluate LLMs’ performance. General multitask evaluations such as MMLU assess knowledge across 57 subjects (from math and history to law and medicine), providing a broad measure of academic and professional understanding ( , ). Medical-specific benchmarks have also emerged. Researchers used United States Medical Licensing Exam (USMLE) question banks to assess models, where GPT-3.5/ChatGPT achieved ~60% accuracy (the passing threshold) without specialized training ( , ). Medical Q&A datasets such as PubMedQA ( , ), HEAD-QA ( , ), MedQA ( , ), MedMCQA ( , ), Medbullets ( , ) have been used to quantify clinical knowledge recall. More recently, Google’s MultiMedQA benchmark combined several of these resources, spanning professional medical exams, research questions, and consumer health queries, to holistically evaluate medical LLM knowledge ( , ). OpenAI’s HealthBench ( , ) focuses on open-ended generation tasks, utilizing 48,562 unique rubric criteria to enable more realistic and granular clinical evaluations.

These benchmarks have driven rapid progresses; for instance, an instruction-tuned PaLM model (Med-PaLM) achieved state-of-the-art (SOTA) performance on multiple-choice questions, reaching 67.6% accuracy on MedQA ( , ). Despite these gains, high scores on structured, exam-style questions do not necessarily translate to real-world clinical competence ( , ). Human evaluations reveal persistent reasoning errors, limited robustness, and a lack of trustworthiness outside narrow formats ( , ). This has reinforced the consensus that new evaluation frameworks are needed, ones that extend beyond exam accuracy to assess real-world clinical competencies ( , ).

In response, researchers have begun creating specialized evaluation sets targeting particular failure modes. MEDEC ( , ) is the first publicly available benchmark for medical error detection and correction in clinical notes. MedHallu ( , ) focuses on medical hallucination detection, comprising question-answer pairs from PubMedQA with deliberately fabricated answers, challenging a model to judge correctness, and results show that even top models struggle to reliably identify incorrect medical statements. In addition, as shown in Table , except for MEDIQA ( , ), a small dataset, the proportion of ADRD-related questions in each benchmark is generally below 3%, mostly under 1%. Collectively, these findings indicate that ADRD-specific evaluation remains underdeveloped. We aim to address this gap by introducing a benchmark focused on the ADRD domain, which has been largely overlooked in prior works.

Natural language processing research in the ADRD field has traditionally been dominated by ADRD detection tasks (e.g., automated screening and diagnosis). For example, many works focus on screening for cognitive impairment ( , ), and numerous studies tackle the diagnosis of dementia from patient speech or text ( , ). A recent systematic review found that about 56% of ADRD-related NLP studies focus on detecting cognitive impairment, such as through classifying transcripts ( , ). In contrast, far fewer studies have explored NLP for ADRD care and support. The same review identifies two emerging task categories beyond detection: caregiver support and patient assistance. However, only $\sim$6% of existing works fell under caregiver support, highlighting the nascent state of this area ( , ).

Nevertheless, there is growing interest in applying NLP and LLMs to ADRD. Recent work explored caregiver-oriented dialogue systems, such as ADQueryAid ( , ), which leverages ADRD-specific knowledge and retrieval augmentation to provide more relevant and supportive responses than general-purpose models, as well as domain-adapted language models such as the Caregiving Language Model (CaLM) ( , ), fine-tuned on dementia care data. Beyond direct Q&A, LLMs have also been applied to analyze caregiver narratives and clinical texts, for example, to identify overburdened informal caregivers from long-term care notes. Collectively, these studies illustrate the emerging interests in language technologies beyond detection tasks, while highlighting the need for rigorous, ADRD-specific evaluation.

Existing medical benchmarks provide very limited coverage of ADRD-related tasks, and the questions for evaluating models in daily caregiving scenarios are even more scarce. As a result, highly relevant ADRD caregiving questions, such as how to respond when a patient becomes agitated at night, are absent from current evaluations, leaving little systematic, quantitative evidence of how LLMs perform in the realities of daily caring. To address this gap, we construct an evaluation set that integrates the fragmented, knowledge-oriented questions from seven existing healthcare benchmarks with a complementary set of daily caregiving questions grounded on the ABC model. Overall, this new benchmark dataset provides an initial testbed for assessing LLM performance across both ADRD clinical knowledge and the practical, human-centered aspects of daily care, enabling more realistic, rigorous, and targeted research on ADRD-related LLM applications.

To identify ADRD-related questions from existing benchmarks, we searched 12 public benchmark datasets using ADRD domain keywords containing “Alzheimer’s”, “Dementia”, “ADRD”, and “AD”. Except for MEDIQA, the proportion of ADRD-related questions in each benchmark is below 3%, and most are under 1%. The proportion of MEDIQA is 13.05%; however, since MEDIQA contains 383 questions, 13.05% leads to 50 questions only. Across all benchmarks, the total number of ADRD-related questions accounts for 0.64% of total non-generation task questions and 0.54% of generation task questions, respectively. In addition, manual checking showed that there were only seven questions in total about caregivers’ tasks. Here, a caregiver can be either formal or informal, defined as a person who provides paid or unpaid assistance with daily living and cognitive support as a family member or close friend, or delivers such care as a non-nurse, non-medical licensed personnel (e.g., community health workers, certified nurse aides). These low percentages show that existing medical benchmarks contain only a tiny proportion of ADRD-relevant content, especially on caregiver tasks, indicating a substantial gap and leaving significant room for future exploration. The results are shown in Table .

We consolidated ADRD questions from seven existing non-generation task benchmarks in Table , including PubMedQA ( , ), HEAD-QA ( , ), MedBullets ( , ), MedMCQA ( , ), MedQA ( , ), MEDEC ( , ), and MedHallu ( , ). These datasets have been the leading healthcare benchmarks to test SOTA LLMs ( , ). This led to a unified question and answer set, named ADRD Unified QA. We retain the fully original benchmark content in its entirety without modification. MEDCALC-BENCH ( , ) is excluded from our benchmark because we focus on evaluating models’ ADRD-related knowledge and reasoning capabilities, whereas this dataset emphasizes numerical calculation and arithmetic reasoning. In addition, ADRD Unified QA does not include generation tasks, as existing evaluation metrics are insufficient to reliably and consistently assess the quality of LLM responses ( , ).

Therefore, ADRD Unified QA consists of two primary question types. The first comprises multiple-choice questions designed to evaluate models’ knowledge of ADRD-related concepts, which were from PubMedQA, HEAD-QA, MedBullets, MedMCQA, and MedQA. The second focuses on error detection, assessing models’ ability to identify hallucinations and factual errors present in textual inputs, which were from MEDEC and MedHallu.

One notable limitation of existing benchmarks is that ADRD questions are almost exclusively framed as clinical knowledge or exam-style reasoning tasks, with almost no coverage of daily caregiving. For instance, a manual check of the seven datasets used in section 3.1 yielded only seven caregiving items. However, managing ADRD in real-world settings extends far beyond diagnosis and pharmacological treatment, and largely revolves around day-to-day caregiving challenges such as behavioral management, communication with patients, safety supervision, and caregiver decision-making in home and community settings.

To address this gap, we constructed an additional ADRD Caregiving QA set based on materials from the Aging Brain Care (ABC) program. The ABC program is a comprehensive, evidence-based model of care for Alzheimer’s disease and related dementias (ADRD) that has been continuously developed, evaluated, and implemented for nearly two decades ( , ). Supported initially by the U.S. Agency for Healthcare Research and Quality, the ABC model was designed as a longitudinal, biopsychosocial intervention to address the complex clinical, behavioral, and caregiver challenges associated with ADRD. In 2006, the program demonstrated efficacy in a randomized controlled trial, showing significant reductions in behavioral and psychological symptoms of dementia, caregiver burden, and inappropriate medication use among underserved populations with high social frailty ( , ). Since then, the ABC program has been continuously operated, refined, and scaled, serving more than 5,000 patient–caregiver dyads ( , ). Recognized by the National Academies of Medicine, ABC integrates interdisciplinary team-based care, non-pharmacologic interventions, caregiver support, medication optimization, care coordination, and community partnerships across the full ADRD trajectory. ABC directly informed the development of an alternative payment approach for comprehensive dementia care, later adopted by the U.S. Centers for Medicare & Medicaid Services through the GUIDE Model ( , ). As such, the ABC program represents a mature, rigorously evaluated, and policy-relevant standard for high-quality ADRD care.

Therefore, the ABC program naturally serves as a solid foundation to extract benchmark questions. The ADRD Caregiving QA set covers essential dimensions of ADRD-related caregiving, such as understanding the condition, effective communication with patients, and behavioral management strategies. We follow a structured abstraction process in which recurring caregiving situations and recommended care strategies were identified from the education content and caregiver handout, and transformed into evaluable question-answer pairs. The resulting QA sets include 120 polar (True/False) QAs and 29 multiple-choice QAs. To ensure data quality, all constructed questions were reviewed and refined for accuracy, clarity, and appropriateness by a senior clinician who served as the lead designer of the ABC program and has been involved in its development and refinement for two decades.

Although modest in size, this question set provides targeted coverage of ADRD daily caring scenarios that are largely absent from existing benchmarks. By integrating these questions with the extracted ADRD Unified QA set, the ADRD-Bench enables a more comprehensive evaluation of LLMs that spans both clinical knowledge and practical caregiving support for ADRD.

To validate the performance of LLMs on the proposed ADRD-Bench, we tested 33 LLMs as listed in Table , including both open-weight LLMs and closed-source LLMs. The open-weight LLMs included general and medical LLMs, varying in size from 3.8B to 235B parameters. This evaluation on a diverse set of models with different parameter scales and deployment settings not only helped establish a comprehensive understanding of the models’ performance but also the overall difficulty level of the benchmark across various model settings.

For open-weight models with 14.8B parameters or fewer, all experiments were conducted on a local workstation with an NVIDIA RTX 6000 Ada Generation GPU (48 GB VRAM). For larger open-weight models with 32B parameters and above, experiments are conducted on an Oracle Cloud Infrastructure server with eight NVIDIA H100 GPUs (80 GB HBM3 memory each). All closed-source general models were evaluated via their official APIs/terminals.

For quantitative evaluation, given that all tasks in our benchmark are formulated as discrete-choice questions, including multiple-choice, True/False, and error sentence detection formats, we adopt exact match accuracy as the primary metric. A model’s prediction is considered correct if and only if the selected option exactly matches the ground-truth label. For each model, we reported accuracies separately for each set of the benchmark and the overall accuracy aggregated across all questions.

For qualitative analysis, we focused on the newly proposed ADRD Caregiving QA set to better understand model failures in daily caregiving scenarios, which have not been covered in the literature. Specifically, we selected questions on which LLMs frequently produced incorrect answers, as indicated by the quantitative results. The selected cases were reviewed by our clinical expert to analyze common error patterns. The qualitative analyses were intended to contextualize the findings and highlight practical limitations of current LLMs in daily caregiving scenarios.

Figure  shows a significant, moderate-to-strong positive correlation (Pearson’s $=r0.5486$, $=p0.0025$) between model size and accuracy on the ADRD Unified QA set. Most smaller models in the 4–7B parameter range achieved relatively low performance (approximately $0.45$–$0.75$), whereas large models (70B+ parameters) demonstrated substantially higher accuracy, mostly $\ge 0.85$. Moreover, most models with more than 30B parameters performed above the overall mean accuracy (0.7972). The best-performing model was Llama-3.1-70B-Instruct ( , ), which achieved an accuracy of $0.9334$.

When comparing different model categories, open-weight medical models (red points) generally tended to outperform open-weight general models (blue points) of comparable size. Across the parameter scale, open-weight medical models were mostly positioned above open-weight general models, highlighting the benefits of domain-specific fine-tuning for clinical question answering and medical error detection. However, an exception was observed for the top-performing model, Llama-3.1-70B-Instruct ( , ), which is a general model and slightly outperformed the strongest medical model, Meditron3-70B ( , ) (0.9312). All closed-source general models achieved performance above the overall mean accuracy, with Gemini 2.5 Pro ( , ) performing the best among them, reaching an accuracy of $0.9142$. Nevertheless, it was surpassed by several open-weight models (e.g., Llama-3.1-70B-Instruct and Meditron3-70B) with significantly fewer parameters. In addition, some open-weight medical LLMs exhibited substantially lower accuracy than open-weight general models with similar parameter counts, indicating that domain-specific fine-tuning does not universally guarantee superior performance.

Overall, these results suggest that model size is an important contributor to performance on the ADRD Unified QA benchmark; however, it does not fully determine model effectiveness. Domain-specific fine-tuning may provide performance advantages in certain cases, but general models can still outperform specialized models, possibly due to stronger general reasoning capabilities and more robust language representations learned from large-scale and diverse pretraining corpora.

On the ADRD Caregiving QA set (Figure ), the performance gap between smaller and larger models was narrower than that observed on the ADRD Unified QA set. This may be attributed to the fact that the questions in this benchmark focus more on daily living and are closer to “common-sense” reasoning, which is generally well covered by large language models. Nevertheless, an upward trend in accuracy with increasing model size remained evident. Pearson’s correlation analysis revealed a significant, moderate-to-strong positive correlation between model size and accuracy ($=r0.5277$, $=p0.0039$). Most smaller models in the 4B–7B parameter range achieved accuracies between $0.75$ and $0.85$, all of which (except for the smallest model, phi3:3.8b-mini-128k-instruct-fp16 ( , )) fell below the overall mean accuracy of $0.8629$. In contrast, the majority of models with more than 14B parameters performed above the average accuracy.

Except for grok-4-1-fast-non-reasoning ( , ) and Gemini 2.5 Pro ( , ), all other closed-source general models achieved accuracies above $0.9$, outperforming the majority of open-weight models. This observation suggests that closed-source general models currently exhibit stronger overall capabilities in ADRD daily caregiving tasks. GPT-5.2 ( , ) achieved the best performance among all evaluated models, with an accuracy of $0.9664$, slightly surpassing the best-performing open-weight medical model, Lingshu-32B ( , ) ($0.9463$), as well as the best-performing open-weight general model, Qwen3-235B-A22B-Instruct-2507 ( , ) ($0.9128$).

Medical models did not demonstrate a clear performance advantage over general models. In a few cases, they underperformed general models of comparable size. This pattern suggested that answering practical daily caregiving related questions relied more on broad commonsense reasoning capabilities typically acquired by large general models, rather than on specialized medical fine-tuning. At the same time, this observation might reflect the fact that current domain-specific fine-tuning efforts primarily focus on medical knowledge and incorporate relatively limited information about daily caregiving.

To understand how models’ performance on clinical knowledge is related to that on caregiving, we analyzed the correlation between accuracies on ADRD Caregiving QA and accuracies on the ADRD Unified QA, as shown in Figure  and . Across all models, the Pearson’s correlation analysis revealed a significant and strong positive correlation between the two QA sets ($=r0.6801$, $=p0.00001$). This indicated that, at an aggregated level, models that performed well on clinical questions also tended to achieve a higher accuracy on the caregiving questions.

However, when examining different model categories separately, we observed that only open-weight medical models exhibited this significant and strong correlation ($=r0.8416$, $=p0.0003$). In contrast, open-weight general models ($=r0.2730$, $=p0.3248$) and closed-source models ($=r0.4678$, $=p0.4269$) showed only weak-moderate, non-significant positive correlations. These results demonstrated that open-weight medical models, which incorporated specific medical fine-tuning, had more consistent performance across both clinical and caregiving questions. A possible cause of the non-significant correlations of the general models could be the relatively small portion of medical data in model training, and thus the models’ performance depends more on general common sense and social reasoning. This contributes to more socially appropriate responses, a key point in daily caregiving. This analysis also aligned with our earlier observation that, on ADRD Caregiving QA, open-weight general models achieved higher accuracy than open-weight medical models at comparable parameter scales. On the other hand, these findings also implied that, to develop healthcare LLMs that fit ADRD care needs, more training data on caregiving tasks should be included to balance the models’ knowledge and reasoning on this aspect.

To better understand large language model (LLM) performance beyond accuracy metrics, we examined the models’ reasoning processes on ADRD Caregiving QA questions for which over $\%75$ of the evaluated models generated incorrect answers. In reviewing these challenging cases, feedback from a clinical expert was incorporated to ensure that the analysis was grounded in real-world ADRD caregiving principles. Two representative cases are presented to illustrate common failure patterns and model behaviors.