Fig. illustrates the overall framework of our proposed FAL-AD, which consists of three core modules: (1) a data augmentation module, responsible for expanding the training data; (2) a federated learning module, used for collaborative model training; and (3) a cross-modal fusion module, used for final classification decision. Each module is described in detail below.

To address the scarcity of AD speech data, we employ a voice data augmentation strategy. While text-to-speech (TTS) synthesis is commonly used for augmentation, TTS systems are typically trained on healthy speech samples and offer limited controllability. Even after adjusting parameters such as speech rate and pause duration, the synthesized speech inevitably exhibits artificial characteristics and fails to capture the authentic pathological patterns of real AD patients. To more effectively generate disease-relevant vocal manifestations while minimizing interference from non-pathological factors (e.g., speaker identity), we introduce voice conversion (VC) technology, as shown in Fig.(a). This method preserves the linguistic and pathological content of speech while transforming speaker-dependent characteristics, thereby enabling the model to focus on clinically relevant features.

Specifically, Our voice conversion is implemented using the CosyVoice 2.0B model []. Let $=Vi(Si,Ci,Yi)$ denote an original speech sample, where $\in Yi\{AD,CN\}$ (CN stands for cognitively normal), $Si$ denotes the speaker characteristics (including timbre and identity-related features), and $Ci$ represents the content aspects (including linguistic content, rhythm, pauses, and pathology-related acoustic features). During augmentation, for source sample $Vi$ and target sample $Vj$ with the opposite category ($\ne YiYj$), we employ voice conversion to recombine the speaker characteristics of the target audio $Sj$ with the content features of the source audio $Ci$, thereby generating a new augmented sample:

where $VC(\cdot )$ denotes the voice conversion function, $\ne YiYj$, and the label of the augmented sample satisfies $=~Yi,jYi$. Each sample in the dataset is alternately treated as the target sample, while a sample from the opposite category is randomly selected as the source sample to perform the aforementioned voice conversion.

Through cross-category voice conversion, we generate both positive and negative class samples for each speaker, significantly enhancing the dataset’s diversity. This strategy also resolves class imbalance issues while ensuring models no longer favor majority classes. Crucially, by maintaining identical speaker timbre across both sample types, the technique forces models to concentrate exclusively on learning pathology-related acoustic features, such as AD-specific prosodic abnormalities and pause patterns, and reduces the risk of model overfitting to irrelevant speaker-specific features. This innovative decoupling of timbre and pathological traits not only ensures precise control over diagnostic characteristics but also substantially improves the model’s accuracy, robustness, and generalization capability to unseen samples.

We adopt Federated Learning (FL) as the foundational paradigm for collaborative training, as Fig.(b) shows. In the standard Federated Averaging (FedAvg) procedure [], the server executes the following steps at each communication round $r$:

However, the standard FL framework solely outputs the final aggregated global model $\omega G$, potentially overlooking superior intermediate models obtained during training. To mitigate this limitation, we introduce an adaptive model selection strategy, which operates as follows:

Our adaptive strategy maximizes performance and adaptability by dynamically selecting the optimal model $\omega i*$ for each client, whether a personalized local update or a generalized global model. The approach remains simple and efficient, introducing no extra hyper-parameters and adding only minimal evaluation and storage overhead. Each client thus deploys a tailored model $\omega i*$ that ensures high performance and local relevance.

To effectively integrate information from speech and text modalities, we build upon the CogniAlign [] framework, as shown in Fig.(c), where we conduct targeted improvements to adapt it for the federated learning environment. The final fusion model achieves fine-grained multimodal interaction through the following mechanisms:

First, as for feature extraction and alignment, we begin with prosodic information modeling, where the Whisper [] automatic speech recognition (ASR) model generates transcriptions with word-level timestamps. According to the timestamps, three types of pause markers (comma, period, and ellipsis) are inserted into the text sequence to capture pause and disfluency patterns []. Subsequently, we use pre-trained models to extract token-level textual features and frame-level acoustic features, and then according to the timestamps, the latter is mean-pooled into word level to complete alignment.

Second, as for the feature fusion, we employ a gated cross-attention mechanism, implemented through a single-layer Transformer encoder to enable deep interaction between modalities []. Specifically, the audio embeddings serve as the Query while the text embeddings provide the Key and Value. The gated cross-modal attention is computed as:

where A and T respectively represent the word-level embedding sequences of audio and text, $\sigma$ denotes the sigmoid function, $Wg$ and $Bg$ are trainable parameters, and $\odot$ represents element-wise multiplication. And then an attention pooling layer dynamically learns the importance of each token through the learned weights:

where $Wa$ and $ba$ are trainable parameters. The final classification is performed by a multilayer perceptron (MLP) on the aggregated representation $h$.

We experiment with the ADReSSo Challenge dataset [], which contains spontaneous speech recordings from 237 subjects (118 AD and 119 CN), describing the ”Cookie Theft” picture. To ensure a fair comparison with existing works [], we adopt a five-fold cross-validation strategy. The dataset is randomly partitioned into five mutually exclusive folds, iteratively using one fold as the test set and the remaining four as the training set. The final reported performance metrics are the average results across all five folds. Note that only the training set is augmented. For both local and federated paradigms, the training set is further randomly divided into three parts to serve as the local data for three clients.

To comprehensively evaluate the efficacy of the FAL-AD framework, we designed multiple comparative experiments from the perspective of learning paradigms, systematically measuring the trade-off between data collaboration and privacy preservation. Under each learning paradigm, model variants with three input modalities, i.e., audio, text, and multi-modal, so as to thoroughly validate the performance of different modalities under various paradigms.

In machine learning, Centralized Learning (CL) represents the ideal scenario of data sharing, and serves as the reference for the performance upper bound. Local Learning (LL) simulates the complete data silo scenario, where each client trains independently using only its local data without any collaboration, representing the performance lower bound. Federated Learning (FL) enables collaboration among clients while keeping data localized. This constitutes a rigorous comparative system. Comparing CL with LL quantifies the performance degradation caused by data silos; comparing LL with FL validates the fundamental benefits of federated collaboration; comparing standard FL with our proposed FAL-AD verifies the added value of our data augmentation and adaptive federated strategies.

All audios are transcribed using the Whisper-large-v3 model [] to obtain word-level timestamps. The audio is down-sampled to 16kHz. Audio and text features are respectively extracted using the frozen pre-trained models facebook/wav2vec2-base-960h [] and distilbert-base-uncased [], with the maximum sequence length set to 200. The multimodal fusion module employs with a hidden dimension of 768 and 12 attention heads. The classifier is a two-layer MLP. The hyper-parameter settings for CL and LL baselines are consistent with CogniAlign. Our FL methods are trained using the AdamW optimizer with a learning rate of 5e-5, weight decay of 0.01, and a fixed batch size of 64, which simulates 3 clients for 30 communication rounds. All hyper-parameters are selected through grid search to ensure optimal model performance.

Overall, our proposed FAL-AD framework achieves state-of-the-art performance on the ADReSSo dataset, as shown in Table , surpassing all existing centralized benchmarks. The experimental results systematically validate its effectiveness across three key aspects: first, the comparison between CL and LL quantifies the severe performance degradation caused by data isolation, highlighting the necessity of collaborative paradigms. Second, FL effectively mitigates this issue, achieving a performance gain of over 10 percentage points compared to isolated training (LL), demonstrating its capability to break down data silos while preserving privacy. Finally, and most significantly, our FL+Aug approach not only matches but exceeds the performance of its centralized counterpart (CL+Aug), achieving a new multi-modal accuracy benchmark of 91.52% and outperforming previous state-of-the-art methods including CogniAlign (90.36%). This paradigm breakthrough reveals a synergistic effect: federated aggregation acts as a regularizer to reduce overfitting, while data augmentation enhances generalization through expanded diversity.

Analysis on Data Augmentation: We further analyze the utility of the proposed voice conversion strategy under different learning paradigms. This strategy brings consistent performance improvements in almost all settings. The benefit is most prominent under the Federated Learning (FL) paradigm, boosting multimodal accuracy from 89.70% to 91.52%. This indicates that data augmentation provides richer and more diverse local data ’fuel’ for federated learning, synergizing with the regularization effect of federated aggregation to push model performance to new heights. The gains under the Local Learning (LL) paradigm are also significant (multimodal accuracy increases from 78.59% to 80.61%), confirming its value as an effective regularization in extremely data-scarce environments. Notably, the improvement in CL paradigm is relatively limited (+0.61%), consistent with our previous finding that models trained on sufficient data are more susceptible to overfitting to the subtle biases introduced by augmented data. This contrast, in turn, demonstrates that our federated learning framework is a superior environment for deploying such data augmentation strategies.

Analysis on Federated Learning: The performance of our FAL-AD surpasses the strong centralized baselines. This may be because that, while centralized training has access to all data, it can easily lead the model to overfit to specific patterns in the training set. The inherent ”local update & global average” iterative mechanism in the FedAvg algorithm effectively constrains the model optimization path, forcing it to converge to a flatter and more robust optimum for all data distributions, thereby achieving superior generalization performance. In addition, the adaptive model selection strategy ensures each client finally deploys its historical best model, maximizing the potential of models from different communication rounds. As shown in Table , while different aggregators exhibit varying affinities for not identically and independently distributed (non-IID) data, our proposed adaptive Federated strategy (aFL) demonstrates significant and consistent advantages. It achieves optimal performance under most configurations, validating its effectiveness as a lightweight, hyperparameter-free personalization solution. Notably, traditional local fine-tuning strategies (pFL) did not bring stable gains, suggesting that mandatory local fine-tuning in heterogeneous data environments may lead the model to deviate from the optimal solution. This proves that aFL possesses strong fault tolerance and stability. The dynamic selection mechanism retains the deployment simplicity of standard FL (sFL) while achieving personalization benefits close to pFL.