Normative modeling learns a healthy reference distribution and quantifies subject-specific deviations to capture heterogeneous disease effects. In Alzheimer’s disease (AD), multimodal neuroimaging offers complementary signals but VAE-based normative models often (i) fit the healthy reference distribution imperfectly, inflating false positives, and (ii) use posterior aggregation (e.g., PoE/MoE) that can yield weak multimodal fusion in the shared latent space. We propose
:
This study uses data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) which are publicly available and can be requested following ADNI Data Sharing and Publications Committee guidelines: https://adni.loni.usc.edu/data-samples/access-data/. Code scripts will be made public upon acceptance.
All ADNI participants provided written informed consent, and study protocols were approved by each local site’s institutional review board.
Brain disorders such as Alzheimer’s disease (AD) affect millions of people worldwide and substantially reduce quality of life for patients and their families
Normative modeling addresses this need by learning the distribution of measurements in a reference population (typically cognitively unimpaired controls) and quantifying how much each individual deviates from that learned ”norm”
However, two persistent limitations reduce the reliability and interpretability of VAE-based normative modeling in AD. First, the normative paradigm depends on learning the healthy (training) distribution sufficiently well that reconstructions for controls are accurate; otherwise, large residuals may occur even for controls, inflating false positives and blurring the separation between normal and abnormal subjects. Empirically, standard, variational, and adversarial autoencoders can underperform on out-of-distribution (OOD) detection when normal and abnormal distributions overlap
Multimodal VAEs can, in principle, address the latter limitation, but in practice they introduce an additional challenge: how to aggregate modality-specific posteriors into a joint latent representation. Common strategies include Product-of-Experts (PoE) and Mixture-of-Experts (MoE). PoE can over-weight a highly precise modality, effectively suppressing weaker modalities and yielding brittle joint inference
To address this, our work proposes a multimodal normative modeling framework designed to (i) learn the healthy distribution more faithfully and (ii) form a more informative multimodal latent representation. We introduce a
Our contributions can be summarized as follows:
We propose
We adopt
We define
To improve interpretability, we
Overall, the goal is to assess whether mmSIVAE with MOPOE can (a) better capture the distribution of a healthy reference population, (b) reconstruct inputs more accurately from multimodal latent representations, and (c) yield deviation measures that are both discriminative for outlier detection and clinically interpretable in AD.
Let be a set of conditionally independent N modalities. The backbone of our model is a multimodal variational autoencoder (mVAE), a generative model of the form , where z is the latent variable and is the prior. mVAE optimizes the ELBO (Evidence Lower Bound) which is a combination of modality-specific likelihood distributions and the KL divergence between the approximate joint posterior and prior .
where is the reconstructed image, is the generated image, and is the hard threshold for constraining the KL divergence.
A key limitation of IntroVAE is its use of a hard threshold m to constrain the KL divergence term. Soft-Introspective VAE (SIVAE;
The product distribution required above are not in general solvable in closed form. However, if we approximate both and as Gaussian, a product of Gaussian experts is itself Gaussian with mean and variance where and are parameters of the i-th Gaussian expert and .
The challenge of approximating inference distribution with POE is that if the inference distribution for a particular modality (expert) is very sharp, then the joint distribution will be dominated by it. In other words, the overconfident but mis-calibrated experts may bias the joint posterior distribution which is undesirable for learning informative latent representations between modalities.
Another way to approximate joint inference is to use the mixture of experts (MoE) form, where the approximate posterior of the joint is represented by the sum of the unimodal posterior distributions. Since MoE is the sum of each expert and not product, the joint posterior distribution is not dominated by experts with high precision as in PoE, but spreads its density over all individual experts. In the MoE setting, each uni-modal posterior is evaluated with the generative model such that the ELBO becomes:
But this approach only takes each unimodal encoding distribution separately into account during training. The aggregation of experts in MoE does not result in a distribution that is sharper than the other experts. Therefore, even if we increase the number of experts, the shared representation does not become more informative as in PoE. Thus there is no explicit aggregation of information from multiple modalities in the latent representation for reconstruction by the decoder networks.
In this section, we introduce multimodal soft-introspective VAE (mmSIVAE), extending SIVAE (
Following Equation (), the encoder and decoder loss functions for mmSIVAE can be written as:
where , and , are hyperparameters. is defined in Equation (). can be defined similarly, but with replacing x in Equation (). Following in Equation , can be written as:
Equation represents a min-max game between the encoders and the decoders. The encoders are encouraged, via the ELBO value, to differentiate between real samples (high ELBO) and generated samples (low ELBO), while the decoders aim to generate samples that “fool” the encoders (Figure ).
We extend the nonparametric analysis of SIVAE
We write the ELBO for real samples and for generated samples as and , respectively, where denotes the collection of modality-specific decoder outputs :
In a nonparametric setting (where and can be any distributions), the encoder and decoder objectives can be written as:
Taking expectations over real samples () and generated samples () yields the complete objectives:
A Nash equilibrium satisfies and for all .
If for all such that , then an optimal encoder response satisfies , and .
Let be defined so that , where is Shannon entropy. If , then is a Nash equilibrium of Eq. ().
We adopt Mixture-of-Product-of-Experts (MOPOE), a generalization of both PoE and MoE, to aggregate unimodal posteriors into a shared latent distribution. Let denote a (non-empty) subset of the modalities and let denote the power set. We define:
MOPOE can be viewed as a hierarchical aggregation: unimodal posteriors for each subset are first merged via PoE, and subset-level approximations are then combined via MoE.
MOPOE yields a valid generalized multimodal ELBO, with PoE and MoE as special cases.
mmSIVAE (Fig. 1) uses separate encoders to map the two modalities to unimodal latent parameters (mean and variance), which are aggregated via MOPOE to obtain shared latent parameters (a joint latent distribution). The shared latent is then passed through modality-specific decoders to reconstruct each modality. We first train the model to characterize the healthy population cohort. We assume disease abnormality can be quantified by measuring how subjects deviate from the joint latent space
Here is a sample from the joint posterior for subject , and and are the mean and covariance of latent positions for healthy controls. We also derive a multivariate feature-space deviation index based on Mahalanobis distance that quantifies how reconstruction errors deviate from the reconstruction-error distribution of controls:
Here is the mean squared reconstruction error between original and reconstructed input for subject at brain region , and and denote the mean and covariance of reconstruction errors in healthy controls.
Here and are the latent value and reconstruction error for test subject at position/region , and and are computed from healthy controls.
For training, we selected 248 cognitively unimpaired (healthy control) subjects from the Alzheimer’s Disease Neuroimaging (ADNI) dataset
Each brain ROI was normalised by removing the mean and dividing by the standard deviation of the healthy control cohort brain regions. We conditioned our model on the age and sex of patients, represented as one-hot encoding vectors, to remove the effects of covariates from the MRI and PET features.
Our proposed mmSIVAE was compared with the following baselines as follows: (i) Unimodal Soft-Instrospective VAE (SIVAE)
All models were implemented in Pytorch and trained using Adam optimizer with hyperparameters as follows: epochs = 500, learning rate = , batch size = and latent dimensions in the range [5,10,15,20]. The encoder and decoder networks have 2 fully-connected layers of sizes and respectively.
Our first objective was to evaluate whether mmSIVAE achieved better reconstruction performance for cognitively unimpaired healthy controls compared to a standard vanilla VAE. Furthermore, we examined whether multimodal methods, which integrate information from multiple modalities in the latent space, demonstrated improved reconstruction accuracy compared to their unimodal counterparts. To address this, we visualized the average reconstruction errors for each cortical and subcortical region across our proposed mmSIVAE and baseline models, including SIVAE, multimodal VAE, and vanilla VAE (Figure ). For a fair comparison, both mmSIVAE and mmVAE employed the MOPOE technique to aggregate multimodal information within the latent space. In contrast, the unimodal methods, SIVAE and vanilla VAE, utilized a single encoder and decoder, with both modalities (MRI and amyloid) concatenated as a single unimodal input.
Across all brain regions, mmSIVAE demonstrated the lowest reconstruction error for both modalities when compared to the baseline models (Figure ). Introspective VAE methods, such as mmSIVAE and SIVAE, outperformed vanilla VAE approaches (mmVAE and VAE) in reconstruction accuracy. Additionally, multimodal methods (mmSIVAE and mmVAE), which integrate information from multiple modalities, exhibited lower reconstruction errors than unimodal VAEs that treated multiple modalities as a single unimodal input (Figure ).
The latent mahalanobis deviations (Eq ) and the feature mahalanobis deviations (Eq ) were estimated independently for the ADS cohort and separate holdout cohort of healthy controls with no cognitive impairment. For both deviation metrics, subjects with statistically significant () deviations
Latent deviations () consistently achieved higher likelihood ratios compared to feature space deviations (), as shown in Table A. This demonstrates that calculating deviations within the multimodal joint latent space enhances normative performance and improves outlier detection compared to regional space deviations. Among the models, mmSIVAE with MOPOE latent space aggregation achieved the highest likelihood ratios, outperforming all baselines. Introspective VAE methods (mmSIVAE and SIVAE) exhibited superior normative performance compared to vanilla VAE-based methods (mmVAE and unimodal VAE). Furthermore, in both mmSIVAE and mmVAE, MOPOE aggregation proved to be more effective for outlier detection than POE or MOE, validating its selection as the preferred aggregation strategy by taking advantage of the strengths of both approaches. In general, increasing the number of latent dimensions to 15 or 20 resulted in higher likelihood ratios, further improving performance.
A similar trend is observed for likelihood ratios generated by feature-space Mahalanobis deviations (), as shown in Table B. Our proposed mmSIVAE achieved the greatest separation between control and disease cohorts, evidenced by the highest Earth Mover’s Distance (34.043) between the distributions of the two cohorts (Figure ). Introspective VAE-based methods (mmSIVAE and SIVAE) demonstrated more pronounced separation (reduced overlap and greater distance) between control and disease cohorts compared to VAE-based methods (mmVAE and VAE), as illustrated in Figure .
Ideally, all latent dimensions can be used to reconstruct the input data and quantify feature-space deviations. The latent dimensions with statistically significant () mean absolute Z-scores (Eq ) indicate the latent dimensions which show deviation between control and disease cohorts. This can provide an interpretation of how latent space deviations can be mapped to deviations in the feature-space. We passed these selected latent vectors through the decoders setting the remaining latent dimensions and covariates to be 0 such that the reconstructions and feature deviations (Eq ) reflect only the information encoded in the selected latent vectors. From Table A, we observed mmSIVAE with MOPOE latent space aggregation had the maximum likelihood ratio for d = 15 (number of latent dimensions). We identified 5 out of 15 latent dimensions (6,7,8,14 and 15) whose mean absolute () and used them for generating the feature-space deviations (Figure A).
We visualized the pairwise differences in the magnitude of abnormal deviations () in each region between amyloid negative CU individuals and disease groups along the ADS: (i) (preclinical AD), (ii) (very mild dementia), and (iii) (mild or more severe dementia). Our aim was to validate the derived regional abnormal deviations by examining whether these deviations showed increased group differences across progressive CDR stages. We quantified group differences using Cohen’s d-statistic effect size, calculated separately for each modality. A higher effect size when comparing regional MRI volumes indicated lower gray matter volume (more atrophy). Similarly, a higher effect size when comparing amyloid or tau uptake indicated elevated SUVR uptake (higher amyloid and tau loads) compared to the amyloid-negative CU group.
Region-level group differences in MRI atrophy were most evident within the temporal, parietal and hippocampal regions which is consistent with the observations in existing literature
Our results show that mmSIVAE—combining soft-introspective training with robust multimodal posterior aggregation—improves reference-distribution fidelity (Fig. ), strengthens deviation-based separation between healthy controls and AD-spectrum cohorts (Table A; Fig. ), and yields clinically meaningful deviation measures with interpretable regional patterns (Fig. ). Below we summarize the main take-home messages and how they address gaps in multimodal normative modeling.
A central challenge in normative modeling is accurately capturing the reference (healthy) distribution: when reconstruction quality is weak on held-out controls, reconstruction-based deviation scores can inflate false positives and reduce downstream separability. Relative to standard VAE training, the introspective objective explicitly pressures the model to improve fidelity on real samples while discouraging trivial solutions that make generated samples appear “too normal,” which helps sharpen the learned reference manifold. Empirically, this is reflected in improved reconstructions on healthy controls (Fig. ) and stronger separation in deviation scores both in the latent and feature space (Table A,B; Fig. ). Methodologically, this suggests that for deviation-based pipelines, optimizing only the vanilla ELBO may be insufficient: training objectives that prioritize reference fidelity can have first-order impact on downstream outlier detection.
How the joint posterior is constructed is not a minor implementation detail: it determines whether the shared latent representation is informative for deviation scoring. PoE aggregation can yield sharp posteriors but may be dominated by a highly precise or miscalibrated modality, suppressing complementary signal and producing brittle inference
In heterogeneous disorders such as AD, subject-level deviations can capture variation that is not well summarized by coarse staging alone. Regional deviation maps provide a transparent view of how abnormalities manifest across MRI- and PET-derived features (Fig. B). Importantly, the clinical take-away is not tied to ADNI-specific labels: the broader implication is that reliable multimodal normative models can produce quantitative phenotypes for stratification, hypothesis generation, and studying links between biomarkers and behavior. More generally, the combination of (i) training for reference fidelity and (ii) robust multimodal posterior aggregation is likely to be beneficial in other healthcare settings where modalities have different noise levels, missingness patterns, or clinical relevance.
This study is evaluated on ADNI, and generalization may be affected by site/scanner effects, demographic differences, and selection bias typical of research cohorts; validating stability under explicit domain shifts and across external cohorts is an important next step. Our deviation scores and rely on Gaussian modeling of reference embeddings and reconstruction errors, and Mahalanobis distances can be sensitive to covariance estimation, especially at higher latent dimensionalities; regularized covariance estimators and uncertainty-aware deviation intervals could improve robustness. While our interpretability pipeline promotes sparsity by selecting significant latent dimensions and mapping them back to region-level effects, thresholds (e.g., ) and multiple-comparison controls influence which effects are highlighted; systematic sensitivity analyses would strengthen interpretability claims. Future work should also extend the framework to additional modalities and longitudinal trajectories, and study principled handling of modality missingness to preserve calibrated multimodal inference in real-world clinical workflows.
We denote the ELBO for real samples by and the ELBO for generated samples by :
Here, the encoder is represented by the approximate joint posterior and the decoder is represented by , with prior . Using conditional independence across modalities and linearity of expectation, the marginal distribution of generated samples for modality is .
By the Radon–Nikodym theorem, we have the equalities:
Moreover, the ELBO satisfies
where and .
We consider a nonparametric setting where and can be any distributions. For , let be a multimodal sample from . The encoder and decoder objectives are:
The complete multimodal Soft-IntroVAE objectives take expectations over real samples () and generated samples ():
A Nash equilibrium point satisfies
Given , let satisfy
Fix an with and consider maximizing the following term in over :
Let and . Then the dependence on is through , whose maximizer is when and when . Under the condition of the lemma, , so the maximum is attained at , i.e., , implying . Substituting into yields .
where is the Shannon entropy of generated samples. Let . Then is a Nash equilibrium of the min–max game in Eq. ().
Dropping the non-positive KL terms gives the upper bound
Noting and , we obtain
The final expectation is constant in , hence maximizing over is equivalent to minimizing , which is achieved by by definition. Finally, with , Lemma 1 implies is a best response to , and is a best response to , so is a Nash equilibrium.
To prove this lemma, first we need to show that is a valid multimodal ELBO, i.e. . Next we will show that MOPOE is a generalized case of both POE and MOE.
In order to prove that is a valid multimodal ELBO, we need to show that . Note that while taking all possible subsets , we denote each subset by for better readability.
MoPoE generates a generalized ELBO with PoE and MoE being special cases. The MVAE architecture proposed by Wu and Goodman, 2018 only takes into account the full subset, i.e., the PoE of all data types. Trivially, this is a MoE with only a single component as follows:
As the PoE of a single expert is just the expert itself, the MMVAE model (Shi et al., 2019) is the special case of MoPoE which takes only into account the N unimodal subsets as follows:
is equivalent to a of the N unimodal posterior approximations for .
Therefore, the proposed MoPoE-VAE is a generalized formulation of the MVAE and MMVAE, which accounts for all subsets of modalities. The identified special cases offer a new perspective on the strengths and weaknesses of prior work: previous models focus on a specific subset of posteriors, which might lead to a decreased performance on the remaining subsets. In particular, the MVAE should perform best when all modalities are present, whereas the MMVAE should be most suitable when only a single modality is observed.
ELBO
Algorithm 1 shows the detailed training procedure of mmSIVAE. Expanding mmSIVAE’s objective function, which is minimized from Equation with the complete set of hyperparameters.
where denotes the summation of reconstruction error of real samples X across the 2 modalities. denotes the KL divergence between the joint posterior and prior .
denote the summation of the reconstruction error of the generated samples across the 2 modalities. denote the KL divergence between the joint posterior of generated samples and the prior .
The hyperparameters and control the balance between inference and sampling quality respectively. When , the optimization is focused on good reconstructions, which may lead to less variability in the generated samples, as latent posteriors are allowed to be very different from the prior. When , there will be more varied samples, but reconstruction quality will degrade.
Each ELBO term in Equations and can be considered as an instance of and can have different and parameters. However, we set them all to be the same, except for the ELBO inside the exponent in Equation . For this term, controls the repulsion force of the posterior for generated samples from the prior. We denote this specific parameter as .
As reported in the SIVAE paper
Finally, we use a scaling constant s to balance between the ELBO and the expELBO terms in the loss, and we set s to be the inverse of the input dimensions
The range of the search was [0.05, 1.0] for and and [, 5] for . Following a hyperparameter tuning by grid search, the final values selected are and . As followed in the SIVAE paper, we set constant and s = 1/number of features = 1/90.