When to Align, When to Predict?

Abstract

Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds and when each fails, a gap that leaves practitioners, especially in scientific domains with heterogeneous instruments and multiple levels of measurement, unable to diagnose why standard methods underperform the best single modality. We study both objectives under a spiked signal-plus-noise model with structured cross-modal nuisance correlation, the ingredient that breaks the classical recovery guarantees, and derive separation ratios that expose complementary failure modes: alignment whitens each modality and fails when nuisance is strongly correlated across views; prediction encodes whatever is cross-predictable through a one-sided whitening, with recovery governed by source-modality quality. The resulting phase diagram partitions multimodal problems into four regimes — Both, CA only, CP only, and Neither — refined by a recovery count that separates partial recovery from complete failure. We present a data-driven procedure to locate real-world datasets in this diagram using a small labeled subsample, identifying the preferred objective and prediction direction before any cross-modal training, and identifying when no objective in the CA/CP family can improve on the stronger modality alone. Experiments on synthetic data, stereo-vision benchmarks, image–caption pairs, and two real scientific domains, astronomy and single-cell multi-omics, validate the predictions in the nonlinear regime, including both faces of the Neither regime. Code to reproduce the results is available at https://github.com/IlayMalinyak/mm_align_vs_pred.

Approach

We study two objectives for multimodal representation learning. The first is cross-alignment (CA), which aligns paired samples in a shared latent space. The second is cross-prediction (CP), which predicts one modality from the other through an encoder–decoder factorization. Both are formalized below:

Let \(f_X : \mathbb{R}^{d_x} \to \mathbb{R}^k\) and \(f_Y : \mathbb{R}^{d_y} \to \mathbb{R}^k\) be encoders producing latent codes \(\mathbf{z}_x^{(i)} := f_X(\mathbf{x}_i)\) and \(\mathbf{z}_y^{(i)} := f_Y(\mathbf{y}_i)\) in a shared latent space of dimension \(k\). Let \(f_D : \mathbb{R}^k \to \mathbb{R}^{d_y}\) be a decoder. The two objectives are

\begin{aligned} \text{(CA)} \quad &\min_{f_X,\, f_Y} \; \tfrac{1}{n} \textstyle\sum_i \| f_X(\mathbf{x}_i) - f_Y(\mathbf{y}_i) \|_2^2 \quad \text{s.t.} \quad \tfrac{1}{n} \textstyle\sum_i f_X(\mathbf{x}_i)\, f_Y(\mathbf{y}_i)^\top = \mathbf{I}_k, \\[6pt] \text{(CP)} \quad &\min_{f_X,\, f_D} \; \tfrac{1}{n} \textstyle\sum_i \| \mathbf{y}_i - f_D(f_X(\mathbf{x}_i)) \|_2^2. \end{aligned}

With linear encoders \(f_X(\mathbf{x}) = W\mathbf{x}\), \(f_Y(\mathbf{y}) = V\mathbf{y}\) for CA, and linear encoder \(f_X(\mathbf{x}) = E\mathbf{x}\) and decoder \(f_D(\mathbf{z}) = D\mathbf{z}\) for CP, both objectives admit closed-form solutions expressible through the SVDs of two modality-coupling matrices:

C := \Sigma_{xx}^{-1/2}\, \Sigma_{xy}\, \Sigma_{yy}^{-1/2}, \qquad A := \Sigma_{yx}\, \Sigma_{xx}^{-1/2},

where \(\Sigma_{xx}, \Sigma_{yy}, \Sigma_{xy}\) denote the (population) (cross-)covariances.

To analyze recovery, we posit a signal-plus-noise model in which each modality decomposes into \(k\) shared signal coordinates and \(d - k\) modality-specific nuisance coordinates.

The singular values of \(C\) and \(A\) split into signal values \(\{\rho_i,\,\tau_i\}_{i=1}^{k}\) and nuisance values \(\{\nu_j,\,\xi_j\}_{j=1}^{d-k}\), given by

\rho_i = \frac{\kappa_i^2}{\sqrt{(\kappa_i^2 + \gamma_i^x)(\kappa_i^2 + \gamma_i^y)}}, \quad \tau_i = \frac{\kappa_i^2}{\sqrt{\kappa_i^2 + \gamma_i^x}}, \quad \nu_j = \frac{\eta_j}{\sqrt{\tilde{\gamma}_j^x\, \tilde{\gamma}_j^y}}, \quad \xi_j = \frac{\eta_j}{\sqrt{\tilde{\gamma}_j^x}}.

CA (resp. CP) fully recovers the shared signal subspace whenever its signal singular values exceed its nuisance singular values. Defining the ratio of signal to nuisance singular values as the separation ratios, we define 4 different recovery regimes — Both, CP only, CA only, and Neither — as described in the following table:

Region	CA recovers?	CP recovers?
Both	✓	✓
CA only	✓	✗
CP only	✗	✓
Neither	✗	✗

Phase diagram with CA and CP recovery boundaries

Each method shine and fall under different scenarios: CA is symmetric and preferable when modality-specific noise is large or uncertain. CP is assymetric, requires the correct source-target direction, and is preferable when signal is strong and the modality-specific noise is weak.
Importatnly, the Neither regime is a carachteristic of real world problems with complementary modaitlies, and low signal-to-noise ratio. This regime requires novel alignment methods, beyond simple CA or CP.

Synthetic Experiments

We validate on dSprites and Shapes3D stereo-view datasets, sweeping jitter (de-alignment) and noise parameters to move across the phase diagram. CA (VICReg) and CP (MSE reconstruction) are trained from scratch; downstream accuracy is measured via frozen linear probe.

CA vs CP accuracy across dSprites and Shapes3D configurations

Linear probe accuracy vs. nuisance alignment \(\nu_{\max} = 1 - \sigma_{\text{jitter}}\). Left: Stereo-dSprites (3-class, grayscale, 100k samples). Right: Stereo-3DShapes (4-class, RGB, 100k samples). In both settings, color represents weak noise levels. In both panels, the trade-off between the methods is clearly seen. CA (solid, circles) peaks at moderate-to-low alignment and collapses at full alignment; CP (dashed, squares) shows the opposite pattern. The crossover at \(\nu_{\max} \approx 0.8\) is consistent across datasets. Lower absolute ceilings in 3DShapes reflect the harder discrimination task.

UMAP embeddings of learned representations of the stereo-dSprites experiment (color = shape, intensity = position). From left to right: CA with aligned noise (\(\sigma_{\text{jitter}}=0\)), CA with misaligned noise (\(\sigma_{\text{jitter}}=0.5\)), CP with aligned noise, and CP with misaligned noise. All experiments have the same modality-specific noise (\(\sigma_{\text{noise}}=0.5\)). Each method succeeds exactly where the other fails, and on failures, the models learn the nuisance.

Real-World Experiments: Astrophysics (LAMOST × Kepler & TESS)

Astrophysical cross-modal results, mean ± std over 5 seeds. Best per row in **bold** (ties within seed std co-bolded). *Kepler (Both, \(\hat{\DCA}{=}1.08\), \(\hat{\DCP}{=}2.29\)):* at least one cross-modal method matches or beats the best unimodal baseline on every target; CP preserves LAMOST's ceiling where LAMOST dominates, CA captures photometric signal where LAMOST is weak. *TESS (Neither, \(\hat{\DCA}, \hat{\DCP} < 1\)):* no cross-modal method beats LAMOST-only on any target. CP_rev (photometry → spectra) fails on tasks where spectra carry the more direct signal (\(\log g\), binarity), but outperforms forward CP on age, where photometric rotation provides a more direct gyrochronological signal — the same source-quality principle in both directions.
Task	CA	CP	CP_rev	LAMOST	Photometry
LAMOST × Kepler — Both regime
Binarity (bal. acc.)	0.802±0.009	0.814±0.006	0.751±0.004	0.814	0.731
log g (R²)	0.956±0.003	0.976±0.001	0.639±0.004	0.977	0.542
Age (R²)	0.620±0.001	0.434±0.006	0.497±0.039	0.431	0.470
LAMOST × TESS — Neither regime
Binarity (bal. acc.)	0.756±0.022	0.763±0.011	0.626±0.010	0.779	0.604
log g (R²)	0.929±0.005	0.939±0.001	−0.312±0.001	0.942	−0.312
Age (R²)	0.431±0.029	0.396±0.064	−0.072±0.004	0.503	−0.037

Astrophysical phase diagnostics: CCA and CP spectra

Each panel shows the sorted singular values (gray bars, left axis) and per-component \(R^2\) against \(\log g\) (teal line, right axis) used by our phase estimation algorithm to classify components as signal (green shading) or nuisance (red shading). Dashed horizontal line: nuisance floor \(\max_j \hat{\nu}_j\) (CCA panels) or \(\max_j \hat{\xi}_j\) (\(\Ab\)-SVD panels); \(\hat{\Delta} > 1\) iff every classified signal singular value exceeds the nuisance floor. Top: LAMOST × Kepler — both decompositions have signal components above the nuisance floor (\(\hat{\DCA} = 1.08\), \(\hat{\DCP} = 2.29\); Both regime). Bottom: LAMOST × TESS — no CCA component predicts \(\log g\) above noise (\(R^2 \approx 0\) across all components, zero signal detected); the \(\Ab\)-SVD has one candidate signal component but below the nuisance floor. Both ratios fall below one (Neither regime). The contrast between the two rows — same LAMOST encoder, same protocol, different photometric instrument — shows that instrument quality determines the signal–nuisance separation and hence the regime placement.

Real-World Experiments: Single-Cell (CITE-seq RNA × Protein)

We test the framework on the NeurIPS 2021 CITE-seq bone-marrow (BMMC) dataset: ~90k cells measured jointly by RNA (frozen scGPT embeddings) and ADT surface-protein abundance (134 proteins). The task is cell-type classification (44 classes, balanced accuracy) under a donor-based split (6 train / 1 val / 2 test donors). CA is VICReg between projected RNA and ADT embeddings; CP predicts ADT from RNA, and CP_rev predicts RNA from ADT.

Our phase estimation places BMMC in the Neither regime (\(\hat{\DCA} = 0.67\), \(\hat{\DCP} = 0.26\)): both spectra show only partial recovery, with the nuisance floor set by donor/batch technical variation. The trained models agree: the ordering CA > CP > CP_rev holds on every seed, but no trained cross-modal head beats simply concatenating the frozen unimodal embeddings.

BMMC cell-type classification, balanced accuracy, mean ± std over 5 seeds. Each column is a separate training run (CA, CP, or CP_rev) with its own unimodal and frozen-concatenation baselines. *Neither* regime: the trained head stays below frozen concatenation in every run.
Representation	CA run	CP run	CP_rev run
RNA only	0.584±0.043	0.584±0.043	0.584±0.043
ADT only	0.558±0.040	0.507±0.075	0.525±0.051
Frozen concat	0.622±0.047	0.608±0.063	0.606±0.051
Trained head	0.606±0.030	0.572±0.034	0.524±0.048

Estimating Your Phase

Given paired embeddings from two modalities and labels for a small subsample, our estimator places a dataset in the phase diagram before any cross-modal training. It computes \(\hat{\DCA}\) from the CCA spectrum (the singular values of \(C\)) and \(\hat{\DCP}\) from the singular values of \(A = \Sigma_{yx}\Sigma_{xx}^{-1/2}\). Each component is classified as signal or nuisance by its cross-validated \(R^2\) against the labels. A ratio above one means that objective recovers the signal. The recovery count \(\hat{r}\) separates partial recovery from complete failure.

import sys; sys.path.insert(0, "src")
from analyze_phase_diagram import estimate_parameters

# X: (n, d_x) source embeddings, Y: (n, d_y) target embeddings (paired rows, centered)
# labels: (n, L) one-hot or continuous labels for the same samples
res = estimate_parameters(X, Y, labels, n_components=50)
print(res["delta_ca_direct"], res["delta_cp_direct"])   # > 1  =>  that objective recovers
print(res["r_hat_ca"], res["r_hat_cp"])                 # recovery counts

Try it interactively in the Colab notebook .