Mitigating Sample-Level Imbalance via Probabilistic Separation for Adaptive Multimodal Fusion

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2510.21797v4 Announce Type: replace-cross Abstract: Multimodal learning faces modality imbalance, where dominant modalities suppress weaker ones due to inconsistent convergence rates. Existing static or heuristic methods overlook sample-level variations in prediction bias and fail to isolate low-quality outlier samples. To address this, we propose a novel framework to quantitatively diagnose and dynamically mitigate modality imbalance at the sample level. We first introduce a Modality Gap metric to quantify prediction discrepancies between unimodal branches. Empirical analysis reveals a distinct bimodal distribution, reflecting the natural coexistence of balanced and imbalanced sample subgroups. We then employ a Gaussian Mixture Model (GMM) to model this gap distribution, leveraging Bayesian posterior probabilities for probabilistic soft separation of subgroups. Next, we construct a two-stage training framework comprising a Warm-up stage and an Adaptive Training stage. In the Adaptive Training stage, a GMM-guided Adaptive Loss dynamically reallocates optimization priorities, imposing stronger modality alignment penalties on imbalanced samples while prioritizing multimodal fusion for balanced ones. Experimental results demonstrate that our method significantly outperforms current state-of-the-art baselines. Furthermore, fine-tuning on a high-quality balanced subset filtered by the GMM serves as an effective data purification strategy.

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2510.21797
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.