Fanyu Kong, Jingyuan Zhang, Yahui Liu, Hongzhi Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Wei Bi, Fuzheng Zhang, Guorui Zhou · ACM International Conference on Multimedia (ACM MM) 2026 · 2026
DOI: 10.1145/3767308.3836506
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
With the proliferation of media data across diverse sources, multimodal information retrieval has emerged as a trending research. However, it faces inherent challenges arising from heterogeneous data sources and complex cross-modal alignment. Although previous studies have revealed modal gaps in feature spaces, a systematic approach to address these challenges remains unexplored. In this work, we introduce Unite, a universal multimodal embedding framework that jointly supports text, image, and video modalities. We systematically investigate two critical yet underexplored aspects: data curation and modality-aware training configurations. Specifically, we provide a comprehensive analysis of how modality-specific data properties (e.g., cross-modal distributional discrepancies, varying levels of semantic granularity) systematically influence downstream task performance across diverse scenarios. Furthermore, we propose Modal-Aware Masked Contrastive Learning (MAMCL), which selectively masks non-target-modal instances based on modality-aware signals, enabling the model to learn more discriminative representations and effectively mitigating cross-modal competition. Extensive experiments on diverse multimodal retrieval benchmarks show that Unite achieves state-of-the-art performance across coarse-grained, fine-grained, and instruction-based retrieval tasks. These results demonstrate that strategic modality-specific data curation and tailored training protocols are pivotal for robust universal multimodal representation learning.
Last synced
This work has 28 recorded citations, but citing papers have not been linked locally yet.
No comments yet — start the discussion below.