MOSAIC: A Multimodal Semantic-Oriented Alignment with Integrated Contrastive Learning for Multimodal Knowledge Graph Construction from Scientific Documents

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Multimodal Knowledge Graphs (MMKGs) offer a promising paradigm for integrating heterogeneous sources into a unified, queryable, semantically structured representation. However, existing MMKG construction pipelines remain predominantly text-centric, extracting information from textual passages while leaving much of the visual and structural knowledge in scientific papers unrepresented. This produces fragmented graphs with disconnected components, isolated singleton nodes, and weak cross-modal connectivity, reducing the effectiveness of retrieval-augmented generation (RAG) systems that rely on interconnected graph traversal for multi-hop reasoning and evidence aggregation. We propose Multimodal Semantic-Oriented Alignment with Integrated Contrastive Learning (MOSAIC), an end-to-end framework for constructing semantically coherent MMKGs from heterogeneous scientific documents. MOSAIC combines four complementary contributions: (i) adaptive clustering via HDBSCAN, which infers data-driven entity boundaries without the brittle ε hyperparameters of conventional DBSCAN; (ii) a confidence-aware cross-modal alignment mechanism applying cosine-similarity gating to selectively invoke Large Language Models (LLMs), reducing spurious alignments and computational overhead; (iii) post-fusion semantic bridging, which links semantically related but structurally disconnected components through cosine-similarity-weighted bridge edges; and (iv) self-supervised contrastive embedding fine-tuning using an InfoNCE-style MultipleNegativesRankingLoss objective to specialise the embedding space for cross-modal entity representations. Empirical evaluation shows MOSAIC substantially improves graph topology and structural coherence, achieving a 110% increase in average clustering coefficient, reducing fragmentation, and strengthening intra-cluster semantic consistency. Evaluation across two challenging benchmark datasets, MMLongBench-Doc (134 documents, 1082 QA pairs) and DocBench (166 documents), shows that the MOSAIC-RAG engine significantly outperforms competing graph-based and dense retrieval systems, and that its advantage over lexical retrieval is concentrated in visually grounded questions, where it is statistically significant on both benchmarks. These results establish the effectiveness of our proposed MOSAIC framework for multimodal document understanding and its generalisability across diverse document categories.