Modal fusion of behaviors and semantics for sequential recommendation with language model

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Recommender systems are crucial for alleviating information overload by filtering content based on user interests and behaviors. While recent efforts to integrate language models (LMs) have shown promise in utilizing rich semantic features, key challenges in alignment and scalability persist due to the complex representation of language tokens and the sparsity of user-item interactions. To address these issues, we propose a novel LM-based recommendation framework IaaMRec, which effectively integrates users’ historical behavioral information with rich semantic knowledge. Different from prior work, IaaMRec employs an Item-Query Transformer to bridge the representational gap between behavioral and semantic modalities. Additionally, the framework is optimized through a multi-task training strategy to ensure alignment with downstream recommendation tasks. Furthermore, we design a bimodal input pipeline for processing behavioral and semantic textual data and introduce a dedicated ranker at the final stage to mitigate the inaccuracies and hallucination problems common in generative LMs. Extensive experiments on five public datasets demonstrate that IaaMRec consistently outperforms state-of-the-art methods.