Lydia Klecan, Carlos R. Rivero · ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 2026
DOI: 10.1145/3799682.3840137
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
Link prediction aims to infer missing (subject, predicate, object) triples in a knowledge graph. To evaluate link prediction, each test triple is transformed into two queries: (subject, predicate, ?) and (?, predicate, object), and the rank of the corresponding correct, singleton answer is recorded. However, current empirical studies largely ignore the fact that multiple test triples may share the same (subject, predicate) or (predicate, object) pairs, resulting in queries with multiple correct answers. These settings naturally correspond to micro and macro averaging in traditional machine learning and information retrieval. Despite its importance, the macro setting has received very little attention. Prior work is limited to a single study over two benchmarking datasets, FB15K-237 and WN18RR, using a limited set of ranking metrics. Other metrics specifically designed to capture multi-answer retrieval behavior, such as Bpref and RBP, have not yet been analyzed in the macro setting. In this paper, we present an empirical study of micro and macro averaging for link prediction evaluation in knowledge graphs. We analyze nine benchmarking datasets to characterize query imbalance and the distribution of correct answers per query. We further evaluate ten pre-trained link prediction models for each dataset under both micro and macro averaging using several metrics, including Bpref and RBP. Our results show that macro averaging tends to yield higher effectiveness scores than micro averaging, consistent with previous observations on FB15K-237. Non-singleton queries, those with multiple answers, exhibit distinct ranking behavior, revealing differences in how models find correct answers beyond the first one. We conclude that macro averaging and non-singleton queries deserve more attention in link prediction evaluation.
ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 0 citations
No comments yet — start the discussion below.