Semantic filtering and greedy matching for open vocabulary visual relationship detection

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Visual Relationship Detection (VRD) is essential for structured scene understanding. However, transformer-based VRD models face severe bottlenecks in open-vocabulary settings, specifically the quadratic explosion of candidate subject–object pairs and the cubic computational cost of standard Hungarian bipartite matching. This paper present a semantic filtering and greedy matching framework for open vocabulary visual relationship detection. First, we decouple spatial localization from semantic identification to enable scalable open-vocabulary detection. YOLO is deployed as a class-agnostic Region Proposal Network (RPN), while RegionCLIP provides zero-shot semantic labeling. To mitigate the combinatorial pair explosion, we employ a sentence embedding network to process natural language queries. This mechanism filters out semantically irrelevant subject–object pairs, significantly reducing the search space prior to relational reasoning. Second, we replace the conventional Hungarian matching with a lightweight greedy assignment strategy. By exploiting the sparsity of visual relationship graphs, this strategy not only decreases computational overhead but also avoids forced global assignments over noisy edges, improving prediction accuracy. Extensive experiments on COCO-Rel, GQA, Open Images Rel, and Visual Genome demonstrate that our proposed pipeline outperforms the standard matching approach, achieving higher recall, faster open-vocabulary inference (via semantic filtering), and significantly accelerated training and evaluation (via greedy matching). Ultimately, this work provides an optimization for transformer-based VRD, achieving better trade-offs between open-vocabulary generalization, computational efficiency, and prediction accuracy.