BM25 and Dense Retrieval Are Complementary for Portuguese Clinical Text: An Empirical Study of Hybrid RAG Across 500 Clinical Queries

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

How should retrieval-augmented generation (RAG) systems be configured for clinical decision support in Portuguese? Despite growing deployment of LLM-based clinical tools in Latin America, no empirical study has evaluated hybrid retrieval strategies for Portuguese medical text. We present an evaluation of 500 clinical queries across 6 medical specialties against a production clinical corpus, comparing BM25, dense, and hybrid retrieval with Reciprocal Rank Fusion (RRF). Four findings emerge: (1) BM25 and hybrid retrieval surface statistically distinct document sets (McNemar chi2 = 39.55, p < 0.001), with 680 sources unique to hybrid and 467 unique to BM25 - confirming complementarity rather than redundancy; (2) dense-only retrieval fails for 22.2% of clinical queries even at the most permissive threshold, making it structurally unsuitable as the sole retrieval path; (3) authority-weighted scoring affects document ranking but not recall, suggesting that evidence hierarchy boosts should be evaluated by ranking quality rather than coverage metrics; (4) inter-annotator agreement for clinical relevance judgments between two independent LLM judges reaches kappa = 0.954, validating automated evaluation methodology for Portuguese medical text. Deterministic citation verification eliminates hallucinated references entirely (461/500 correct vs. 1/500 for prompt-based, Fisher's exact p < 0.001). These findings inform retrieval architecture decisions for clinical RAG systems operating in Portuguese and potentially other non-English clinical domains.