A Comparative Study of RAG System Performance Using Milvus and Qdrant Vector Databases for Legal Information Retrieval and Question Answering in the Context of Thai Law

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

This study presents an empirical comparative evaluation of Retrieval-Augmented Generation (RAG) system performance for legal information retrieval and question answering in the context of Thai law. The main objective of this research is to analyze, evaluate, and compare the functional effectiveness of two prominent open-source vector databases, namely Milvus and Qdrant, in supporting specialized legal information retrieval and question-answering workflows. To execute a standardized evaluation, the experiment utilized an extensive benchmark dataset comprising a 3-year historical collection (2016-2018) of Thai Bar Examination essay questions in Criminal Procedure Law. These test questions were systematically evaluated against official ground-truth answer keys that were validated by legal domain experts. The complete active text of the Thai Criminal Procedure Code served as the authoritative external knowledge base for data indexing and retrieval. In terms of system architecture, the n8n low-code platform was integrated for automated workflow orchestration. The operational framework combined Google Gemini Embedding (models/gemini-embedding-001) for semantic vectorization of legal documents with the Google Gemini Chat Model (models/gemini-2.5-flash) for generative response synthesis. The qualitative and quantitative assessment of system-generated responses was conducted by legal experts across three standard criteria: (1) accuracy, completeness, and contextual relevance of the answers; (2) legal reasoning logic, supporting justification, and fact-to-statute subsumption capabilities; and (3) accuracy, precision, and completeness of statutory section citations. The experimental results indicate that both vector databases effectively enable the RAG architecture to fulfill legal information retrieval and question-answering tasks with high performance. Notably, both systems demonstrated zero statutory citation hallucinations and achieved equal accuracy in legal citation retrieval. However, a detailed comparative analysis focusing on answer accuracy, contextual relevance, and logical legal reasoning revealed that Milvus slightly outperformed Qdrant in overall performance. Based on these empirical findings, this paper offers three actionable recommendations for legal technology (LegalTech) development and software implementation: (1) System-driven tool selection, where Milvus is recommended for core systems requiring deep legal reasoning and detailed subsumption—such as judicial decision-support tools or contract analysis—while Qdrant is ideal for lightweight applications prioritizing low latency, rapid retrieval, and resource efficiency; (2) Implementation of hierarchy-aware legal chunking strategies, using section-based chunking combined with parent-child chunking techniques to preserve section-level logical context; and (3) Expansion of future research benchmarks to include substantive law codes and supreme court judgment databases to establish a sustainable evaluation framework for the Thai justice system.