LangBiTe: model-driven bias testing of text-to-text large language models

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Large language models (LLMs) have rapidly gained popularity across a wide range of applications, from customer support to content generation and decision support systems. However, LLMs have also been found to exhibit social biases, reflecting and amplifying stereotypes and prejudices present in their training data. Such biases can lead to harmful outcomes, particularly in sensitive domains. Addressing these risks is essential to ensure that the adoption of LLMs contributes positively to society without reinforcing existing inequalities. To facilitate a continuous, robust ethical assessment of text-to-text LLMs, we propose LangBiTe, a model-driven solution to configure and automate the testing of ethical biases. LangBiTe may uncover biases embedded within the LLM-based components of a software system and thus motivate adjustments, or the selection of a different LLM in line with the ethical requirements. The model-driven approach makes both the requirements specification and the test generation platform-independent and provides end-to-end traceability between the requirements and their assessment. Moreover, LangBiTe includes capabilities for an LLM-assisted generation of bias-testing datasets. We have implemented an open-source tool set, available on GitHub, to support the application of our approach. Finally, we present a series of case studies where LangBiTe was applied to unveil biases in several, popular online text-to-text LLMs, thus demonstrating the effectiveness of its diverse functionalities.