Designing Classifact: Towards a Transparent and Secure Platform for Stakeholder-in-the-Loop Data Annotation

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

AI systems depend on human judgment, yet many annotation workflows are either designed for data-science specialists or managed through external commercial platforms. The first approach may demand more technical skills than relevant stakeholders possess. The second can require organizations to transfer data, expertise, and governance to an outside provider. Both can limit the involvement of people who understand what data means in its real-world context. Human-in-the-loop approaches introduce human judgment. Stakeholder-in-the-loop annotation focuses on selecting and organizing people whose contextual knowledge fits the AI application. This paper presents Classifact, a transparent and secure platform for stakeholder-in-the-loop data annotation and validation. Using an interdisciplinary design-science approach, we developed Classifact with practitioners and researchers from AI and data, operational practice, UX and engagement, privacy and governance, security, and lived-experience or project settings. The platform enables non-specialists to set up missions, supports cooperation among data scientists, business or context experts, annotators, and data protection officers, and preserves data governance through role-based access, audit trails, and controlled or local deployment. Its game-like interface supports rapid and deliberate annotation tasks. Classifact supports the selection and organization of stakeholder groups for annotating AI inputs or validating AI outputs. Formative feedback and a proof-of-concept pilot assessed whether non-specialists could set up and run an annotation mission, complete the end-to-end workflow, and sustain annotation activity. In a comparison with Labelbox on the same task, Classifact produced more annotations and required less mission-setup time under the documented study conditions. The workshops provide an operational comparison under differing conditions, rather than a controlled causal comparison. The results demonstrate feasibility and practical potential. Annotation quality, stakeholder representation, inclusion, fairness, and downstream AI performance form later evaluation stages. Classifact contributes a practical design for human-centred, transparent, secure, and ethically governed stakeholder-in-the-loop data annotation.