Fikra Benchmark Phase 1: Comparative Evaluation of Four LLM Configurations Across Reasoning, Instruction Following, Mathematics, and Tool Use

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

This paper reports Phase 1 of an author-conducted evaluation of four language-model configurations served through the Fikra API; the author operates the Fikra API (see Disclosure). The evaluation uses a frozen 500-task suite assembled from MMLU-Pro, GPQA-Diamond, GSM8K, IFEval, LiveCodeBench, BFCL, and a Fikra-owned FRES set. Four configurations were evaluated under a common generation protocol, producing 2,000 planned model-task evaluations. Of these, 1,894 completed successfully (94.7%), with 395 of the 500 frozen tasks (including tasks from the two unscored components) having successful results for all four configurations. The final scored analysis covers GPQA-Diamond, GSM8K, MMLU-Pro, IFEval, and BFCL; LiveCodeBench and FRES are explicitly excluded because reliable evaluation environments were not established. The completed results show heterogeneous capability profiles rather than a single uniform ordering across task families. Observed scores range from 46-88% on GPQA-Diamond, 84-94.67% on GSM8K, 67.74-90% on MMLU-Pro, 68-86.67% on strict IFEval, and 54-86% on BFCL. Execution measurements also vary substantially, including median total latency from 2.70 s to 16.07 s across the four configurations. The study documents both capability results and practical evaluation limitations, including incomplete coverage and evaluator compatibility issues. The resulting artifact package provides the experiment manifest, benchmark artifact, raw results, diagnostics, and analysis records needed to reproduce or extend the study.Competing interests: the author operates the Fikra API evaluated in this study; the FRES set is Fikra-owned. Code and data: https://github.com/Fikra-API/fikra-benchmark