Tiny Language Models: A Comprehensive Study

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Large language models have pushed the boundaries of what natural language processing can do, but that progress has come at a cost: BERT, GPT-3, PaLM and their peers demand computing resources that most developers and organisations simply don't have. Tiny Language Models have emerged as a practical answer to this problem — compact, efficient versions of these architectures that can run on a phone, a microcontroller, or any other device where memory and power are tight. This paper looks at how such models are built and trained, where they are already being used, and where their limits lie. We weigh their real advantages — faster inference, a smaller memory footprint, lower energy draw — against the accuracy and generalisation they inevitably give up in exchange. Alongside a direct comparison with full-scale LLMs, we argue that TLMs have a genuine role to play in making AI accessible on modest hardware, and we close with some thoughts on where the field might go next: hybrid architectures, better quantisation, and models tuned for specific domains.