The Context Collapse Paradox: Why Data Saturation Induces Cognitive Noise in Enterprise LLMs

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Modern Large Language Models (LLMs) boast effective context windows spanning from 32,000to over 1,000,000 tokens. In production enterprise architectures, this expansion has driven a naivedesign pattern: piping uncurated historical databases, transcripts, and document revisions directlyinto the active prompt. In this work, we demonstrate that uncurated context scaling triggersContext-Entropy Collapse (CEC), an architectural state where retrieval recall rises while outputfidelity severely degrades. We mathematically establish that scaling the token sequence length Nmonotonically inflates the softmax denominator, dispersing probability mass across backgroundnoise and eroding attention allocated to negative system constraints. Furthermore, we analyzevector-space collisions where cosine similarity fails to differentiate between temporally distinct records.To resolve this, we present the Dynamic Minimalist Context Filter (DMCF), an upstream,deterministic engine that executes O(1) temporal interval validation, hierarchical authority pruning,and cross-encoder score thresholding. DMCF enforces the principle of Minimum Viable Context(MVC), reducing prompt noise by up to 96.8% while guaranteeing zero-speculation adherence