Ivo Nowak · Annals of Computer Science and Information Systems 2026 · 2026
DOI: 10.15439/2026f6447
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
Reinforcement learning is typically treated as a uniform optimization process that does not explicitly exploit global structure, leading to slow and inefficient information propagation.In contrast, dynamic programming achieves high efficiency by leveraging structured propagation of value information.In this paper, we argue that such structure is implicitly encoded in its learning dynamics.Focusing on distributional reinforcement learning, we show that the temporal evolution of return variance reveals a propagation process across the state space.We introduce a temporal learning indicator t * (s) that captures when a state is most strongly affected by propagated updates.Empirically, this signal induces an ordering over states consistent with dynamic programming-style propagation, where information spreads from terminal regions outward.Based on this insight, we propose StructRL, a framework that exploits this emergent structure to guide exploration and sampling.In a deterministic single-goal gridworld setting, this leads to substantially faster and more stable convergence compared to a standard baseline.We further validate StructRL on Taxi 10×10 environment, a standard sparse-reward benchmark, achieving 169 ± 25 successful episodes out of 300 versus 13 ± 12 for a C51 baseline -a twelve-fold improvement, evaluated over 3 independent seeds.Our results support a new perspective: reinforcement learning can be understood as a structured propagation process, and learning dynamics themselves provide actionable information to control it.This opens a pathway toward more sample-and energy-efficient reinforcement learning.
No comments yet — start the discussion below.