Yanming Kang, Giang Tran, Hans De Sterck · Neural Networks 2026 · 2026
DOI: 10.1016/j.neunet.2026.109847
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
While Transformer networks benefit from a global receptive field, their quadratic cost relative to sequence length restricts their application to long sequences and high-resolution inputs. We introduce Fast Multipole Attention (FMA), a divide-and-conquer mechanism for self-attention inspired by the Fast Multipole Method from n-body physics. FMA reduces the time and memory complexity of self-attention from 𝒪 ( 𝑛 2 ) to 𝒪 ( 𝑛 l o g 𝑛 ) and 𝒪 ( 𝑛 ) while preserving full-context interactions. FMA contains a learned hierarchy with 𝒪 ( l o g 𝑛 ) levels of resolution. In this hierarchy, nearby tokens interact at full resolution, while distant tokens engage through progressively coarser, learned basis functions. We have developed both 1D and 2D implementations of FMA for language and vision tasks, respectively. On autoregressive and bidirectional language modeling benchmarks, the 1D variant either matches or outperforms leading efficient attention baselines with substantially lower memory use. With linear complexity, the 2D variant demonstrates superior performance over strong vision transformer baselines in classification and semantic segmentation tasks. Our results confirm that the multilevel attention implemented by FMA allows Transformer-based models to scale to much longer sequences and higher-resolution inputs without loss in accuracy. This provides a principled, physics-inspired approach for developing scalable neural networks suitable for language, vision, and multimodal tasks. Our code will be available at [Details omitted for double-anonymized reviewing].
Last synced
This work has 2 recorded citations, but citing papers have not been linked locally yet.
No comments yet — start the discussion below.