Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention

Zyphra released Zyphra Hierarchical Attention (ZHA), a sparse attention method that uses a tree-structured indexer to search the KV cache instead of scanning the full context, and its paper shows it can be distilled into frozen pretrained models with about 4B tokens. On Qwen3.6-35B-A3B, ZHA adds 81,920 trainable parameters and stays within a few points of dense attention on seven reasoning, coding and retrieval benchmarks, with the wide tree reaching 96.2% on RULER-13 at 256K versus 96.0% dense, while MSA fell to 77.1%. Zyphra released no optimized kernels, so speedups such as 373 to 417x over dense at 1B tokens come from an H100 BF16 roofline model rather than measurements.