Zyphra AI Introduces Zyphra Hierarchical Consideration (ZHA): A Path to True Subquadratic Consideration

Zyphra has launched Zyphra Hierarchical Consideration (ZHA), a sparse consideration methodology constructed round a tree-structured indexer. ZHA targets a recognized hole in sparse consideration: the indexer that picks tokens nonetheless scans the entire context. Zyphra’s technical paper exhibits ZHA will be distilled into frozen pretrained fashions with about 4B tokens. On Qwen3.6-35B-A3B it stays near dense consideration on reasoning, agentic coding and 256K retrieval benchmarks.

TL;DR

Dimension: It’s a methodology that provides 81,920 trainable parameters to Qwen3.6-35B-A3B (about 0.00024%). Examined as much as 256K context.
Runs on: Retrofits frozen pretrained fashions after a 4B-token KL warmup at 32K context. Speedups come from a modeled H100 BF16 roofline. No optimized kernels launched. An FP4 router variant was additionally examined.
Efficiency: Inside just a few factors of dense consideration on 7 reasoning, coding and retrieval benchmarks.
Greatest: LongBench v2 hits 63.4% (deep tree) vs 58.0% dense, utilizing about 1.1% of keys.
Worst: On NoPE-based Nemotron-3.5-Lightning-30B-A3B, deep-tree RULER-13 drops 25.3 factors under dense.

What’s Zyphra Hierarchical Consideration (ZHA)?

ZHA is a sparse consideration methodology from Zyphra that finds related keys by looking a tree. It retains the total KV cache. The tree solely decides which cached keys every question reads.

Fastened-state strategies like SSMs and sliding home windows saturate at lengthy contexts. Positive-grained sparse strategies like DeepSeek Sparse Consideration rating each previous token with an indexer. The ultimate consideration is sparse, however the indexer nonetheless grows with the total context.

Close to 1M tokens, cited work exhibits indexing takes 60 to 90% of layer time. ZHA makes the search itself sparse.

How does ZHA’s tree indexer work?

Constructing the tree

ZHA splits keys into contiguous chunks and compresses every chunk right into a abstract key. Summaries are grouped and compressed once more, degree by degree. With chunk measurement 4, 6 ranges cowl 4,096 tokens per prime node.

Summaries come from a small set of discovered reference queries. Every reference question offers a tangent-plane estimate of a bit’s log consideration mass. The max over these planes approximates how a lot consideration any future question would give the chunk.

How does top-p routing choose tokens?

At decode time, a question scores the top-level nodes. It retains the smallest set holding a fraction p of estimated consideration mass. It then expands solely these kids, repeating till it reaches uncooked KV entries.

High-p makes the finances adaptive. A single-fact lookup touches few branches. A question that wants many scattered details expands extra. The primary token is all the time saved as an consideration sink.

How does ZHA carry out in opposition to dense consideration and MSA?

Zyphra workforce transformed Qwen3.6-35B-A3B with 2 timber: deep (chunk 4, 6 ranges) and large (chunk 256, 1 degree). Each use 16 reference queries and p = 0.95. It additionally educated a MiniMax Sparse Consideration (MSA) indexer on 40B tokens, 10x ZHA’s finances.

Benchmark (Qwen3.6-35B-A3B)	Dense	ZHA deep	ZHA large	MSA
AIME 2026	93.1	91.2	90.8	91.7
SWE-bench Verified	67.4	67.4	68.4	63.6
Terminal-Bench 2.1	49.9	46.7	53.3	46.1
MMLU-Professional	84.9	85.3	85.2	85.3
GPQA Diamond	84.8	85.0	85.0	83.1
RULER-13 (256K)	96.0	93.3	96.2	77.1
LongBench v2 (pondering)	58.0	63.4	62.4	49.3

Supply: arXiv 2610.42112, Determine 5. All numbers self-reported by Zyphra.

The most important hole is on retrieval. MSA drops to 77.1% on RULER-13, whereas each ZHA timber keep above 93%. On LongBench v2, the deep tree and MSA learn related key shares (1.1% vs 1%). The deep tree scores 14.1 factors increased.

Throughout 28 tree geometries, RULER accuracy held regular when attending to 10 to 30% of keys. At mounted compression, deeper timber saved accuracy at increased sparsity. A 4-parameter mannequin predicts RULER accuracy from geometry with 1.44-point RMSE.

<div class="hd"><div><p class="tag">Zyphra Analysis · Explainer</p><h3>How ZHA searches the KV cache with a tree</h3><p class="sub">Zyphra Hierarchical Consideration, evaluated on Qwen3.6-35B-A3B (arXiv 2610.42112)</p></div></div>

<div class="tabs">
<button class="tb on" data-p="p1">1 · The way it works</button>
<button class="tb" data-p="p2">2 · Benchmarks</button>
<button class="tb" data-p="p3">3 · Keys attended</button>
<button class="tb" data-p="p4">4 · Pace mannequin</button>
<button class="tb" data-p="p5">5 · Caveats</button>
</div>

<div class="pn on" id="zp1"><div class="field">
<svg id="ztree" viewBox="0 0 800 300" position="img" aria-label="ZHA tree routing diagram"></svg>
<p class="cap" id="zcap"></p>
<div class="ctrl"><button class="btn" id="zprev"> Again</button><button class="btn pri" id="znext">Subsequent step </button><button class="btn" id="zplay">Auto-play</button><div class="dots" id="zdots"></div></div>
</div></div>

<div class="pn" id="zp2"><div class="field">
<div class="sel" id="zbsel"></div>
<div id="zbars"></div>
<p class="word">Qwen3.6-35B-A3B, p = 0.95, R = 16. Deep = chunk 4, 6 ranges. Vast = chunk 256, 1 degree. MSA = top-16 blocks of 128, educated on 40B tokens. Self-reported by Zyphra.</p>
</div></div>

<div class="pn" id="zp3"><div class="field">
<div class="sel" id="zksel"></div>
<div id="zkbars"></div>
<p class="word">Share of KV keys attended throughout decoding Decrease is sparser. High-p lets the deep tree learn far fewer keys on retrieval than on diffuse reasoning duties.</p>
</div></div>

<div class="pn" id="zp4"><div class="field">
<svg id="zlat" viewBox="0 0 800 300" position="img" aria-label="Modeled decode latency vs sequence size"></svg>
<div class="grid2" model="margin-top:10px">
<div class="kpi"><div class="n">~48K</div><div class="t">tokens the place modeled ZHA overtakes dense consideration</div></div>
<div class="kpi"><div class="n">~0.48M</div><div class="t">tokens the place modeled ZHA overtakes MSA</div></div>
<div class="kpi"><div class="n">373 to 417x</div><div class="t">projected speedup over dense at 1B tokens</div></div>
<div class="kpi"><div class="n">52 to 58x</div><div class="t">projected speedup over MSA at 1B tokens</div></div>
</div>
<p class="word">Roofline mannequin: H100, BF16, 1 layer, batch measurement 1, ZHA R = 1, C = 8, p = 0.75. Not measured kernels. Curves are schematic, anchored to the reported crossovers.</p>
</div></div>

<div class="pn" id="zp5"><div class="lim" id="zlim">
<div class="li"><b>Speedups are modeled, not measured.</b><div class="d">Zyphra launched no optimized kernels. Latency past 256K comes from a roofline projection. At 256K, frontier settings mannequin just one.93x to 2.28x over dense.</div></div>
<div class="li"><b>NoPE fashions lose retrieval accuracy.</b><div class="d">On Nemotron-3.5-Lightning-30B-A3B, RULER-13 at p = 0.95 drops 25.3 factors (deep) and 10.2 factors (large) under dense.</div></div>
<div class="li"><b>High-p should scale with depth.</b><div class="d">A set per-level p shrinks efficient protection as timber develop deeper. Too excessive a p removes pruning and returns to quadratic value.</div></div>
<div class="li"><b>Small fashions wrestle.</b><div class="d">Fashions beneath 1B parameters transformed poorly. Outcomes improved sharply at 4B and above.</div></div>
<div class="li"><b>Terminal-Bench 2.1 dip.</b><div class="d">The deep tree scores 46.7% vs 49.9% dense. The large tree scores 53.3%.</div></div>
</div><p class="word">Faucet a card to increase.</p></div>

<div class="ft"><span>Sources: <a href="https://arxiv.org/abs/2610.42112" goal="_blank" rel="noopener">arXiv 2610.42112</a> · <a href="https://zyphra.com/our-work/zyphra-hierarchical-attention" goal="_blank" rel="noopener">Zyphra weblog</a></span><span class="mtp">© Marktechpost</span></div>

/* ———- tree ———- */
var NS=’http://www.w3.org/2000/svg’,T=R.querySelector(‘#ztree’);
operate el(n,a,p)
var L3=[],L2=[],L1=[],ED=[];
var y3=40,y2=125,y1=205,y0=270;
for(var i=0;i<4;i++){L3.push({x:100+i*200,y:y3});}
for(var i=0;i<16;i++){L2.push({x:40+i*48,y:y2,p:Math.flooring(i/4)});}
for(var i=0;i<64;i++){L1.push({x:28+i*11.7,y:y1,p:Math.flooring(i/4)});}
L2.forEach(operate(n,i){ED.push({a:L3[n.p],b:n,lv:2,ci:i,pi:n.p})});
L1.forEach(operate(n,i){ED.push({a:L2[n.p],b:n,lv:1,ci:i,pi:n.p})});
ED.forEach(operate(e){e.el=el(‘line’,{x1:e.a.x,y1:e.a.y,x2:e.b.x,y2:e.b.y,stroke:’#2a2a2a’,’stroke-width’:1,class:’edge’})});
var kv=[];for(var i=0;i<64;i++){kv.push(el(‘rect’,{x:24+i*11.7,y:y0-8,width:8,peak:16,rx:2,fill:’#222′,class:’node’}))}
L1.forEach(operate(n){n.el=el(‘circle’,{cx:n.x,cy:n.y,r:4,fill:’#333′,class:’node’})});
L2.forEach(operate(n){n.el=el(‘circle’,{cx:n.x,cy:n.y,r:8,fill:’#333′,class:’node’})});
L3.forEach(operate(n){n.el=el(‘circle’,{cx:n.x,cy:n.y,r:14,fill:’#333′,class:’node’})});
[‘Level 3′,’Level 2′,’Level 1′,’KV cache’].forEach(operate(t,i){var t1=el(‘textual content’,{x:4,y:[y3,y2,y1,y0][i]-20,fill:’#9a9a96′,’font-size’:11,’font-family’:’Inter,sans-serif’});t1.textContent=t;});
var q=el(‘g’,{});var qc=el(‘circle’,{cx:400,cy:-30,r:13,fill:’#E7620F’},q);var qt=el(‘textual content’,{x:400,y:-26,’text-anchor’:’center’,’font-size’:12,’font-weight’:700,fill:’#0A0A0A’,’font-family’:’Inter,sans-serif’},q);qt.textContent=”q”;
var K3=[1],K2=[5,6],K1=[21,22,25],keep0={};K1.forEach(operate(c){for(var j=0;j<4;j++)keep0[c*4+j]=1});
var C={off:’#333′,sc:’#5a3a22′,kp:’#E7620F’,e:’#2a2a2a’,ek:’#E7620F’};
var caps=[
‘<b>Step 1 · Build.</b> Raw keys are grouped into chunks. Each chunk is compressed into summary keys using 16 learned reference queries. Summaries are compressed again, level by level.’,
‘<b>Step 2 · Score the top.</b> A new query scores only the top-level summaries, not every token.’,
‘<b>Step 3 · Top-p prune.</b> It keeps the smallest set of nodes holding a fraction p of estimated attention mass. The rest are dropped as whole regions.’,
‘<b>Step 4 · Descend.</b> Only children of kept nodes are scored. Branches can sit far apart in the context.’,
‘<b>Step 5 · Attend.</b> Exact attention runs over the selected raw KV entries, plus the first token (sink) and the current chunk. The full KV cache is still stored.’];
var st=0,dots=R.querySelector(‘#zdots’);caps.forEach(operate(){var d=doc.createElement(‘span’);d.className=”dot”;dots.appendChild(d)});
operate paint(){
var all=[].concat(L3,L2,L1);all.forEach(operate(n){n.el.setAttribute(‘fill’,C.off);n.el.setAttribute(‘opacity’,1)});
ED.forEach(operate(e){e.el.setAttribute(‘stroke’,C.e);e.el.setAttribute(‘opacity’,1)});
kv.forEach(operate(ok,i){ok.setAttribute(‘fill’,’#222′)});
qc.setAttribute(‘cy’,-30);qt.setAttribute(‘y’,-26);
if(st===0){all.forEach(operate(n){n.el.setAttribute(‘fill’,’#4a4a46′)});kv.forEach(operate(ok){ok.setAttribute(‘fill’,’#5a5a56′)});}
if(st>=1){qc.setAttribute(‘cy’,14);qt.setAttribute(‘y’,18);L3.forEach(operate(n){n.el.setAttribute(‘fill’,C.sc)});}
if(st>=2){L3.forEach(operate(n,i){if(K3.indexOf(i)>=0)n.el.setAttribute(‘fill’,C.kp);else n.el.setAttribute(‘opacity’,.35)});}
if(st>=3){L2.forEach(operate(n,i){if(K3.indexOf(n.p)>=0)n.el.setAttribute(‘fill’,K2.indexOf(i)>=0?C.kp:C.sc);else n.el.setAttribute(‘opacity’,.25)});
ED.forEach(operate(e){if(e.lv===2&&K3.indexOf(e.pi)>=0&&K2.indexOf(e.ci)>=0)e.el.setAttribute(‘stroke’,C.ek);});
L1.forEach(operate(n,i){if(K2.indexOf(n.p)>=0)n.el.setAttribute(‘fill’,K1.indexOf(i)>=0?C.kp:C.sc);else n.el.setAttribute(‘opacity’,.25)});
ED.forEach(operate(e){if(e.lv===1&&K2.indexOf(e.pi)>=0&&K1.indexOf(e.ci)>=0)e.el.setAttribute(‘stroke’,C.ek);});}
if(st>=4){kv.forEach(operate(ok,i));}
R.querySelector(‘#zcap’).innerHTML=caps[st];
dots.querySelectorAll(‘.dot’).forEach(operate(d,i){d.classList.toggle(‘on’,i===st)});
}
var tm=null;
R.querySelector(‘#znext’).onclick=operate(){st=(st+1)%caps.size;paint()};
R.querySelector(‘#zprev’).onclick=operate(){st=(st+caps.length-1)%caps.size;paint()};
R.querySelector(‘#zplay’).onclick=operate(){var b=this;if(tm){clearInterval(tm);tm=null;b.textContent=”Auto-play”;return}b.textContent=”Pause”;tm=setInterval(operate(){st=(st+1)%caps.size;paint()},2200)};
paint();

/* ———- benchmarks ———- */
var BD={‘AIME 2026′:[93.1,91.2,90.8,91.7],’SWE-bench Verified’:[67.4,67.4,68.4,63.6],’Terminal-Bench 2.1′:[49.9,46.7,53.3,46.1],’MMLU-Professional’:[84.9,85.3,85.2,85.3],’GPQA Diamond’:[84.8,85.0,85.0,83.1],’RULER-13 (256K)’:[96.0,93.3,96.2,77.1],’LongBench v2′:[58.0,63.4,62.4,49.3]};
var SN=[‘Dense’,’ZHA deep’,’ZHA wide’,’MSA’],SC=[‘#6b6b6b’,’#E7620F’,’#f2b27a’,’#4a7bd0′];
var cb=’RULER-13 (256K)’;
operate chips(id,keys,cur,fn){var w=R.querySelector(id);w.innerHTML=”;keys.forEach(operate(ok){var c=doc.createElement(‘button’);c.className=”chip”+(ok===cur?’ on’:”);c.textContent=ok;c.onclick=operate(){fn(ok)};w.appendChild(c)})}
operate bars(id,names,vals,cols,max,suf){var w=R.querySelector(id);w.innerHTML=”;names.forEach(operate(n,i){var r=doc.createElement(‘div’);r.className=”row”;r.innerHTML='<span class="lbl">’+n+'</span><div class="trk"><div class="bar" model="background:’+cols[i]+’"></div></div><span class="val">’+(suf===’%’&&vals[i]>=10?vals[i].toFixed(1):vals[i])+suf+'</span>’;w.appendChild(r)});
requestAnimationFrame(operate(){requestAnimationFrame(operate(){w.querySelectorAll(‘.bar’).forEach(operate(b,i){b.model.width=Math.max(1.5,vals[i]/max*100)+’%’})})})}
operate drawB(){chips(‘#zbsel’,Object.keys(BD),cb,operate(ok){cb=ok;drawB()});bars(‘#zbars’,SN,BD[cb],SC,100,’%’)}
var KD={‘AIME 2026′:[5.7,47,8.7],’MMLU-Professional’:[16,81,85],’GPQA Diamond’:[9,60,21],’RULER-13 (256K)’:[2.8,42,0.76],’LongBench v2′:[1.1,21,1]};
var ck=’RULER-13 (256K)’;
operate drawK(){chips(‘#zksel’,Object.keys(KD),ck,operate(ok){ck=ok;drawK()});bars(‘#zkbars’,[‘ZHA deep’,’ZHA wide’,’MSA’],KD[ck],[‘#E7620F’,’#f2b27a’,’#4a7bd0′],100,’%’)}

/* ———- latency ———- */
var LT=R.querySelector(‘#zlat’);
operate drawL(){LT.innerHTML=”;var x0=60,x1=780,yb=260,yt=20;
operate X(l){return x0+(l-3)/6*(x1-x0)}operate Y(l){return yb-(l+2)/5*(yb-yt)}
operate e(n,a){var t=doc.createElementNS(NS,n);for(var ok in a)t.setAttribute(ok,a[k]);LT.appendChild(t);return t}
for(var i=3;i<=9;i++){e(‘line’,{x1:X(i),y1:yt,x2:X(i),y2:yb,stroke:’#1f1f1f’});var t=e(‘textual content’,{x:X(i),y:yb+18,’text-anchor’:’center’,fill:’#9a9a96′,’font-size’:11,’font-family’:’Inter,sans-serif’});t.textContent=”10^”+i;}
var t2=e(‘textual content’,{x:x0-8,y:yt+4,’text-anchor’:’finish’,fill:’#9a9a96′,’font-size’:11,’font-family’:’Inter,sans-serif’});t2.textContent=”slower”;var t3=e(‘textual content’,{x:x0-8,y:yb,’text-anchor’:’finish’,fill:’#9a9a96′,’font-size’:11,’font-family’:’Inter,sans-serif’});t3.textContent=”sooner”;
var lab=e(‘textual content’,{x:(x0+x1)/2,y:yb+36,’text-anchor’:’center’,fill:’#9a9a96′,’font-size’:11,’font-family’:’Inter,sans-serif’});lab.textContent=”Sequence size (tokens, log scale)”;
operate curve(f,col,sprint,identify,ly){var d=”;for(var l=3;l<=9;l+=.05){d+=(d?’L’:’M’)+X(l).toFixed(1)+’ ‘+Y(f(l)).toFixed(1)}var p=e(‘path’,);var len=p.getTotalLength();if(!sprint){p.model.strokeDasharray=len;p.model.strokeDashoffset=len;p.getBoundingClientRect();p.model.transition=’stroke-dashoffset 1.4s ease’;p.model.strokeDashoffset=0;}var tt=e(‘textual content’,{x:X(9)-14,y:ly,’text-anchor’:’finish’,fill:col,’font-size’:12,’font-weight’:700,’font-family’:’Inter,sans-serif’});tt.textContent=identify;}
var c48=Math.log10(48000),c48m=Math.log10(480000);
operate zha(l){return -1.35+0.1*(l-3)+0.035*(l-3)*(l-3)}
operate dense(l){return Math.max(-1.75,zha(c48)+0.85*(l-c48))}
operate msa(l){return Math.max(-1.6,zha(c48m)+0.75*(l-c48m))}
curve(dense,’#9a9a96′,’5 5′,’Dense’,Y(dense(9))-10);curve(msa,’#4a7bd0′,’8 4′,’MSA’,Y(msa(9))-10);curve(zha,’#E7620F’,”,’ZHA’,Y(zha(9))+20);
[[c48,’~48K’],[c48m,’~0.48M’]].forEach(operate(c){e(‘line’,{x1:X(c[0]),y1:yt,x2:X(c[0]),y2:yb,stroke:’#E7620F’,’stroke-opacity’:.5,’stroke-dasharray’:’2 4′});var t=e(‘textual content’,{x:X(c[0])+4,y:yt+12,fill:’#ff8a3d’,’font-size’:11,’font-family’:’Inter,sans-serif’});t.textContent=c[1];});
}

/* ———- caveats ———- */
R.querySelectorAll(‘.li’).forEach(operate(li){li.addEventListener(‘click on’,operate(){li.classList.toggle(‘open’);setTimeout(submit,30)})});
window.addEventListener(‘load’,submit);setTimeout(submit,200);setTimeout(submit,1200);
})();
</script>
</div>
</physique></html>” model=”width:100%;max-width:880px;peak:600px;border:0;show:block;margin:0 auto;overflow:hidden” scrolling=”no” loading=”lazy”>
window.addEventListener(“message”,operate(e){if(e.information&&e.information.sort===”mtp-zha-h”){var f=doc.getElementById(“mtp-zha-frame”);if(f)f.model.peak=e.information.mtpH+”px”;}});

How briskly is ZHA?

Zyphra launched no kernels. It modeled decode latency for one layer at batch measurement 1 with a roofline. The light-weight setting makes use of 1 reference question, chunk 8 and p = 0.75.

In that mannequin, ZHA beats dense consideration close to 48K tokens and MSA close to 0.48M tokens. At 1B tokens, projected speedups attain 373 to 417x over dense and 52 to 58x over MSA. The weblog submit provides 32 to 35x and 4.7 to five.1x at 10M tokens.

At 256K, the modeled features are modest. Pareto-frontier settings attain 1.93x to 2.28x over dense, at 92.7 to 96.4% RULER-13. Routing and choice add their very own value.

How is ZHA educated onto present fashions?

The bottom mannequin stays frozen. Solely the reference queries prepare. A KL loss matches dense consideration mass per chunk at each degree. Qwen and Nemotron runs used 3.98B tokens over 1,900 steps at 32K context.

Zyphra experiences fashions beneath 1B parameters struggled throughout conversion. Outcomes improved sharply at 4B and above.

ZHA vs MSA, DSA and NSA

Characteristic	ZHA (Zyphra)	MSA (MiniMax)	DSA (DeepSeek)	NSA (DeepSeek)
Indexer	Multi-level tree of compressed keys	Flat block indexer per GQA group	Lightning indexer, token degree	Compression + choice + sliding window
Choice rule	High-p per degree (adaptive)	High-16 blocks of 128 tokens	High-2,048 tokens	16 blocks of 64 tokens, 512 window
Indexer value	Sublinear per question (claimed)	Quadratic index department	Scores each cached key	Not disclosed
Added to pretrained mannequin	Sure, ~4B tokens, spine frozen	Sure, 400B-token CPT	Continued coaching of V3.1-Terminus	Native pretraining
Largest mannequin proven	35B MoE (Qwen3.6-35B-A3B)	109B MoE, 6B lively	685B MoE	27B MoE, ~3B lively
Reported speedup	Modeled solely (H100 roofline)	14.2x prefill, 7.6x decode at 1M (H800)	Not disclosed	Speedups at 64K, determine solely