Two frontier open-weight fashions shipped inside a day of one another this week. Z.ai launched GLM-5.3-Flash, a 320B-parameter multimodal MoE mannequin with 18B lively parameters. Alibaba’s Qwen crew launched Qwen3.8-Flash-Subsequent, a 125B mannequin with 6B lively parameters that previews the Qwen4 structure.
The 2 groups designed these techniques independently. But their configs learn like near-copies of one another. Each use a 3:1 hybrid of linear and full consideration. Each choose context with a compressed indexer capped at 2048 tokens. Each widen the residual stream into 4 gated branches. Each prepare with the Muon optimizer, with fused parameter matrices break up earlier than orthogonalization. This text walks by way of the shared recipe, the one level of disagreement, and the one lab that dissents.
The Two Releases in Temporary
GLM-5.3-Flash is the primary natively multimodal mannequin within the GLM-5 sequence, launched below the MIT license on Hugging Face. Z.ai examined it anonymously as Ox Alpha on OpenRouter, the place it grew to become the most well-liked mannequin of the week. It was skilled on a 30T-token multimodal corpus and serves a 1M-token context window. Z.ai says it outperforms GLM-5.2 throughout benchmarks at one-tenth the value, whereas approaching Claude Opus 4.8 on coding and agentic benchmarks. Record pricing is $0.15 per million enter tokens and $0.50 per million output tokens.
Qwen3.8-Flash-Subsequent performs the function Qwen3-Subsequent performed for Qwen3.5: an early public preview of the subsequent structure household. The mannequin card lists a 125B essential mannequin plus a further 51B n-gram embedding desk, with 6B parameters activated per token. Native context is 262,144 tokens, extensible to 1M with YaRN. The Qwen crew studies that coaching required solely about one-ninth the compute of Qwen3.7-Plus. The accompanying technical report is titled “On the Design of Qwen3.8-Subsequent Structure: Analysis, Effectivity, and Coaching Stability.”
Convergence Level 1: Three of Each 4 Consideration Layers Are Linear
GLM-5.3-Flash stacks 45 layers: 34 linear-attention layers and 11 full-attention layers, per the shipped config. Qwen3.8-Flash-Subsequent stacks 48 layers in a repeating block of three Gated DeltaNet layers plus 1 Qwen Sparse Consideration layer, per the vLLM recipe. Each land on the identical 3:1 ratio.
The linear layers are a budget layers. As an alternative of a KV cache that grows with the textual content, they compress all historical past right into a fixed-size recurrent state. Compute per token stays fixed no matter context size. GLM makes use of Kimi Delta Consideration (KDA), the linear-attention design launched by Moonshot AI’s Kimi Linear, which applies a fine-grained per-channel decay gate. Qwen makes use of its personal Gated DeltaNet (GDN), which gates on the per-head degree. Totally different gating granularity, similar delta-rule household, similar job.
The remaining quarter of layers do exact long-range retrieval. GLM makes use of NoPE multi-head latent consideration (MLA) within the DeepSeek model. Qwen makes use of grouped-query consideration inside QSA. That is the place the KV cache really lives, and the place the second shared trick is available in.
Convergence Level 2: Compress 4x, Rating, Hold 2048 Tokens
Neither mannequin lets its full-attention layers attend over your complete context. Each connect a small realized indexer that scores chunks of historical past and retains solely the winners. The parameters match nearly precisely.
GLM’s sparse layers use a 32-head lightning indexer with top-2048 choice, descended from DeepSeek’s DSA. To chop indexer price at 1M-token contexts, Z.ai introduces IndexPool, which compresses 4 indexer key vectors into one by way of weighted pooling earlier than scoring. Qwen’s QSA operates at micro-block granularity: the compressed light-weight indexer scores 4-token blocks and retains the highest 512 blocks, which is precisely 2048 tokens. So each fashions compress the context 4x earlier than scoring, and each cap the eye price range at 2048 tokens. Qwen credit QSA with as much as 7.6x prefill and 4.9x decoding speedups over full consideration at 1M tokens.
The mixed impact on GLM’s facet is giant. In contrast with the total GLM-5.3 mannequin, Z.ai studies the Flash structure cuts consideration compute by roughly 3x and KV cache dimension by 4.4x, whereas almost halving lively parameters (18B vs 32B) and layer depend (45 vs 92).
Convergence Level 3: 4 Residual Streams As an alternative of One
Each fashions abandon the one residual stream that has outlined transformers since 2017. Each widen it into 4 parallel branches, with gates controlling what every block reads again and writes out.
GLM adopts Manifold-Constrained Hyper-Connections (mHC), a DeepSeek-originated design, configured with 4 branches within the shipped weights. Qwen wrote its personal variant, Gated Residual, which modulates circulation by way of 4 widened streams through an element-wise data-dependent learn gate and a per-branch scalar write gate. Per the Qwen crew, Gated Residual removes the additional branch-mixing step utilized by Hyper-Connections, lowering memory-access overhead, and the gate suppresses activation outliers effectively sufficient to permit FP8 residual storage. Notably, the Qwen crew ablated each approaches and located them roughly equal in high quality. Two labs, two implementations, one an identical conclusion: 4 gated streams beat one.
Convergence Level 4: Muon, With Fused Matrices Cut up Per Part
Each fashions prepare with the Muon optimizer. And each apply the identical refined refinement: fused projection matrices are break up into their impartial transformations earlier than Muon orthogonalizes them. Qwen paperwork splitting fused QKV, SwiGLU, and GDN projections, assigning Muon to real 2-D linear maps and AdamW to embeddings, routers, and low-rank parameters. Qwen additionally refit its scaling legal guidelines for the brand new structure and dropped batch-size warmup totally, after measuring that warmup price 18.8% extra optimizer steps with out bettering outcomes.
The place They Disagree: Positional Encoding
The one clear break up is rotary place embeddings within the full-attention layers. GLM-5.3-Flash drops them: the config units qk_rope_head_dim = 0, making its sparse MLA layers absolutely NoPE. Place data flows implicitly by way of the recurrent linear layers.
Qwen tried the identical factor and saved RoPE. In keeping with the Qwen3.8-Subsequent technical report, NoPE produced no measurable distinction throughout pre-training. The failure surfaced later: after post-training, the NoPE variant usually didn’t cease producing. That may be a helpful cautionary consequence for the sector. Pre-training loss curves can conceal behavioral defects that solely seem after RLHF-stage tuning.
The Broader Convergence, and the One Dissenter
This recipe is just not restricted to 2 labs. DeepSeek pioneered the sparse-indexer-plus-2048-budget sample with DSA in DeepSeek-V3.2-Exp, and mHC is a DeepSeek design now delivery in GLM. Moonshot’s Kimi contributed KDA, the precise linear-attention layer GLM adopted. Chinese language open fashions are visibly cross-pollinating structure elements and converging on shared settings.
The notable dissenter is MiniMax. Throughout M2 improvement, the crew extensively examined linear and sliding-window consideration at scale and located extreme deficits in multi-hop reasoning, particularly past 32K context after SFT. M2 shipped with full softmax consideration in each layer. For M3, MiniMax adopted MiniMax Sparse Consideration (MSA), which sparsifies softmax consideration through block choice however contains no linear-attention layers in any respect. So the sector has not absolutely settled. Z.ai, Qwen, DeepSeek, and Kimi are betting {that a} 3:1 linear hybrid preserves reasoning. MiniMax’s ablations say it doesn’t, no less than for his or her stack.
Key Takeaways
- GLM-5.3-Flash (34:11) and Qwen3.8-Flash-Subsequent (36:12) independently landed on the identical 3:1 linear-to-full consideration ratio.
- Each compress context 4x and cap sparse consideration at a 2048-token price range, a sample DeepSeek’s DSA began.
- Each exchange the one residual stream with 4 gated branches; Qwen ablated its Gated Residual in opposition to mHC and located them equal.
- They break up on positional encoding: GLM drops RoPE (NoPE), whereas Qwen saved it after NoPE fashions didn’t cease producing post-training.
- MiniMax is the dissenter: its scaled ablations discovered linear consideration hurts multi-hop reasoning, so M3 makes use of sparse softmax consideration solely.
Sources: GLM-5.3-Flash on Hugging Face, Z.ai GLM-5.3-Flash docs, GLM-5 Technical Report (arXiv:2602.15763), Qwen3.8-Flash-Subsequent on Hugging Face, Qwen3.8-Flash-Subsequent GitHub, NVIDIA Technical Weblog, MiniMax-M2 Report (arXiv:2605.26494), and MiniMax Sparse Consideration (arXiv:2606.13392)
Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its reputation amongst audiences.
