Liquid AI has launched two open-weight bidirectional encoders, LFM2.5-Encoder-230M and LFM2.5-Encoder-350M. Each are masked language fashions constructed on the LFM2 hybrid spine. Each carry an 8,192-token context.
Encoders sit beneath classifiers, intent routers, security filters, and PII detectors. These jobs run constantly, normally with out a GPU, and more and more on longer inputs. BERT established the category. ModernBERT pushed its accuracy, velocity, and context. Liquid AI’s argument is that the LFM2 structure continues that line, as a result of its value grows extra slowly as inputs get longer.
How a decoder turned an encoder
The encoders are usually not skilled from scratch. They’re initialized from the LFM2.5-230M and LFM2.5-350M decoder backbones, then transformed with three modifications:
- First, the causal consideration masks is changed with a bidirectional one, so each token attends to either side.
- Second, the LFM2 brief convolutions are made non-causal utilizing symmetric middle padding, so every token’s convolution mixes in neighbours on either side.
- Third, the mannequin is skilled with a masked language modeling goal at a 30% masks charge. That’s denser than BERT’s 15%, following proof Liquid AI cites {that a} greater masks charge helps at this scale.
Coaching runs in two levels:
- Stage one establishes common language competence with a short-context MLM goal on a big net corpus at 1,024 tokens.
- Stage two extends context to eight,192 tokens on the complete knowledge combine, strengthening factual, authorized, and multilingual competence.
Architecturally, the spine interleaves gated short-convolution blocks with grouped-query consideration, the identical design described within the LFM2 technical report. Each checkpoints use a hidden measurement of 1024 and a 65,536-token vocabulary, and help 15 languages. The license is the LFM Open License v1.0.
The embed beneath breaks down the conversion, the complete rating desk, the per-task scores, and the CPU latency figures.
The benchmark outcomes
Liquid AI evaluated 14 fashions on 17 duties pulled from GLUE, SuperGLUE, and multilingual classification. Each mannequin is totally fine-tuned per activity, and the reported rating is that fine-tuned mannequin’s consequence.
LFM2.5-Encoder-350M posts a 17-task imply of 81.02 (±1.00), rating fourth. The three fashions forward of it are all bigger. They’re XLM-R XL at 3.5B (83.06), ModernBERT-large at 395M (81.68), and XLM-R giant at 560M (81.34). The highest mannequin is sort of 10 instances its measurement.
LFM2.5-Encoder-230M posts 79.29 (±1.02), rating sixth. It beats ModernBERT-base at 78.19 and each EuroBERT mannequin within the desk, together with EuroBERT-610M (75.87) and EuroBERT-2.1B (72.19). Each new encoders additionally rating above Liquid AI’s personal retrieval siblings, LFM2.5-ColBERT-350M (76.18) and LFM2.5-Embedding-350M (75.68). That hole is the acknowledged cause Liquid AI constructed a general-purpose encoder as a substitute of reusing the retrievers.
The methodology is the extra fascinating half, and it’s open-sourced underneath Apache-2.0. Each mannequin is loaded with fp32 grasp weights and bf16 autocast, so the desk compares fashions quite than quantity codecs. Each mannequin makes use of the identical AdamW recipe, taken from the EuroBERT card. Studying charge is chosen per mannequin and activity throughout 10 charges and three seeds. Scores are then reported because the imply over 5 recent seeds that by no means touched choice. The transformers model is pinned to 4.56.2 so dependency drift isn’t an uncontrolled variable.
Use Circumstances and Deployment Environments
The discharge names three settings. Edge and embedded units come first. A automobile’s onboard compute or an industrial controller has no spare GPU, and can’t afford a cloud spherical journey. Regulated and on-premise methods in finance, healthcare, and authorized, the place paperwork are lengthy, delicate, and can’t depart in-house infrastructure. And high-volume cost-sensitive pipelines, the place a small encoder acts as an inexpensive first move in entrance of a bigger mannequin.
Liquid AI additionally places a helpful quantity on the context window: 8,192 tokens is roughly 13 to fifteen pages. One ahead move covers a full contract or an entire affected person report.
To indicate what a fine-tuned encoder seems like, the analysis group shipped 5 demos. Every runs in a CPU-only Hugging Face House. They cowl zero-shot immediate routing, zero-shot coverage linting, and spell checking. A PII detector handles 40 data varieties throughout 16 languages. A bonus masked-diffusion demo runs the encoder as a chatbot that generates by iteratively unmasking.
Getting them operating
Each encoders load by way of transformers. The physique is uncovered as Lfm2BidirectionalModel and masked-LM loading makes use of Lfm2BidirectionalForMaskedLM. Each are wired by way of auto_map, so trust_remote_code=True is required on each load name.
A base encoder produces general-purpose representations, not activity outputs, so fine-tuning is obligatory. Liquid AI’s fine-tuning tutorial walks by way of lengthy authorized paperwork at an 8k context configuration. The mannequin choice steerage is easy: 350M when accuracy issues most, 230M for tighter {hardware} or greater throughput.
Key Takeaways
- Liquid AI launched LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, open-weight bidirectional encoders with 8,192-token context.
- Each convert LFM2.5 decoders into encoders through bidirectional masking, non-causal brief convolutions, and 30% MLM.
- LFM2.5-Encoder-350M ranks 4th of 14 fashions at 81.02; the three above it are all bigger.
- At 8K tokens on CPU, the 230M runs one ahead move in ~28s versus over 90s for ModernBERT-base.
Try the Liquid AI weblog publish, the Hugging Face write-up, and the mannequin playing cards for LFM2.5-Encoder-230M and LFM2.5-Encoder-350M.
Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its recognition amongst audiences.
