GLM-4.1V-Pondering: Advancing Basic-Goal Multimodal Understanding and Reasoning

Imaginative and prescient-language fashions (VLMs) play an important position in right now’s clever techniques by enabling an in depth understanding of visible content material. The complexity of multimodal intelligence duties has grown, starting from scientific problem-solving to the event of autonomous brokers. Present calls for on VLMs have far exceeded easy visible content material notion, with rising consideration on superior reasoning. Whereas latest works present that long-form reasoning and scalable RL considerably improve LLMs’ problem-solving talents, present efforts primarily give attention to particular domains to enhance VLM reasoning. The open-source group presently lacks a multimodal reasoning mannequin that outperforms conventional non-thinking fashions of comparable parameter scale throughout various duties.

Researchers from Zhipu AI and Tsinghua College have proposed GLM-4.1V-Pondering, a VLM designed to advance general-purpose multimodal understanding and reasoning. The strategy then introduces Reinforcement Studying with Curriculum Sampling (RLCS) to unlock the mannequin’s full potential, enabling enhancements throughout STEM downside fixing, video understanding, content material recognition, coding, grounding, GUI-based brokers, and lengthy doc understanding. Researchers open-sourced GLM-4.1V-9B-Pondering, which units a brand new benchmark amongst equally sized fashions. It additionally delivers aggressive, and in some instances superior efficiency in comparison with proprietary fashions like GPT-4o on difficult duties corresponding to lengthy doc understanding and STEM reasoning.

GLM-4.1V-Pondering comprises three core elements: a imaginative and prescient encoder, an MLP adapter, and an LLM decoder. It makes use of AIMv2-Large because the imaginative and prescient encoder and GLM because the LLM, changing the unique 2D convolutions with 3D convolutions for temporal downsampling. The mannequin integrates 2D-RoPE to help arbitrary picture resolutions and side ratios, and course of excessive side ratios over 200:1 and excessive resolutions past 4K. Researchers prolong RoPE to 3D-RoPE within the LLM to enhance spatial understanding in multimodal contexts. For temporal modeling in movies, time index tokens are added after every body token, with timestamps encoded as strings to assist the mannequin perceive real-world temporal gaps between frames

Throughout pre-training, the researchers use quite a lot of datasets, combining massive educational corpora with interleaved image-text information wealthy in information. By together with pure textual content information, the mannequin’s core language capabilities are preserved, leading to higher move@ok efficiency than different state-of-the-art pre-trained base fashions of comparable measurement. The supervised fine-tuning stage transforms the bottom VLM into one able to lengthy CoT inference utilizing a curated long-CoT corpus throughout verifiable, like STEM issues, and non-verifiable duties corresponding to instruction following. Lastly, the RL part employs a mixture of RLVR and RLHF to conduct large-scale coaching throughout all multimodal domains, together with STEM downside fixing, grounding, optical character recognition, GUI brokers, and plenty of extra.

GLM-4.1V-9B-Pondering outperforms all competing open-source fashions below 10B parameters in Basic VQA duties masking each single-image and multi-image settings. It achieves the best efficiency on difficult STEM benchmarks, together with MMMU_Val, MMMU_Pro, VideoMMMU, and AI2D. Within the OCR and Chart domains, the mannequin units new state-of-the-art scores on ChartQAPro and ChartMuseum. For Lengthy Doc Understanding, GLM-4.1V-9B-Pondering outperforms all different fashions on MMLongBench, whereas establishing new state-of-the-art ends in GUI Brokers and multimodal Coding duties. Lastly, the mannequin exhibits strong Video Understanding efficiency, outperforming VideoMME, MMVU, and MotionBench benchmarks.

In conclusion, researchers launched GLM-4.1V-Pondering, which represents a step towards general-purpose multimodal reasoning. Its 9B-parameter mannequin outperforms bigger fashions just like the one which exceeds 70B parameters. Nevertheless, a number of limitations stay, corresponding to inconsistent enhancements in reasoning high quality by RL, instability throughout coaching, and difficulties with advanced instances. Future developments ought to give attention to enhancing supervision and analysis of mannequin reasoning, with reward fashions evaluating intermediate reasoning steps whereas detecting hallucinations and logical inconsistencies. Furthermore, exploring methods to forestall reward hacking in subjective analysis duties is essential to realize general-purpose intelligence.

Take a look at the Paper and GitHub Web page. All credit score for this analysis goes to the researchers of this venture.

Sajjad Ansari is a closing 12 months undergraduate from IIT Kharagpur. As a Tech fanatic, he delves into the sensible functions of AI with a give attention to understanding the affect of AI applied sciences and their real-world implications. He goals to articulate advanced AI ideas in a transparent and accessible method.