MAROKO133 Eksklusif ai: Anthropic's Claude Code can now read your Slack messages and

🔗 Sumber: venturebeat.com

📌 MAROKO133 Hot ai: Z.ai debuts open source GLM-4.6V, a native tool-calling vision

Chinese AI startup Zhipu AI aka Z.ai has released its GLM-4.6V series, a new generation of open-source vision-language models (VLMs) optimized for multimodal reasoning, frontend automation, and high-efficiency deployment.

The release includes two models in "large" and "small" sizes:

1. GLM-4.6V (106B), a larger 106-billion parameter model aimed at cloud-scale inference

2. GLM-4.6V-Flash (9B), a smaller model of only 9 billion parameters designed for low-latency, local applications

Recall that generally speaking, models with more parameters — or internal settings governing their behavior, i.e. weights and biases — are more powerful, performant, and capable of performing at a higher general level across more varied tasks.

However, smaller models can offer better efficiency for edge or real-time applications where latency and resource constraints are critical.

The defining innovation in this series is the introduction of native function calling in a vision-language model—enabling direct use of tools such as search, cropping, or chart recognition with visual inputs.

With a 128,000 token context length (equivalent to a 300-page novel's worth of text exchanged in a single input/output interaction with the user) and state-of-the-art (SoTA) results across more than 20 benchmarks, the GLM-4.6V series positions itself as a highly competitive alternative to both closed and open-source VLMs. It's available in the following formats:

• API access via OpenAI-compatible interface

• Try the demo on Zhipu’s web interface

• Download weights from Hugging Face

• Desktop assistant app available on Hugging Face Spaces

Licensing and Enterprise Use

GLM‑4.6V and GLM‑4.6V‑Flash are distributed under the MIT license, a permissive open-source license that allows free commercial and non-commercial use, modification, redistribution, and local deployment without obligation to open-source derivative works.

This licensing model makes the series suitable for enterprise adoption, including scenarios that require full control over infrastructure, compliance with internal governance, or air-gapped environments.

Model weights and documentation are publicly hosted on Hugging Face, with supporting code and tooling available on GitHub.

The MIT license ensures maximum flexibility for integration into proprietary systems, including internal tools, production pipelines, and edge deployments.

Architecture and Technical Capabilities

The GLM-4.6V models follow a conventional encoder-decoder architecture with significant adaptations for multimodal input.

Both models incorporate a Vision Transformer (ViT) encoder—based on AIMv2-Huge—and an MLP projector to align visual features with a large language model (LLM) decoder.

Video inputs benefit from 3D convolutions and temporal compression, while spatial encoding is handled using 2D-RoPE and bicubic interpolation of absolute positional embeddings.

A key technical feature is the system’s support for arbitrary image resolutions and aspect ratios, including wide panoramic inputs up to 200:1.

In addition to static image and document parsing, GLM-4.6V can ingest temporal sequences of video frames with explicit timestamp tokens, enabling robust temporal reasoning.

On the decoding side, the model supports token generation aligned with function-calling protocols, allowing for structured reasoning across text, image, and tool outputs. This is supported by extended tokenizer vocabulary and output formatting templates to ensure consistent API or agent compatibility.

Native Multimodal Tool Use

GLM-4.6V introduces native multimodal function calling, allowing visual assets—such as screenshots, images, and documents—to be passed directly as parameters to tools. This eliminates the need for intermediate text-only conversions, which have historically introduced information loss and complexity.

The tool invocation mechanism works bi-directionally:

• Input tools can be passed images or videos directly (e.g., document pages to crop or analyze).

• Output tools such as chart renderers or web snapshot utilities return visual data, which GLM-4.6V integrates directly into the reasoning chain.

In practice, this means GLM-4.6V can complete tasks such as:

• Generating structured reports from mixed-format documents

• Performing visual audit of candidate images

• Automatically cropping figures from papers during generation

• Conducting visual web search and answering multimodal queries

High Performance Benchmarks Compared to Other Similar-Sized Models

GLM-4.6V was evaluated across more than 20 public benchmarks covering general VQA, chart understanding, OCR, STEM reasoning, frontend replication, and multimodal agents.

According to the benchmark chart released by Zhipu AI:

• GLM-4.6V (106B) achieves SoTA or near-SoTA scores among open-source models of comparable size (106B) on MMBench, MathVista, MMLongBench, ChartQAPro, RefCOCO, TreeBench, and more.

• GLM-4.6V-Flash (9B) outperforms other lightweight models (e.g., Qwen3-VL-8B, GLM-4.1V-9B) across almost all categories tested.

• The 106B model’s 128K-token window allows it to outperform larger models like Step-3 (321B) and Qwen3-VL-235B on long-context document tasks, video summarization, and structured multimodal reasoning.

Example scores from the leaderboard include:

• MathVista: 88.2 (GLM-4.6V) vs. 84.6 (GLM-4.5V) vs. 81.4 (Qwen3-VL-8B)

• WebVoyager: 81.0 vs. 68.4 (Qwen3-VL-8B)

• Ref-L4-test: 88.9 vs. 89.5 (GLM-4.5V), but with better grounding fidelity at 87.7 (Flash) vs. 86.8

Both models were evaluated using the vLLM inference backend and support SGLang for video-based tasks.

Frontend Automation and Long-Context Workflows

Zhipu AI emphasized GLM-4.6V’s ability to support frontend development workflows. The model can:

• Replicate pixel-accurate HTML/CSS/JS from UI screenshots

• Accept natural language editing commands to modify layouts

• Identify and manipulate specific UI components visually

This capability is integrated into an end-to-end visual programming interface, where the model iterates on layout, design intent, and output code using its native understanding of screen captures.

In long-document scenarios, GLM-4.6V can process up to 128,000 tokens—enabling a single inference pass across:

• 150 pages of text (input)

• 200 slide decks

• 1-hour videos

Zhipu AI reported successful use of the model in financial analysis across multi-document corpora and in summarizing full-length sports broadcasts with timestamped event detection.

Training and Reinforcement Learning

The model was trained using multi-stage pre-training followed by supervised fine-tuning (SFT) and reinforcement learning (RL). Key innovations include:

• Curriculum Sampling (RLCS): Dynamically adjusts the difficulty of training samples based on model progress

• Multi-domain reward systems: Task-specific verifiers for STEM, chart reason…

  Konten dipersingkat otomatis.

  🔗 Sumber: venturebeat.com

  ---