MAROKO133 Eksklusif ai: Together AI's ATLAS adaptive speculator delivers 400% inferen

📌 MAROKO133 Hot ai: Together AI's ATLAS adaptive speculator delivers 400% inf

Enterprises expanding AI deployments are hitting an invisible performance wall. The culprit? Static speculators that can't keep up with shifting workloads.

Speculators are smaller AI models that work alongside large language models during inference. They draft multiple tokens ahead, which the main model then verifies in parallel. This technique (called speculative decoding) has become essential for enterprises trying to reduce inference costs and latency. Instead of generating tokens one at a time, the system can accept multiple tokens at once, dramatically improving throughput.

Together AI today announced research and a new system called ATLAS (AdapTive-LeArning Speculator System) that aims to help enterprises overcome the challenge of static speculators. The technique provides a self-learning inference optimization capability that can help to deliver up to 400% faster inference performance than a baseline level of performance available in existing inference technologies such as vLLM.. The system addresses a critical problem: as AI workloads evolve, inference speeds degrade, even with specialized speculators in place.

The company which got its start in 2023, has been focused on optimizing inference on its enterprise AI platform. Earlier this year the company raised $305 million as customer adoption and demand has grown.

"Companies we work with generally, as they scale up, they see shifting workloads, and then they don't see as much speedup from speculative execution as before," Tri Dao, chief scientist at Together AI, told VentureBeat in an exclusive interview. "These speculators generally don't work well when their workload domain starts to shift."

The workload drift problem no one talks about

Most speculators in production today are "static" models. They're trained once on a fixed dataset representing expected workloads, then deployed without any ability to adapt. Companies like Meta and Mistral ship pre-trained speculators alongside their main models. Inference platforms like vLLM use these static speculators to boost throughput without changing output quality.

But there's a catch. When an enterprise's AI usage evolves the static speculator's accuracy plummets.

"If you're a company producing coding agents, and most of your developers have been writing in Python, all of a sudden some of them switch to writing Rust or C, then you see the speed starts to go down," Dao explained. "The speculator has a mismatch between what it was trained on versus what the actual workload is."

This workload drift represents a hidden tax on scaling AI. Enterprises either accept degraded performance or invest in retraining custom speculators. That process captures only a snapshot in time and quickly becomes outdated.

How adaptive speculators work: A dual-model approach

ATLAS uses a dual-speculator architecture that combines stability with adaptation:

The static speculator – A heavyweight model trained on broad data provides consistent baseline performance. It serves as a "speed floor."

The adaptive speculator – A lightweight model learns continuously from live traffic. It specializes on-the-fly to emerging domains and usage patterns.

The confidence-aware controller – An orchestration layer dynamically chooses which speculator to use. It adjusts the speculation "lookahead" based on confidence scores.

"Before the adaptive speculator learns anything, we still have the static speculator to help provide the speed boost in the beginning," Ben Athiwaratkun, staff AI scientist at Together AI explained to VentureBeat. "Once the adaptive speculator becomes more confident, then the speed grows over time."

The technical innovation lies in balancing acceptance rate (how often the target model agrees with drafted tokens) and draft latency. As the adaptive model learns from traffic patterns, the controller relies more on the lightweight speculator and extends lookahead. This compounds performance gains.

Users don't need to tune any parameters. "On the user side, users don't have to turn any knobs," Dao said. "On our side, we have turned these knobs for users to adjust in a configuration that gets good speedup."

Performance that rivals custom silicon

Together AI's testing shows ATLAS reaching 500 tokens per second on DeepSeek-V3.1 when fully adapted. More impressively, those numbers on Nvidia B200 GPUs match or exceed specialized inference chips like Groq's custom hardware.

"The software and algorithmic improvement is able to close the gap with really specialized hardware," Dao said. "We were seeing 500 tokens per second on these huge models that are even faster than some of the customized chips."

The 400% speedup that the company claims for inference represents the cumulative effect of Together's Turbo optimization suite. FP4 quantization delivers 80% speedup over FP8 baseline. The static Turbo Speculator adds another 80-100% gain. The adaptive system layers on top. Each optimization compounds the benefits of the others.

Compared to standard inference engines like vLLM or Nvidia's TensorRT-LLM, the improvement is substantial. Together AI benchmarks against the stronger baseline between the two for each workload before applying speculative optimizations.

The memory-compute tradeoff explained

The performance gains stem from exploiting a fundamental inefficiency in modern inference: wasted compute capacity.

Dao explained that typically during inference, much of the compute power is not fully utilized.

"During inference, which is actually the dominant workload nowadays, you're mostly using the memory subsystem," he said.

Speculative decoding trades idle compute for reduced memory access. When a model generates one token at a time, it's memory-bound. The GPU sits idle while waiting for memory. But when the speculator proposes five tokens and the target model verifies them simultaneously, compute utilization spikes while memory access remains roughly constant.

"The total amount of compute to generate five tokens is the same, but you only had to access memory once, instead of five times," Dao said.

Think of it as intelligent caching for AI

For infrastructure teams familiar with traditional database optimization, adaptive speculators function like an intelligent caching layer, but with a crucial difference.

Traditional caching systems like Redis or memcached require exact matches. You store the exact same query result and retrieve it when that specific query runs again. Adaptive speculators work differently.

"You can view it as an intelligent way of caching, not storing exactly, but figuring out some patterns that you see," Dao explained. "Broadly, we're observing that you're working with similar code, or working with similar, you know, controlling compute in a similar way. We can t…

Konten dipersingkat otomatis.

🔗 Sumber: venturebeat.com

📌 MAROKO133 Hot ai: 1,360-pound NUMO ground robot deployed by Ukraine to support t

Designed to minimize the risks faced by troops on the frontlines, Ukraine’s NUMO ground robot features a modular design that accommodates a range of attachments, from turrets and relay masts to wire cutters and mine-clearing equipment. This flexibility allows it to perform multiple roles while keeping soldiers out of harm’s way. 

Developed by Lviv-based autonomous systems company Tank Bureau, NUMO can carry up to 661 pounds of cargo and weighs 1,367 pounds when fully loaded. 

The system can be deployed in under 10 minutes, features a 9-inch ground clearance, and is capable of navigating challenging terrains such as forests, wetlands, and concealed positions, making it a versatile asset for modern combat operations.

Multiple communications and combat roles

Established in 2024, Tank Bureau specializes in designing and producing unmanned ground vehicles, operating its own research, manufacturing, and maintenance facilities throughout Ukraine. The company has announced plans to expand its lineup, aiming to introduce several additional robotic platforms by the end of the year.

The NUMO ground robot offers extended operational flexibility, with a range of up to 16 miles on two batteries and 28 miles on four, reaching speeds of 4 miles per hour. The vehicle supports six communication systems – Wi-Fi, DTC, Silvus, Starlink, LTE, and SineLink – with a maximum control range of roughly 4,921 feet. It also features the “Droid Box” module, a combined hardware and software suite that enables optional communication capabilities.

NUMO also forms the base of the Droid TW reconnaissance and assault system, equipped with either a KT-7.62 (PKT) or M2 Browning 12.7mm machine gun. The Droid TW-12.7 variant, co-developed by Tank Bureau and DevDroid, is currently deployed across 30 Ukrainian brigades.

Ground robots can deliver, evacuate, and operate in danger zones

Nazar Priymak, director of Tank Bureau, explained that ground robotic systems are designed to take on tasks traditionally handled by soldiers, including delivering supplies, performing evacuations, and operating in environments that are too dangerous for humans.

Priymak also explained that NUMO is already being successfully deployed by the military in active missions, and its formal codification underscores the system’s proven operational readiness and practical value on the battlefield.

Additionally, Ukraine has already fully integrated the DELTA battlefield management system across all levels of its Armed Forces, allowing commanders to coordinate operations more efficiently and track battlefield developments in real time. Since its deployment, the system has verified over 130,000 destroyed or damaged enemy targets, providing critical intelligence for ongoing operations and helping optimize resource allocation.

Earlier this year, Ukrainian assault forces carried out what they claim to be the world’s first fully unmanned offensive operation, capturing Russian troops using only ground robots and aerial drones. The operation took place in the Kharkiv region of eastern Ukraine, representing a milestone in the deployment of autonomous and remotely operated systems in frontline combat.

🔗 Sumber: interestingengineering.com