llm-d 1.0 aims to ease production AI inference at scale

Contributors to the llm-d project have released llm-d 1.0, software designed to support large language model inference in production environments.

The release marks a milestone for the open-source project, founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA and now governed by the Cloud Native Computing Foundation. Its backers say the new version is aimed at organisations that need to run AI workloads at scale while controlling infrastructure use and operational complexity.

llm-d focuses on distributed inference, the part of AI deployment that serves models to users and applications once training is complete. It is built for modern infrastructure environments and supports multiple model servers, including vLLM and SGLang.

Version 1.0 adds multi-tenant quality of service, zero-downtime operations, a stable configuration API and what the project describes as agent-aware serving. The additions are intended to help operators manage fleets handling different workloads at the same time, including latency-sensitive user requests and less predictable agent-based tasks.

A central issue for AI operators is the growing resource demand of agentic workloads, where systems repeatedly access and reuse context across multiple turns and parallel tasks. In these settings, managing context can consume more resources than producing the final output, increasing pressure on latency, throughput, memory use and cost.

The software already includes prefix-cache and load-aware routing, KV cache offloading, prefill and decode disaggregation, expert parallelism, and batch and asynchronous inference. Version 1.0 is positioned as a more stable base for day-to-day production operations, including traffic management during upgrades and maintenance.

Ecosystem uptake

The project now has more than 500 contributors from more than 25 organisations, spanning model developers, cloud providers, infrastructure vendors and enterprises running distributed inference systems.

Examples released alongside the launch point to the cost and throughput gains supporters believe the software can deliver. In one use case, IBM Research said open frontier models and llm-d allowed it to explore quantum error-correction hardware at 11.5 times lower inference cost than an all-proprietary set-up, with a prefix-cache hit rate of about 98% and a blended serving cost of roughly USD $0.11 per million tokens.

In another demonstration, researchers used llm-d to deploy GLM 5.2, a roughly 753-billion-parameter open-weight model, on NVIDIA GPUs. On an agentic benchmark with hundreds of sessions running in parallel, the set-up delivered more than 6.6 million output tokens per minute at peak and served up to 3,000 concurrent coding agents with zero preemptions.

The project's contributors said that, at current cloud rental rates, the deployment would have produced per-token costs five to 10 times lower than equivalent commercial API pricing. They said the biggest savings appeared in input-heavy traffic patterns, which are common in agentic workloads.

Chris Aniszczyk, Chief Technology Officer at CNCF, described the release as part of a wider shift in cloud-native computing. "AI inference is quickly becoming a core cloud native workload. llm-d 1.0 helps organizations meet this shift at scale by bringing together open source inference capabilities with the requirements that matter in scaling production to then meet the demands of agentic workloads. It's an important step toward making high performance open source inference more operationally mature and accessible across the cloud native ecosystem," he said.

Backers from the founding organisations framed the release as a response to the operational issues that arise after initial AI experimentation. These include balancing competing traffic across shared GPU fleets, maintaining service during software changes and making better use of existing hardware.

Brian Stevens, Senior Vice President and AI Chief Technology Officer at Red Hat, said: "As organizations move beyond AI experimentation and into full-scale production, the bottleneck shifts from generating answers to managing the economics, operational reliability, and extreme infrastructure demands of complex agentic workloads. Open source collaboration is essential to solving these day-two operational challenges at scale. With llm-d 1.0, the community is delivering a standardized, hardened core for distributed inference to empower enterprises to optimize hardware performance, address operational costs, and build high-density AI architectures on an open foundation."

Industry support

Supporters also include cloud operators and software companies using the project in live environments. DigitalOcean said it relies on llm-d as distributed inference software for its serverless inference and managed agents services, while JetBrains said the software now runs in production behind its Junie Lite plan and handles a significant share of that traffic.

Some users emphasised flexibility across mixed infrastructure. Roblox said the project gives it a Kubernetes-native foundation for disaggregated serving and cache-aware routing across cloud and on-premises GPUs, without tying it to a single inference engine.

Priya Nagpurkar, Vice President of AI Native Systems at IBM Research, said: "At IBM, we see firsthand that the challenge of enterprise AI is no longer just building powerful models but running them efficiently and reliably at scale. llm-d 1.0 represents an important milestone in creating an open foundation for distributed inference to help organizations get more value from their AI infrastructure and more control on the AI capabilities they depend on. As agentic workloads become more prevalent, open technologies like llm-d will be critical to improving performance, lowering costs, and accelerating innovation across the AI ecosystem."