Amazon Net Companies introduced Amazon SageMaker HyperPod Inference Gateway on September 18, 2026, a Kubernetes-native, GPU-aware routing system for big language mannequin inference that deploys as a single managed add-on for Amazon EKS on current HyperPod infrastructure. AWS stated the gateway can scale back first-token latency by as much as 82%.
The Routing Drawback Behind the Gateway
Based on AWS, default Kubernetes load-balancing algorithms similar to round-robin and least-connections don’t have any visibility into GPU state: which pods have saturated KV caches, that are mid-way by way of long-context generations, and which have already got the LoRA adapter a request wants loaded in reminiscence. The corporate stated requests pile up behind busy pods whereas idle capability stays unused, first-token latency spikes above 4 seconds throughout site visitors bursts, utilization turns into uneven and unpredictable, and operators over-provision to compensate. AWS described a situation through which a chatbot consumer ready 4.4 seconds for a primary token as an alternative sees it in underneath 800 milliseconds.
Two-Tier Structure
The gateway makes use of a two-tier design constructed on Kubernetes-native primitives. AWS stated it makes use of real-time GPU indicators to position each inference request on the best-suited pod. Tier 1 installs immediately on every HyperPod or EKS cluster because the amazon-sagemaker-hyperpod-inference add-on and consists of three elements, all constructed on the open-source Gateway API Inference Extension. Envoy Gateway, a layer-7 proxy, terminates incoming HTTPS site visitors and exposes a single non-public endpoint per cluster. The Physique-Based mostly Router inspects every incoming OpenAI-compatible request physique, extracts the mannequin subject, and routes the request to the proper mannequin pool, so one gateway can serve a number of fashions.
The Endpoint Picker consumes real-time Prometheus metrics from each model-serving pod and applies a weighted scoring algorithm throughout scorers overlaying KV cache utilization, queue depth, LoRA adapter residency, prefix cache hit fee, and operating requests. Every scorer carries a configurable weight, permitting routing conduct to be tuned for a selected workload, similar to latency-sensitive chat versus throughput-optimized batch.
Tier 2, the World Inference Router, is listed as coming quickly. AWS stated it’ll add fleet-wide coordination throughout a number of clusters and areas, with cross-cluster failover, world fee limiting, and cost-aware site visitors shaping. Tier 2 builds on prime of Tier 1, whereas every cluster’s per-cluster gateway continues to deal with native routing.
Deployment, Failure Dealing with, and Observability
Deployment consists of a single aws eks create-addon command and one declarative InferenceGatewayConfig customized useful resource that defines fashions and routing conduct, with current mannequin server deployments found by way of pod labels. AWS stated the set up requires no sidecars, no service mesh, and no software code adjustments. The gateway exposes a regular OpenAI-compatible endpoint over HTTP; in keeping with AWS, current shopper code works unchanged, with no SDK adjustments and no SigV4 signing for inference site visitors.
For workloads serving fine-tuned LoRA adapters on a shared base mannequin, the Endpoint Picker’s LoRA Affinity Scorer routes adapter requests to pods that have already got the requested adapter resident in GPU reminiscence; if no pod has it loaded, the request goes to the pod with probably the most obtainable capability. AWS stated this eliminates adapter swap latency.
Documented failure behaviors cowl pod failure, pool exhaustion, cluster failure, and regional failure. On pod failure, the Endpoint Picker excludes pods with stale metrics and routes to wholesome pods, recovering routinely when metrics resume. On pool exhaustion, the gateway returns HTTP 429 with a Retry-After header whereas autoscaling provides capability. On cluster failure, the World Inference Router detects a stale heartbeat and redirects site visitors inside 35 seconds, with gradual ramp-up when the cluster is reintroduced. On regional failure, cross-region routing prompts routinely, which AWS stated carries greater latency however no availability influence.
The gateway emits metrics on the pod, pool, cluster, and fleet ranges: KV cache utilization, queue depth, operating requests, and adapter residency by way of Prometheus on the pod degree; request totals, period histograms, and token counts by way of Prometheus and Grafana on the pool degree; common KV cache, error fee, and P99 latency by way of Amazon CloudWatch on the cluster degree; and routing selections, failover occasions, and fee restrict hits by way of CloudWatch on the fleet degree.
AWS-Reported Benchmark Outcomes
AWS stated it benchmarked 4 fashions starting from 8B to 235B parameters on p5.48xlarge situations with H100 GPUs and g5 situations with A10G GPUs. All site visitors was routed by way of inside Utility Load Balancers, matching the trail a manufacturing request travels, with a devoted shopper node group producing managed load and mannequin servers remoted on a separate server node group. Each consequence makes use of the gateway’s default routing configuration with no tuning and is measured in opposition to a Kubernetes round-robin baseline on the identical mannequin replicas, in keeping with AWS.
Within the reported outcomes, a mixed-generation GPU fleet lower time-to-first-token P95 and P99 latency by 97% every for Llama-3.1-8B, with an 8% throughput improve, and by 98% and 97% for Qwen3-32B, with a 50% throughput improve. Beneath bursty site visitors, Llama-3.1-70B posted P95 and P99 reductions of 94% and 98% with 12% greater throughput, whereas Qwen3-235B confirmed comparable P95 latency and an 89% decrease P99. With shared immediate prefixes, Llama-3.1-8B P95 and P99 latency fell 26% and 43%.
AWS stated that on a totally uniform fleet underneath regular site visitors the gateway performs on par with round-robin, and it outlined comparable outcomes as variations inside run-to-run variance. The corporate stated the enhancements are largest the place round-robin struggles most: blended {hardware}, bursty demand, and shared immediate prefixes.
Availability and Roadmap
AWS describes the gateway as conformant with the Kubernetes Gateway API and its Inference Extension, configured by way of a single customized useful resource definition, and suitable with any OpenAI-compatible mannequin server, together with vLLM, SGLang, and TGI. Administration works by way of kubectl, GitOps, Helm, and ArgoCD, with set up, upgrades, and rollback dealt with by way of the EKS add-on lifecycle.
Tier 1 per-cluster routing is obtainable as of September 18, 2026, in areas the place the inference add-on is obtainable. Past the World Inference Router, AWS’s named roadmap gadgets embody canary site visitors splitting, which is able to route a proportion of site visitors to new mannequin variations utilizing InferenceModelRewrite customized assets, and circulate management that classifies requests as Crucial, Commonplace, or Sheddable with per-band admission management.
