Kubernetes autoscaling for AI inference in 2026: HPA vs KEDA vs external metrics

Summary. In Datadog's 2025 State of Containers and Serverless report, 64% of organisations run the Horizontal Pod Autoscaler, but only 20% of those scale on anything other than CPU and memory. For AI inference that default is expensive. GPU nodes sit at roughly 5% average utilisation across production clusters, per Cast AI's 2026 optimisation data, while a single NVIDIA H100 lists at about $6.88 per GPU-hour on an AWS p5.48xlarge (eight GPUs, roughly $55 per hour, us-east-1, mid-2026). CPU rarely tracks demand for a large language model server, so a CPU-based HPA scales up after users are already queued and scales down too slowly to reclaim idle GPUs. This guide shows when to use the HPA, KEDA, external metrics, the Vertical Pod Autoscaler and Karpenter, with working YAML for queue-depth scaling and scale-to-zero on Kubernetes.

Why CPU-based autoscaling fails GPU inference

An LLM server built on vLLM is bound by GPU memory, batch occupancy and request queue depth, not by the CPU that the HPA reads by default. A pod can hold a full continuous batch, keep every user waiting, and still report 30% CPU. The HPA sees no pressure and adds no replicas. The signal that matters, the number of requests waiting in the engine, never reaches the autoscaler.

Datadog's engineering team put the problem plainly: "Many workloads are constrained by queue backlog, concurrency limits, or tail latency rather than raw CPU consumption. For these services, resource metrics don't reflect demand accurately enough, and scaling on them can mean scaling too late." That is the whole case for custom and external metrics.

The cost side is just as sharp. Pavan Madduri, a senior cloud platform engineer at Grainger and a CNCF Golden Kubestronaut, wrote on the CNCF blog in May 2026: "I wanted KEDA to scale on the signals that matter for GPU workloads: utilization, memory, temperature, and power draw." At $6.88 per GPU-hour, a handful of replicas held open overnight for no traffic is real money. Scaling GPU pods to demand, and to zero when there is none, is one of the highest-use moves in any AI infrastructure budget.

The four autoscalers, and what each one moves

Kubernetes autoscaling in 2026 is not one tool. It is a stack of four that move different things, and inference needs at least two of them working together.

The mental model that keeps teams out of trouble: KEDA or the HPA scales pods, Karpenter or the Cluster Autoscaler scales nodes, and the VPA only ever touches CPU and memory requests. Running the VPA in Auto mode on the same resource an HPA scales on will make the two fight. Keep them on separate signals.

Level 1: the HPA with resource metrics (the baseline you outgrow)

The default HPA is one manifest and no extra components. It reads CPU or memory from the metrics-server and holds a target average.

            apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-infer-cpu
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-infer
  minReplicas: 1
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
          

This is fine for a stateless API tier. For a GPU model server it is the trap described above: CPU is the wrong proxy. You keep the HPA, but you feed it a better metric.

Level 2: external and custom metrics through the Prometheus Adapter

The HPA can read two richer metric types. Custom metrics are attached to a Kubernetes object (requests per second per pod). External metrics come from outside the cluster or from an engine's own counters (a queue depth, a broker lag). The Prometheus Adapter implements both the Kubernetes Custom Metrics API and the External Metrics API, exposing any Prometheus series to the HPA.

Once the adapter is installed and mapping vllm:num_requests_waiting into the external metrics API, the HPA scales on the queue itself:

            apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-infer-queue
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-infer
  minReplicas: 1
  maxReplicas: 12
  metrics:
    - type: External
      external:
        metric:
          name: vllm_num_requests_waiting
        target:
          type: AverageValue
          averageValue: "5"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120
          

Two details make this production-grade. First, the target is five waiting requests per replica, a demand signal, not a CPU percentage. Second, the behavior block is asymmetric on purpose: scale up now (zero-second window, up to two pods a minute) because a rising queue means users are already waiting, and scale down slowly (a five-minute window, one pod every two minutes) because a cold GPU pod can take minutes to load weights. AWS documents exactly this pattern for vLLM on Amazon EKS, using queue depth as the primary demand signal and p95 latency as an SLO guardrail.

Level 3: KEDA for event-driven scaling and scale-to-zero

The HPA cannot go below one replica. For inference that runs in bursts, holding one idle H100 replica around the clock is the single biggest avoidable line on the bill. KEDA, a CNCF graduated project, closes that gap. It installs as an operator, extends the external metrics API with more than 60 built-in scalers, and manages an HPA for you behind a single ScaledObject. Set minReplicaCount: 0 and the workload drops to zero pods when the trigger is quiet.

            apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-infer
spec:
  scaleTargetRef:
    name: llm-infer
  minReplicaCount: 0
  maxReplicaCount: 12
  cooldownPeriod: 300
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        metricName: vllm_num_requests_waiting
        query: sum(vllm:num_requests_waiting)
        threshold: "5"
          

For an HTTP model endpoint, true scale-to-zero needs the KEDA HTTP add-on. It runs a proxy in the request path: when the service has zero pods, the proxy holds the incoming request, triggers a scale-up, and forwards the call once a pod is ready. That removes the "first request after idle gets a connection refused" problem, at the cost of one cold-start wait. Red Hat's validation of KServe with KEDA found the SLO-driven approach scaled to full cluster capacity to hold the latency target and reached the highest request success rate in its test, 86.9%, where Knative's concurrency metric left expensive hardware idle and still missed the target. For GPU-native signals, a KEDA external scaler can read NVIDIA utilisation, memory, temperature and power draw directly through NVML, so you can scale on the GPU itself rather than a proxy.

Level 4: node autoscaling with Karpenter

Pod autoscaling is pointless if a new replica has nowhere to land. When KEDA adds a pod and no GPU node is free, a node autoscaler must provision one. In 2026 the practical choice on AWS is Karpenter: it calls cloud APIs directly and brings a node online in roughly 45 to 90 seconds, against 4 to 8 minutes for the Cluster Autoscaler backed by traditional autoscaling groups. Karpenter also collapses the node-group sprawl that GPU, ARM and x86 mixes create into a few NodePool definitions, consolidates workloads for better bin-packing, and handles Spot with automatic fallback.

            apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference
spec:
  template:
    spec:
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g6.12xlarge", "p5.48xlarge"]
  limits:
    nvidia.com/gpu: "16"
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
          

Off AWS, or on a genuinely multi-cloud estate, the Cluster Autoscaler is still the more consistent option until Karpenter's other cloud providers mature. Either way, node scaling is the second half of the loop: KEDA scales the pods on queue depth, the node autoscaler scales the GPUs under them, and the consolidationPolicy reclaims a node the moment its pods drain. This pairing is what makes scale-to-zero actually save money instead of just parking idle pods on paid nodes.

A reference decision path

For a production LLM inference service in 2026, the combination that holds up is: KEDA on the engine's own queue-depth metric with minReplicaCount: 0, an asymmetric behaviour policy (fast up, slow down), the KEDA HTTP add-on if the endpoint must serve the first request after idle, and Karpenter provisioning GPU NodePools underneath. Keep a VPA in Off (recommendation-only) mode to right-size the CPU sidecars without fighting the HPA. Reserve the plain resource-metric HPA for the stateless CPU tiers around the model, where it does its job well.

India-specific considerations

For teams running inference from Indian regions or on rented GPU capacity, the economics are sharper still, because on-demand H100 hours are billed the same whether a replica serves traffic or idles. Scale-to-zero and consolidation matter more when capacity is scarce and reserved commitments are harder to justify. Data-handling adds a second constraint: an inference service that processes personal data must satisfy the Digital Personal Data Protection Act 2023, so autoscaling logs, request buffers in the HTTP proxy, and any queue telemetry should be scoped so they do not persist prompt content. For a wider view of rented accelerator economics, see our breakdown of India GPU cloud rental pricing and the cloud FinOps playbook for Indian teams.

How this fits the rest of your platform

Autoscaling is one layer of running AI on Kubernetes. It sits on top of the scheduling and driver work covered in our guide to GPU ResourceClaims and Dynamic Resource Allocation, and it assumes a current control plane, which is why the Kubernetes 1.35 and containerd 2.0 upgrade path matters before you tune scaling behaviour. For the capacity-planning context around all of this, our AI compute capacity guide covers how to budget GPU supply against demand.

FAQ

How eCorpIT can help

eCorpIT is a Gurugram-based, CMMI Level 5 and ISO 27001:2022 certified engineering organisation that builds and runs AI infrastructure on Kubernetes for teams that cannot afford idle GPUs. Our senior platform engineers design queue-depth and scale-to-zero autoscaling, wire up Prometheus and KEDA, and pair pod scaling with Karpenter so capacity tracks demand. If you are spending on GPU hours you are not using, talk to our platform engineering team about an autoscaling and FinOps review.

References

1. Scaling Kubernetes workloads on custom metrics, Datadog, 2026.

1. State of Containers and Serverless report, Datadog, 2025.

1. Autoscale AI inference with HPA and KEDA on Amazon EKS, AWS documentation, 2026.

1. GPU autoscaling on Kubernetes with KEDA: building an external scaler, Pavan Madduri, CNCF blog, 27 May 2026.

1. Kubernetes GPU autoscaling and the 5% utilisation problem, Cast AI, 2026.

1. Prometheus Adapter: external metrics documentation, Kubernetes SIGs, 2026.

1. Combining KServe and KEDA for optimised generative AI inference, Red Hat Developer, 21 April 2026.

1. Scale to zero using KEDA on GKE, Google Cloud documentation, 2026.

1. KEDA: Kubernetes Event-driven Autoscaling, CNCF, 2026.

1. Karpenter vs Cluster Autoscaler in 2026, Cast AI, 2026.

1. AWS H100 P5 pricing 2026, GetDeploying GPU price index, mid-2026.

_Last updated: 30 July 2026._