Kubernetes DRA hits GA in 1.35: migrating GPU workloads off the device plugin in 2026

Summary. Dynamic Resource Allocation reached GA in Kubernetes v1.35, and NVIDIA moved its GPU driver into the Kubernetes SIGs organisation as kubernetes-sigs/dra-driver-nvidia-gpu, Apache-2.0, 672 stars and 12 releases with v0.4.1 shipping on 30 June 2026. The API group is resource.k8s.io/v1 and it introduces four objects: DeviceClass, ResourceSlice, ResourceClaim and ResourceClaimTemplate. This matters because on-demand H100 capacity ran between $3.00 and $6.98 per GPU per hour across AWS, Google Cloud and Azure as of April 2026 according to CloudZero, and the device plugin's all-or-nothing allocation is a direct cause of idle GPU time. One caveat the launch coverage missed: the driver's own README states the GPU kubelet plugin is disabled by default in the Helm chart and that GPU allocation features are not yet officially supported. Only ComputeDomains is. Read that before you plan a migration.

That gap between "GA" and "supported" is the whole story here, so start with it.

GA does not mean the GPU path is supported

Kubernetes v1.35 promoted DRA itself to GA. That is a statement about the scheduler and the resource.k8s.io/v1 API, not about any particular vendor driver.

The NVIDIA driver is a separate artifact with its own maturity. Its README, in the kubernetes-sigs organisation, splits the driver into two kubelet plugins that can be enabled or disabled independently: gpu-kubelet-plugin and compute-domain-kubelet-plugin. On ComputeDomains, which is the abstraction for Multi-Node NVLink, the README says "Officially supported." On GPUs it says the opposite: "While some GPU allocation features can be tried out, they are not yet officially supported. Hence, the GPU kubelet plugin is currently disabled by default in the Helm chart installation."

So the headline feature most teams want, fine-grained GPU allocation and sharing, is the one shipping off by default. Turning it on is a single Helm value, but you are opting into an unsupported surface, and that belongs in your risk register rather than your rollout plan.

The README also states the driver targets Kubernetes 1.32 or newer with DRA enabled, which means you do not need 1.35 to run it. You need 1.35 for the API to be GA and stable underneath you.

The four objects, and the storage analogy that makes them click

Writing on the CNCF blog on 1 July 2026, ChengHao Yang, a CNCF Ambassador, gives the cleanest mental model for the API: "DRA is modeled after Storage in Kubernetes - PersistentVolumeClaim and PersistentVolumeClaimTemplate (the latter only existing inside StatefulSet), with DeviceClass playing roughly the role of StorageClass."

ResourceSlice has a size constraint worth knowing before you plan a large node pool. Per the CNCF post, devices on the same node managed by the same driver belong to one pool, and "when the device count exceeds what fits in a single object (up to 128 entries, or 64 if any device uses taints or counters), the driver splits the Pool across multiple ResourceSlices." The scheduler uses .spec.pool.generation and .spec.pool.resourceSliceCount to decide whether it has the complete and latest device list for a node. If you are writing tooling that reads slices directly, handle the multi-slice case or you will silently see a partial inventory.

Each device in a slice carries attributes including architecture, product name and driver version, plus a capacity block. In the post's lab, an RTX A5000 reports architecture: Ampere, productName: NVIDIA RTX A5000, driverVersion: 580.126.20 and memory: 23028Mi, while a Tesla T10 reports architecture: Turing and memory: 16Gi. Those attributes are what your selectors match on.

What the manifests actually look like

Here is a ResourceClaim that two containers in the same pod can share, from the CNCF post's lab:

            apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: must-nvidia-gpu
spec:
  devices:
    requests:
    - name: gpu
      exactly:
        deviceClassName: gpu.nvidia.com
        count: 1
          

The pod then references it by name under resourceClaims, and each container opts in via resources.claims. Both containers see the same physical device.

The interesting part is selection. DRA lets you express device requirements in CEL against the attributes in the ResourceSlice, with an ordered fallback list. This request prefers an A5000 and falls back to a T10:

            apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: first-a5000
spec:
  spec:
    devices:
      requests:
      - name: gpu
        firstAvailable:
        - name: a5000
          deviceClassName: gpu.nvidia.com
          selectors:
          - cel:
              expression: device.attributes["gpu.nvidia.com"].productName == "NVIDIA RTX A5000"
        - name: fallback-t10
          deviceClassName: gpu.nvidia.com
          selectors:
          - cel:
              expression: device.attributes["gpu.nvidia.com"].productName == "Tesla T10"
          

Or match on capacity rather than model, which is usually the more durable rule:

                  - name: gpu
        firstAvailable:
        - name: gt20g
          deviceClassName: gpu.nvidia.com
          selectors:
          - cel:
              expression: device.capacity["gpu.nvidia.com"].memory.isGreaterThan(quantity("20Gi"))
          

This is the concrete win over the device plugin. Yang puts the general case plainly: "Compared to the Device Plugin, DRA now offers a much cleaner usage model that lets developers and cluster admins allocate devices more precisely. There's no longer a need to colocate the same kind of device on the same node, nor to write complex rules in nodeSelector or Affinity."

Time slicing survives the move, configured through opaque driver parameters inside the claim:

                config:
    - requests: ["ts-gpu"]
      opaque:
        driver: gpu.nvidia.com
        parameters:
          apiVersion: resource.nvidia.com/v1beta1
          kind: GpuConfig
          sharing:
            strategy: TimeSlicing
            timeSlicingConfig:
              interval: Long
          

Yang flags this one hard, and the flag matters more than the snippet: "As of June 2026, neither the NVIDIA official documentation nor the NVIDIA DRA Driver GPU wiki contains any tutorials on Time Slicing. The configuration below is adapted from demo/specs/quickstart/v1/gpu-test5.yaml, supplemented by reading parts of the source code; the Feature Gate part draws from third-party articles. The setup may change in future releases." It also requires the TimeSlicingSettings feature gate. Undocumented plus feature-gated is not a combination to build a platform commitment on this quarter.

Device plugin versus DRA

The device plugin is not deprecated by any statement in the sources here, and the CNCF post is careful about that too, calling it "legacy" and "original" while still demonstrating a fallback to its time slicing behaviour. Treat this as a migration you choose, not one you are forced into.

The two failure modes to test before you commit

The CNCF lab surfaces two behaviours that will bite a production rollout, and neither is obvious from the API docs.

Rolling updates lose your preferred device. With a Deployment using the A5000-preferred template above, deleting the pod does not get you the A5000 back. In Yang's words: "The Deployment default strategy.type is RollingUpdate; while the old Pod is Terminating, its ResourceClaim hasn't been released yet. The Deployment controller immediately creates a new Pod and a new ResourceClaim from the ResourceClaimTemplate. Since the A5000 is still held by the old Pod, the new claim falls back to T10." If your inference service has a preferred accelerator, a routine restart can quietly downgrade it. Test your update strategy against a scarce device, not an abundant one.

Unsatisfiable claims pend rather than fail. When no device matched the 20 GiB capacity selector, the pod stayed Pending with a scheduler event reading 0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 cannot allocate all claims. still not schedulable, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling. That message names claim allocation explicitly, which is a genuine improvement for debugging, but your alerting needs to treat a pending GPU pod as a page rather than a transient.

Looking further out, Yang notes that "Starting with K8s v1.36, device health reporting is also available, so Pods no longer simply show Error - we can tell whether the failure stems from the device or from the application." That is a reason to plan the DRA migration and the 1.36 upgrade as one piece of work rather than two.

The cost case, and where it is weaker than it looks

The argument for finer-grained allocation is idle capacity. CloudZero's April 2026 comparison is blunt about where GPU money goes: "GPU idle time is the single largest hidden cost in AI workloads. An H100 instance running idle overnight costs the same as one running a training job." The same piece puts hidden cost drivers at 30 to 60 percent beyond the published hourly rate.

Figures as published by CloudZero on 17 April 2026 and updated 27 April 2026, calculated from full instance prices. The same source separately answers the direct question with a narrower band, $3.00 to $6.98 per GPU per hour on demand across the three hyperscalers as of April 2026, and $1.95 to $2.50 on spot for AWS and Google Cloud. The two figures in one article are a fair warning about how much region, configuration and discount model move the number. Price your own workload.

The clearest DRA-shaped saving in that data is a packaging problem rather than a rate problem: AWS offers the A100 only in an 8-GPU configuration, p4d.24xlarge, "so you're paying for all eight GPUs even if your workload only uses one or two." Fine-grained claims and shared devices are exactly the tool for that. Erik Peterson, co-founder and CTO of CloudZero, frames the visibility gap the same way: "AI experiments in your business are icebergs."

Be honest about the limit, though. DRA improves how you request and share devices. It does not buy you capacity, and it does not reduce your hourly rate. If your GPUs are idle because a training pipeline stalls between stages, scheduling semantics will not fix that. Teams already working through GPU spend as a FinOps problem and EC2 capacity block pricing should treat DRA as one lever among several.

A migration order that does not break production

Working from the sources above, this is the sequence that carries the least risk.

1. Get to Kubernetes v1.35 first. The CNCF lab ran v1.35.3 with containerd 2.2.2 on Ubuntu 24.04. If you are still planning that upgrade, the 1.35 and containerd 2.0 migration path is the prerequisite, not a parallel task.

1. Install the driver with ComputeDomains only if you run Multi-Node NVLink, since that is the officially supported half. Leave the GPU plugin at its default until you have a test cluster.

1. On a test cluster, set gpuResourcesEnabledOverride: true and disable the device plugin in the GPU Operator values, which the lab does with devicePlugin: enabled: false. Run both paths on separate node pools rather than flipping a shared pool.

1. Port one workload that has a real device preference, so the selector logic gets exercised. Then delete its pod during a rolling update and confirm which device it lands on.

1. Only then move batch and training jobs. If those already use gang scheduling and the Workload API, validate that claim allocation and gang admission interact the way you expect under contention.

1. Keep model weight distribution out of scope for this migration. OCI image volumes for model weights is a separate change and stacking both makes a rollback ambiguous.

India-specific considerations

Indian teams renting GPU capacity rather than owning it feel the packaging problem more sharply, because the fixed-configuration instances that dominate on-demand catalogues are priced in dollars while budgets are set in rupees. A single A100 fine-tuning run that needs one device but must rent eight is the exact waste DRA addresses, and it is a larger share of a smaller budget.

The practical constraint is version lag. Managed Kubernetes offerings in Indian regions do not always reach a new minor release on the same schedule as US regions, and DRA GA requires v1.35. Check your provider's supported version list for your specific region before committing a quarter to this. If you are running self-managed clusters to get there faster, the operational load of a managed Kubernetes AI platform is worth weighing against the scheduling gain.

FAQ

How eCorpIT can help

eCorpIT runs GPU and AI workloads on Kubernetes for teams that would rather ship models than debug schedulers. Our senior engineering teams handle version upgrades, driver rollouts and the migration sequencing described above, including the test-cluster work that catches claim-release behaviour before it reaches production. If you are weighing a DRA migration against your current device plugin setup, talk to us about a scoped assessment.

References

1. Kubernetes SIGs, kubernetes-sigs/dra-driver-nvidia-gpu, README and repository, retrieved 20 July 2026.

1. ChengHao Yang, CNCF Ambassador, Understanding dynamic resource allocation in Kubernetes, CNCF blog, 1 July 2026.

1. Kubernetes, Dynamic Resource Allocation concepts, documentation.

1. Kubernetes, Kubernetes v1.33 DRA updates, Kubernetes blog.

1. Kubernetes, Kubernetes 1.35 release page.

1. NVIDIA, DRA Driver for NVIDIA GPUs documentation site, installation guide.

1. Kubernetes SIGs, gpu-kubelet-plugin source.

1. Kubernetes SIGs, compute-domain-kubelet-plugin source.

1. Kubernetes SIGs, dra-driver-nvidia-gpu releases, v0.4.1 dated 30 June 2026.

1. Lyne Carolyne, Cloud GPU Pricing Comparison: AWS Vs Azure Vs GCP For AI Workloads (2026), CloudZero, 17 April 2026, updated 27 April 2026.

1. CloudZero, A guide to AI FinOps, source of the Erik Peterson quote.

1. CNCF, Kubernetes project page.

Last updated: 20 July 2026.