Kubernetes gang scheduling in v1.36: the Workload API rewrite your v1.35 manifests will not survive

Summary. Kubernetes v1.35 shipped the Workload API and native gang scheduling on 29 December 2025 under scheduling.k8s.io/v1alpha1. On 22 April 2026, v1.36 replaced that API group with v1alpha2 and, in the maintainers' own words, "completely replacing the previous v1alpha1 API version". The workloadRef field on the Pod is gone, swapped for schedulingGroup. podGroups became podGroupTemplates, and PodGroup is now a standalone runtime object rather than a block nested inside Workload. If you wrote manifests against the December 2025 blog post, they do not apply to the release you are probably running. The reason to care is arithmetic: an AWS p5.48xlarge carries 8 NVIDIA H100 GPUs and lists at $55.04 per hour on-demand in us-east-1, about $6.88 per GPU-hour. Start 3 of a 4-pod training gang and the 3 that did start burn $165.12 an hour producing nothing, because a distributed training job with a missing worker is not a slow job, it is a stopped one. Cast AI measured average GPU utilisation at 5% across tens of thousands of clusters in its 2026 report. Every workload-aware scheduling feature in v1.36 is alpha, so this is a staging-cluster exercise, not a production migration.

The problem gang scheduling solves

Distributed training is all-or-nothing by nature. A PyTorch or TensorFlow job with 4 workers needs all 4 running before step one begins. The default kube-scheduler does not know that. It places pods one at a time, independently, and is perfectly happy to bind 3 of your 4 workers and leave the fourth pending because the cluster ran out of GPUs.

Those 3 pods now hold GPUs. They are not training. They are waiting for a sibling that may never arrive, and while they wait they block the very resources another job would need to finish and free capacity. That is the classic resource deadlock, and on GPU hardware it is expensive in a way CPU deadlock never was.

The Kubernetes maintainers put the general case plainly in the v1.35 announcement: "Without gang scheduling, a Job might be partially scheduled, consuming resources without being able to run, leading to resource wastage and potential deadlocks."

Laurent Gil, co-founder and president of Cast AI, framed the economics in the company's 2026 report announcement on 21 April 2026: "A GPU sitting idle costs dollars per hour. A CPU sitting idle costs cents. And 95% of GPU capacity is doing nothing."

What a stall actually costs

Using the published us-east-1 on-demand rate for p5.48xlarge, 8 H100s at $55.04 per hour:

The bottom row is the entire point. Gang scheduling does not make your job faster. It makes your failures free.

The v1.35 implementation bounded this with a timeout: if only a subset of the group was scheduled within 5 minutes, the scheduler rejected all pods in the group and returned them to the queue, freeing the reserved resources. So even the first implementation capped a partial-start stall at roughly $13.76 in the 3-of-4 case, against an unbounded deadlock without it.

What v1.35 shipped

The v1.35 release, written up by Maciej Skoczeń and Dominik Marciński of Google, delivered three things.

The Workload API arrived in the scheduling.k8s.io/v1alpha1 group as a machine-readable description of how a group of pods should be scheduled, as distinct from Jobs which describe what to run. Gang scheduling arrived as an initial implementation instructing kube-scheduler to place gang pods all-or-nothing. And opportunistic batching arrived as a beta feature, on by default, which speeds up scheduling of identical pods by reusing feasibility calculations across pods with identical scheduling requirements. Opportunistic batching needs no Workload API and no opt-in, so most clusters on v1.35 got it for free.

The v1.35 gang mechanism held pods at a Permit gate. The scheduler blocked pods until the Workload existed, the pod group existed within it, and the number of pending pods met minCount. Then it checked whether it could place the whole group, opened the gate if so, and rejected the lot on a 5-minute timeout if not.

What v1.36 changed, and why your manifests broke

This is the part most write-ups have not caught up with. Kubernetes v1.36 did not extend v1alpha1. It replaced it.

The split is the design idea. Workload is now a static template. PodGroup is the runtime object that carries the actual scheduling policy and a status. The scheduler reads PodGroup directly and never has to watch or parse Workload, which also lets status updates shard per replica instead of contending on one object.

A Workload in v1.36 looks like this:

            apiVersion: scheduling.k8s.io/v1alpha2
kind: Workload
metadata:
  name: training-job-workload
  namespace: some-ns
spec:
  podGroupTemplates:
  - name: workers
    schedulingPolicy:
      gang:
        minCount: 4
          

Controllers then stamp out runtime PodGroups from that template:

            apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  name: training-job-workers-pg
  namespace: some-ns
spec:
  podGroupTemplateRef:
    workload:
      workloadName: training-job-workload
      podGroupTemplateName: workers
  schedulingPolicy:
    gang:
      minCount: 4
          

And pods point at the PodGroup, not the Workload:

            apiVersion: v1
kind: Pod
metadata:
  name: worker-0
  namespace: some-ns
spec:
  schedulingGroup:
    podGroupName: training-job-workers-pg
          

If you are copying YAML from a December 2025 tutorial into a v1.36 cluster, workloadRef is the line that will fail you first.

The PodGroup scheduling cycle

v1.36 gave kube-scheduler a dedicated PodGroup scheduling cycle instead of reserving resources pod by pod, which is what risked the deadlock in the first place.

When the scheduler pops a PodGroup member off the queue, it fetches the rest of that group's queued pods, sorts them deterministically, and runs an atomic cycle: take a single snapshot of cluster state to avoid race conditions, attempt to find node placements for every pod in the group using the standard filter and score phases, then apply the decision atomically.

On success the schedulable pods move to binding together, and any leftover unschedulable pods return to the queue to wait for capacity so they can join the ones already placed. On failure nothing binds and the whole group returns to the queue after a backoff.

One behaviour worth internalising: if pods are added to a PodGroup after others are already scheduled, the cycle evaluates the new pods while accounting for the existing ones, and pods already assigned to nodes keep running. The scheduler will not unassign or evict them even if the group later fails to meet its requirements. Gang scheduling protects the start of a job, not its continued existence.

Where the algorithm gives up

The maintainers are unusually candid about the limits of this first cycle, and this is the section to read before you plan around it:

Read that table as a design constraint, not a bug list. Native gang scheduling in v1.36 is built for the identical-worker case, which happens to be exactly what distributed training looks like. Push heterogeneous pods or inter-pod affinity through it and you are outside what the algorithm promises.

Topology-aware scheduling

Placing training workers randomly across a cluster adds network latency between them, and on a distributed job the slowest link sets the pace. v1.36 lets you constrain a PodGroup to a topology domain:

            apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  name: topology-aware-workers-pg
spec:
  schedulingPolicy:
    gang:
      minCount: 4
  schedulingConstraints:
    topology:
    - key: topology.kubernetes.io/rack
          

The scheduler runs a three-phase placement algorithm for this: generate candidate placements, which are subsets of nodes theoretically feasible for the group, via a new PlacementGenerate extension point; evaluate each candidate to confirm the group actually fits; then score the feasible ones through a new PlacementScore extension point and pick the best.

Two current gaps. Topology-aware scheduling does not trigger preemption to satisfy constraints, though the maintainers plan to integrate the two. And only one topology level is supported today; multiple levels, soft constraints and deeper DRA integration are named as future work.

Workload-aware preemption

Default preemption evaluates victims node by node, which cannot help a gang that needs room across several nodes at once. v1.36 adds workload-aware preemption, which treats the entire PodGroup as a single preemptor unit and searches the whole cluster, so it can preempt pods from multiple nodes simultaneously to open space for the whole group.

It adds two fields to the PodGroup API. A PodGroup priority overrides the priority of the individual pods forming the group. A disruptionMode dictates whether pods in the group can be preempted independently or must go all-or-nothing.

            apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  name: victim-pg
spec:
  priorityClassName: high-priority
  priority: 1000
  disruptionMode: PodGroup
          

The catch: in v1.36 these fields are respected only by workload-aware preemption. Default pod-by-pod preemption ignores them. So a PodGroup you marked all-or-nothing can still be taken apart one pod at a time by the ordinary preemption path. Extending support to other disruption sources is stated future work, not current behaviour.

DRA: one ResourceClaim for a whole PodGroup

Dynamic Resource Allocation reached general availability in v1.34, letting pods make detailed requests for GPUs, TPUs and NICs. v1.36 makes PodGroup a replicable unit for a ResourceClaimTemplate: Kubernetes generates one ResourceClaim for the entire PodGroup regardless of pod count, and a pod whose spec.resourceClaims matches its PodGroup's resolves to the group's claim instead of getting its own.

The scaling detail matters more than it first appears. Previously kube-scheduler could only list individual pods in a ResourceClaim's status.reservedFor, which is capped at 256 items. A single PodGroup reference in status.reservedFor can now represent far more than 256 pods. If you have ever hit that ceiling on a large training run, this is the change that removes it.

Letting the Job controller do it for you

Hand-writing Workload and PodGroup objects is not an operating model either. v1.36 ships the first phase of Job controller integration behind the WorkloadWithJob feature gate. With it on, the Job controller creates a Workload and matching runtime PodGroup for each qualifying Job, sets .spec.schedulingGroup on every pod it creates, and sets the Job as owner so the generated objects are garbage-collected with it.

It only kicks in for Jobs with a fixed, well-defined shape:

A Job that qualifies needs nothing unusual in it:

            apiVersion: batch/v1
kind: Job
metadata:
  name: training-job
  namespace: job-ns
spec:
  completionMode: Indexed
  parallelism: 4
  completions: 4
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: registry.example/trainer:latest
          

Jobs that miss any condition schedule pod by pod, exactly as before. And if you set schedulingGroup yourself because a higher-level controller manages the workload, the Job controller leaves your pod template alone, which makes the gate safe to switch on in clusters already running an external batch system. Elastic Jobs and other controllers are tracked in KEP-5547 and are not covered yet.

Native gang scheduling versus the incumbents

Native gang scheduling is alpha. Volcano and Kueue have been doing this in production for years, and the honest answer for most teams in July 2026 is that the incumbents still win on maturity.

InfraCloud's comparison of Apache YuniKorn, Volcano and Kueue, published 14 May 2025 before the Workload API existed, describes Volcano's gang scheduling via a minAvailable field and Kueue as a job-queueing layer that works with the default scheduler through spec.suspend rather than scheduling pods itself. That division is why teams commonly run Kueue for quota and admission with Volcano underneath for the placement guarantee.

Here is where native scheduling changes that calculus over time. Volcano and YuniKorn solve gang scheduling by not being the default scheduler, which means a second scheduler binary, a second set of failure modes, and workloads that must opt in by name. If kube-scheduler does it natively, that whole layer disappears. That is worth real money in operational overhead, but not until it leaves alpha.

Turning it on

Everything below is alpha in v1.36 and off by default.

You also need the scheduling.k8s.io/v1alpha2 API group enabled. On managed platforms this is usually the wall. Managed control planes rarely expose alpha feature gates or alpha API groups, so check with your provider before assuming a v1.36 cluster can turn these on. Amazon EKS lists 1.35 and 1.36 in standard support, but standard support is not the same thing as alpha gate access. Budget for a self-managed or local cluster to evaluate this.

What v1.37 is expected to bring

The maintainers named their v1.37 targets: graduating the Workload and PodGroup APIs to beta, minCount mutability to support elastic jobs, multi-level workload hierarchies to express JobSet and LeaderWorkerSet shapes, beta for topology-aware scheduling and workload-aware preemption, and a unified controller integration API. Priority and order are explicitly subject to change.

The minCount mutability item is the one to watch. Fixed gang size is what forces the current Job controller restriction that completions equals parallelism. Make minCount mutable and elastic training jobs become expressible.

India-specific considerations

For Indian ML teams, three things sharpen the point.

GPU capacity is the scarce input, and it is billed in dollars and paid in rupees, so a deadlocked training run costs more here than the dollar figure suggests. The $165.12 an hour a 3-of-4 stall burns is a rupee problem before it is a scheduling one, and unlike a rate discount, it is waste you can remove without negotiating with anyone.

Capacity scarcity makes the deadlock worse, not better. Gang scheduling matters most exactly when GPUs are hard to get, because that is when a partially-started job is most likely to sit waiting for a fourth worker that no one can free. Teams renting scarce accelerator capacity have the strongest case for all-or-nothing placement and the least room to absorb a stall.

Data residency shapes which regions you can train in, and the Digital Personal Data Protection Act 2023, which received presidential assent on 11 August 2023, governs personal data used in training sets. Topology constraints and residency constraints interact: pinning a gang to a rack is a placement decision, and pinning it to a jurisdiction is a legal one. We design applications aligned with DPDP Act 2023 requirements so the second is settled before the first. Our note on GPU spend as the top FinOps concern covers the wider cost picture, and our write-up of the EC2 Capacity Blocks GPU price rise covers why accelerator rates stopped falling. Teams weighing build versus buy can compare our managed Kubernetes and AI platform service.

The verdict

Native gang scheduling is the right direction and it is not ready. Every feature here is alpha, the API broke between two consecutive releases, managed control planes will not let you enable the gates, and the placement algorithm makes no promise for heterogeneous groups. If you need gang scheduling in production this quarter, run Volcano.

What changed in v1.36 is the shape of the destination. The Workload and PodGroup split, the atomic scheduling cycle, PodGroup-level DRA claims and Job controller integration together describe a Kubernetes where distributed training does not need a second scheduler. That is worth tracking closely and worth prototyping in a staging cluster now, because when it hits beta the teams that already know the object model will move first.

The API broke once between v1.35 and v1.36. Assume it can break again before beta, and do not build tooling on v1alpha2 that you cannot afford to rewrite.

FAQ

How eCorpIT can help

eCorpIT is a CMMI Level 5, MSME-certified, senior-led engineering organisation in Gurugram that builds and runs GPU training platforms on Kubernetes for Indian and global teams. We benchmark your current scheduler against gang-scheduled placement, quantify what partial starts cost you per hour, and prototype the v1alpha2 Workload and PodGroup model in staging while keeping Volcano carrying production. We design applications aligned with DPDP Act 2023 requirements, so training-data residency is settled before placement. To scope a GPU scheduling review, contact our team.

References

1. Kubernetes v1.36: Advancing Workload-Aware Scheduling - Maciej Skoczen et al., Kubernetes Blog, 13 May 2026.

1. Kubernetes v1.35: Introducing Workload Aware Scheduling - Maciej Skoczen and Dominik Marcinski, Kubernetes Blog, 29 December 2025.

1. KEP-4671: Gang Scheduling - Kubernetes Enhancements.

1. Kubernetes Release History - Kubernetes.

1. Kubernetes v1.36: Haru - Kubernetes Blog, 22 April 2026.

1. Kubernetes v1.35: Timbernetes (The World Tree Release) - Kubernetes Blog, 17 December 2025.

1. Review release notes for Kubernetes versions on standard support - Amazon EKS User Guide.

1. 2026 State of Kubernetes Resource Optimization: CPU at 8%, Memory at 20%, and Getting Worse - Laurent Gil, Cast AI, 21 April 2026.

1. Cast AI's 2026 State of Kubernetes Optimization Report Reveals GPU Utilization at 5% - Cast AI press release, 21 April 2026.

1. Batch Scheduling on Kubernetes: Comparing Apache YuniKorn, Volcano.sh, and Kueue - Rahul Kadam, InfraCloud, 14 May 2025.

1. p5.48xlarge pricing and specs - Vantage EC2 instance pricing.

1. The Digital Personal Data Protection Act, 2023 - Ministry of Electronics and Information Technology, Government of India.

Last updated: 16 July 2026.