Kubernetes OCI image volumes hit stable in v1.36: ship model weights without rebuilding images

Summary. The image volume source went stable in Kubernetes v1.36, enabled by default, ending a graduation that began as alpha in v1.31 in August 2024 and passed through beta in v1.33 (April 2025) before becoming on-by-default in v1.35 (December 2025). It lets a pod mount the contents of an OCI object, read-only, at a mount path, with no rebuild of the container image that consumes it. For teams serving models, that separates a 40 GB weights artifact from a 500 MB model-server image on independent release cycles. KEP-4639 names this case directly: mounting "large language model weights or machine learning model weights in a pod alongside a model-server". You need containerd v2.1 or later, or CRI-O, and you need to understand three constraints the announcements underplay. Storage cost is not the win. At Amazon ECR's $0.10 per GB-month, registry deduplication means baking weights into images was never costing you what you assumed. The win is decoupling, pull behaviour, and rollback granularity.

What actually shipped, and when

The timeline matters, because a lot of published material about this feature is describing an older version of it.

The Kubernetes documentation for image volumes now carries the header "FEATURE STATE: Kubernetes v1.36 [stable] (enabled by default)". The v1.35 release announcement puts the intermediate step plainly: "The image volume type has been in beta since v1.33 and is enabled by default in v1.35."

One wrinkle worth knowing if you go reading the source material. KEP-4639's own metadata still lists beta at v1.34, which contradicts both the feature-gate table and the v1.33 beta announcement. The KEP's implementation history explains the drift: the beta target moved from v1.33 to v1.34, then to "beta (enabled by default) in v1.35", and finally GA was retargeted to v1.36 in January 2026. The documentation and the release blogs are correct. The KEP metadata is stale.

Sascha Grunert of Red Hat, who co-authored KEP-4639 and wrote the Kubernetes release announcements for the feature, marked the beta graduation this way: "SIG Node is proud and happy to deliver this feature graduation as part of Kubernetes v1.33."

The YAML

This is the canonical example from the Kubernetes documentation, unchanged since beta:

            apiVersion: v1
kind: Pod
metadata:
  name: image-volume
spec:
  containers:
  - name: shell
    command: ["sleep", "infinity"]
    image: debian
    volumeMounts:
    - name: volume
      mountPath: /volume
  volumes:
  - name: volume
    image:
      reference: quay.io/crio/artifact:v2
      pullPolicy: IfNotPresent
          

Applied to model serving, the shape becomes:

            apiVersion: v1
kind: Pod
metadata:
  name: model-server
spec:
  containers:
  - name: server
    image: registry.example.com/vllm-server:1.9.2   # never rebuilt for a new model
    args: ["--model", "/models/weights"]
    volumeMounts:
    - name: weights
      mountPath: /models
  volumes:
  - name: weights
    image:
      reference: registry.example.com/models/llama-70b:2026-06-01
      pullPolicy: IfNotPresent
          

The server image and the weights artifact now have separate tags, separate release cadences, and separate rollbacks. Changing the model is a pod spec edit, not a build.

The API surface is two fields

ImageVolumeSource has exactly two fields, reference and pullPolicy. That is the whole API.

reference is the OCI object to mount. The current documentation describes it as optional, "to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets", which is a useful detail if you template these.

pullPolicy accepts Always, Never, or IfNotPresent, and defaults to Always if the :latest tag is specified, or IfNotPresent otherwise. The failure semantics of each are specific, and they are the part that will page you:

Never pin a weights volume to a mutable :latest tag. That combination gives you Always, which means every pod start is a registry round trip for tens of gigabytes, and a registry hiccup is a Failed pod rather than a slow one.

Three constraints the announcements underplay

It is read-only, permanently

The documentation is unambiguous: the OCI object "gets mounted in a single directory (spec.containers[*].volumeMounts[*].mountPath) and will be mounted read-only". The KEP explains the reasoning and the roadmap: "For security reasons, ro (read-only) option is set by default. Having rw (read-write) support will require a follow-up enhancement."

For weights this is correct behaviour and not a limitation. For anything that expects to write into its data directory, including some model servers that build an index or cache on first load, this will fail. Check before you migrate. Point the writable cache at an emptyDir and the weights at the image volume.

One correction if you read the v1.31 alpha announcement: it stated the volume is mounted non-executable (noexec). That requirement was dropped in June 2025 during the beta retarget. Do not design around it.

The volume re-resolves on pod recreation

This is the behaviour that surprises people, and the docs state it directly: "The volume gets re-resolved if the Pod gets deleted and recreated, which means that new remote content will become available on Pod recreation. A failure to resolve or pull the image during Pod startup will block containers from starting and may add significant latency. Failures will be retried using normal volume backoff and will be reported on the Pod reason and message."

Read that twice if you are running a mutable tag. A pod rescheduled at 03:00 onto a fresh node can come back with different weights than its neighbours. That is a silent correctness bug in a serving fleet, not a deployment event. Pin to an immutable tag or a digest.

This is also why ImageVolumeWithDigest, alpha in v1.35, matters. It records the resolved image digest in the pod status, so you can tell which bytes a running pod actually has.

fsGroupChangePolicy does nothing here

The docs note that spec.securityContext.fsGroupChangePolicy "has no effect on this volume type". If your model server runs as a non-root user and expects group ownership fixups on the mount, verify permissions on a real cluster before you roll it out. The AlwaysPullImages admission controller, for what it is worth, does work on this volume source the same way it works for container images.

The cost argument is not the one you have been told

Here is where most write-ups on this feature get it wrong, and it is worth being precise because the wrong version is expensive to act on.

The usual claim is that baking model weights into a container image wastes registry storage, because every rebuild stores another copy. That is not how OCI registries work. Registries deduplicate by layer digest. If your weights sit in a layer that does not change between builds, thirty tagged builds of your server image reference the same weights layer, and the registry stores those bytes once.

Run the arithmetic at Amazon ECR's published price of $0.10 per GB-month for private repository storage. A 40 GB weights layer shared across thirty tags costs $4.00 a month, not $120. Splitting it into a separate artifact costs the same $4.00. Storage is a wash. In-region transfer is free anyway: data moving between ECR and EC2, Lambda, App Runner, or Fargate in the same Region carries no charge.

So if the storage saving is roughly zero, why do this at all? Four reasons that are actually real.

Rebuild coupling disappears. Patching your server image for a CVE means a rebuild. If the weights are in that image, every rebuild has to move 40 GB through your CI, and every model change has to rebuild the server. The two artifacts have completely different change rates: your server image might ship weekly, your weights quarterly. Coupling them means the fast one drags the slow one.

Layer invalidation stops being a trap. The dedup that saves you storage only holds while the weights layer is byte-identical. Reorder a COPY in the Dockerfile, change a base image, or rebuild in a way that alters layer digests, and you have just written another 40 GB to the registry. The saving is real but fragile, and it depends on Dockerfile discipline that nobody audits.

Rollback gets granular. With weights in the image you roll back both or neither. Split, you can roll the model back to last quarter's tag while keeping this week's security-patched server.

Init containers and startup scripts go away. The v1.35 release announcement makes this the headline benefit: the feature "lets you package and deliver data-only artifacts such as configs, binaries, or machine learning models using standard OCI registry tools", and "you can fully separate your data from your container image and remove the need for extra init containers or startup scripts".

That last one is the honest sell. Most teams serving models today have an init container that curls weights from S3 into an emptyDir. That init container is code you wrote, with credentials you rotate, retry logic you maintain, and no layer caching. Deleting it is the win.

What you need on the nodes

The feature is only as available as your runtime. The v1.35 announcement is explicit: "Please note that using this feature requires a compatible container runtime, such as containerd v2.1 or later." CRI-O has supported the initial feature since v1.31 and added beta support in v1.33.

The failure mode on an old runtime is described in the KEP and is not obvious from the pod events: "Pods using the new VolumeSource combined with a not supported container runtime version will fail to run on the node, because the Mount.host_path field is not set for those mounts."

If you run a managed cluster, check the runtime version before the Kubernetes version. A control plane on v1.36 with nodes on containerd v1.7 gives you an API that accepts the pod spec and nodes that cannot honour it.

Mounting one subdirectory with subPath

The beta release added the ability to mount a subdirectory rather than the whole artifact, which matters when one artifact carries several model variants. From the v1.33 announcement: "This allows end-users to mount a certain subdirectory of an image volume, which is still mounted as readonly (ro). This means that non-existing subdirectories cannot be mounted by default. As for other subPath and subPathExpr values, Kubernetes will ensure that there are no absolute path or relative path components part of the specified sub path. Container runtimes are also required to double check those requirements for safety reasons."

              volumeMounts:
  - name: weights
    mountPath: /models
    subPath: llama-70b-int4
          

A non-existent subdirectory fails at container creation, surfaced through kubelet events. That is a deployment-time failure rather than a silent empty mount, which is the behaviour you want.

Watching it in production

The v1.33 beta announcement documents three kubelet metrics for image volumes: kubelet_image_volume_requested_total ("Outlines the number of requested image volumes"), kubelet_image_volume_mounted_succeed_total ("Counts the number of successful image volume mounts"), and kubelet_image_volume_mounted_errors_total ("Accounts the number of failed image volume mounts").

Check the exact names against your own cluster before you build alerts on them. KEP-4639's metadata lists a different naming scheme for the same three counters, without the kubelet_ prefix, and the two have not been reconciled in the source material. Scrape the endpoint and read what is actually there.

The alert that matters is on the error counter. A rising errors_total with healthy pods means nodes are failing to pull weights and backing off, which shows up as capacity that never arrives rather than as a crash.

India-specific considerations

For teams running model-serving clusters from Indian regions, two things follow from the mechanics above.

Registry locality does most of the work on cost. Transfer between ECR and compute in the same Region is free, so a weights artifact in an India region pulled by nodes in that Region carries the $0.10 per GB-month storage charge and no egress. The expensive pattern is a registry in one Region serving nodes in another, which turns every node scale-up into cross-Region transfer of tens of gigabytes. If you serve from Mumbai, publish to Mumbai.

The second is a data governance point rather than a cost one. A weights artifact is not personal data, but fine-tuned weights derived from customer records sit closer to it than most teams admit, and adapters trained on support transcripts are a live question under the Digital Personal Data Protection Act 2023 (DPDP). Image volumes make this easier to reason about, not harder: the artifact has a digest, a registry, and an access policy, rather than living inside a container image that also contains your application. Treat the weights registry as a system holding derived data and apply the same access controls you apply in production.

Teams building on this will find the related mechanics in our guides to gang scheduling for GPU jobs with the Workload API, in-place pod resize for rightsizing, and pod certificates and constrained impersonation in v1.35.

Migrating off an init container

The realistic path from a curl-from-S3 init container is four steps, and only the first is interesting.

Package the weights as an OCI artifact and push them to your registry. Pin the tag to something immutable, dated or digest-based, never :latest. Point the volume at it with pullPolicy: IfNotPresent. Delete the init container, the bucket credentials it used, and the retry logic you wrote.

Test one thing before you roll out: kill a pod and let it reschedule onto a node that has never pulled the artifact. That path exercises the cold pull, the backoff, and the startup latency the docs warn "may add significant latency". Measure it, then set your readiness probe and pod disruption budget against the number you measured rather than the number you hoped for.

The feature is stable, on by default, and the API is two fields. The work is in your tags.

FAQ

How eCorpIT can help

eCorpIT builds and operates Kubernetes platforms for teams serving models in production. Our senior engineering teams handle the parts of this migration that are not in the documentation: packaging weights as OCI artifacts, setting registry locality so node scale-ups do not cross Regions, replacing init-container download scripts and their long-lived credentials, and validating that your node runtime actually supports the API your control plane accepts. We design these systems aligned with DPDP 2023 requirements where fine-tuned weights carry derived customer data. If you are serving models on Kubernetes and your model release still means an image rebuild, talk to our team.

References

1. Kubernetes documentation, Use an Image Volume With a Pod

1. Kubernetes v1.33: Image Volumes graduate to beta

1. Kubernetes 1.31: Read Only Volumes Based On OCI Artifacts (alpha)

1. KEP-4639: OCI VolumeSource, kubernetes/enhancements

1. VolumeSource: OCI Artifact and/or Image, enhancements issue #4639

1. Kubernetes v1.35: Timbernetes (The World Tree Release)

1. Kubernetes v1.35 Sneak Peek

1. Kubernetes v1.31: Elli release announcement

1. Amazon ECR pricing

1. Amazon ECR FAQs

1. Understanding data transfer costs for AWS container services

1. KEP-4639 writeable, container isolated volume content, containerd issue #11853

1. Support Kubernetes image volume mounting by using standard OCI layer media types, kitops issue #1144

Last updated: 16 July 2026.