On this page · 11 sections
Summary. PyTorch 2.13, released around July 11, 2026, brings FlexAttention to Apple Silicon through the Metal (MPS) backend, with hand-written Metal kernels for the sparse prefill and decode paths and up to 12x speedups over scaled dot-product attention (SDPA) on sparse patterns. The release landed 3,328 commits from 526 contributors since PyTorch 2.12, with a live Q&A scheduled for July 22, 2026. For anyone running models on a Mac, this closes a gap that used to push custom attention work onto NVIDIA hardware. A Mac with 546 GB/s of memory bandwidth (M4 Max, 2026) already generates roughly 60 tokens per second on Llama 3.1 8B at 4-bit, the newer M5 Max pushes that to about 614 GB/s (near 12% more), and a machine around $1,799 can hold a 33B model in unified memory. FlexAttention makes the attention step on that hardware programmable without writing Metal by hand.
If you build or fine-tune transformers on a MacBook or a Mac Studio, this is the most useful line item in the PyTorch 2.13 release. It is not a headline model or a new training recipe. It is a kernel change that quietly removes a reason to keep a separate NVIDIA box around for experimentation.
What actually shipped in PyTorch 2.13
FlexAttention on MPS is the change most Mac developers will feel first, but it arrived alongside a few other performance items worth knowing before you upgrade.
| Feature | What it does | Who it helps |
|---|---|---|
| FlexAttention on Metal (MPS) | Hand-written Metal kernels for sparse prefill and decode, including grouped-query attention and captured buffers, up to 12x over SDPA on sparse patterns | Anyone training or serving attention models on Apple Silicon |
| CuTeDSL Inductor backend | A second high-performance code path alongside Triton for key GPU operations, with faster compilation | Kernel authors and heavy torch.compile users on CUDA |
nn.LinearCrossEntropyLoss |
Fuses the final projection and the loss into one operation, cutting peak GPU memory by up to 4x for large-vocabulary language-model training | Teams pretraining or fine-tuning LLMs |
| Deterministic FlexAttention backward (CUDA) | A reproducible gradient path for FlexAttention | Research reproducibility and debugging |
| Scale of the release | 3,328 commits from 526 contributors since 2.12 | Context for how much changed |
The Apple Silicon piece matters because the Metal path now covers both phases of inference. The release blog describes hand-written Metal kernels for the sparse prefill and the decode path, with support for grouped-query attention (GQA) and captured buffers. That combination is what real decoder-only models need, not just a toy causal mask.
Why attention was the weak spot on a Mac
Attention is the part of a transformer that scales badly. A dense attention mask over a sequence of length N holds N by N entries, so memory and compute grow with the square of the context. Fused kernels such as FlashAttention solved this on NVIDIA GPUs by never materialising the full score matrix. On Apple Silicon, that class of fused kernel was thin on the ground, so many people fell back to SDPA with a dense mask, or wrote a bespoke Metal kernel for the one pattern they needed.
The FlexAttention authors framed the underlying tension well. In the original FlexAttention announcement, Team PyTorch (Driss Guessous, Yanbo Liang, Joy Dong, and Horace He at Meta) wrote that fused attention "has come with a loss of flexibility, as you can no longer try out a new attention variant by writing a few PyTorch operators. You often need to write a new custom kernel." That is the exact problem Mac users hit: you either accepted slow generic attention, or you signed up to maintain Metal by hand.
Two numbers explain why the fix is worth caring about. Apple Silicon inference is memory-bandwidth bound, so raw bandwidth largely sets your token rate. An M4 Max runs at 546 GB/s and generates roughly 60 tokens per second on Llama 3.1 8B at 4-bit quantisation, according to Apple Silicon LLM benchmarks. Cut wasted work in the attention step and you get closer to that bandwidth ceiling instead of leaving it on the table.
What FlexAttention is, in one paragraph
FlexAttention is a single API that lets you express a custom attention pattern as plain Python and have torch.compile turn it into a fused kernel. You supply two small callables. A mask_mod(b, h, q_idx, kv_idx) returns a boolean for whether a query position may attend to a key position. A score_mod adjusts the raw attention score before the softmax, which is how biases such as ALiBi or soft-capping are expressed. The compiler traces these functions, infers the block-sparsity pattern, and generates a kernel that does not materialise the full mask. The FlexAttention documentation and the MLSys 2025 paper cover the programming model in full.
The practical win is that one API covers many variants. Causal masking, document or block masking for packed sequences, sliding-window attention, PagedAttention, and additive biases all become a few lines instead of a new kernel each.
How to use it on Apple Silicon
Install PyTorch 2.13 and confirm the MPS backend is available. FlexAttention lives in torch.nn.attention.flex_attention.
import torch
from torch.nn.attention.flex_attention import flex_attention, create_block_mask
device = "mps" if torch.backends.mps.is_available() else "cpu"
# A causal mask expressed as a predicate, not a materialised NxN tensor.
def causal(b, h, q_idx, kv_idx):
return q_idx >= kv_idx
block_mask = create_block_mask(causal, B=None, H=None, Q_LEN=4096, KV_LEN=4096, device=device)
q = torch.randn(1, 16, 4096, 128, device=device, dtype=torch.float16)
k = torch.randn(1, 16, 4096, 128, device=device, dtype=torch.float16)
v = torch.randn(1, 16, 4096, 128, device=device, dtype=torch.float16)
# Compile once, then call in the hot path.
flex = torch.compile(flex_attention)
out = flex(q, k, v, block_mask=block_mask)
The create_block_mask helper builds a block-sparse representation: a query and key block is treated as sparse only if every position inside it is masked. That block view is what lets the kernel skip whole regions instead of checking element by element, and it is where the speedup on sparse patterns comes from.
A score bias looks like this. ALiBi, which penalises attention to distant tokens, is a two-line score_mod:
def alibi(score, b, h, q_idx, kv_idx):
bias = (q_idx - kv_idx) * get_slope(h) # per-head slope
return score - bias
out = flex(q, k, v, score_mod=alibi)
Two habits keep this fast. Compile the call once and reuse it, because tracing on every forward pass throws away the benefit. And build the BlockMask once per shape and cache it, since the mask does not change between steps for a fixed sequence layout.
What "up to 12x" really means
The 12x figure in the PyTorch 2.13 notes is a ceiling on sparse patterns, not a blanket speedup. The more of the attention matrix your mask removes, the more work the block-sparse kernel skips, and the larger the gap over SDPA with a dense mask. A long sequence with document masking, where most cross-document pairs are masked out, sits near the top of that range. A short, fully dense causal attention over 512 tokens will see little to no gain, because there is almost nothing to skip.
So the honest read is: FlexAttention on MPS helps most when your attention is both long and sparse. That covers packed training batches, retrieval-augmented prompts with many chunks, and sliding-window decoders. It does less for short-context, dense workloads, where SDPA is already close to optimal.
Remember the hardware ceiling underneath all of this. On a memory-bandwidth-bound Mac, the attention kernel is one term in the budget. An M4 Max at 546 GB/s and about 60 tokens per second on Llama 3.1 8B is limited as much by bandwidth as by the kernel, per ML Journey's Apple Silicon benchmarks. FlexAttention removes waste; it does not raise the bandwidth wall.
When to reach for it, and when not to
| Scenario on Apple Silicon | Recommended approach | Why |
|---|---|---|
| Standard causal LM, short context | SDPA | Already close to optimal; nothing to skip |
| Long context with document or block masking | FlexAttention (MPS) | Skips masked blocks, up to 12x on sparse patterns |
| Custom score bias (ALiBi, soft-capping) | FlexAttention score_mod |
Expresses the bias with no custom kernel |
| One fixed pattern, maximum control | Hand-written Metal kernel | Full control, at a higher maintenance cost |
| MLX-first project | MLX attention | Stays inside the MLX stack you already use |
The decision is mostly about sparsity and maintenance. If your pattern is standard and dense, SDPA is fine. If it is custom or sparse and you do not want to own a Metal kernel, FlexAttention is the reason to be on 2.13.
Sharp edges to expect
FlexAttention is powerful, and it has edges. Block sizes matter: the block-sparse machinery assumes block-aligned shapes, and unusual block sizes have historically caused correctness and performance surprises, which the FlexAttention docs and open issues discuss. Compile time is real, so a cold torch.compile adds latency to the first call; warm it up before you benchmark. And not every mask benefits, so measure your own pattern rather than trusting the 12x headline. A quick A/B against SDPA on your real sequence length and mask is the only number that matters for your model.
For production serving specifically, treat the Mac as a development and low-concurrency target. A single-user Mac is excellent for building and testing attention variants; multi-user serving at scale still points to GPU servers, a tradeoff we cover in our guide to running local LLMs versus cloud inference for mobile apps and in the cost of self-hosting versus API inference.
India-specific considerations
For teams in India weighing local Mac development against renting cloud GPUs, the maths is about capital versus running cost. A Mac around $1,799, roughly ₹1.5 lakh at mid-2026 rates, holds a 33B-class model in unified memory and needs no per-hour GPU bill, which suits a small team iterating on attention code all day. Cloud GPUs win for bursty multi-user serving where you pay only for the hours you use.
There is a data-protection angle too. Running inference on-device keeps prompts and user data off third-party APIs, which supports the data-minimisation expectations under the Digital Personal Data Protection Act 2023. For product teams handling personal data, an on-device path built and tested on Apple Silicon can be a cleaner privacy story than routing every request to a hosted model, a theme we develop in our note on choosing between on-device and cloud AI and in the guide to Apple's on-device foundation models for Swift developers.
FAQ
How eCorpIT can help
eCorpIT is a Gurugram-based, senior-led engineering organisation (founded 2021, CMMI Level 5, MSME certified) that builds and ships AI features on Apple Silicon and in the cloud. We help teams decide where inference should run, port attention and model code to the MPS backend, and design on-device paths that keep user data local and aligned with DPDP requirements. If you are weighing a Mac-based development setup against cloud GPUs, or need help moving a model to production, talk to our team about your workload. You can also read how we approach private LLM deployment in India and iOS and Swift application development.
References
- PyTorch 2.13 release blog - PyTorch
- PyTorch 2.13.0 release notes - GitHub
- Best Mac for local AI 2026: Apple Silicon chips ranked - Local AI Master
- Local LLMs on Apple Silicon Mac 2026 guide - SitePoint
- LM Studio on M4 Max: verified MLX performance data - Markaicode
_Last updated: July 18, 2026._
