FlexAttention in PyTorch: Custom and Sparse Attention Without Writing Kernels
October 12, 2026 · IntelliSensei Team
Sliding windows, document masking for packed training, ALiBi and soft-capping — written as small Python functions and compiled into a fused FlashAttention-style kernel. How BlockMask sparsity works, what speedups to expect, and the sharp edges to avoid.
Read more