Operator fusion has become a key optimization for deep learning,
which combines multiple deep learning operators to improve data reuse and reduce global
memory transfers.
However, existing tensor compilers struggle to fuse complex reduction computations
involving loop-carried dependencies, such as attention mechanisms.

This paper introduces Neptune, a tensor compiler for advanced operator fusion for
sequences of reduction operators.
Neptune presents a new approach for advanced operator fusion, which intentionally
breaks some existing dependencies and compensates by constructing algebraic correction expressions
that allow the kernel to produce the correct result. Applying Neptune’s advanced operator fusion to a plain attention operator generates operators
equivalent to FlashAttention and FlashDecoding.

On ten attention-based benchmarks, Neptune, starting from a plain attention code
and a high-level scheduling template, outperforms existing compilers
like Triton, TVM, and FlexAttention, including Triton-based implementations of FlashAttention.
Across four different GPU architectures from NVIDIA and AMD,
Neptune-generated kernels have an average speedup of 1.35$\times$ over the next best alternative,
with up to $2.65 \times$ speedup on Nvidia GPUs and up to $3.32 \times$ on AMD GPUs,
demonstrating its effectiveness for deep learning workloads.