FastKernels: Benchmarking GPU Kernel
Generation in Production

LLM-based agents for GPU kernel generation are advancing rapidly, but the benchmarks they optimize against evaluate kernels in isolation, with synthetic inputs and weak baselines, rewarding sandbox speedups that break or vanish in real inference systems.

We introduce FastKernels, a benchmark of 384 tasks drawn from 47 representative architectures across 8 categories, whose kernels suffice to reimplement 94.6% (472/499) of HuggingFace Transformers architectures with outputs matching the native implementations. Each task mirrors the interface of the corresponding production module and is scored against the kernels production frameworks ship. Tasks form a compositional hierarchy, from primitives to full models, in which higher-level modules import lower-level ones.

Candidates are scored at the kernel level and end to end inside the models they come from, on the production execution path. MacroEval aggregates calibrated correctness, coverage, and speedup into a leaderboard. Seeding it with five representative agents (6,900 agent-hours), we find that kernel-level speedups of 1.6–6.6× shrink to at most 1.25× end to end, only 20% of winning kernel sets run correctly as-is, and kernel-level scores mis-rank agents: Claude Code matches or beats KDA at every level in isolation, yet KDA scores 3× higher end to end.

LLM-based kernel-generation agents now achieve strong scores on isolated benchmarks, with state-of-the-art systems autonomously compiling, profiling, and refining CUDA or Triton code. Yet these gains often fail to transfer into production inference frameworks such as vLLM and SGLang: kernels that score well in sandboxes routinely regress once they encounter real serving interfaces, compilation stacks, and workloads.

The root cause is benchmark–production misalignment. Current benchmarks rely on synthetic inputs, single-GPU isolated kernels, simplified interfaces, and independent task levels that do not compose into full inference pipelines. They therefore reward kernels that are fast in isolation but brittle in deployed systems, hiding interface mismatches, compilation-stack conflicts, and correctness degradation that only appears at model scale.

FastKernels closes this gap by scoring every candidate inside the models its tasks come from, on the production execution path (continuous batching, chunked prefill, CUDA graphs, torch.compile, tensor parallelism), and by giving each task the interface of the corresponding production module so winning kernels transfer into vLLM and SGLang.

FastKernels organizes 384 tasks into four levels of increasing scope: 108 L1, 155 L2, and 74 L3 kernel tasks, shared across architectures, plus 47 end-to-end L4 models. Higher-level baselines import lower-level modules, so an agent can reuse an optimized linear operator inside an MLP or transformer block. In standalone mode each candidate is benched over production children; in composed mode every candidate file is active at once. The gap is the composition effect.

Before any agent is scored, every architecture is checked against the production framework for its family (vLLM and SGLang for serving, diffusers for SDXL, FLA for linear attention, timm for vision encoders, and the corresponding upstream repositories elsewhere). On the full set—47 scenarios, 184 workloads, tensor parallel 1–8—gains concentrate in linear attention (1.45× throughput) and recommendation (1.51×). The other categories sit near parity. References are pinned per release and GPU; the LLM rows use vLLM 0.26.0.

The largest throughput gains are on architectures whose reference is a research repository or a non-serving library: RetNet 3.40×, ColBERTv2 3.13×, Pi0 3.09×, BGE-M3 2.35×, and CosyVoice3 2.30×. Mainstream LLM serving stays at parity with the hardened frameworks (Llama-3.1 0.95×, Mixtral 1.06×, GPT-OSS 0.95×).

Random tensors yield a near-uniform gate (Gini 0.232). Random token IDs over-concentrate load and pick different hot experts from real requests (Gini 0.480 vs. 0.339; only 4 of the top-16 hot experts in common). The kernel tier can replay golden inputs captured from real end-to-end executions, and the end-to-end tier runs real requests, so data-dependent kernels are scored on the tensors production traffic actually creates.

Model families expose different correctness signals—tokens, embeddings, labels, rankings, trajectories, audio, video—and different throughput–latency tradeoffs. MacroEval calibrates architecture-specific correctness to a common [0, 1] scale, measures end-to-end speedup, and macro-averages across families so no single architecture dominates the leaderboard. The end-to-end table above is this score on the default set.

For each request, a discrepancy is mapped through two thresholds fixed before evaluation. The lower threshold g_i is the 95th percentile of a noise run: the production reference repeated in a fresh process with a cold compile cache. The upper threshold is f_i = 0.5. An item is valid when its calibrated correctness is at least 0.9; crashes, NaNs, and invalid items receive no speedup credit and are penalized through coverage.

The default leaderboard score multiplies three components: macro-geometric speedup S_macro (a λ = 0.5 blend of throughput and latency), macro-calibrated correctness C_macro, and macro-coverage. The three components are also reported separately, along with throughput-only and latency-only rankings. KDA stays first when the correctness thresholds are halved or doubled.

FastKernels requires Python 3.10+, CUDA 12+, and a recent NVIDIA GPU (Hopper / Blackwell tested; Ampere supported for a subset of kernels). Clone the repository and install the fastkernels CLI:

git clone https://github.com/Snowflake-AI-Research/fastkernels.git
cd fastkernels
pip install .

Baselines live in fastkernels/tasks/baseline/, organized by level. Capture the shapes a real workload actually feeds each operator, then scaffold a candidate with the baseline's class name and forward signature:

fastkernels list --map
fastkernels capture minimal
fastkernels create-stubs --architecture llama

Drop the replacement in fastkernels/tasks/candidate/L<level>/<op_name>.py. The kernel tier checks correctness and speedup against the production baseline; the end-to-end tier is what the leaderboard reports:

fastkernels bench --target rms_norm
fastkernels bench --standalone          # one module, production children
fastkernels eval default                # 11-model MacroEval set

Full Documentation Code