KernelShift

KernelShift

Benchmarking Kernel Optimization Across Evolving Hardware

Shuaiwei Huang1*, Weihua Du1*, Allen Wu1*, Ivan Fox2, Manupa Karunaratne2, Yiming Yang1, Sean Welleck11Carnegie Mellon University · 2Arm Ltd. · *Equal contribution

Each new generation of Arm processors adds instruction-set extensions, shifting over time, and the fastest code has to be rewritten to use them. KernelShift gives agents 144 real optimization tasks from four open-source libraries and runs their code on the real processors. We provide a benchmarking pipeline and a set of environments for kernel optimization on evolving hardware, and show that agents beat library code on established extensions but still struggle with newer ones.

1 · Task

Start from portable C++

for (int m = 0; m < M; ++m)
  for (int n = 0; n < N; ++n) {
    float sum = bias[n];
    for (int k = 0; k < K; ++k)
      sum += A[m*K+k] * B[k*N+n];
    C[m*N+n] = sum;
  }

A matrix multiply from KleidiAI, one of 144 tasks

2 · Agent

Rewrite it for one hardware extension

NeonSVESVE2SME2

Evaluated on various hardware/intrinsics

3 · Score

Timed against the expert code

Opus 4.8 on this task, Apple M4

Motivation

New hardware changes how fast code has to be written

Arm keeps adding instruction-set extensions. Some add new instructions; others, like SME2, bring a new programming model, so fast code has to be reorganized, not just recompiled. Expert libraries take time to catch up with each one, and existing kernel benchmarks test a fixed target rather than a sequence of hardware changes. KernelShift asks whether coding agents can make this shift.

Neon
Every 64-bit Arm processor

The base instruction set. Each instruction handles a few numbers at once.

Intrinsics work on fixed 128-bit registers (4 floats).
SVE
AWS Graviton3 (2022)

New instructions: longer rows, and one piece of code works for any row length.

Intrinsics work on vectors of any length, from 128 to 2048 bits.
SVE2
AWS Graviton4 (2024)

More instructions, same programming model: more kinds of operations on the same rows.

Same intrinsics style as SVE, with more operations.
SME2
Apple M4 (2024)

A new programming model: compute on a grid of numbers in a special mode, so code has to be organized differently.

Intrinsics work on 2D tiles.

The shape of data a processor handles at once

Neona short row
SVE / SVE2a longer row / more operators
SME / SME2a whole grid
Neon, SVE and SVE2 work on a row of numbers. SME and SME2 work on a whole grid, so code has to be written in a different way, with a different grammar.

Agents pick up new instructions well. When the programming model changes, they fall behind.

Benchmarking pipeline

kgen: an agent builds the tasks, people check them

kgen is our command-line toolkit for turning code from open-source libraries into benchmark tasks, with skills for agents to follow step by step. A coding agent runs it; people make the few decisions and verify the result.

Environments

Any agent, measured on the real processor

Agents connect to the KernelShift server. Every agent gets the same toolkit, and each call runs on a machine with the target processor.

Agent harnessone interface for all

Claude CodeCodexnanobottool-call loop
Agent skillsdocumentation · memory
call

KernelShift serverone on each target instance

CompileEvaluatetest & timeDisassembleread assembly
runs on

Target processorswhere the code is run and timed

AWS Graviton3SVE
AWS Graviton4SVE2
Apple M4SME2
results: correctness + performance
Correctness

Every test input must pass

Code is timed only after it gives the right answer on every test input of the task.

Timing

Same machine, same clock

The agent's code and the expert code are timed on the same machine, the same way.

Integrity

One core, no shortcuts

Each run gets one core and its own process.

Key results

Strong on familiar extensions, weak on a new programming model

Scores below compare the agent's code with the code the library ships for that processor.

Established extension (SVE)

Agents beat the code popular libraries ship

The best tested agent (GLM-5.3 with nanobot) is 2.7× faster than ncnn's own code, and the best tested agents are about 1.95× faster on llama.cpp and 1.56× on Arm SIMD Loops. Arm's hand-tuned KleidiAI code is still ahead. A compiler on its own reaches at most half the library's speed.

Full results · AWS Graviton3 (SVE)

Each agent gets 40 tool calls per task. Score: speed of its best correct code relative to the library's code, averaged over tasks (1.0 = same speed, higher is better). The small number is the share of tasks where the agent wins. Select a library to sort.

# Agent Model

From the paper's Table 2 (SIMD Loops without the two in-place sorts). Underlined: best in column. Grey rows: no agent loop.

Real LLM inference

The gains carry over to running an LLM

We put agent-written code for the matrix multiplications of two LLMs into llama.cpp on AWS Graviton4. Compared with the original llama.cpp, prompt processing gets 1.3 to 1.6 times faster and token generation 1.04 to 1.1 times faster.

prompt processingtoken generationoriginal llama.cpp = 1.0
CPU threads

Moving to a newer extension (SVE → SME2)

Agents struggle more with the programming-model shift to SME2

Each model's speed relative to the expert code on the same six KleidiAI kernels, first on SVE, then on SME2, which brings a new programming model (mean of three runs). Three of the five models fall to less than half; GLM-5.3, the strongest of them on SVE, goes from 0.46× to 0.21×. The two Claude models do not drop.

Get started

Try one task locally, or run an agent in one command

Install, then test and time one task
# Python 3.12+ and clang++
git clone https://github.com/cmu-l3/kernel-shift
cd kernel-shift
pip install -r requirements.txt
hf download l3lab/KernelShift \
    --repo-type dataset --local-dir bench-trace

python -m bench.cli bench --dataset simd-loop --isa sve \
    --definition loop_001 --solution reference-scalar_loop_001
Run an agent on a cloud Arm instance
# starts the instance, runs the agent, copies results back
python3 test_scripts/bench_fleet.py \
    --harness claude-code --model claude-opus-4-8 \
    --dataset simd-loop --isa sve \
    --definitions "loop_001 loop_002"

Demo

Watch an agent improve a kernel

Real runs from our experiments. Each dot is a version the agent tested. Select a key step to see what the agent changed and read its code.

best so farkey stepother tested versionlines changed since the previous step
    
              

    Speed is relative to the library's own code on the same processor (1.0× = as fast).

    Citation

    BibTeX

    Placeholder until the review period ends