Arm keeps adding instruction-set extensions, and fast kernels have to be rewritten for each one. KernelShift measures whether coding agents can do that rewrite. It has two parts: a benchmarking pipeline (kgen) that collects tasks from open-source libraries, and a set of environments that check and time each kernel on real Neon, SVE, SVE2 and SME2 processors. The benchmark has 144 kernels from four libraries; the current evaluation set uses 99 of them.
Each task is a kernel from a production library with a reference implementation, workloads taken from real models, and an expert implementation to beat. Through MCP tools, the agent compiles its kernel, tests and times it, and reads the generated assembly, all on the target processor. A kernel scores only if it is correct on every workload, and its score is the speedup over the expert implementation.
Every harness talks to a KernelShift server over MCP and gets the same tools. The server runs on a cloud instance with the target processor, one per target, and the harness reaches it through an SSH tunnel. Each call compiles, checks and times the code right there, and the result goes back to the agent.
The tasks live in the Hugging Face dataset l3lab/KernelShift. Its TASKS.md lists, for every definition, the expert baseline on each hardware tier and whether it runs there.
Getting started
Installation
You need Python 3.12 or later and clang++. Apple clang works on macOS.
Each line is one workload: its id, the verdict, the minimum and 5th-percentile time, and the largest absolute and relative error against the reference.
Scalar and Neon code runs on any 64-bit Arm processor. SVE needs Graviton3, SVE2 needs Graviton4 and SME2 needs Apple M4. An SVE kernel run on Apple silicon fails with RUNTIME_ERROR ... SIGILL.
A task is four kinds of file in bench-trace/. Every evaluation writes a fifth: the trace.
bench-trace/
definitions/<op_type>/<name>.json # what to compute, with a Python reference
workloads/<op_type>/<name>.jsonl # concrete sizes and inputs, one per line
solutions/<dataset>/<author>/<op_type>/<name>.json # an implementation
tensors/<op_type>/<name>/*.npy # inputs captured from real model runs
Definition
The shapes, dtypes and tags of one kernel, and a NumPy reference that defines correct output. Axes are either var (set per workload) or const (baked into the kernel). This is rms_norm_fp32_d2048 from llama.cpp:
Source files plus a build spec. Every task ships two: reference-scalar, plain C++ that is the agent's starting point, and the library's own kernel (for example baseline-llamacpp), which the agent has to beat. The agent edits only kernel.cpp; the harness adds the binding files.
The record of one solution evaluated on one workload: the verdict, the correctness metrics, the timing samples, and the timing protocol they were taken with. Traces from different protocols are never compared.
Tags that change evaluationcorrectness:sqnr switches the check from element-wise tolerances to a signal-to-quantization-noise floor, and e2e:<model> marks the kernels spliced into llama.cpp for the end-to-end runs.
Concepts
Processors and extensions
Each instruction-set extension is a separate target, chosen with --isa. It compiles with its own flags and runs on an EC2 instance whose processor has those instructions. Settings come from config/kernel_contracts.yaml.
Extension
Compile flags
Features
Instance (processor)
Neon
-march=armv8-a
128-bit Advanced SIMD
c7g.xlarge (Graviton3)
SVE
-march=armv8.2-a+sve
sve, dotprod, i8mm; 256-bit vectors
c7g.xlarge (Graviton3)
SVE2
-march=armv9-a+sve2
sve, sve2, dotprod, i8mm; 128-bit vectors
c8g.xlarge (Graviton4)
SME2
-mcpu=apple-m4
sme2; 512-bit streaming vectors, ZA
mac-m4.metal (Apple M4)
Each extension includes the one before
A solution that declares isa_features: ["sve"] also runs on the SVE2 target, because SVE2 is a strict superset of SVE. The rule stops at SME2.
Why SME2 uses -mcpu=apple-m4
No Graviton processor implements SME, so the SME2 target runs on Apple M4. Its cores run SVE instructions only in streaming mode. With -march=armv9-a+sme2, plain SVE code compiles and then dies with SIGILL; -mcpu=apple-m4 rejects it at compile time, which gives the agent an error it can act on.
CostAn M4 needs an EC2 Dedicated Host, which AWS bills for at least 24 hours. Provisioning never allocates one: allocate it yourself and release it when done.
Concepts
Scoring
A version counts only if it is correct on every workload. Among correct versions, the score is the speedup over the expert kernel.
Correctness
Each workload's output is compared with the definition's reference. Most kernels use element-wise absolute and relative tolerances set per op type. Quantized kernels tagged correctness:sqnr must reach a signal-to-quantization-noise floor set relative to what the reference implementation itself scores on that workload. evaluate stops at the first failing workload.
Timing
10 warmup samples, then 50 timed samples; the score uses the minimum.
Each sample repeats the kernel until it lasts at least 10 ms (inner_iters: auto), so short calls are not lost in timer noise.
The expert kernel is timed the same way on the same processor.
Each run uses one core and its own process.
Speedup@40
For one kernel, the speedup of a version is the geometric mean over workloads of expert time divided by kernel time. Speedup@40 is the best speedup among correct versions within the first 40 tool calls. A benchmark result is the geometric mean of Speedup@40 over kernels, reported with the share of kernels above 1.0.
Every kernel has a scoreA session starts by compiling the scalar starting point unchanged and evaluating it, so even a run that never finds a faster version is scored.
Concepts
Datasets
The Hugging Face dataset l3lab/KernelShift holds 144 definitions. 99 of them are the current evaluation set.
--dataset is ncnn, simd-loop, llama.cpp or kleidiai. Without --definitions every definition of the dataset runs. --max-iterations (default 40) caps the compile, evaluate and disassemble calls per kernel.
Harnesses
Harness
--harness
Needs
Claude Code
claude-code
claude CLI, logged in
Codex
codex
codex CLI, logged in
Cline
cline
cline CLI, --model, an endpoint in its config
nanobot
nanobot
nanobot CLI and an API key in its config
Built-in tool-call loop
own
--model, any litellm model id
One completion, no tools
single-shot
--model
Results
Each kernel gets agent-runs-<author>/<definition>/, where the author is <harness>-<model>-<isa>:
v1.cpp, v2.cpp, …: every version the agent compiled;
trajectory.jsonl: one line per tool call. A passing evaluate carries time_speedup_geomean.
Spot by defaultInstances are spot unless you pass --on-demand. A Linux instance shuts itself down within about 2 hours of its driver stopping.
Guides
Bring your own harness
Any MCP client can be the agent. Connect it to a server you start by hand, or add an adapter so bench_fleet.py runs it like the built-in harnesses.
Connect any MCP client
shell
python3 skills/launch/launch_session.py launch --isa sve --dataset ncnn --author my-agent
This starts (or reuses) an instance for the extension, copies the code over, builds the dataset and starts the server. When it prints tunnel up: http://127.0.0.1:8888/mcp, connect your client to that URL over streamable HTTP. The tools are listed in MCP tools.
Give the client a tool-call timeout of at least 1500 s: one evaluate can take up to 20 minutes.
Ctrl-C stops the server and the tunnel. The instance keeps running until you tear it down.
Copy the results back, then remove the instance. Its label is <dataset>-<author> and its address is in provisioning/eval_config.json:
No call budgetA server started this way does not limit tool calls. bench_fleet.py stops each kernel after 40 compile, evaluate and disassemble calls; keep your agent to the same budget to compare with its results.
Add a harness to bench_fleet
Write the agent's instructions in skills/<name>/<name>-kernel-session/SKILL.md. It becomes the agent's system prompt.
Subclass HarnessAdapter in test_scripts/harness_adapters/<name>.py and implement run_job(job, *, endpoint, author, log_path) and sandbox_runtime_paths(). Point the agent at endpoint with a tool timeout of at least 1500 s. claude_code.py is a complete example.
Export the class from test_scripts/harness_adapters/__init__.py, and register it in test_scripts/bench_fleet.py: the import, ADAPTER_CLASSES, the branch in run_fleet and the --harness choices.
The task prompt is the adapter's prompt_template, filled with the definition, dataset, expert author, extension and call budget.
From Python
eval/mcp_client.py wraps a running server as a client whose tool schemas are in litellm format. eval/evaluator.py holds the tool-call loop behind --harness own, a starting point for an agent of your own.
Guides
Add a kernel with kgen
A coding agent writes the task files by following four skills; kgen checks them and puts them in place. People confirm the definition, choose the expert library, and approve hand-picked sizes.
The four skills
They are in .claude/skills/, so Claude Code finds them when it runs in the repository.
Skill
Writes
gen-definition
The definition: shapes, dtypes, tags and the reference.
gen-workload
The workloads. For quantized kernels, also captured inputs and the SQNR floor.
gen-candidate-solution
reference-scalar, the plain C++ starting point.
gen-baseline-solution
The expert solution, which calls the library's own kernel.
Commands
shell
python -m kgen show <similar-definition> --full # start from a task like it
python -m kgen add definition def.json
python -m kgen add workload workloads.jsonl --definition <name>
python -m kgen add solution reference-scalar.json
python -m kgen add solution baseline.json
python -m kgen check <name>
add checks a file and writes it to its place in bench-trace/. To overwrite a file, put --replace before the kind: python -m kgen add --replace workload …
What check verifies
The files: schema, names, workloads that set exactly the definition's variable axes and inputs, and tensor files with the declared shape and dtype. --static-only stops here.
The reference runs and returns outputs of the declared shapes.
reference-scalar and the expert solution build, run and match the reference. This part runs only on an AArch64 machine.
check times nothing. On the target processor, python -m kgen autovec --isa <isa> --definitions <name> flags an expert solution that is more than 2× slower than the auto-vectorized starting point.
Quantized kernelsKernels tagged correctness:sqnr also need real inputs and an SQNR floor: run kgen capture pytorch, then kgen calibrate --author reference-scalar --baseline-author <author> on an AArch64 machine.
Guides
End to end in llama.cpp
Put an agent's matrix-multiply kernels into llama.cpp and time a whole model against stock llama.cpp with llama-bench, on Graviton4 (SVE2).
Models
--model
Weights (Q4_K_M)
Smallest instance
qwen3.5-4b
unsloth/Qwen3.5-4B-GGUF
c8g.xlarge
llama-3.1-8b
bartowski/Meta-Llama-3.1-8B-Instruct-GGUF
c8g.4xlarge
Their kernels are the 18 llama.cpp definitions tagged e2e:<model>. Other models are in config/e2e_models.json.
Run it
First run an agent on the model's definitions with bench_fleet.py --dataset llama.cpp --isa sve2. Then, with that run's author name:
shell
# on your machine: build stock and patched llama.cpp on the instance, download the model
python -m e2e evaluate --model qwen3.5-4b --steps deploy --label <label> --agent-build
# on the instance, in ~/kernelshift
~/venv/bin/python3 -m e2e evaluate --model qwen3.5-4b --steps kernel_set --author <author> --rows-abi
~/venv/bin/python3 -m e2e evaluate --model qwen3.5-4b --steps measure --threads 1 4 16
~/venv/bin/python3 -m e2e evaluate --model qwen3.5-4b --steps report
What it measures
Only matrix multiplies with q4_K, q5_K or q6_K weights are replaced. Everything else is stock llama.cpp.
Three builds take turns: stock, stock without weight repacking and KleidiAI, and that build with the agent's kernels. Speedup is against stock.
Before timing, every kernel must be seen running inside llama.cpp, and a kernel that fails to load stops the run. --perplexity <text file> also runs llama-perplexity.
Results go to e2e-runs/llamacpp-<model>-sve2/e2e_results.json.
Guides
AWS setup
bench_fleet.py and launch_session.py start instances with Terraform. You provide the account, a security group and an SSH key; the code creates a key pair and the instances, and copies the repository over.
What you need
An AWS account in us-west-2, where the machine images are pinned, with its default VPC.
A security group that allows inbound SSH (port 22). The MCP server listens only on 127.0.0.1 and is reached through an SSH tunnel.
~/.ssh/id_rsa and ~/.ssh/id_rsa.pub. For another key, set TF_VAR_ssh_key_files.
terraform, rsync and ssh on your machine, plus the AWS CLI for Apple M4.
Configure
shell
cp provisioning/workspaces.json.example provisioning/workspaces.json # set aws_region and security_group_id
terraform -chdir=terraform init
aws_profile, account_id and namespace are optional.
Apple M4
An M4 instance needs a Dedicated Host, and AWS bills a host for at least 24 hours. The code never allocates one: it uses the newest free mac-m4.metal host in your region. Allocate it yourself and release it when done:
bench_fleet.py destroys its instance after a successful run. After a failed run the instance stays: a Linux instance shuts itself down about 2 hours later (--watchdog-minutes), a Mac stays up. launch_session.py never shuts one down on its own.
Without --label, --teardown destroys every instance listed in provisioning/eval_config.json.
Reference
MCP tools
The tools every harness sees. They behave the same across harnesses and processors; the server runs on the instance being tested.
Tool
Arguments
What it does
Returns
check_progress
definition
Looks up the best result so far on this definition.
Its version, metrics and code.
compile
definition, code
Compiles the agent's kernel.cpp; the harness and binding files are added automatically.
A version number, or the compiler error.
evaluate
definition, version
Checks correctness on all workloads, stopping at the first failure; if correct, times the kernel. A new best result is saved at once.
Errors, or speedups.
disassemble
definition, version, optional fn
Disassembles the kernel entry symbol, or fn.
Up to 300 lines of AArch64 assembly.
Call budget
compile, evaluate and disassemble each count as one call; check_progress does not. Over the budget, a call returns MAX_ITERATIONS_EXCEEDED. bench_fleet.py sets 40 per kernel; a server started by hand has no budget. See Bring your own harness.
Reference
CLI
python -m bench.cli checks and times solutions on the machine you run it on. Global flags go before the subcommand.
Times the expert solutions and stores the results.
--dataset, --isa, --definition
retime
Times stored solutions again against freshly timed experts.
--dataset, --isa, --author or --index, --pass-id, --out
--dataset is ncnn, llama.cpp, simd-loop or kleidiai. --isa is neon, sve, sve2 or sme2; SIMD Loops needs it, because its expert solution differs per extension.
Building the ncnn and llama.cpp experts needs their source checkouts: set NCNN_ROOT and LLAMA_CPP_ROOT.
Wrong root, empty outputA missing --root directory is created empty, so a wrong path prints empty lists rather than an error.
test_scripts/bench_fleet.py and skills/launch/launch_session.py: run agents. See Run an agent.
Reference
Configuration
Settings live in config/, in three files under provisioning/ and eval/, and in environment variables.
File
What it sets
config/kernel_contracts.yaml
Each extension's compile flags and instance type, correctness tolerances, the timing protocol, the expert authors, MCP tool timeouts and banned source patterns.
config/dataset_builds.json
How ncnn and llama.cpp are built on an instance.
config/e2e_models.json
The models for end-to-end runs.
config/rsync_allowlist.json
What is copied to an instance.
provisioning/workspaces.json
Your AWS region, security group and profile. Copy it from the .example.
provisioning/eval_config.json
The instances that are up. Provisioning writes it; do not create it yourself.
eval/llm_providers.json
API keys and endpoints for the own and single-shot harnesses. Copy it from the .example; empty values fall back to environment variables.
.env
Environment variables, loaded by provisioning, launch_session.py and the harness adapters.
Timing protocolChanging warmup, repeat, inner_iters or target_sample_ns in eval_defaults changes the timing protocol: every stored expert timing is collected again.