KernelShiftDocs

Getting started

Introduction

Arm keeps adding instruction-set extensions, and fast kernels have to be rewritten for each one. KernelShift measures whether coding agents can do that rewrite. It has two parts: a benchmarking pipeline (kgen) that collects tasks from open-source libraries, and a set of environments that check and time each kernel on real Neon, SVE, SVE2 and SME2 processors. The benchmark has 144 kernels from four libraries; the current evaluation set uses 99 of them.

Each task is a kernel from a production library with a reference implementation, workloads taken from real models, and an expert implementation to beat. Through MCP tools, the agent compiles its kernel, tests and times it, and reads the generated assembly, all on the target processor. A kernel scores only if it is correct on every workload, and its score is the speedup over the expert implementation.

How a run works

Every harness talks to a KernelShift server over MCP and gets the same tools. The server runs on a cloud instance with the target processor, one per target, and the harness reaches it through an SSH tunnel. Each call compiles, checks and times the code right there, and the result goes back to the agent.

Agent harnessClaude Code · Codex · nanobot · tool-call loop · …
KernelShift serveron the target instancecompile · evaluate · disassemble
Target processorGraviton3 (SVE) · Graviton4 (SVE2) · Apple M4 (SME2) · …

results: correctness + performance

Where the kernels come from

SourceKernelsExpert reference
ncnnConv2D, depthwise Conv2D, pooling (FP32, INT8)ncnn's Neon kernels
llama.cppGEMM (FP32 to 4-bit), RMSNorm, attention, MoEggml CPU kernels
Arm SIMD LoopsStandalone loops loop_001 … loop_223Arm's hand-written SVE, SVE2 and SME2 versions
KleidiAIGEMM and convolution micro-kernelsArm's vendor micro-kernels

The tasks live in the Hugging Face dataset l3lab/KernelShift. Its TASKS.md lists, for every definition, the expert baseline on each hardware tier and whether it runs there.