Established extension (SVE) Agents beat the code popular libraries ship The best tested agent (GLM-5.3 with nanobot) is 2.7× faster than ncnn's own code, and the best tested agents are about 1.95× faster on llama.cpp and 1.56× on Arm SIMD Loops. Arm's hand-tuned KleidiAI code is still ahead. A compiler on its own reaches at most half the library's speed. Full results ▾ best tested agent (named under each library)compiler onlylibrary's code = 1.0 Full results · AWS Graviton3 (SVE) Each agent gets 40 tool calls per task. Score: speed of its best correct code relative to the library's code, averaged over tasks (1.0 = same speed, higher is better). The small number is the share of tasks where the agent wins. Select a library to sort. # Agent Model ncnn 23 tasks llama.cpp 26 SIMD Loops 24 KleidiAI 6 From the paper's Table 2 (SIMD Loops without the two in-place sorts). Underlined: best in column. Grey rows: no agent loop.
Real LLM inference The gains carry over to running an LLM We put agent-written code for the matrix multiplications of two LLMs into llama.cpp on AWS Graviton4. Compared with the original llama.cpp, prompt processing gets 1.3 to 1.6 times faster and token generation 1.04 to 1.1 times faster. prompt processingtoken generationoriginal llama.cpp = 1.0 CPU threads 1416
Moving to a newer extension (SVE → SME2) Agents struggle more with the programming-model shift to SME2 Each model's speed relative to the expert code on the same six KleidiAI kernels, first on SVE, then on SME2, which brings a new programming model (mean of three runs). Three of the five models fall to less than half; GLM-5.3, the strongest of them on SVE, goes from 0.46× to 0.21×. The two Claude models do not drop. SVESME2expert code = 1.0