Benchmarks
Test Setup
Section titled “Test Setup”On a read-only grafana/grafana TypeScript and Go checkout, gpt-6.1-sol (medium) answered 3 multi-hop architecture questions 3 times per tool in each of 3 client environments, and a blind grader checked each answer against 12 required facts (7 Core + 5 Extended).
Questions and expected routes
Section titled “Questions and expected routes”git-sync-job-dispatch- Question: “Trace every segment of the message-bus path from creating a Git repository sync job to dispatching it to the worker that runs it.”
- Expected route (
Core7·Extended5): Sync requests from the UI, controller, or webhook persist a job resource on the API server rather than queuing in memory; after KV commit, a NATS watch notification triggers the job informer queue-group subscription, enqueues the job in the driver queue, claims it, and dispatches it to the sync worker. Extended facts cover degraded start with relist fallback, dual host wiring, ignoring live update notifications, best-effort publish failures, and claim retry/cooldown handling.
alertmanager-config-autosync- Question: “Trace every segment connecting the Kubernetes-style API to automatic syncing of external Alertmanager configuration.”
- Expected route (
Core7·Extended5): The settings UI reads and JSON-patches the singleton configuration resource, admission validation checks the target datasource, periodic polling pre-processing fetches external Alertmanager configuration over datasource proxy HTTP, persists it, and writes sync conditions back to resource status for the UI. Extended facts cover hash-based deduplication, disable cleanup, blocking manual imports (409) while active, management-mode branching, and failure reason /404classification.
scope-annotation-filter- Question: “Trace every segment connecting dashboard scope filters to saving and querying dashboard annotations.”
- Expected route (
Core7·Extended5): Scope names extracted from the scope-filter toggle and dashboard scope variable flow through three entry points—datasource queries, panel-context create/update, and the annotation list panel—into the Kubernetes-style annotation client’s query parameters and request body, reaching the annotation app’s custom search route and store-backend filtering. Extended facts cover legacy REST omission and per-backend rejection or ignore behavior when scopes are unsupported.
Compared tools and pinned versions
Section titled “Compared tools and pinned versions”| Tool & version | Upstream project / package | Pinned commit | Tools used |
|---|---|---|---|
codemap-search 0.14.2 |
buYoung/mcp (codemap-search) |
e727d20c6 |
codemap-search tools only (Jev filter off) |
Graphify 0.9.95 |
Graphify-Labs/graphify (PyPI graphifyy) |
92741d16 |
Graphify + extra tools (read, grep, find, ls, shell) |
zvec-grep 0.2.2 |
zvec-ai/zvec-grep (npm @zvec/zvec-grep) |
b1a9148e |
zvec-grep + extra tools (read, grep, find, ls, shell) |
codebase-memory-mcp 0.12.1 |
DeusData/codebase-memory-mcp | a7158ac2 |
codebase-memory-mcp + extra tools (read, shell) |
rg 15.2.0 (baseline) |
BurntSushi/ripgrep | e89fff89 |
Built-in tools only (rg, read, grep, find, ls, shell; no MCP) |
CodeGraph 1.6.2 |
colbymchenry/codegraph (npm @colbymchenry/codegraph) |
6560052a |
CodeGraph + extra tools (grep, read, find, ls, shell) |
Serena 2.1.0 (excluded) |
oraios/serena (PyPI serena-agent) |
5e4f5cd1 |
Go/TypeScript LSP MCP (excluded before run) |
Serena 2.1.0 was excluded before the benchmark runs because indexing Grafana’s Go and TypeScript LSP took roughly 34–70 minutes per run.
Agent clients, models, and execution environments
Section titled “Agent clients, models, and execution environments”| Component / Environment | Pinned version & details |
|---|---|
| Target codebase | grafana/grafana commit c0a598f5a49abf2014f67e806075d8ad267c9efb, 2026-10-26, 24,115 tracked TypeScript and Go files |
| Solver model | gpt-6.1-sol (medium), 3 independent runs per question per tool/environment |
| Blind grader model | gpt-6-astra (high), no tools or repository access; checks 12 required facts per question |
pi direct |
pi 1.4.0 + @buyong/pi-mcp-adapter 0.1.2; turn-by-turn MCP calls plus repo-confined read, grep, find, ls (no bash) |
pi codemode |
pi 1.4.0 + @buyong/pi-mcp-adapter 0.1.2 + QuickJS codemode; runs JS to batch and filter calls (10k-token per-call output cap) |
Codex code mode |
Codex CLI 0.164.0; runs JS exec/wait cells calling MCP tools and a repo-confined read-only shell (10k-token per-call output cap; codemap-search 0.14.2 uses 100k) |
Grafana index footprint and build time
Section titled “Grafana index footprint and build time”| Tool | Indexed files / graph entities | Index disk size | Index build time (pi envs / Codex code mode prep) |
|---|---|---|---|
codemap-search 0.14.2 |
19,757 files |
136.5–138.3 MiB |
12.96 s / 16.18 s |
codebase-memory-mcp 0.12.1 |
21,430 tracked files (525,952 nodes; 1,562,241 pi / 1,562,181 Codex edges) |
1,915.9–1,916.9 MiB |
34.35 s / 56.95 s |
CodeGraph 1.6.2 |
17,094 files (314,436 nodes, 1,035,428 edges) |
1,194.3 MiB (+ 1.9 MiB state) |
88.82 s / 101.00 s |
zvec-grep 0.2.2 |
22,642 files (175,403 entities) |
961.9 MiB (+ 32.0 MiB model cache) |
147.68 s / 154.22 s |
Graphify 0.9.95 |
19,823 AST-scanned code files (160,008 nodes, 867,900 edges) |
626.8 MiB |
311.73 s / 366.08 s |
rg 15.2.0 (baseline) |
None (live disk search) | 0 MiB |
0 s |
Serena 2.1.0 (excluded) |
Go + TypeScript LSP | — | ~34–70 min (excluded before run) |
Summary
Section titled “Summary”Fulfilled rate (%)
Mean total tokens / answer (k)
| Tool | pi direct |
pi codemode |
Codex code mode |
All-core (7/7) answers (3 envs) | Mean total tokens / answer | Mean uncached total / answer | Mean tool calls / answer | Built-in / fallback calls / answer | Cited test/gen |
|---|---|---|---|---|---|---|---|---|---|
codemap-search 0.14.2 |
96/108 | 97/108 | 98/108 | 5/9 · 6/9 · 6/9 | 946k · 829k · 1,333k | 98k · 101k · 133k | 45.1 · 39.2 · 50.4 | 0.0 · 0.0 · 0.0 | 0 · 0 · 0 |
Graphify 0.9.95 |
93/108 | 88/108 | 94/108 | 3/9 · 2/9 · 5/9 | 1,008k · 746k · 1,183k | 93k · 93k · 108k | 55.8 · 66.0 · 70.6 | 16.8 · 22.9 · 11.2 | 8 · 9 · 8 |
zvec-grep 0.2.2 |
89/108 | 71/84 (7 ans) | 87/108 | 3/9 · 3/7 · 4/9 | 784k · 608k · 869k | 77k · 79k · 96k | 45.6 · 55.6 · 20.9 | 38.6 · 48.4 · 15.7 | 4 · 2 · 4 |
codebase-memory-mcp 0.12.1 |
87/108 | 84/108 | 85/108 | 3/9 · 3/9 · 2/9 | 1,331k · 1,190k · 1,827k | 89k · 113k · 144k | 73.9 · 94.1 · 103.1 | 5.0 · 6.3 · 2.1 | 7 · 9 · 6 |
rg 15.2.0 |
86/108 | 87/108 | 89/108 | 3/9 · 1/9 · 5/9 | 555k · 505k · 866k | 68k · 76k · 104k | 63.7 · 66.6 · 20.3 | 63.7 · 66.6 · 20.3 | 6 · 6 · 5 |
CodeGraph 1.6.2 |
78/108 | 85/108 | 86/108 | 2/9 · 3/9 · 3/9 | 1,727k · 1,724k · 3,309k | 132k · 142k · 194k | 24.7 · 26.3 · 36.3 | 2.8 · 4.6 · 3.4 | 5 · 5 · 4 |
- Triple order (
·): Each triple listspi direct·pi codemode·Codex code mode. - Statistical significance (
p < 0.05, question-stratified permutation test):codemap-search 0.14.2scored significantly higher thanrg 15.2.0,CodeGraph 1.6.2, andcodebase-memory-mcp 0.12.1in all3environments.- It also scored significantly higher than
Graphify 0.9.95inpi codemodeandzvec-grep 0.2.2inCodex code mode. - None of the other
4open-source MCP servers significantly outperformedrg 15.2.0in any environment (p >= 0.05).
- Run notes:
codemap-search 0.14.2completed every run using its own MCP tools only (0.0built-in/fallback calls,0answers citing test or generated files).*Inpi codemode,zvec-grep 0.2.2crashed twice withSIGSEGVon one question, completing7of9answers (71/84,84.5%).
See Detailed Results for per-environment k-unit token distributions, pairwise p-values, per-question/per-fact breakdowns, and subtool call tables.