agentgw-llm-bench

Serving-fitness benchmark for llama.cpp / llama-swap gateways: throughput, capability, tool-call sanity, quant integrity (KL), and a calibrated judge layer - one command, unattended, verdict as exit code.

One command (the pipeline)

# configure once (copy the template and edit)
cp bench/pipeline.example.yaml bench/pipeline.yaml

# offline validation of config + tools + task files (no network)
python3 bench/pipeline.py --config bench/pipeline.yaml --dry-run

# the real thing: bench -> judge -> validity -> report -> sink
JUDGE_API_KEY=sk-... python3 bench/pipeline.py --config bench/pipeline.yaml

The pipeline runs, in order: bench phases (telemetry, warm-up, single-shot perf, lm-eval, EvalPlus coding, judge-response generation) -> optional KL quant check -> judge pass 2 (loads alone; two-pass separation preserved) -> optional cross-judge validity pass -> merge + report -> sink. Judging checkpoints after every block and resumes after crashes - a 429 at 3am no longer discards paid tokens.

Exit codes (cron/CI contract): 0 all PASS (or completed, no thresholds active) · 1 WARN only · 2 any FAIL · 3 pipeline error.

Triggering it

Whatever you already have - the exit-code contract is the integration:

# nightly 03:00, results + log persisted
0 3 * * * cd /opt/agentgw-llm-bench && JUDGE_API_KEY=... python3 bench/pipeline.py \
    --config bench/pipeline.yaml >> bench/results/pipeline.log 2>&1

# systemd timer unit (pipeline.service + pipeline.timer)
[Service]
Type=oneshot
WorkingDirectory=/opt/agentgw-llm-bench
Environment=JUDGE_API_KEY=...
ExecStart=/usr/bin/python3 bench/pipeline.py --config bench/pipeline.yaml

CI: fail the job when exit code >= 2.

Where the report goes (sink)

• sink: file (default): bench/results/<run>/report.md is the source of truth; one summary line per run appended to bench/results/pipeline.log.
• sink: webhook: POSTs a JSON summary to $BENCH_WEBHOOK_URL (env name configurable). Schema:

  {"run_id": "...", "tag": "pipeline", "finished_utc": "...",
   "verdict_per_model": {"model": "PASS|FAIL|null"},
   "exit_code": 0, "summary": [{"model": "...",
   "throughput_by_ctx": {"512": {"gen_tps": 24.9, "prefill_tps": 980.0,
   "ttft_any_s": 0.4, "vram_peak_gib": 21.0}, "8192": {"...": "..."},
   "32768": {"gen_tps": null, "prefill_tps": null, "ttft_any_s": null,
   "vram_peak_gib": null}},
   "capacity": {"peak_aggregate_tps": 140.0, "best_n": 4, "levels":
   [{"n": 1, "aggregate_tps": 52.0, "ttft_ms_p95": 900.0, "itl_ms_p95": 20.0},
   {"n": 4, "...": "..."}]},
   "lmeval": {"ifeval": 0.62}, "judge_checklist_rate": 0.71}],
   "report_path": "/abs/path/report.md", "kl_divergence": {...}}
  

  A plain-text text field is included for Slack-style receivers. POST failure is a warning only - it never changes the exit code.

Judge validity without humans

Judge metrics are advisory by default - they inform, they never gate serving decisions; objective metrics do the gating. To know whether the judge's verdicts mean anything, the pipeline periodically re-grades the checklist with a second, different-family judge (cross_judge: in the config; runs on the first-ever run and then every every_n_runs):

• agreement >= 75%: judge trusted for this era
• agreement < 75%: loud warning in report + pipeline log - do not trust judge-layer numbers until the judges are reconciled (divergent items are listed with justifications)

Two run modes (identical code, no branching)

# bare stdlib (any box with python3): smoke path - telemetry, warm-up,
# tools smoke, single-shot throughput, judge gen, KL
python3 bench/run_bench.py --smoke --phases telemetry,warmup,tooluse,throughput,judge

# tools environment (.venv or docker): full suite
.venv/bin/python3 bench/run_bench.py --phases telemetry,warmup,perf,lmeval,coding,judge --repeats 2

# docker (canonical deployment)
docker build -t agentgw-bench:$(git rev-parse --short HEAD) .
docker run --rm -e BASE_URL=http://192.168.1.219:8000/v1 -e JUDGE_API_KEY=... \
    -v $PWD/results:/app/bench/results agentgw-bench:<tag> \
    --config bench/pipeline.yaml

The venv: python3 -m venv .venv && .venv/bin/pip install "lm-eval[api]" evalplus bfcl-eval soundfile (+ apk add libsndfile on Alpine/musl hosts). Torch guardrail: none of these pull torch; the Dockerfile hard-fails the build if torch ever appears via a transitive upgrade.

Phases

phase	tool	measures	notes
telemetry	(stdlib)	VRAM/RAM/power from /metrics	VRAM-hog detection before any load
warmup	(stdlib)	swap-in seconds	llama-swap loads the model on first request
tooluse	(stdlib) + BFCL	structured tool_calls parse; BFCL simple/parallel/irrelevance	smoke check catches the --jinja trap; BFCL wiring needs one live validation - recipe emitted per run
throughput	(stdlib)	single-shot TTFT/prefill/gen t/s @--tp-targets (default 512/8k/32k: llama-bench-style anchor + long-context checks)	randomized seeded prefixes (no prefix-cache flattery); target > model ctx -> null, run survives
perf	(stdlib)	single-shot TTFT/prefill/gen t/s (extra pass)	same exact timer as throughput; headline thresholds read the ~1k point
lmeval	lm-eval [api]	IFEval, MATH500, GSM8K (generation tasks)	/v1/completions, no chat template, --lmeval-limit per task (default 300, 0=full); MMLU-Pro needs prompt logprobs this gateway won't serve - opt in only if yours does; GPQA opt-in (--include-gpqa, HF-gated)
coding	EvalPlus	HumanEval+ pass@1, canonical non-thinking lane	raw /v1/completions, greedy, codegen stop list; --coding-limit N to sample; >25% empty completions fails the phase loudly
coding_thinking	EvalPlus	HumanEval+ pass@1, thinking mode (opt-in)	chat with enable_thinking, vendor sampling (temp 1.0, top_p 0.95), 8k budget - roughly 10x slower than the canonical lane; same guardrail
judge	(stdlib) 2-pass	open-ended quality (ADVISORY)	binary checklist; checkpointed + resumable
kl	(stdlib)	sym-KL/JS/top-1 agreement Q4 vs Q6	teacher-forced native /completion top-20 dists; same-tokenizer pairs only
cross-judge	(stdlib)	judge validity	different-family judge agreement, >= 75% required

--repeats 2 runs perf-ish phases block-reversed (model order reversed on the second pass) so thermal drift cannot masquerade as model identity; disagreements >10% are flagged in the report, never silently averaged.

Methodology baked in

• Reproducibility tuple in every results.json: endpoint, tool versions, python, seeds, repeats, perf config, gateway model list, image digest.
• Error bars: binomial SE per accuracy; cross-model deltas below 2*SE are annotated as noise in the report ("a 3-point win at n=400 is nothing").
• Transcripts gate: transcripts.md dumps raw responses (incl. lm-eval --log_samples); the report instructs: read ~50 before believing a delta - most first-pass differences are extraction artifacts.
• Percentiles, never means, for perf reporting.
• Resilience: 429 responses honor Retry-After (clamped 1-120s, default 15, 8 attempts); error messages include the response body snippet (the difference between self-diagnosing a bad model slug and guessing); --request-delay paces judge calls under hosted rate limits.

Thresholds / verdicts

bench/thresholds.yaml ships fully commented. Run a baseline, look at the numbers, uncomment your bars - every later run gets PASS/FAIL verdicts per model (speed, capability, tooluse, KL) in report.md and the pipeline exit code. Judge metrics are advisory and should not gate serving decisions. Results history in bench/results/ gives the regression line for the fleet.

Host-side companion (optional, needs shell on the fleet box)

bench/host_checks.sh: llama-bench / rocm-smi logging / llama-perplexity two-quant triage, run next to llama.cpp; fold results into a run dir as host_checks.json and the report renders them.

Frozen datasets (provenance)

bench/tasks/ holds the frozen subsets (seed 42, sources + SHA-256 in manifest.json): HumanEval-50, GSM8K-50, MMLU-100 (57 subjects), MMLU-dev (5-shot), IFEval-50 (stdlib-verifiable types only). Regenerate only deliberately: .venv/bin/python3 bench/prepare_tasks.py (needs requests/pandas/pyarrow). Judge checklists live in judge_tasks.jsonl.

llama-swap constraints honored

Models bench sequentially; first request pays swap-in (warm-up phase measures it); num_concurrent=1 everywhere; thinking models' reasoning_content parsed separately from verdict text.

Webapp API (bench/api.py, stdlib only)

BENCH_API_TOKEN=secret python3 bench/api.py        # 0.0.0.0:8088
# then from any webapp:
#   POST /api/bench {"phases":"warmup,perf"}   -> blocks, returns report JSON
#   POST /api/runs {"phases":"perf"}           -> 202 {run_id, status_url}
#   GET  /api/runs/<id>                        -> status/exit_code/report path
#   GET  /api/runs/<id>/report                 -> JSON summary (see schema below)
#   POST /api/runs/<id>/cancel                 -> terminate
#   GET  /api/health | /api/runs
# One active run at a time (llama-swap constraint); concurrent POST -> 409.
# Bearer auth only if BENCH_API_TOKEN is set. "pipeline": true invokes the
# full pipeline (judge + verdicts + exit codes) instead of a phase subset.

GET /api/runs/<id>/report returns the compact JSON report (application/json, same shape as the webhook payload: exit_code, verdict_per_model, per-model summary rows with headline perf + capability metrics - including throughput_by_ctx (TTFT/prefill/gen t/s per context size) and capacity (aggregate t/s + p95 TTFT/ITL per concurrency level; the price-per-token basis) - kl), built from the run's results.json once the run finishes; 404 before that. Run-level vram_baseline (GiB occupied before the run; null if not captured) qualifies all vram_peak_gib values - if the card wasn't empty at start, llama-swap got a handicapped card. The human-readable report.md stays on disk in the run dir; stdout-style logging goes to <run_id>.api.log in bench/results/.

POST /api/bench is the single-request contract for webapps: send the run body, the response IS the report JSON (connection stays open for the run's duration - set the HTTP client timeout accordingly). POST /api/runs + GET /api/runs/<id>/report remains the async alternative. If the client disconnects mid-run (Ctrl+C, webapp abort), the run is terminated server-side.

Tests

python3 -m unittest discover -s bench/tests   # works with system python3 AND the venv

Known gaps / TODO

• BFCL live wiring (recipe emitted per run; needs one validation session)
• no git remote yet - copy changed files to clones manually