Voice PC Assistant Benchmark Report

Deterministic text-only benchmark of the local voice-driven PC assistant path. The harness used ToolCallingAgent, the real desktop tool registry, a real controlled browser, and local fixture pages. STT, TTS, public web drift, and raw model-only prompting were excluded from scored results. All runs were measured on an NVIDIA GeForce RTX 4080 Super test system.

Final run: 20260505-000652-loop5-all Date: May 5, 2026 Hardware: NVIDIA GeForce RTX 4080 Super Tasks: 10 deterministic PC-assistant workflows Target: reliable 8/10 or better

Executive Summary

Best Overall

qwen3.5-4b-q4_k_m

8/10

Repeated post-fix scores: 8/10, 9/10, 8/10.

Fastest But Weakest

gemma-4-e4b-it-q4_k_m

3/10

Median latency was lowest, but accuracy suffered from guessed fixture paths and early give-ups.

Middle Result

granite-4.1-8b-q4_k_m

6/10

Good on several browser tasks, but often used too many steps and timed out on simple workflows.

Measurement Story

Loops 1 and 2 intentionally used Qwen only as a fast probe model to harden shared assistant and harness behavior. They were not comparative model runs, because the failures being fixed affected every model path: stale memory, recovery handling, fixture routing, and max-step policy. After those shared issues were fixed, the final comparison measured Qwen, Granite, and Gemma on the same deterministic 10-task suite, with one llama.cpp server running at a time.

Run Stage Qwen Granite Gemma What Changed Or Was Learned
baseline-qwen
probe run		 4/10 not run not run Qwen-only probe to expose shared product failures: stale memory, immediate stop on recoverable tool errors, and public web shortcuts.
loop1-qwen
probe run		 5/10 not run not run Still Qwen-only by design. Isolated benchmark memory per case and tightened deterministic fixture context before comparing models.
loop2-qwen
probe run		 8/10 not run not run Final Qwen-only hardening probe. Allowed recovery after failed tool observations, required real navigation, and raised max steps to 12.
loop3-all
first comparison attempt		 9/10 6/10 partial not reached First all-model attempt confirmed Qwen strength and exposed a report-printing Unicode crash before Gemma could run.
loop5-all
final comparison		 8/10 6/10 3/10 Final complete measurement: Qwen met target reliability, Granite was usable but step-heavy, and Gemma was fast but inaccurate.

Model Outcome Story

Qwen: Reliable Enough

8/10 final

Qwen was the only model to repeatedly meet the 8/10 target after shared assistant fixes. Its main remaining failure was video follow-up drift.

Granite: Accurate In Places, Step-Heavy

6/10 final

Granite passed official docs, video seeking, recovery, file lookup, and time reasoning, but ran out of steps on several workflows.

Gemma: Fast But Not Reliable

3/10 final

Gemma had the best latency profile, but repeatedly guessed local paths and stopped after 404 pages instead of recovering through the home page.

Final Model Comparison

Model Score Median Task P90 Task Total LLM Failed Tools Read
qwen3.5-4b-q4_k_m 8/10 6421 ms 14978 ms 31121 ms 3 Best accuracy; still weak on video follow-up drift.
granite-4.1-8b-q4_k_m 6/10 5886 ms 19456 ms 25306 ms 0 Competent, but step-heavy and prone to max-step stops.
gemma-4-e4b-it-q4_k_m 3/10 2514 ms 4701 ms 11711 ms 1 Fastest; accuracy not strong enough for PC operation.

“Not run” in the probe rows means “not part of that hardening stage,” not that the models failed or were skipped during final scoring. Granite and Gemma were scored once the shared benchmark path was stable enough to make the comparison meaningful.

Case-Level Results

Task Qwen Granite Gemma What It Tested
Browser Link Following PASS 3622 ms FAIL 19456 ms PASS 3431 ms Navigate from home page, click Reference Domains, answer maintainer.
Search Result Disambiguation PASS 6107 ms PASS 5263 ms FAIL 2031 ms Choose official NVIDIA result over blog/forum decoys.
Video Result Disambiguation PASS 6735 ms PASS 5013 ms FAIL 2089 ms Find the music video, not an interview or trailer.
Video Seeking FAIL 14978 ms PASS 6509 ms FAIL 2489 ms Use follow-up memory and seek/read the local video state at 1:30.
Sponsored And Shorts Filtering FAIL 5082 ms FAIL 5253 ms FAIL 4442 ms Avoid sponsored and Shorts decoys, open full walkthrough.
Form Fill From Evidence PASS 11393 ms FAIL 20307 ms FAIL 928 ms Read customer record, fill transfer form, submit, report code.
Multi-Step Documentation Navigation PASS 8863 ms FAIL 6930 ms FAIL 2539 ms Open docs, follow setup guide, answer latency setting follow-up.
Error Recovery PASS 37016 ms PASS 8986 ms FAIL 12101 ms Handle stale 404 link and recover via site navigation.
File/Index Lookup PASS 5110 ms PASS 3466 ms PASS 4701 ms Compare dated guide entries and read newest first instruction.
Mixed Browser Plus Time Reasoning PASS 3847 ms PASS 3527 ms PASS 2421 ms Combine schedule page evidence with the local time tool.

Run Progression

Run Model(s) Score Median Latency Key Learning
20260504-235450-baseline-qwen Qwen 4/10 8500 ms Initial harness exposed stale refs, public web drift, and unrecovered tool failures.
20260504-235718-loop1-qwen Qwen 5/10 7824 ms Memory isolation helped, but the agent still masked recovered work after earlier failures.
20260504-235946-loop2-qwen Qwen 8/10 6537 ms Recovery behavior, clearer local-site policy, and a 12-step cap reached the target.
20260505-000209-loop3-all Qwen, Granite partial Qwen 9/10 Not final Run crashed only while printing Unicode to the Windows console; results confirmed qwen strength.
20260505-000652-loop5-all Qwen, Granite, Gemma 8/10, 6/10, 3/10 6421 / 5886 / 2514 ms Final comparison: qwen was reliable enough; Gemma was fast but inaccurate.

Behavior Notes

Qwen

  • Best balance of tool choice, concise answers, and recovery.
  • Remaining risk: video follow-ups can drift to public YouTube if the prior turn only identifies the local video instead of opening it.
  • Error recovery can be slow: the stale-link case took 37016 ms and 9 tool calls.

Granite

  • No failed tool calls in the final run, but several tasks exhausted the step budget.
  • Passed video seeking where qwen failed in the final run.
  • Some scorer misses came from punctuation variants such as nonstandard hyphen characters.

Gemma

  • Very fast median latency, but often guessed paths like /docs or /videos.
  • Frequently stopped after 404 pages instead of returning to the test home page.
  • Not reliable enough for the current PC assistant path despite speed.

Artifacts

Artifact Purpose
scripts/benchmark_pc_assistant_loop.py Benchmark runner: serves fixtures, starts one llama.cpp model at a time, drives ToolCallingAgent, records JSON.
benchmarks/fixtures/pc_assistant/ Deterministic local pages for browser, docs, video, form, recovery, file, and schedule tasks.
benchmarks/results/*summary.json Machine-readable summaries for each run.
benchmarks/results/*-{model}.json Full per-model traces: prompts, replies, tool calls, observations, timings, and case outcomes.
notebook.md Human-readable loop notes and improvement observations.

Verification after code changes: pytest reported 44 passed tests. Port 8080 was clear after the benchmark run.