CLM 8B System One Benchmark

Sun Sep 27 2026

CLM-8B — Contrastive Language Model (System One)

Encoder port: 8091 (Qwen3-8B, vLLM pooling runner — internal, not for direct use) API port: 8700 Venv: /data/vllm_env/venv (shares the existing vLLM install; contrastive-lm depends on vllm>=0.6) Model: Contrastive-LM/CLM-v0.1-8B — released 2026-09-23. Frozen Qwen/Qwen3-8B encoder + a ~75MB trainable projection head (auto-downloaded to ~/.cache/clm/ on first run).

CLM doesn’t generate text — it scores a fixed list of candidate answers/actions against a state/question via contrastive embeddings, similar to a reranker. Good fit for action selection, verification, and agent routing where you already have a candidate set.

Two processes: a vLLM server running Qwen3-8B in --runner pooling mode (produces embeddings only, no text generation) feeds the small clm-serve API, which applies the trained projection heads and does the actual ranking/scoring.

--gpu-memory-utilization 0.16 on the encoder is deliberately low (~20GB target) — this box also runs Ollama, ComfyUI, and Jupyter concurrently; the default 0.9 tries to grab ~115GB of the 128GB unified pool and starves everything else (verified: dropped free RAM to 1.3GB and pushed the box into swap). Raise it only if you’ve checked current headroom with free -h first.

Installation

# contrastive-lm depends on vllm>=0.6, already satisfied by the existing venv
/data/vllm_env/venv/bin/pip install contrastive-lm

# encoder weights (~16GB, one-time)
cd /data/setup && ./scripts/download_model.sh Qwen/Qwen3-8B

Install the systemd services

cp /data/setup/clm-encoder.service /data/setup/clm-serve.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now clm-encoder
systemctl enable --now clm-serve
curl http://localhost:8700/health

Start manually (without systemd)

export HF_HOME=/data/hf_cache CUDA_VISIBLE_DEVICES=0

/data/vllm_env/venv/bin/vllm serve Qwen/Qwen3-8B \
  --served-model-name qwen3-8b --runner pooling \
  --max-model-len 2048 --port 8091 --gpu-memory-utilization 0.16 &

/data/vllm_env/venv/bin/clm-serve --port 8700 \
  --emb-url http://127.0.0.1:8091/v1/embeddings &

First start of clm-serve downloads the ~75MB checkpoint to ~/.cache/clm/CLM_v0.1-8B.pt.

API Examples

Health check

curl http://localhost:8700/health

Rank candidates (the common case — action/answer selection)

curl -X POST http://localhost:8700/v1/rank \
  -H "Content-Type: application/json" \
  -d '{
    "context": "The web server went down and is throwing a 502 bad gateway error.",
    "question": "What should be done?",
    "answers": [
      "sudo systemctl restart nginx",
      "git pull origin main && docker-compose up --build",
      "cat /var/log/syslog | grep ERROR"
    ]
  }'

Returns ranked: [{rank, candidate, prob}], highest probability first.

System-one Q&A (typed questions over a state)

curl -X POST http://localhost:8700/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Customer: my invoice was charged twice",
    "questions": {
      "urgency": {"type": "noul", "instructions": "Is this urgent?"},
      "department": {"type": "choice", "instructions": "Which team?",
        "criteria": {"billing": "Charges, invoices", "support": "General help"}}
    }
  }'

Web playground

http://localhost:8700/

Internals

• The encoder is a plain vLLM pooling server — Qwen3-8B with --convert embed behavior auto-selected via --runner pooling; no chat template, no sampling. Supported tasks: ['token_embed', 'embed'] in its startup log confirms this.
• clm-serve embeds state and each candidate answer separately (via the encoder’s /v1/embeddings), then scores with the frozen contrastive heads. States/actions can be pre-embedded and cached — repeated candidates skip the encoder entirely (action-cache in the CLI args).
• clm-serve needs a CUDA device for the projection heads themselves (--device, defaults to cuda if available) — separate from the encoder’s GPU use.

Benchmarking

scripts/clm_benchmark.py runs a 20-item hand-written quality test set (scripts/clm_testset.json — sysadmin/incident actions, support routing, factual QA, hard near-duplicate distractors, negation, code review, severity triage) through /v1/rank, a cold-vs-warm latency sweep at 5/10/25/50/100 candidates, a concurrent-throughput sweep, and samples GPU util/power/temp + host CPU/RAM throughout via a background thread (nvidia-smi + psutil, every 0.2s).

/data/vllm_env/venv/bin/python scripts/clm_benchmark.py

Results as of 2026-09-27 (Qwen3-8B encoder, qwen3-8b/clm-latest, GB10, sharing the box with Ollama/ComfyUI/Jupyter):

Quality — Top-1 accuracy: 85% (17/20), MRR 0.908, mean P(correct) 0.735

Failure pattern: all 3 misses were against short, label-style candidates (“billing” vs “engineering”, “P0” vs “P3”) where the model picked the more dramatic/action-heavy option regardless of fit — e.g. ranked “P0 - critical, all-hands” at prob 0.971 for a cosmetic Safari-only CSS bug, and “restart the database service” at 0.914 over the actually-correct “du -sh … ” for a disk-full error. Rich, information-dense candidate phrasing (full commands, full sentences) was reliably scored correctly; bare categorical labels were not. Practical takeaway: phrase candidates as full descriptive actions/sentences, not short labels, when precision matters.

Speed (single request, sequential):

Cold = candidates never seen before (must embed context + every candidate). Warm = candidate set already in the server’s action-cache (only the state/context needs embedding — this is the intended production usage: pre-embed a fixed action space once, rank against it near-instantly after). The action-cache is keyed by exact text and persists for the life of the clm-serve process — a naive re-run of a “cold” benchmark with fixed/deterministic candidate strings will silently measure warm performance on the second run onward. The benchmark script works around this with a random per-run ID baked into generated text.

Throughput (concurrent requests against a warm 20-candidate cache):

Throughput plateaus around ~1000-1200 req/s from concurrency 4 upward — it doesn’t scale much further with more concurrent clients, consistent with clm-serve running as a single process/worker (GIL- or single-CUDA-stream-bound for the projection-head math on warm/cached requests). Still ample for interactive agent-routing use cases; don’t expect it to scale linearly by throwing more concurrent clients at one instance.

Resource usage (sampled every 0.2s during each phase):

Power draw stays low (<45W) throughout — this workload never comes close to saturating GB10’s power envelope; it’s latency/serialization-bound, not compute-bound. RAM sits flat at ~59GB/45% (the two CLM processes + everything else already running on the box) — no memory growth observed across the whole benchmark run, i.e. no cache-related leak under this load.

Raw per-item results land in scripts/clm_benchmark_results.json (not committed — treat as a local scratch artifact; rerun the script to regenerate).

Use Case Validation

The generic benchmark above tests CLM in the abstract. These three scripts test the specific use cases this model is actually proposed for, against real material from this box (its own docs, synthetic agent scenarios, and real LLM-generated code fixes). Net result: CLM-8B is much more failure-prone in practice than the generic benchmark suggested — every use case needs its candidate set engineered around the same weakness (surface-level lexical/topical overlap between candidates breaks it), and one use case (fine-grained code verification) doesn’t work at all.

Use Case 1 — Agent tool/action selection

scripts/clm_usecase_agent_actions.py + ..._testset.json — 14 tools (phrased as full descriptive sentences), 20 sequential agent-task scenarios, one correct tool each.

• Top-1 accuracy: 50% (10/20), MRR 0.625
• Systematic bias, not random error: “run the test suite and report which tests failed” alone was picked top-1 in 6/20 cases (30%) despite being correct only once. Together with “open a pull request” (4/20, correct once) and “check the service’s live logs” (4/20, correct twice), 3 of 14 tools absorbed 70% of all decisions regardless of the actual task description. This is the same “dramatic/generic candidate wins regardless of fit” pattern as the P0-severity and qwen3-tts findings elsewhere in this doc — it generalizes across all three use cases, not a one-off.
• Latency correction to the main benchmark’s headline number: real agent loops have a novel state every turn (only the action list is fixed/cached). Measured per-decision latency here was ~78ms, not ~1-2ms — the original “150-340x cache speedup” number was measured with both state and candidates repeated verbatim, which never happens in real usage. The actual benefit of pre-caching a fixed action space is avoiding re-embedding N candidates every turn (a real ~2-6x win at this action-space size per the main benchmark’s cold-sweep numbers), not the 200x+ figure.
• Verdict: usable as a soft prior/pre-filter over tool choices, not as a sole decision-maker — 50% top-1 with a strong bias toward a few tools is not safe to act on directly.

Use Case 2 — Verifier/reranker for coder-LLM output

scripts/clm_usecase_coder_testset.json (frozen fixture) + clm_usecase_coder_verifier.py — 8 canonical bug patterns (off-by-one, null-deref, N+1 query, race condition, bad regex, SQL injection, wrong exception type, resource leak), each with 3 candidate fixes (1 correct, 2 plausible-wrong) and CLM asked to pick the correct one.

Candidates were generated once via Ollama (qwen2.5-coder:latest, deepseek-coder-v2:16b, already resident — no new model load) at varying temperature. Notable side-finding: both models fixed these canonical bugs correctly essentially every time — genuine wrong candidates had to be hand-authored (2 of 3 per scenario) because the models rarely produced a bad fix naturally, even at temperature 1.0. One exception used as-is: deepseek-coder-v2 really did leave the bad-regex bug completely unfixed on one run.

• Top-1 accuracy: 12.5% (1/8) — below the 33% chance rate for 3 candidates. MRR 0.458.
• Only correctly picked when the candidates diverged sharply in approach (n-plus-1: the correct fix restructures the whole query into a batched IN (...), the wrong ones stay structurally close to the N+1 original — prob 0.911 on the correct one).
• Every other miss involved candidates differing by only 1-2 lines (a missing with lock:, a swapped exception type, a regex character class tweak) — CLM ranked the wrong variant higher, sometimes with high confidence (prob 0.744 on a still-broken off-by-one fix).
• Verdict: do not use CLM-8B as a code-fix verifier at this granularity. Its embeddings can’t reliably resolve single-line/single-token differences that flip correctness — this needs execution/testing or an actual diff-aware checker, not embedding similarity. This contradicts the model card’s SOTA-verifier claims for DeepSWE/Terminal-Bench-style tasks; the discrepancy is plausibly that CLM’s own trained checkpoint was tuned on tasks with more behaviorally-distinct candidates than “same function, one line different.”

Use Case 3 — Semantic routing over this repo’s docs

scripts/clm_usecase_doc_routing.py + two testset variants — the 7 docs in docs/ as candidates, 14 natural-language queries (2 per doc).

• v1 (bare functional descriptions, e.g. “text-to-speech service … port 8883”): 42.9% accuracy, MRR 0.679. The 3 TTS docs share heavy vocabulary (“text-to-speech”, “voice cloning”, “port”); qwen3-tts won several queries it had no business winning (e.g. beat Whisper 0.511 vs 0.086 on “how do I transcribe audio into text” — CLM matched on the shared token “text” rather than resolving that transcription is the reverse direction of TTS).
• v2 (same queries, candidates rewritten to state direction explicitly — “Converts spoken audio into written text” vs “Converts written text into spoken audio”): 71.4% accuracy, MRR 0.821 — confirms the fix. Remaining misses are exclusively between the 3 near-identical TTS docs (Chatterbox vs F5 vs Qwen3-TTS) — distinguishing “smallest” or “supports Chinese” needs attribute-level reasoning embeddings don’t reliably carry.
• Verdict: viable for routing over a corpus of topically-distinct documents; degrades sharply (and predictably) once several candidates cover the same functional category — rewrite candidates to state input/output direction and distinguishing attributes explicitly before trusting this for anything beyond a handful of clearly-distinct docs.

Resource usage across all three (measured, not estimated)

GPU power stayed under 30W throughout all three use-case runs — consistent with the main benchmark, this is a serialization/latency-bound workload, never compute-bound. The Use Case 2 generation step (spinning up Ollama’s qwen2.5-coder + deepseek-coder-v2:16b to produce real candidate fixes) is the one part of this exercise that materially stressed the box: deepseek-coder-v2:16b loaded at 49GB resident (its 64k-token context window, far above its 8.9GB on-disk size) and pushed total system RAM to 109GB/121GB (90%) with swap engaged, alongside the always-on CLM services and everything else already running. Stopped both Ollama models immediately after generation (ollama stop <model>) rather than waiting for the default keep-alive timeout; RAM recovered to 52GB/121GB within seconds. Takeaway: don’t run large-context Ollama models on this box at the same time as anything else memory- heavy without an explicit unload step — Ollama’s keep-alive alone left it resident for minutes with no other consumer of the memory in sight.

Troubleshooting

Low free memory / swapping after starting the encoder:

free -h

Lower --gpu-memory-utilization on the encoder (0.16 ≈ 20GB target; drop further if tight).

clm-serve reports embedder: false on /health: the vLLM encoder isn’t up yet or --emb-url doesn’t match its actual port (8091 here, not the CLM default of 8090 — that port is already used by traefik on this box).

Service not starting:

journalctl -u clm-encoder --since "5 min ago" --no-pager
journalctl -u clm-serve --since "5 min ago" --no-pager