voice-bench/README.md

Voice Bench

A browser tool for exploring the Vox Day / Christopher Nuttall SFT packages and generating scenes from them against the joint CPT+SFT model, including a STYLE-swap control for testing voice conditioning across authors.

Full training report (CPT + SFT setup, results, and evaluation data): https://claude.ai/code/artifact/d3415f56-3f9b-40de-aa70-ac0992549ced

What's here

• voice-bench.html — the tool itself. It's a single self-contained file with no install step: open it in a browser (double-click it, or File > Open). It needs no server of its own — it just needs a model server to talk to (see below).
• start-model.sh — starts the model server (llama-server, from llama.cpp) that voice-bench.html sends generation requests to.

All 641 training/holdout packages across both SFT rounds are already embedded inside voice-bench.html — nothing else needs downloading for browsing. That's the original 208-package SFT (one book per author: A Sea of Skulls, Hour of the Wolf) plus the 433-package scale-up SFT (four more books per author, all held out of both CPT and the original SFT). Use the Book dropdown to switch between them; each package's card also shows which book it came from and whether it was used in training or held out.

One-time setup (per machine)

1. Get the model file

Two candidate checkpoints are available. Separability and length-calibration metrics lean slightly toward epoch 3 (see JOINT_TRAINING_CLOSEOUT_REPORT.md for the numbers), but not decisively enough to settle it on metrics alone — live testing through voice-bench.html is what should decide between them. Download both, run one at a time (or both at once on different ports — see step 3), and compare generations on real held-out packages before picking one.

• Qwen3-30B-A3B-VoxDay-Nuttall-SFT-ep3-Q8_0.gguf (~31GB): https://drive.google.com/open?id=1QZT488Uu1b9Pxoie9Hs2lnxtOhMD4Pwr
• Qwen3-30B-A3B-VoxDay-Nuttall-SFT-ep2-Q8_0.gguf (~31GB): https://drive.google.com/open?id=1q1w_XMp6oRmfx8xqXDV-HvDx_krVxQtD

Memory needed: ~31GB for the weights plus ~3GB of KV cache at the full 32768-token context, so ~35GB total. Confirmed working on a DGX Spark; should comfortably fit a 48GB L40 as well. If a machine has less than that, lower CTX_SIZE (below) before anything else.

2. Build llama.cpp with CUDA support

Needs an NVIDIA GPU with the CUDA toolkit and cmake already installed.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

This produces llama.cpp/build/bin/llama-server, which is what start-model.sh runs.

3. Start the model server

Point MODEL_PATH at whichever checkpoint you're testing:

MODEL_PATH=/path/to/Qwen3-30B-A3B-VoxDay-Nuttall-SFT-ep3-Q8_0.gguf \
LLAMA_SERVER=/path/to/llama.cpp/build/bin/llama-server \
./start-model.sh

Leave this running in a terminal — it's the process voice-bench.html talks to. If you'd rather not set environment variables every time, edit the defaults at the top of start-model.sh instead.

To compare epoch 2 and epoch 3 directly, run both at once on different ports (memory permitting — see below):

MODEL_PATH=/path/to/Qwen3-30B-A3B-VoxDay-Nuttall-SFT-ep2-Q8_0.gguf PORT=8200 ./start-model.sh
MODEL_PATH=/path/to/Qwen3-30B-A3B-VoxDay-Nuttall-SFT-ep3-Q8_0.gguf PORT=8201 ./start-model.sh

then switch voice-bench.html's "Server" field between :8200 and :8201 on the same packet to compare their generations side by side.

4. Open voice-bench.html

Open the file in a browser. The "Server" field in the top-right corner defaults to http://127.0.0.1:8200, which is correct if the model server is running on the same machine as the browser. If it's running on a different machine on your network, change it to that machine's address instead, e.g. http://192.168.1.50:8200 — and make sure that machine's firewall allows inbound connections on the port (sudo ufw allow from <your-subnet> to any port 8200 proto tcp on Ubuntu).

Troubleshooting

• "SERVER UNREACHABLE" in the page header — the model server isn't running yet, is still loading the model (can take a minute for a 31GB file), or the Server field points at the wrong address/port.
• Out of memory when starting the server — re-run with a smaller context, e.g. CTX_SIZE=8192 ./start-model.sh.
• Opening the HTML file from a typed path does something odd in Firefox — use File > Open (Ctrl+O) instead of typing the path into the address bar, or paste the full file:///home/you/path/voice-bench.html URL.