Joey Grasty
7ac0b36ae6
Metrics lean toward ep3 but aren't decisive enough to call it outright. README now presents both checkpoints as candidates and walks through running them side-by-side (different ports) for direct comparison in voice-bench.html, rather than naming one as current production.
4.2 KiB
Voice Bench
A browser tool for exploring the Vox Day / Christopher Nuttall SFT packages and generating scenes from them against the joint CPT+SFT model, including a STYLE-swap control for testing voice conditioning across authors.
Full training report (CPT + SFT setup, results, and evaluation data): https://claude.ai/code/artifact/d3415f56-3f9b-40de-aa70-ac0992549ced
What's here
voice-bench.html— the tool itself. It's a single self-contained file with no install step: open it in a browser (double-click it, or File > Open). It needs no server of its own — it just needs a model server to talk to (see below).start-model.sh— starts the model server (llama-server, from llama.cpp) thatvoice-bench.htmlsends generation requests to.
The 208 training/holdout packages themselves are already embedded inside
voice-bench.html — nothing else needs downloading for browsing.
One-time setup (per machine)
1. Get the model file
Two candidate checkpoints are available. Separability and length-calibration
metrics lean slightly toward epoch 3 (see JOINT_TRAINING_CLOSEOUT_REPORT.md
for the numbers), but not decisively enough to settle it on metrics alone —
live testing through voice-bench.html is what should decide between
them. Download both, run one at a time (or both at once on different
ports — see step 3), and compare generations on real held-out packages
before picking one.
Qwen3-30B-A3B-VoxDay-Nuttall-SFT-ep3-Q8_0.gguf(~31GB): https://drive.google.com/open?id=1QZT488Uu1b9Pxoie9Hs2lnxtOhMD4PwrQwen3-30B-A3B-VoxDay-Nuttall-SFT-ep2-Q8_0.gguf(~31GB): https://drive.google.com/open?id=1q1w_XMp6oRmfx8xqXDV-HvDx_krVxQtD
Memory needed: ~31GB for the weights plus ~3GB of KV cache at the full
32768-token context, so ~35GB total. Confirmed working on a DGX Spark;
should comfortably fit a 48GB L40 as well. If a machine has less than that,
lower CTX_SIZE (below) before anything else.
2. Build llama.cpp with CUDA support
Needs an NVIDIA GPU with the CUDA toolkit and cmake already installed.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
This produces llama.cpp/build/bin/llama-server, which is what
start-model.sh runs.
3. Start the model server
Point MODEL_PATH at whichever checkpoint you're testing:
MODEL_PATH=/path/to/Qwen3-30B-A3B-VoxDay-Nuttall-SFT-ep3-Q8_0.gguf \
LLAMA_SERVER=/path/to/llama.cpp/build/bin/llama-server \
./start-model.sh
Leave this running in a terminal — it's the process voice-bench.html
talks to. If you'd rather not set environment variables every time, edit the
defaults at the top of start-model.sh instead.
To compare epoch 2 and epoch 3 directly, run both at once on different ports (memory permitting — see below):
MODEL_PATH=/path/to/Qwen3-30B-A3B-VoxDay-Nuttall-SFT-ep2-Q8_0.gguf PORT=8200 ./start-model.sh
MODEL_PATH=/path/to/Qwen3-30B-A3B-VoxDay-Nuttall-SFT-ep3-Q8_0.gguf PORT=8201 ./start-model.sh
then switch voice-bench.html's "Server" field between :8200 and :8201
on the same packet to compare their generations side by side.
4. Open voice-bench.html
Open the file in a browser. The "Server" field in the top-right corner
defaults to http://127.0.0.1:8200, which is correct if the model server is
running on the same machine as the browser. If it's running on a different
machine on your network, change it to that machine's address instead, e.g.
http://192.168.1.50:8200 — and make sure that machine's firewall allows
inbound connections on the port (sudo ufw allow from <your-subnet> to any port 8200 proto tcp on Ubuntu).
Troubleshooting
- "SERVER UNREACHABLE" in the page header — the model server isn't running yet, is still loading the model (can take a minute for a 31GB file), or the Server field points at the wrong address/port.
- Out of memory when starting the server — re-run with a smaller
context, e.g.
CTX_SIZE=8192 ./start-model.sh. - Opening the HTML file from a typed path does something odd in
Firefox — use File > Open (Ctrl+O) instead of typing the path into the
address bar, or paste the full
file:///home/you/path/voice-bench.htmlURL.