Nibmeister/entropy-sampler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix ABI fragility: compile bindings against the real llama.h instead of hand-copied ctypes structs

Browse Source 
llama_capi.py previously hand-copied struct field order/types from one
specific llama.cpp commit's llama.h into Python ctypes Structures. A
different llama.cpp build could silently reorder or resize those fields and
corrupt memory rather than raising any error.

Replaced with a cffi "API mode" extension (build_capi.py) that #includes the
user's actual llama.h and links against their actual libllama.so. Struct
layout now comes from real compilation - a genuinely incompatible field
fails the build loudly instead of corrupting memory at runtime. Verified
byte-for-byte identical generation output against the prior ctypes
implementation at a fixed seed, plus the fork/peek logit round-trip check
(0.000000 max diff).

Requires a one-time `python build_capi.py` setup step (needs a C compiler,
which building llama.cpp itself already requires).
This commit is contained in:
Joey Grasty
2026-07-11 21:21:29 -05:00
parent e25ee046a3
commit 8c844c08e5
7 changed files with 232 additions and 187 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
README.md
View File

@@ -2,7 +2,7 @@
A prototype implementation of the "Future-Entropy Sampler" technique described in
[countbayesie's *Making LLMs Better at Creative Writing Using Entropy*](https://www.countbayesie.com/blog/2026/7/1/making-llms-better-at-creative-writing-using-entropy)
(2026-07-01), built directly against llama.cpp's C API via `ctypes`.
(2026-07-01), built directly against llama.cpp's C API.
Instead of sampling straight from `p(w | c)`, this sampler forks the model's KV
cache once per top-k candidate token, decodes one token into each fork to see
@@ -29,9 +29,10 @@ relying on it for anything beyond experimentation.
llama.cpp's `llama_memory_seq_cp`/`llama_memory_seq_rm` plus multi-sequence
batched `llama_decode` make the fork/peek/discard cheap: all `top_k` candidate
forks are decoded in a single batched call, not `top_k` sequential ones.
`llama-cpp-python` wasn't used - `llama_capi.py` binds the C API directly via
`ctypes`, which means the struct layouts in that file must match whatever
`libllama.so` you point it at (see Known limitations).
`llama-cpp-python` wasn't used - `llama_capi.py` binds the C API directly,
via a small `cffi` extension (`build_capi.py`) compiled against your actual
`llama.h` + `libllama.so`, so the struct layouts always match the exact
build you point it at (see **Setup**).
## Requirements
@@ -59,6 +60,24 @@ export ENTROPY_SAMPLER_MODEL=/path/to/your-model.gguf
`ENTROPY_SAMPLER_MODEL` is just the default - every entry point also accepts
`--model` to override it per run.
Then build the bindings once (requires a C compiler - you already need one to
have built llama.cpp itself):
```
python build_capi.py
```
This compiles a small extension directly against your `llama.h` and
`libllama.so`, so the struct layouts it uses come from real compilation
against your actual build rather than a hand-copied guess. It looks for
`llama.h` next to `LLAMA_CPP_LIB` (`<repo>/build/bin/libllama.so` ->
`<repo>/include/llama.h`, llama.cpp's normal layout) and for `ggml.h` in the
sibling `<repo>/ggml/include`; set `LLAMA_CPP_INCLUDE` / `LLAMA_CPP_GGML_INCLUDE`
explicitly if your layout differs. Re-run this after rebuilding llama.cpp
against a version that might have changed the structs/functions this project
uses - if something here is now incompatible, this step fails with a compiler
error, not a silent runtime crash.
## Usage
**One-shot CLI**, for testing a single generation with a given configuration:
@@ -106,11 +125,6 @@ constants:
## Known limitations
- **ABI-fragile**: `llama_capi.py`'s struct definitions are hand-copied from
one specific llama.cpp commit's `llama.h`. A different llama.cpp version
can silently reorder/resize struct fields and corrupt memory instead of
raising a clean error. Check the struct layouts against your build's
`llama.h` if you see crashes or garbage output.
- **Single request at a time**: one `EntropySampler` holds one `llama_context`
and one KV cache; `entropy_server.py` serializes requests with a lock. Not
built for concurrent multi-user serving.
@@ -142,7 +156,8 @@ constants:
| File | Purpose |
|---|---|
| `llama_capi.py` | `ctypes` bindings over `libllama.so` |
| `build_capi.py` | One-time build step: compiles `llama_capi.py`'s bindings against your `llama.h`/`libllama.so` |
| `llama_capi.py` | Bindings over `libllama.so`, backed by the extension `build_capi.py` compiles |
| `entropy_sampler.py` | Core `EntropySampler` class: `prime`/`step`/`generate`/`generate_stream` |
| `entropy_cli.py` | One-shot parameterized CLI |
| `entropy_server.py` | OpenAI-compatible HTTP server |