Nibmeister/entropy-sampler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix ABI fragility: compile bindings against the real llama.h instead of hand-copied ctypes structs

Browse Source 
llama_capi.py previously hand-copied struct field order/types from one
specific llama.cpp commit's llama.h into Python ctypes Structures. A
different llama.cpp build could silently reorder or resize those fields and
corrupt memory rather than raising any error.

Replaced with a cffi "API mode" extension (build_capi.py) that #includes the
user's actual llama.h and links against their actual libllama.so. Struct
layout now comes from real compilation - a genuinely incompatible field
fails the build loudly instead of corrupting memory at runtime. Verified
byte-for-byte identical generation output against the prior ctypes
implementation at a fixed seed, plus the fork/peek logit round-trip check
(0.000000 max diff).

Requires a one-time `python build_capi.py` setup step (needs a C compiler,
which building llama.cpp itself already requires).
This commit is contained in:
Joey Grasty
2026-07-11 21:21:29 -05:00
parent e25ee046a3
commit 8c844c08e5
7 changed files with 232 additions and 187 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
fork_peek_test.py
View File

@@ -88,7 +88,7 @@ def main():
n_ctx_pos = len(prompt_tokens) # next free position on seq 0
logits_ptr = C.lib.llama_get_logits_ith(ctx, -1)
base_logits = np.ctypeslib.as_array(logits_ptr, shape=(n_vocab,)).copy()
base_logits = C.as_float_array(logits_ptr, n_vocab)
base_probs = softmax(base_logits)
top_idx = np.argsort(-base_probs)[:TOP_K_BASE]
@@ -105,7 +105,7 @@ def main():
decode_single_seq_batch(ctx, [w], start_pos=n_ctx_pos, seq_id=fork_seq)
fork_logits_ptr = C.lib.llama_get_logits_ith(ctx, -1)
fork_logits = np.ctypeslib.as_array(fork_logits_ptr, shape=(n_vocab,)).copy()
fork_logits = C.as_float_array(fork_logits_ptr, n_vocab)
fork_probs = softmax(fork_logits)
top_future_idx = np.argsort(-fork_probs)[:TOP_N_FUTURE]
@@ -123,7 +123,7 @@ def main():
# same candidate directly on seq 0 must reproduce identical logits
decode_single_seq_batch(ctx, [w], start_pos=n_ctx_pos, seq_id=0)
check_ptr = C.lib.llama_get_logits_ith(ctx, -1)
check_logits = np.ctypeslib.as_array(check_ptr, shape=(n_vocab,)).copy()
check_logits = C.as_float_array(check_ptr, n_vocab)
max_diff = float(np.max(np.abs(check_logits - fork_logits)))
print(f"\nseq0-vs-fork logit max abs diff (expect ~0): {max_diff:.6f}")