Nibmeister/auto-antislop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Complete FTPO training run; document orchestration bug fix and thermal shutdown

Browse Source 
Ran the full FTPO fine-tune to completion: 4h12m wall-clock, early-stopped at
step 380/750 by the configured chosen_win=0.85 threshold, train_loss 4.0->2.19.
Output is a merged 16-bit model plus LoRA adapter.

Also fixes an UnboundLocalError in core/orchestration.py when resuming a run
with generation disabled (--generation-step-enabled false), which is needed to
re-run just the finetune stage against an already-generated dataset.

Documents an unrelated platform incident: an uncapped multi-hour finetune
triggered a thermal shutdown mid-run with no checkpoint saved, requiring a
restart from step 0. Capping GPU clocks/power before the retry let it complete
without issue.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141RdLKiMeXLMXsWyb4U3B5
This commit is contained in:
Joey Grasty
2026-09-04 20:52:55 -05:00
parent c66cf7a947
commit 3d05fa42bf
2 changed files with 48 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
core/orchestration.py
View File

@@ -387,7 +387,7 @@ def orchestrate_pipeline(config: Dict[str, Any], experiment_dir: Path, resume_mo
start_iter_idx = max_found_iter + 1
logger.info(f"Resuming from iteration {start_iter_idx}.")
# Log presence of existing ban lists if resuming past iter 0
if start_iter_idx > 0:
if start_iter_idx > 0 and generation_enabled:
if banned_ngrams_json_path.exists(): logger.info(f"Resuming with existing n-gram ban list: {banned_ngrams_json_path}")
else: logger.info("No existing n-gram ban list found to resume with for subsequent iterations.")
if banned_slop_phrases_json_path.exists(): logger.info(f"Resuming with existing slop phrase ban list: {banned_slop_phrases_json_path}")