diff --git a/DGX_SPARK_SETUP.md b/DGX_SPARK_SETUP.md index fedd566..47017d2 100644 --- a/DGX_SPARK_SETUP.md +++ b/DGX_SPARK_SETUP.md @@ -131,8 +131,8 @@ serving 50 concurrent generation threads. ## 6. Code patches needed for current library versions The repo (as of the commit we tested against) predates some of the exact library versions that -`pip install` resolves to today. Three small patches were needed — none are DGX-Spark-specific, -they'd bite on any platform once these versions are current on PyPI: +`pip install` resolves to today. Four small patches were needed — none are DGX-Spark-specific, +they'd bite on any platform in the same circumstances: **`utils/vllm_manager.py`** — vLLM 0.26.0 removed the `--disable-log-requests` flag (replaced by an opt-in `--enable-log-requests`, off by default already). Delete the line: @@ -155,6 +155,23 @@ training, so all-zero/all-text is correct): - the reference-model forward pass, `self.ref_model is None` branch (inside `null_ref_context()`) - the reference-model forward pass, `self.ref_model is not None` branch +**`core/orchestration.py`** — resuming a run with generation disabled +(`--generation-step-enabled false`, used to re-run just the finetune stage against an +already-generated dataset) crashes: +``` +UnboundLocalError: cannot access local variable 'banned_ngrams_json_path' where it is not associated with a value +``` +`banned_ngrams_json_path` / `banned_slop_phrases_json_path` are only ever assigned inside the +`if generation_enabled:` block, but a later resume-logging block references them unconditionally. +Guard that block with the same flag: +```python +if start_iter_idx > 0 and generation_enabled: + if banned_ngrams_json_path.exists(): ... +``` +(There's a second, non-fatal instance of the same root cause — `iter0_output_file_for_dpo` +referenced before assignment when loading stats from a prior completed run — it's caught and +logged as a warning rather than raised, so it doesn't block anything, but it's the same bug shape.) + ## 7. FTPO fine-tuning was slow because of fixed-length padding, not raw compute With `finetune_batch_size: 1` / `gradient_accumulation_steps: 16`, we measured ~750 optimizer steps @@ -210,18 +227,40 @@ run. If you need it faster still, the other lever is `finetune_max_train_examples` (fewer total steps, less data coverage) — or just accept the ~8h runtime and let it run in the background. +## 8. Sustained full-GPU load can trigger a thermal shutdown + +An unthrottled attempt at the full finetuning run caused the machine to shut down from heat partway +through (no OOM/crash signature in the logs — the process and GPU state simply vanished when the +box power-cycled). No checkpoint had been written yet, so the run restarted from step 0 — FTPO +doesn't checkpoint mid-run by default, so a mid-run shutdown here means losing all progress so far. + +Fix: cap GPU clocks/power before starting a sustained multi-hour job (finetuning, not the bursty +generation step): +```bash +nvidia-smi -pl # if supported on this platform +# or check nvidia-smi -q -d CLOCK,POWER for current caps and headroom +``` +With clocks capped, the retry ran the full ~4h12m to completion without incident — GPU temperature +held in the 71-81°C range under sustained 90%+ utilization throughout. + ## Validated results Ran the full pipeline against `unsloth/gemma-3-4b-it` (2 iterations, 1200 prompts each): - Iteration 0 (baseline, no bans): completed in 28m24s, `repetition_per_100k_chars` = 160 - Iteration 1 (with ban lists from iteration 0's analysis): completed in 1h31m43s (slower — active - backtracking around bans), `repetition_per_100k_chars` = 56 — a real, measured reduction in slop -- FTPO training: confirmed working end-to-end (750 steps, 12,000 preference pairs) after the - patches in §6. At the original `finetune_max_seq_length: 4000`, steady-state was ~160-185s/step - (~34h for the full run). After the fix in §7 (`finetune_max_seq_length: 1280`), measured - ~40s/step over a 30-minute validation run — a confirmed ~4.3x speedup, ~8.3h extrapolated for the - full 750 steps. Not run to full completion. + backtracking around bans), `repetition_per_100k_chars` = 56 — a real, measured reduction in slop. + Both numbers are from a run predating the `refusal_detector.py` fix in §6, so they reflect + n-gram/phrase-ban slop reduction only, not refusal-aware filtering. +- FTPO training: **ran to completion.** With the fixes from §6 and §7 + (`finetune_max_seq_length: 1280`), trained on the 12,000-example preference-pair dataset (quota- + sampled from 58,369 raw pairs) at `finetune_batch_size: 1` / `gradient_accumulation_steps: 16`. + Stopped early by the configured `finetune_early_stopping_wins: 0.85` threshold — `chosen_win` + (fraction of chosen completions preferred over rejected) crossed 0.85 at step 380 of a possible + 750 (1 epoch). `train_loss` fell from ~4.0 to 2.19 over the run. Total wall-clock time + (training + LoRA save + 16-bit merge): **4h12m32s**, steady-state ~40s/step throughout — matching + the §7 prediction. Output: LoRA adapter and a merged 16-bit model (8.1GB) under + `finetuned_model_ftpo_exp01/`. ## Quick-reference: full env setup diff --git a/core/orchestration.py b/core/orchestration.py index 20c4997..7911b84 100644 --- a/core/orchestration.py +++ b/core/orchestration.py @@ -387,7 +387,7 @@ def orchestrate_pipeline(config: Dict[str, Any], experiment_dir: Path, resume_mo start_iter_idx = max_found_iter + 1 logger.info(f"Resuming from iteration {start_iter_idx}.") # Log presence of existing ban lists if resuming past iter 0 - if start_iter_idx > 0: + if start_iter_idx > 0 and generation_enabled: if banned_ngrams_json_path.exists(): logger.info(f"Resuming with existing n-gram ban list: {banned_ngrams_json_path}") else: logger.info("No existing n-gram ban list found to resume with for subsequent iterations.") if banned_slop_phrases_json_path.exists(): logger.info(f"Resuming with existing slop phrase ban list: {banned_slop_phrases_json_path}")