tap-model-training

What this skill is for

muvon/octomind-embed is the bi-encoder that decides which capability auto-activates from a user's prompt. It is fine-tuned from BGE-small-en-v1.5 on the triggers authored in capabilities/*/config.toml and the semantic(phrase) rules in skills/*/SKILL.md.

Whenever those source files change, the model is stale. This skill walks through the retrain-and-publish loop and the recovery paths.

The cross-encoder reranker (muvon/octomind-rerank) is intentionally not maintained. Production validation showed it doesn't help at the current capability count (~75 labels); the bi-encoder + a well-tuned runtime gate is the production path.

When to retrain

You MUST retrain (and republish) when the supervision set changes:

Standard retrain flow

ALWAYS run inside tmux or nohup — sessions get disconnected.

cd model
uv sync  # if env not warm

tmux new -s train
ANTHROPIC_API_KEY=sk-ant-... bin/train --llm --resume --iterations 2
# Ctrl+B then D to detach. Reattach with: tmux attach -t train

What runs:

1. augment_llm.py --resume — paraphrases ONLY the new (label, trigger) pairs against data/intents.jsonl. Skips already-augmented ones. Cost: ~$0.10–$2 depending on how many new triggers. Time: 3–5 min.
2. build_dataset.py — rebuilds pairs.jsonl, triplets.jsonl, holdout.jsonl from capabilities/ + skills/ + LLM intents + OOS sink + multi-turn surfaces + realistic typo augmentation.
3. Train iter 1 with CachedMNRL + GISTEmbed + Matryoshka. ~3-6h on GPU, 6-12h on CPU.
4. Re-mine triplets with iter-1 model (retrieval-aligned + NV-Retriever positive-aware filter).
5. Train iter 2. Same time as iter 1.
6. eval.py against holdout (in-distribution).
7. eval_gate.py against data/eval_real.jsonl (real-user distribution — informational at this step, blocks at publish).
8. export_onnx.py — produces checkpoints/embed-<ts>/onnx/ for ORT-based consumers.

When it finishes, publish:

HF_TOKEN=hf_... bin/publish

bin/publish runs eval_gate.py against eval_baselines.json first and aborts on regression (top1_acc drops >1pt, null_fpr rises >1pt, any per-cap recall drops >5pts). Use --force only for documented emergencies.

After production rollout, record the new baseline so the NEXT publish gates against it:

uv run python scripts/eval_gate.py --model muvon/octomind-embed \
    --write-baseline eval_baselines.json

Resuming an interrupted training

SentenceTransformerTrainer auto-checkpoints at the end of each epoch (save_strategy="epoch", save_total_limit=2). Recovery:

# 1. check what got saved
ls checkpoints/embed-*/checkpoint-*

If you see checkpoint-N/ dirs, there's state. Resume:

uv run python scripts/train.py --resume

That picks the latest checkpoints/embed-*/ dir and continues from the highest-numbered checkpoint-N. Use --resume-from <path> for an explicit run dir.

Mid-iter-1 interruption — finish iter 1, then continue iter 2 manually:

# 1. finish iter 1
uv run python scripts/train.py --resume

# 2. re-mine triplets with iter-1 model
ITER1=$(ls -td checkpoints/embed-*/ | head -1 | sed 's:/$::')
uv run python scripts/build_dataset.py --neg-embed-model "$ITER1"

# 3. train iter 2 fresh
uv run python scripts/train.py

# 4. eval + export
ITER2=$(ls -td checkpoints/embed-*/ | head -1 | sed 's:/$::')
uv run python scripts/eval.py        --run "$ITER2"
uv run python scripts/eval_gate.py   --model "$ITER2"
uv run python scripts/export_onnx.py --run "$ITER2"

--resume works only for the modern train.py path (CachedMNRL + GIST + Matryoshka). --legacy has no checkpoint-resume.

Calibrating the runtime gate

After every retrain, recalibrate the runtime threshold/margin against eval_real:

RUN=$(ls -td model/checkpoints/embed-*/ | head -1 | sed 's:/$::')
uv run python model/scripts/calibrate_thresholds.py --model "$RUN" --target-fpr 0.05

It sweeps τ × δ over data/eval_real.jsonl, prints the Pareto front, and recommends the operating point with highest gate_acc whose null_fpr ≤ target. Copy-paste the recommended constants into:

• octomind/src/mcp/core/capability.rs:772 → AUTO_ACTIVATE_THRESHOLD
• octomind/src/mcp/core/capability.rs:781 → AUTO_ACTIVATE_MARGIN
• octomind/src/mcp/core/skill.rs:66 → SEMANTIC_DEFAULT_THRESHOLD
• octomind/src/mcp/core/skill.rs:78 → SEMANTIC_MARGIN

The threshold often drops after fine-tuning (0.55 → ~0.45) because the FT model puts every matched positive well above the floor; the margin becomes the binding constraint, not the floor.

Common mistakes

1. Using --skip-build when triggers changed. The build is what converts new triggers into training surfaces. --skip-build reuses the previous pairs.jsonl, so new triggers don't show up in the training set, and the resulting checkpoint inherits the old weaknesses. The new run is wasted compute. Always rebuild after a catalog change.

2. Forgetting --llm after adding triggers. --llm is what kicks off augment_llm.py to generate user-style paraphrases of the new triggers. Without it, the new triggers only get rule-template expansion, which is much narrower. Use --llm --resume to skip already-paraphrased triggers.

3. Running outside tmux/nohup. Long training (5h+) over SSH dies when the connection drops. Always wrap in tmux or nohup.

4. Skipping the publish gate. bin/publish --force exists for emergencies. Don't use it as habit — silent regressions ship.

5. Not recording the new baseline after promotion. If you don't run eval_gate.py --write-baseline after a successful publish, the next publish has nothing to gate against and any future regression slips through.

6. Editing trigger lists without thinking about overlap. When two capabilities share vocabulary (e.g. all messaging-*, all legal-*, webfetch vs scraping), the model gets confused. Use PLATFORM-specific or JURISDICTION-specific vocabulary in each cap's triggers — not the same generic verbs across the family.

Key files

• model/bin/train — full pipeline entrypoint.
• model/bin/publish — gate-protected HF upload.
• model/configs/default.yaml — loss stack, batch sizes, eval paths.
• model/scripts/build_dataset.py — surface generation, OOS sink, multi-turn, typo aug, hard-negative mining.
• model/scripts/train.py — modern trainer (CachedMNRL + GIST + Matryoshka) + --resume flag.
• model/scripts/eval_gate.py — publish-time regression gate.
• model/scripts/calibrate_thresholds.py — runtime τ/δ sweep.
• model/scripts/build_eval_seed.py — frozen real-user eval set generator. Hand-edit data/eval_real.jsonl to add production-observed failure modes.
• model/eval_baselines.json — recorded baseline; the publish gate reads this.
• model/data/intents.jsonl — LLM paraphrases. augment_llm.py --resume skips entries already here.

When NOT to retrain

If the symptom is "auto-activation picks the wrong capability for X but the triggers are correct", retraining usually won't fix it. Three other levers first:

1. Run calibrate_thresholds.py to see if the current τ/δ in the Rust constants matches the FT model's distribution.
2. Add the failing prompt to data/eval_real.jsonl with the correct label, then re-run eval_gate.py to confirm it's an actual model problem and not an eval-distribution gap.
3. Check the trigger lists for the competing capabilities — if two neighbors share generic vocabulary, the fix is differentiating triggers, not retraining alone.

Retrain when the triggers themselves change, not when production behavior is unexpected.