Fine-Tune Embedding Model: Octomind Embed
skillFine-tune, evaluate and publish the octomind-embed model that routes requests to capabilities and agents. Use when triggers or agent descriptions change.
instructions
Overview
muvon/octomind-embed is the bi-encoder octomind uses for two decisions: scoring a user message against capability triggers and skill semantic() phrases, and scoring a delegated task brief against each reachable agent's "<title>. <description>". Since 2026-09 it is ibm-granite/granite-embedding-30m-english (6 layers, 384-dim, CLS-pooled) given a short contrastive fine-tune; the runtime loads onnx/model_quantized.onnx, a 30 MB graph with int8-stored weights and fp32 arithmetic. Whenever triggers, semantic phrases, or agent titles and descriptions change, the model is stale. This skill walks through retraining, calibration, publishing, and recovery.
The cross-encoder reranker is intentionally not trained or shipped: at the current capability count it does not improve routing over a well-calibrated bi-encoder.
Mental model
Both decisions have the same shape: rank the candidates by cosine, accept the winner only if it clears an absolute threshold and beats the runner-up by a margin, otherwise abstain. For capabilities the score is the mean of the top-3 trigger cosines; for agents it is the cosine to the agent text. The model, the corpus, and the two pairs of constants are one system: retraining moves the cosine scale, so the constants must be recalibrated on the graph the runtime loads and shipped together with the weights.
The fine-tune is a nudge with two phases. First 105 steps over all four training files in turn (capability pairs and triplets, agent pairs and triplets), then 105 steps over the agent files alone: capability batches past about 50 start to cost accuracy on real prompts, while agent batches keep improving routing. bin/train then sweeps a blend of the fine-tuned and base weights (WiSE-FT soup) and keeps whichever of the fine-tune and the blends scores best on the capability gate plus routing handoff; in 2026-11 that was the unblended fine-tune.
Rules
When to retrain
| Change | Retrain? |
|---|---|
Added or removed a capability (capabilities/<name>/config.toml) | yes |
Edited triggers = [...] | yes |
Added or edited semantic(...) rules in any SKILL.md | yes |
Edited the _oos curated phrases in build_dataset.py | yes |
Added or removed an agent, or edited its # Title: / # Description: | yes |
| Agent prompt, skill body, or docs change with none of the above touched | no |
Threshold/margin change in octomind capability.rs / skill.rs / tap_route.rs | no — rebuild octomind |
Standard retrain flow
Run from model/, inside tmux or nohup so a dropped connection doesn't kill it. Without a GPU the trainer uses the CPU; on a machine other jobs share, set OMP_NUM_THREADS to a measured count (4 beat 12 by 3-6x on a loaded 12-core box):
cd model && uv sync
tmux new -s train
ANTHROPIC_API_KEY=... bin/train --llm --resumeWhat runs (bin/train):
build_eval_seed.pyonce, ifdata/eval_real.jsonlis missing.augment_llm.py --resume— LLM paraphrases for new(label, trigger)pairs only, appended todata/intents.jsonl.augment_agents.py— LLM-written task briefs per agent in five styles, train and eval split written by different models, appended todata/agent_briefs.jsonlfor whatever is missing. To replace an agent's briefs after its description changed, delete its lines first.build_dataset.py—pairs.jsonl,triplets.jsonl,holdout.jsonl,raw_triggers.jsonl(the un-augmented corpus the runtime embeds),agent_pairs.jsonl,agent_triplets.jsonl,raw_agents.jsonl(the agent texts the runtime embeds), with the_oossink label, multi-turn surfaces, and hard-negative mining.train.py— CachedMNRL + Matryoshka:train.max_stepsover all four files (round_robin), thentrain.agent_stepsover the agent files alone. Capability files train atloss.scale(10), agent files atloss.agent_scale(50).model_soup.py+compare_bases.py— sweepsoup.alphasagainst the base and keep the best of the fine-tune and the blends by capability gate plus routing handoff.eval.py(holdout),eval_gate.py(eval_real, informational here) andeval_route.py(does a brief reach its agent).export_onnx.py(fp32 graph + int8-stored graph, checked against each other), theneval_gate.py,eval_route.py,dump_scores.pyandcalibrate_cpus.pyon the graph the runtime loads.
Other flags: --skip-build (reuse data/), --skip-export, --ft checkpoints/<dir> (redo soup, eval, export, and calibration on an existing run), --iterations N (re-mine capability triplets from the previous round's mistakes), --legacy (diagnostic model.fit path).
Calibrating the runtime gates
bin/train prints the operating points for the machine it ran on. Confirm them on every CPU kind before rollout: dump the scores once per kind, then calibrate over all dumps.
uv run python scripts/dump_scores.py --model checkpoints/<run>/onnx --out checkpoints/<run>/scores/arm64.json
uv run python scripts/calibrate_cpus.py checkpoints/<run>/scores/*.json
uv run python scripts/eval_runtime_gate.py --model checkpoints/<run>/onnx # octomind's test fixtures, no Rust builddump_scores.py needs only numpy, pyyaml, tokenizers and onnxruntime, so it runs in a bare container (its docstring has the Docker command for x86 on an ARM Mac). calibrate_cpus.py picks, per decision, the point with the best accuracy on the worst machine among those that keep false activations under the cap on every machine (capability null false-positive rate 0.03; routing wrong handoffs 0.02, foreign handoffs 0.03). Copy the values into:
octomind/src/mcp/runtime/capability.rs—AUTO_ACTIVATE_THRESHOLD,AUTO_ACTIVATE_MARGINoctomind/src/mcp/runtime/skill.rs—SEMANTIC_DEFAULT_THRESHOLD,SEMANTIC_MARGINoctomind/src/mcp/orchestration/tap_route.rs—ROUTE_THRESHOLD,ROUTE_MARGINmodel/configs/default.yaml—eval.runtime_threshold,eval.runtime_margin,eval.route_threshold,eval.route_margin(the publish gate andeval_route.pyscore at these)
The 2026-11 model runs at 0.68 / 0.05 for capabilities and 0.65 / 0.05 for routing, and scores identically on ARM, x86 and x86 with VNNI. The graph shipped 2026-09 quantized activations per batch and scored the same text up to 0.08 apart between ARM and x86; the per-CPU dumps are the check that this has not come back.
Publishing
HF_TOKEN=hf_... bin/publish # latest checkpoint; --run <dir> to choosebin/publish runs eval_gate.py against eval_baselines.json on the fp32 weights and again on onnx/model_quantized.onnx, and aborts on regression (top-1 accuracy drop > 0.01, null FPR rise > 0.01, any per-capability recall drop > 0.05). It warns if onnx/model_quantized.onnx is missing, since the runtime prefers it. --force/--skip-gate bypass the gate for documented emergencies only; --dry-run prints without uploading.
Octomind's CI and every fresh install download the current weights, while an existing install keeps its cached copy (a cached HF repo is not re-checked). So publish and the octomind commit that carries the recalibrated constants go out together. After the rollout is confirmed, record the baseline the next publish gates against:
uv run python scripts/eval_gate.py --model muvon/octomind-embed --write-baseline eval_baselines.jsonResuming an interrupted run
The trainer checkpoints every epoch (save_total_limit=2). If checkpoints/embed-*/checkpoint-N/ exists:
uv run python scripts/train.py --resume # latest run dir
uv run python scripts/train.py --resume-from <run-dir> # explicitIf iteration 1 was interrupted: finish it with --resume, re-mine with build_dataset.py --neg-embed-model <iter1-dir>, train iteration 2 with train.py, then bin/train --ft <iter2-dir> for soup, eval, export, and calibration. --resume covers the modern train.py path only, not --legacy.
Examples
Example 1: new triggers not picked up
❌ bin/train --skip-build after editing triggers — the build is what turns triggers into training surfaces, so the new run inherits the old weaknesses.
✅ bin/train --llm --resume — rebuild, augment only the new triggers, retrain.
Example 2: wrong capability wins but triggers look right
Retraining rarely fixes this. First run calibrate_cpus.py on fresh dumps to see whether the shipped thresholds still fit the model; add the failing prompt to data/eval_real.jsonl with its correct label and rerun eval_gate.py; then compare the competing capabilities' triggers — shared generic vocabulary is fixed by differentiating triggers, not by more training.
Example 3: publishing without a baseline update
Publishing and forgetting --write-baseline leaves the next publish gating against the previous model, so a later regression can slip through. Record the baseline once the rollout is confirmed.
Checklist
- Trigger, semantic-phrase or agent-description change → full rebuild with
--llm --resume, not--skip-build? - Training run inside tmux or nohup?
- Winner evaluated on holdout,
eval_realand the agent briefs; ONNX graphs exported and the small one scored? - Thresholds calibrated over dumps from every CPU kind and copied into
capability.rs,skill.rs,tap_route.rs, andconfigs/default.yaml? eval_runtime_gate.pyreports PASS for octomind's fixtures?bin/publishpassed the gate without--force, and the octomind commit with the new constants goes out with it?- Baseline rewritten after the rollout was confirmed?
Composition / References
model/README.md— pipeline, dataset, loss stack, eval setmodel/configs/default.yaml— base model, training, soup alphas, runtime-mirror constantsmodel/bin/train,model/bin/publish— entry pointstap-capability-authoringskill — writing triggers