OctoHub
Completion Chaining • Prometheus Metrics • Multi-Tenant Keys
The proxy that sits in front of every model Octomind Hub and Octomind Cloud serve — one OpenAI-style endpoint, 20+ upstream providers, and every request recorded in your own database. Completion chaining replays prior turns automatically, including reasoning blocks. Deploy in minutes: one binary, one config file, schema created on first connection.
Key Features
Completion Chaining
Pass previous_completion_id and OctoHub replays the prior turns automatically — including reasoning/thinking blocks, DeepSeek-compatible — so clients don't have to resend history or reconstruct provider-specific message formats themselves.
Model Mapping & Load Balancing
Map short model names to a list of provider:model targets; OctoHub picks randomly across them for simple load balancing and failover. Per-provider concurrency limits throttle any upstream independently, so one slow provider never starves the rest.
Multi-Tenant API Key Management
Issue and revoke per-client keys with per-key model allow-lists, without ever exposing the real provider credentials. Two independent auth layers: client keys hit /v1/completions and friends, a separate master key gates the entire /v1/admin/* surface.
Observability Built In
Structured JSON or pretty logs, Prometheus metrics on a separate port, and X-Request-Id correlation across every hop. Full request/response bodies land in your own SQLite, MySQL, or PostgreSQL database — nothing routes through a third party.
Usage Analytics
Aggregated stats by API key and time bucket — hour, day, week, month — so cost and consumption are visible per team, service, or developer without building your own reporting layer on top of raw logs.
Ops-Friendly by Default
GET /health for load balancers, SIGHUP to hot-reload octohub.toml with zero downtime, and an octohub-admin.sh wrapper for day-to-day admin tasks. Automatic schema creation on first connection — no migration step to run before your first request.
Why OctoHub?
Teams share raw provider API keys with no per-service tracking.
Issue per-client virtual keys that never expose the real provider credentials, each with its own model allow-list and usage attribution.
No visibility into AI spending until the invoice arrives.
Aggregated usage stats by key and time bucket, plus Prometheus metrics on a dedicated port — know what is being spent before the bill, not after.
Switching providers means touching every client integration.
Clients call a short model name; you remap that name to any provider:model behind the scenes. Zero client changes when a provider changes.
A config change means restarting the proxy and dropping in-flight requests.
SIGHUP hot-reloads octohub.toml in place. GET /health and structured logs make it safe to wire into a real deployment from day one.
When to Use OctoHub
Team LLM Gateway
Give every developer and service its own tracked API key routed through one gateway. See who uses what, how much it costs, and enforce per-key model allow-lists.
Provider Failover Without Client Changes
Map one model name to several providers behind the scenes. When one goes down, requests route to the next automatically — clients never see the outage or change a line of code.
Audit Trail for Every Completion
Every request and response is stored in your own database. Replay a conversation, debug a bad completion, or produce an audit trail without depending on the provider retaining anything.
Cost Visibility Across the Org
One pane of glass for AI spending: aggregated usage by key and time bucket, per-provider concurrency limits to cap runaway load, Prometheus metrics to alert on before the invoice arrives.
Install
Build from Source
git clone https://github.com/muvon/octohub && cd octohub && cargo build --releaseTech Stack
Built in the Open
OctoHub is open source under the Apache 2.0 license. Contributions, issues, and stars are welcome.