The managed model layer for AI agents
GoThink
Not every call your agent makes needs a frontier model.
Token bills, hallucinations, regressions, fallback sprawl — every production
agent hits the model layer eventually. GoThink runs that layer for you: a cheaper
open-source model answers first, we check the answer, and escalate to a frontier
model only when it's actually needed. Your code keeps calling
/v1/chat/completions — the savings happen behind the URL.
Day-1 design targets vs frontier + RAG baseline — verified per-customer in PoC reports, not claimed in advance.
- Cost ratio
- ≤ 50%
- Quality ratio
- ≥ 95%
- Latency ratio
- ≤ 1.2×
- Traced
- 100%
of baseline spend
of baseline quality
P50; batch callers may relax to 2.0×
of requests, replayable by trace_id
Twenty seconds.
The whole idea — one line of config, a cheaper model that answers first, and the receipts. Behind a drop-in gateway.
You run the agent. We run the model layer.
Shipping a production agent is already a full-time job. Keeping the model layer cheap, accurate, and stable — traces, evals, routing, fallback, retraining — is a second one. GoThink quietly takes that second job, so your team can stay on the product.
- Production teams
- Your agents are live, and the model bill is climbing faster than usage.
- Quality-sensitive
- You want frontier-level answers — just not on every call, and not at frontier prices.
- Lean by choice
- You'd rather not build and staff a model-ops team just to keep costs sane.
One line. Your agent code stays exactly as it is.
Point your existing OpenAI client at GoThink — nothing else changes. We run the
model layer behind the same /v1/chat/completions your code already
calls.
export OPENAI_API_BASE="https://api.openai.com/v1"
export OPENAI_API_KEY="$OPENAI_API_KEY"
# agent loop: N frontier calls per taskexport OPENAI_API_BASE="https://gothink.example.com/v1"
export OPENAI_API_KEY="$GOTHINK_API_KEY"
# agent loop unchanged — routing happens behind the URLPOST /v1/chat/completionsOpenAI-compatible, drop-inPOST /v1/reasonnative structured output: root_cause / evidence / fixGET /v1/traces/{id}full audit and replay of any decision
How it routes
Behind the URL, every request takes the same path: pull the relevant context, let a cheaper model answer, check that answer, then decide — ship it, or escalate to a frontier model. Every decision is recorded.
-
L1
Retrieve
Pull the relevant docs for the request, so the answer comes from facts — not guesses.
-
L2
Compress
Trim the context to what matters, so you're not paying for tokens that don't change the answer.
-
L3
Reason
A cheaper, fine-tuned open-source model writes the answer.
-
L4–5
Verify
Automated checks score the answer — is it grounded, consistent, and complete? — into one confidence number.
-
L6–7
Route
The router reads that number and decides. Both outcomes are first-class:
accept_ossThe answer passed the check. It ships — at the cheaper cost.
fallbackThe answer didn't pass. The request escalates to a frontier model — and the reason is recorded.
-
L8
Trace
Every decision is saved under a
trace_id— fully auditable and replayable. -
L9
Distill
Good escalations become training data — so the cheaper model keeps getting better.
Every answer ships with its evidence.
Every response carries a gothink block that tells you which model
answered, what it cost, and why. Nothing about the decision is hidden.
{
"choices": [{ "message": { "role": "assistant", … } }],
"gothink": {
"trace_id": "tr_9f2e84c1",
"router_reason": "accept_oss",
"model_used": "qwen2.5-coder-32b-rag",
"retrieved_sources": ["paas/node-engines", "vercel/build"],
"fallback_used": false
}
}trace_id- Replay the full decision chain any time via
GET /v1/traces/{id}. router_reason- Why the gateway accepted the OSS answer — or why it escalated.
model_used- Which specialist actually answered. No silent model swaps.
retrieved_sources- The corpus documents that grounded this answer.
fallback_used- Whether a frontier model was involved at all.
Every fallback makes the next answer a little better.
Every time we escalate to a frontier model and the answer checks out, that becomes a training example. The cheaper model learns the cases it used to miss — so over time fewer requests need escalating, and your cost keeps dropping. Quality is re-checked against the same benchmark every round.
Frontier models are the temporary teacher, not the permanent infrastructure. Independence from top-tier LLMs is what the loop earns over time — it is not a day-one claim.
How we start working together.
No big rebuild, no long contract to find out whether it works. We start small, prove it on your own traffic, and only then talk about running it for you.
- 1 · BenchmarkSend a sample of sanitized queries. We show you cost, quality, and fallback on your own workload — free, and with no integration.
- 2 · PilotA short, scoped engagement. We tune the model layer to your traffic and hand you a clear before / after report.
- 3 · ManagedOnce the numbers hold, we run the layer month over month — improving it quietly while you keep shipping.
Priced as a share of what you save.
No per-token games. Two simple lines, both scoped to your real workload during the pilot — so you see the number before you commit to it.
- Usage
- You pay a fraction of your current frontier bill — typically around half. When your cost drops, ours drops with it.
- Managed
- A flat monthly fee for running the layer: evals, routing, retraining, regression checks, and a regular ROI report.
- Pooled
- The default. Your anonymized traces help improve the shared model for your domain — and you pay less for choosing it.
- Private
- Your data stays isolated and never trains a shared model. Priced as a private deployment.
Every figure is measured on your own traffic during the pilot — never quoted blind, and never charged per token.
Benchmark it on your own workload.
- 1Send 30 sanitized queries from your agent's real traffic.
- 2We run GoThink against a frontier + RAG baseline on the same set.
- 3You get a report: cost, quality, fallback rate, latency — every trace attached.