Task Router API
One endpoint makes ten routing decisions in a single forward pass. One more says whether the model is up.
PugPort's production instance runs behind a Cloudflare Tunnel; the server itself listens on
loopback only. Examples use https://<your-host>.
- Auth
- None; rate-limited per client IP
- Format
- JSON, UTF-8
- Request cap
- 64 KiB
- Rate
- 30 per IP per minute
- Queue
- 1 running, 8 waiting
POST /v1/judge
Send the routing state; get one answer per head. The body is exactly
{ "state": … }, and the state has exactly five keys. Unknown keys,
duplicate keys at any depth, NaN or Infinity, and invalid UTF-8 are rejected with 400.
Attachments travel as metadata only: kind, name and an optional MIME type, never the file. PugPort redacts credentials and removes file paths before it sends anything.
Request
curl https://<your-host>/v1/judge \
-H 'Content-Type: application/json' \
--data '{
"state": {
"message": "Translate hello to French.",
"recent_context": "",
"reference_text": "",
"attachments": [],
"candidate_executors": []
}
}' The state, as a TypeScript type
type RouterState = {
message: string; // up to 32,768 characters
recent_context: string; // up to 32,768 characters
reference_text: string; // up to 32,768 characters
attachments: { // up to 32 items, metadata only
kind: string; // up to 64 characters
name: string; // up to 512 characters
mime_type?: string; // up to 256 characters
}[];
candidate_executors: string[]; // up to 32, each up to 128 characters
};
// The body is exactly { "state": RouterState }, at most 64 KiB. When the state is long
A state longer than the model's token budget is packed rather than refused. Fields shrink
by weight (message 6, reference text 2, attachments 2, recent context 1, executors 1).
The message and reference text keep their beginning and end around an
[... omitted ...] marker; recent context keeps its end.
Response
| modelVersion | Checkpoint name. pugport-laya-v4 is the internal name of Task Router v4. |
|---|---|
| modelSha256 | SHA-256 of the loaded model.safetensors, computed at startup. |
| device | mps or cpu. |
| latencyMs | End to end on the server, queue time included. |
| region | eu for the EU/EEA, Switzerland and the UK, else other. |
| raw.answers | One object per head: type: "choice", the choice, and probabilities over every option. |
| raw.usage | Packing report: whether input was compacted, token counts before and after, the budget, which fields were cut. |
raw carries the model's prediction unmodified. PugPort accepts it only when all ten
heads answer, each choice is a listed option, no answer abstained, and each head's
probabilities sum to 1 within 0.02.
Shape of a response; values elided
HTTP/1.1 200 OK
content-type: application/json
cache-control: no-store
{
"modelVersion": "pugport-laya-v4",
"modelSha256": "<sha256 of model.safetensors>",
"device": "mps",
"latencyMs": …,
"region": "other",
"raw": {
"answers": {
"task_type": {
"type": "choice",
"choice": "translation",
"probabilities": { "conversation": …, "coding": …, … }
},
"difficulty": { "type": "choice", "choice": "simple", "probabilities": { … } },
… // eight more heads
},
"usage": {
"input_compacted": false,
"original_state_tokens": …,
"packed_state_tokens": …,
"state_token_budget": …,
"compacted_fields": []
}
}
} GET /healthz
Reports the model version, the device and how many requests are waiting. It never runs
the model and does not count against the rate limit. Until the model has loaded, and
while the service shuts down, it answers 503 unavailable with
Retry-After: 1.
Request and response
curl https://<your-host>/healthz
HTTP/1.1 200 OK
{ "modelVersion": "pugport-laya-v4", "device": "mps", "queueLength": 0 } Status codes
Every error is a fixed code in { "error": … }. No traces, no echoed
input. Every response is no-store.
| Status | error | Retry-After | When |
|---|---|---|---|
| 400 | invalid_request | – | Malformed JSON or invalid UTF-8; duplicate keys at any depth; NaN or Infinity; a top level other than exactly { state }; a state that breaks a field rule; an invalid Content-Length. |
| 404 | not_found | – | Any other path or method. |
| 413 | body_too_large | – | Content-Length over 65,536 bytes, or a streamed body that grows past it. |
| 429 | rate_limited | Seconds until a slot frees | More than 30 judge requests from one client IP in a rolling 60 s. |
| 503 | queue_full | 1 | One inference is running and eight are waiting. |
| 503 | queue_expired | 1 | The request waited more than 2.5 s for the model and was dropped unrun. |
| 503 | rate_capacity | 1 | The rate limiter is already tracking 4,096 active clients. |
| 503 | unavailable | 1 | The model is still loading, or the service is shutting down. |
| 500 | inference_failed | – | The model call failed, or its output was over 256 KiB. |
| 500 | request_failed | – | Any other server error. No trace is returned. |
A full queue
HTTP/1.1 503 Service Unavailable
content-type: application/json
cache-control: no-store
retry-after: 1
{ "error": "queue_full" } A client that disconnects is logged as 499 and gets no response; a waiting job is removed from the queue at once.
Limits
The rate limiter keys on Cloudflare's client IP header, which the server trusts only because it listens on loopback behind the tunnel. Counters live in memory and reset when the process restarts.
| Request body | 64 KiB (65,536 bytes), whitespace included, checked while streaming |
|---|---|
| Text fields | message, recent_context, reference_text: up to 32,768 characters each |
| Attachments | Up to 32; kind ≤ 64, name ≤ 512, optional mime_type ≤ 256; no other keys |
| Executors | Up to 32 strings, each ≤ 128 characters |
| Rate | 30 judge requests per client IP per rolling 60 s; requests later rejected as invalid still count |
| Concurrency | 1 running inference, 8 waiting; a wait over 2.5 s is dropped |
| Response | Prediction output capped at 256 KiB; PugPort's client enforces the same cap |
Client example
The model is advice, so the client must never wait on it. This is the pattern PugPort
uses: a hard deadline (8 s in production), the model version and weight hash pinned, any
failure answered with null so the caller routes by its own rules, and a 30 s
pause after a failure.
PugPort also caps the response at 256 KiB and refuses redirects. However the answer arrives, it can never widen permissions: user pins, approved rules, quotas, budgets and read-only limits always win.
TypeScript, Node 18+ or any modern runtime
const JUDGE_URL = "https://<your-host>/v1/judge";
const MODEL_VERSION = "pugport-laya-v4";
const MODEL_SHA256 = "<sha256 of model.safetensors>";
const DEADLINE_MS = 8_000; // PugPort's production deadline
const BACKOFF_MS = 30_000; // pause after any failure
let pausedUntil = 0;
/** Ten choices, or null: the caller then routes by its rules. */
export async function judge(state: RouterState): Promise<Record<string, string> | null> {
if (Date.now() < pausedUntil) return null;
try {
const res = await fetch(JUDGE_URL, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ state }),
signal: AbortSignal.timeout(DEADLINE_MS),
redirect: "error",
});
if (!res.ok) throw new Error("HTTP " + res.status); // 429, 503, 4xx, 5xx
const body = await res.json();
if (body.modelVersion !== MODEL_VERSION || body.modelSha256 !== MODEL_SHA256) {
throw new Error("unexpected model");
}
const choices: Record<string, string> = {};
for (const [head, answer] of Object.entries<any>(body.raw.answers)) {
if (answer.type !== "choice" || answer.abstention === "abstained") {
throw new Error("no usable answer for " + head);
}
choices[head] = answer.choice;
}
return choices;
} catch {
pausedUntil = Date.now() + BACKOFF_MS;
return null;
}
} Design decisions
A small model on one machine, built so that it can be slow or down without the app noticing.
One model slot, a short queue
One inference runs at a time on one worker thread, with room for eight more. The next request gets 503 queue_full at once. On a single Apple-silicon host, memory and latency stay predictable.
Drop stale work
A job that waited over 2.5 s is answered 503 queue_expired without running. The client has its own deadline and a rule-based fallback, so a fast refusal beats a late answer. A client that disconnects leaves the queue at once.
Weights pinned by hash
At startup the server hashes model.safetensors and passes that hash to the model loader. Every response reports modelVersion and modelSha256, and the client rejects any pair it was not built against.
Nothing kept from requests
One log line per request: timestamp, status, latency, packed token count, country. No bodies, predictions, IP addresses or exception text are stored, model output streams are silenced, and errors are fixed codes.
Warm after idle
After 60 s without a prediction, the server runs a one-word synthetic prediction through the same slot, so the next real request skips the cold start. It writes no log line and counts against no rate limit.
Strict in, packed to fit
Bad input is refused before it reaches the model. A state longer than the model's token budget is packed field by field, keeping both ends of the message and the tail of the context, and raw.usage says what was cut.