Waytrine

Task Router API

One endpoint makes ten routing decisions in a single forward pass. One more says whether the model is up.

PugPort's production instance runs behind a Cloudflare Tunnel; the server itself listens on loopback only. Examples use https://<your-host>.

Auth
None; rate-limited per client IP
Format
JSON, UTF-8
Request cap
64 KiB
Rate
30 per IP per minute
Queue
1 running, 8 waiting

POST /v1/judge

Send the routing state; get one answer per head. The body is exactly { "state": … }, and the state has exactly five keys. Unknown keys, duplicate keys at any depth, NaN or Infinity, and invalid UTF-8 are rejected with 400.

Attachments travel as metadata only: kind, name and an optional MIME type, never the file. PugPort redacts credentials and removes file paths before it sends anything.

Request

curl https://<your-host>/v1/judge \
  -H 'Content-Type: application/json' \
  --data '{
    "state": {
      "message": "Translate hello to French.",
      "recent_context": "",
      "reference_text": "",
      "attachments": [],
      "candidate_executors": []
    }
  }'

The state, as a TypeScript type

type RouterState = {
  message: string;          // up to 32,768 characters
  recent_context: string;   // up to 32,768 characters
  reference_text: string;   // up to 32,768 characters
  attachments: {            // up to 32 items, metadata only
    kind: string;           // up to 64 characters
    name: string;           // up to 512 characters
    mime_type?: string;     // up to 256 characters
  }[];
  candidate_executors: string[]; // up to 32, each up to 128 characters
};

// The body is exactly { "state": RouterState }, at most 64 KiB.

When the state is long

A state longer than the model's token budget is packed rather than refused. Fields shrink by weight (message 6, reference text 2, attachments 2, recent context 1, executors 1). The message and reference text keep their beginning and end around an [... omitted ...] marker; recent context keeps its end.

Response

200 OK, application/json
modelVersionCheckpoint name. pugport-laya-v4 is the internal name of Task Router v4.
modelSha256SHA-256 of the loaded model.safetensors, computed at startup.
devicemps or cpu.
latencyMsEnd to end on the server, queue time included.
regioneu for the EU/EEA, Switzerland and the UK, else other.
raw.answersOne object per head: type: "choice", the choice, and probabilities over every option.
raw.usagePacking report: whether input was compacted, token counts before and after, the budget, which fields were cut.

raw carries the model's prediction unmodified. PugPort accepts it only when all ten heads answer, each choice is a listed option, no answer abstained, and each head's probabilities sum to 1 within 0.02.

Shape of a response; values elided

HTTP/1.1 200 OK
content-type: application/json
cache-control: no-store

{
  "modelVersion": "pugport-laya-v4",
  "modelSha256": "<sha256 of model.safetensors>",
  "device": "mps",
  "latencyMs": …,
  "region": "other",
  "raw": {
    "answers": {
      "task_type": {
        "type": "choice",
        "choice": "translation",
        "probabilities": { "conversation": …, "coding": …, … }
      },
      "difficulty": { "type": "choice", "choice": "simple", "probabilities": { … } },
      …  // eight more heads
    },
    "usage": {
      "input_compacted": false,
      "original_state_tokens": …,
      "packed_state_tokens": …,
      "state_token_budget": …,
      "compacted_fields": []
    }
  }
}

GET /healthz

Reports the model version, the device and how many requests are waiting. It never runs the model and does not count against the rate limit. Until the model has loaded, and while the service shuts down, it answers 503 unavailable with Retry-After: 1.

Request and response

curl https://<your-host>/healthz

HTTP/1.1 200 OK
{ "modelVersion": "pugport-laya-v4", "device": "mps", "queueLength": 0 }

Status codes

Every error is a fixed code in { "error": … }. No traces, no echoed input. Every response is no-store.

StatuserrorRetry-AfterWhen
400 invalid_request – Malformed JSON or invalid UTF-8; duplicate keys at any depth; NaN or Infinity; a top level other than exactly { state }; a state that breaks a field rule; an invalid Content-Length.
404 not_found – Any other path or method.
413 body_too_large – Content-Length over 65,536 bytes, or a streamed body that grows past it.
429 rate_limited Seconds until a slot frees More than 30 judge requests from one client IP in a rolling 60 s.
503 queue_full 1 One inference is running and eight are waiting.
503 queue_expired 1 The request waited more than 2.5 s for the model and was dropped unrun.
503 rate_capacity 1 The rate limiter is already tracking 4,096 active clients.
503 unavailable 1 The model is still loading, or the service is shutting down.
500 inference_failed – The model call failed, or its output was over 256 KiB.
500 request_failed – Any other server error. No trace is returned.

A full queue

HTTP/1.1 503 Service Unavailable
content-type: application/json
cache-control: no-store
retry-after: 1

{ "error": "queue_full" }

A client that disconnects is logged as 499 and gets no response; a waiting job is removed from the queue at once.

Limits

The rate limiter keys on Cloudflare's client IP header, which the server trusts only because it listens on loopback behind the tunnel. Counters live in memory and reset when the process restarts.

Request body 64 KiB (65,536 bytes), whitespace included, checked while streaming
Text fields message, recent_context, reference_text: up to 32,768 characters each
Attachments Up to 32; kind ≤ 64, name ≤ 512, optional mime_type ≤ 256; no other keys
Executors Up to 32 strings, each ≤ 128 characters
Rate 30 judge requests per client IP per rolling 60 s; requests later rejected as invalid still count
Concurrency 1 running inference, 8 waiting; a wait over 2.5 s is dropped
Response Prediction output capped at 256 KiB; PugPort's client enforces the same cap

Client example

The model is advice, so the client must never wait on it. This is the pattern PugPort uses: a hard deadline (8 s in production), the model version and weight hash pinned, any failure answered with null so the caller routes by its own rules, and a 30 s pause after a failure.

PugPort also caps the response at 256 KiB and refuses redirects. However the answer arrives, it can never widen permissions: user pins, approved rules, quotas, budgets and read-only limits always win.

TypeScript, Node 18+ or any modern runtime

const JUDGE_URL = "https://<your-host>/v1/judge";
const MODEL_VERSION = "pugport-laya-v4";
const MODEL_SHA256 = "<sha256 of model.safetensors>";
const DEADLINE_MS = 8_000;   // PugPort's production deadline
const BACKOFF_MS = 30_000;   // pause after any failure

let pausedUntil = 0;

/** Ten choices, or null: the caller then routes by its rules. */
export async function judge(state: RouterState): Promise<Record<string, string> | null> {
  if (Date.now() < pausedUntil) return null;
  try {
    const res = await fetch(JUDGE_URL, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ state }),
      signal: AbortSignal.timeout(DEADLINE_MS),
      redirect: "error",
    });
    if (!res.ok) throw new Error("HTTP " + res.status); // 429, 503, 4xx, 5xx
    const body = await res.json();
    if (body.modelVersion !== MODEL_VERSION || body.modelSha256 !== MODEL_SHA256) {
      throw new Error("unexpected model");
    }
    const choices: Record<string, string> = {};
    for (const [head, answer] of Object.entries<any>(body.raw.answers)) {
      if (answer.type !== "choice" || answer.abstention === "abstained") {
        throw new Error("no usable answer for " + head);
      }
      choices[head] = answer.choice;
    }
    return choices;
  } catch {
    pausedUntil = Date.now() + BACKOFF_MS;
    return null;
  }
}

Design decisions

A small model on one machine, built so that it can be slow or down without the app noticing.

One model slot, a short queue

One inference runs at a time on one worker thread, with room for eight more. The next request gets 503 queue_full at once. On a single Apple-silicon host, memory and latency stay predictable.

Drop stale work

A job that waited over 2.5 s is answered 503 queue_expired without running. The client has its own deadline and a rule-based fallback, so a fast refusal beats a late answer. A client that disconnects leaves the queue at once.

Weights pinned by hash

At startup the server hashes model.safetensors and passes that hash to the model loader. Every response reports modelVersion and modelSha256, and the client rejects any pair it was not built against.

Nothing kept from requests

One log line per request: timestamp, status, latency, packed token count, country. No bodies, predictions, IP addresses or exception text are stored, model output streams are silenced, and errors are fixed codes.

Warm after idle

After 60 s without a prediction, the server runs a one-word synthetic prediction through the same slot, so the next real request skips the cold start. It writes no log line and counts against no rate limit.

Strict in, packed to fit

Bad input is refused before it reaches the model. A state longer than the model's token budget is packed field by field, keeping both ends of the message and the tail of the context, and raw.usage says what was cut.

Task Router case study Training, the sealed real-traffic test, results →