From playground to production in one afternoon

Step 0 — pick the right decision

Good first candidates are decisions you already make badly or manually: a hand-written routing regex, a human re-reading the same policy, a "please review" flag nobody reviews. Bad candidates are decisions where being wrong is expensive and unmeasurable — leave those scripted-to-a-human for now, with the typed decision as a first-pass filter.

Write the state as the data you already have: the ticket text, the form payload, the tool call about to be executed. Do not pre-summarize it — the model will do the reading.

Hour 1 — run ten real examples in the playground

The playground runs the real model, free, in the browser. Paste examples from your backlog — real tickets, not hypotheticals — until ten of them come out the way a careful colleague would decide.

• Uncalibrated answers? Tighten the criteria with concrete true/false wording.
• Ambiguous cases getting confident answers? Your criteria are too coarse — add a level to the scale.
• Missing context? Put the missing fact into the state rather than hoping the model knows it.

Hour 2 — copy the JSON into a key-gated call

The JSON you see in the playground is the JSON your server should send. Exactly three things differ in production: the Authorization header, the endpoint path (/api/v1/systemone instead of the playground's browser endpoint), and the quota (free plans run 20 requests per minute; Starter starts at 120/min). Upgrade plans raise both concurrency and requests-per-minute — 3 concurrent at Starter, 10 at Pro.

async function decide(state, questions) {
  const resp = await fetch("https://jevai.website/api/v1/systemone", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.JEV_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ state, questions }),
  });
  if (resp.status === 429 || resp.status === 502 || resp.status === 529) {
    // exponential backoff here — never retry 401/402/422
  }
  return resp.json();
}

Hour 3 — threshold, flag, ship

Decide what a number means before wiring it up: below 0.8 confidence, send the item to a person instead of deciding. Ship behind a feature flag that sends, say, 5% of production traffic to the typed decision for the first day, and log the request_id alongside your old decision so you can diff them.

• Log request_id, the score or noul value, and the action your policy took.
• Set a timeout that tolerates occasional cold starts (the first call after an idle period can take ~3 s; the rest cluster at 400–600 ms).
• Alert on 402 (credits low) — it is the one failure a retry loop cannot fix.

Common first-day failures and their fixes

• Answers inconsistent across similar cases — the criteria descriptions are vague or contradictory. Rewrite them as policy lines.
• 429s during a burst — free allows 20 req/min. Move to Starter (120/min, 3 concurrent) or queue your bursts client-side.
• Everything routes to the top option — the criteria texts lean the same way. Balance them.
• One credit charged where you expected per-question billing — billing is per input token with a 1-credit minimum per request, so batching questions sideways is cheaper than splitting. See the credits post for the math.

Pick the threshold from data, not instinct

After the flag runs at 5% for a day, you own the distribution. Write the low-confidence band (anything below 0.8, between 0.8 and 0.9, and above 0.9) into a scratch sheet, and you will typically find decisions bimodal — one clear pile of confident answers and a tail of stragglers at 0.5–0.7. That tail is where human review happens. Then move the threshold to where your queue can afford to absorb review; on a shared inbox, a 15% review rate may exceed the budget, and the right action is tightening the criteria — not accepting 15% silently.

Escalation when you are not sure

• Confidence below threshold — send to a human with the state and the full distribution attached.
• Request failed (429/502/529 after retries) — default to the legacy path, not to skipping. A decision missing is a bug, not a risk.
• Answer keys missing from the answers map — treat as a failure, log it. (Should not happen; the schema guarantees one answer per question name.)

Wording patterns that move accuracy

• Criteria as policy lines, not synonyms: "true: Explicitly time-sensitive or business-critical; false: No urgency expressed" beats "true: urgent; false: not urgent". The model reads the criteria, so write them for a careful temp who has never seen your product.
• Name the fields you reference: "Does `proposed_tool_call` match what `user_request` asked for?" lets the model ground its judgment; "Is this suspicious?" makes it guess.
• One decision per question: do not ask "is this urgent AND should I page AND not a duplicate?" — the answer to a compound question is un-interpretable.
• Scales with a red line: when the action changes at a level, name that level in the criteria of related questions rather than hoping the score alone encodes it.