Forma

The problem

Ask a model for a line chart of monthly revenue and it usually comes back looking fine. Sometimes it's a bar chart instead, or the spec won't draw in the library you use, or the numbers aren't the ones you gave it. Nothing errors. Someone has to look at every chart to catch it, and when an agent is the one making the chart, nobody looks.

Rules have the same problem. Ask what a standard requires and you get an answer that sounds right and points to nothing you can check.

What I built

Forma checks what a machine produces against versioned rules before anyone relies on it. I call it executable knowledge: rules written so a machine can run them.

Two oracles are live. chart answers in Vega-Lite, gnuplot, Plotly, ECharts or Chart.js, and vector-graphics in TikZ, PicTeX or MetaPost. You ask in plain words through the API, the customer app or an agent over MCP. A valid answer carries verified_by, the version of the rules it passed. Otherwise you get the reason it was refused, or an error if the fault was Forma's. The check covers structure: the right kind of chart, well formed and drawable. It can't tell whether you asked for the right thing. Every oracle is registered in one catalog, and the API, the MCP tools and the customer app all list from it.

The other half, still being built, does the same for standards. An edition of a standard moves through this:

Source → Verify → Extract → Build rules → Review → Publish → Ask · Comply · Act

Fetching, verifying and extracting sources works today. Building the rules, reviewing them and publishing an edition come next. A published edition can be used three ways. Ask reads it, citing the source lines for each value that has them. Comply checks your facts requirement by requirement and returns MET, NOT_MET or UNKNOWN, citing the line each one comes from. If you leave a fact out, the rules that need it come back UNKNOWN rather than NOT_MET, and if the decision can't run without it, the request is refused. Act is the same check run by an agent before it does anything: only a met decision lets it go ahead, and a failure comes back as the rule it broke.

For now all three run in Studio, an internal app for turning standards into rules, against a hand-built sample of RFC 862. The public API for them is designed, and wiring a published edition into an oracle is the next step.

How it works

Everything runs on AWS EKS. Customers reach the marketing site, the docs, the customer app, the Forma API and the MCP server; the account API sits behind the customer app. The team's tools are only reachable on a private network: Studio, Hopper for tuning and promoting each oracle's revisions, and Pulse, which shows every service, the version it runs, recent deploys and the open previews.

It ships as three separate releases. The services release covers the apps and the team's tools: on each merge to main it rebuilds only the images whose inputs changed, records the release against the commit, and rolls itself back if an upgrade fails. Ready pull requests that change a service get their own preview environment. The MCP server is built in its own repository and deploys on a schedule; after each deploy it checks the pods are running the new image and healthy, and rolls back if not. Keycloak is a release of its own, with separate realms for external and internal users, so a failed app deploy doesn't roll sign-in back with it.

The highlighted path follows a request from an AI agent to a checked answer.

A few of the decisions:

• Pass or refuse. An answer either passes the rules or is refused with the reason. I didn't want a confidence score, because then the reader still has to decide.
• Say exactly what "passed" means. The check confirms a chart is well formed, drawable and the kind you asked for. For the numbers it only confirms each one appears in your request; labels and dates aren't compared. By default, a request with no values is refused rather than given made-up ones.
• Read the standard once, with a person reviewing. Interpreting a standard is the fuzzy part, so it belongs at build time, reviewed before an edition is published; that step is the one being built now. After that, answers are deterministic: the same facts get the same outcome and the same citation.
• Decision types belong to the domain, rules to the edition. Each edition binds its own rules to a decision type shared across its domain, so two editions can be compared on the same inputs.
• Every oracle configuration is a revision. Revisions don't change once made, and a new one is evaluated against the live one before it is promoted. Someone can override that, but the override and its reason go on the record, so every change can be explained or rolled back.
• Credits are reserved, then settled. Two calls racing for the last credit can't both spend it.
• Separate identities. External and internal users sign in through different Keycloak realms, API keys and MCP agents have identities of their own, and each customer-facing service connects to Postgres with its own role and only the grants it needs.

What was hard

Making answers fast. At the worst point, with the model served from the cheapest hosts, the typical chart took 26 seconds and one in ten took close to two minutes. I measured where the time went. Forma's own code took a few dozen milliseconds. Almost all of the rest was generating the answer, which is roughly a short fixed start plus the answer's length divided by how fast tokens come out.

That pointed at three things. The check was being started fresh for every answer, at about 0.7 seconds each time; keeping it running brought that to 10 to 25 milliseconds. The cheap hosts produced 19 to 55 tokens a second; routing the same model to faster hardware brought it to around 800. The third is length. In one heatmap the data rows were 85% of the answer, so the same request took 6 seconds or 18 depending on how many rows came back. Making answers shorter without losing anything is what I'm working on next.

I also tried sending less. The instructions for a language are long, and most requests need only part of them, so I built a version that picks the pieces each request needs. A hand-picked shorter set halved the time on a hard geometry figure, but only one run in four passed, against three in four with the full set. Picking the pieces automatically did worse: on the same figure, it never passed in production replays. So that oracle still gets the full set.

What I learned

The rule version on each answer mattered more than I expected. It tells a customer exactly what was checked, and it kept me honest about what Forma actually promises.