Test a Stripe refund agent

Your agent reads a support ticket and issues refunds in Stripe. RigorRun sends it tickets, lets it work with its own key, then reads Stripe with a key of its own and decides each case on what Stripe holds — never on what the agent says it did.

Every case creates its own customer and payments, fresh for each attempt, so no account needs resetting and nothing one case does is judged as the next one’s: a refund a slow agent makes on an earlier case’s payment after that case timed out is shown where it landed, and never counted there.

1. Try it on the twin — no keys, about a minute

The twin is a local copy of the parts of Stripe’s API the cases use. It runs on your machine and accepts any test-mode key.

npx rigorrun stripe twin                      # leave it running; prints http://127.0.0.1:12112
npx rigorrun stripe init --twin --yes         # in another terminal

init lists the refund policy’s rules and asks you to confirm each one (--yes confirms all). Only a rule you confirm can fail your agent: a rule you say no to is left out of the suite, and init names it. Each ticket’s own outcome — the refund it is owed, or none — is always checked. It then creates the project and its suite — seven tickets and a $1.00 canary — and writes rigorrun-stripe/ticket.example.json: exactly what your agent will be sent, with example ids.

2. Or your Stripe test mode

export STRIPE_TEST_KEY=sk_test_…              # RigorRun's key, not your agent's
npx rigorrun stripe init --safety staging

The key is read from the environment, so it never sits in your shell history. Before anything is stored, RigorRun checks its prefix (sk_test_ or rk_test_), then asks Stripe (GET /v1/balance) and stops unless Stripe answers livemode: false. A live key is refused without being sent anywhere, and so is any object that comes back in live mode. The key goes into this machine’s secret store, never into the project file.

Use an account or Sandbox that nothing else is using while a run is going: the reads include every refund made in the account since a case began.

3. Your agent, as a black box

No change to your agent’s code: RigorRun sends each ticket to the endpoint your agent already serves and reads Stripe itself. The ticket arrives as rigorrun/task/1, or in your own request shape with --body-template (see a black-box agent); the agent does the work with its own Stripe test key and answers with a sentence. It imports nothing from RigorRun. examples/stripe-support-agent is a complete one, and a black-box agent describes the envelope.

npx rigorrun agent add --project <id> --name my-agent \
  --black-box http://127.0.0.1:8787/ --claim-path message

Point the agent at the same place RigorRun uses: https://api.stripe.com with a test key of its own, or, for the local twin, its Stripe base URL at the twin (any sk_test_… key works there). The example agent reads STRIPE_BASE_URL; Stripe’s own SDKs take the address in code:

new Stripe(key, { host: '127.0.0.1', port: 12112, protocol: 'http' }); // stripe-node, twin only

StripeClient(key, base_addresses={"api": "http://127.0.0.1:12112"})  # stripe-python, twin only

4. The canary

npx rigorrun stripe canary --project <id> --agent my-agent

One case: a $1.00 payment the customer asks back in full. It shows the agent can be reached, can find the order, and that its refund lands where RigorRun reads — and the shape of every verdict that follows:

Canary · my-agent · PASS
  agent said             Refunded $1.00 on order RR-ORD-3EA44AD3.
  The Stripe twin shows  Refund re_9qwF… of $1.00 on ch_PAko… (a $1.00 charge), succeeded.

Each ticket gets 5 minutes to be answered. For a slower agent — a local model can need several — add --case-timeout <ms> to canary, run or gate.

5. The gate

npx rigorrun gate --project <id> --report report.html

Runs every case and exits 0 when the agent clears the bar, 1 when it does not. The report shows, for each case, what the agent said beside what Stripe holds: an agent that sent $25.00 as 25 fails with Refund re… of $0.25 on ch_… (a $25.00 charge)_.

--case <id> runs only the cases you name; such a run never becomes the baseline. gate --case is never a release verdict: it exits 3 (inconclusive) unless a named case fails, and its --json says which cases ran (selectedCases) out of how many (suiteCaseCount).

6. In CI

Use the GitHub Action or npx rigorrun gate with the project store committed or restored. The key arrives as RIGORRUN_SECRET__STRIPE_TEST_KEY (the twin’s as RIGORRUN_SECRET__STRIPE_TWIN_KEY); RigorRun reads it when nothing is stored.

What a verdict can and cannot say

• PARTIAL, by design. RigorRun reads the case’s own customers and payments, the refunds and disputes on them, and every refund made in the account since the case began. Changes anywhere else are not checked. Every verdict prints what its reads covered.
• And nothing else. Besides the refunds, every case checks that its customers are still there, that no new payment was taken from them, and that its payments still carry their order reference. This is always checked; it is what the policy’s “and nothing else” means.
• State only. A black-box agent’s calls are not seen, so nothing is checked about their order; everything about what Stripe holds afterwards is.
• A pending refund counts as made. As the pre-registered oracle counts it. The verdict is sealed when the case ends; if Stripe later fails or cancels the refund, it stays PASS.
• Nobody else may write to the account during a run. A refund made by anyone while a case runs, on a payment no case created, is read as the agent’s, because a black-box agent’s calls cannot be told apart from anybody else’s. A refund on a payment RigorRun created for another case is that case’s: shown, never counted here.
• A failed case can leave objects behind. Stripe has no undo. If creating a case’s records fails part-way, the harness failure names what was already created; nothing deletes it.
• The twin is a simulation. A verdict against it is marked so, and is good evidence about your agent’s logic and none about Stripe. Confirm a release against test mode.
• Live mode is never used. Not by RigorRun’s key, which must be a test key Stripe confirms, and not as a fallback.