> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sesameterminal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Runbook: incidents

# Runbook: incidents

This runbook is for the account owner. It says what to do when something in sesame-bun
looks wrong while trading. Each section starts with the steps to take now, then how to
find out what happened.

Related: [user guide](../guides/user/trading.md) (what the controls and reject messages mean),
[Session Key setup](session-key-setup.md) (rotation and revocation), and the
[secrets and logging policy](../security/secrets-and-logging.md#5-rotation) (compromise
response). Where behaviour is not fixed yet it says **defined by implementation**.

## First steps for any incident

When you are not sure what is happening, do these in order. Each one is safe on its own.

1. **Cancel All:** `Mod+Shift+X` (Ctrl+Shift+X on Windows and Linux), or the top-bar
   **Cancel All**. It works in SAFE and is never blocked by the risk engine.
2. **Switch to SAFE** with the header switch.
3. **Check polymarket.com** for open orders and recent trades on the account. It is the
   reference when the app and the venue disagree.
4. If the app itself may be at fault, set `LIVE_TRADING=false` and restart. The app starts
   in SAFE and with trading off; positions, orders and cancels still work.

Write down the time (UTC) and what you saw before you change anything else. When you
share logs, check them for secrets first
([live verification §3, step 5](live-verification.md#3-if-a-step-fails)) and never share
`.env`.

## 1. A key may have leaked

Treat the Session Key as leaked if its private key, or the `.env` holding it, appeared
anywhere outside the app host: a log, a commit, a screenshot, a chat, an email, a ticket
or an agent session; or if the host may be compromised.

A leaked Session Key may be able to move funds out of the Deposit Wallet: the wallet
contract does not stop it, only Polymarket's relayer does
([ADR 0012](../adr/0012-session-key-scope.md)). Even if it cannot, it can lose them by
trading: it can sell your positions into an attacker's low bids or buy an attacker's high
asks (security finding F5). That is why the wallet holds only the working balance set in
[ADR 0013](../adr/0013-working-balance-cap.md), and why the key is revoked at once.

**Now:**

1. Cancel All, switch to SAFE, set `LIVE_TRADING=false` and restart the app.
2. **Revoke the Session Key** from the owner machine
   ([Session Key setup §6](session-key-setup.md#6-revocation)). This stops the key from
   trading and cancels its open orders. Revoking only the CLOB API credentials is not
   enough: the key can derive new ones.
3. On polymarket.com, check positions and history for trades you did not make.

**Then:**

1. Rotate every secret the host held: create a new Session Key
   ([Session Key setup §5](session-key-setup.md#5-rotation)), a new UI password hash
   (`task hash-password`), a new `SESSION_SECRET`, and the Telegram bot token if set.
   Clear any `POLYMARKET_CLOB_*` values; they belonged to the old key.
2. If the host may be compromised, rebuild it before putting the new key on it.
3. Compare the audit log with Polymarket's history (`task audit-verify`, then the
   comparison in [live verification step 15](live-verification.md#step-15-audit-log)).
   Trades in Polymarket's history that are not in the audit log were not made by the app.
4. If the leaked key's scope turns out to be broader than `CLOB`, move the funds with the
   owner key on polymarket.com.
5. Record the date and reason of the rotation.

After revocation the app's credential state becomes `failed` with `auth_rejected`; when
exactly is defined by implementation. How fast revocation takes effect at the venue is
UNVERIFIED (secrets policy §5).

## 2. Fills or orders you did not make

The app compares the account's activity on Polymarket (from the Data API, which sees
every entry on the wallet) with its own audit log. It reports as foreign activity:

* an order, fill or position change it did not place;
* a withdrawal or other outgoing pUSD transfer, a split, a merge or a conversion. The app
  never does any of these, so each one is either you on polymarket.com or someone holding
  the key (security finding P3R-02).

Redeems and incoming transfers are not reported. On foreign activity the app raises an
alert and switches to SAFE, and `GET /api/trading` `reasons` shows `foreign_activity`
(security finding F5). The flag is stored, so a restart does not clear it.

**Now:**

1. Cancel All. The app is already SAFE; keep it there.
2. Ask yourself whether the activity was yours: a trade, withdrawal, transfer, split,
   merge or conversion on polymarket.com, on the mobile app, or with another tool or
   Session Key. A Session Key sees only its own orders, so trading from anywhere else on
   the same wallet looks foreign to the app. Compare the entry's time and amount in
   polymarket.com's history with what you did.

**If it was yours:** arm again with the UI password. Arming after foreign activity asks
for it: the switch opens a password dialog, or send it as `password` in the body of
`POST /api/trading/arm` (HTTP 403 `step_up_required` without it, 429 `rate_limited` after
too many attempts). A correct password arms and clears `foreign_activity`. A stolen
browser session alone cannot re-arm. To avoid the pause, do not trade or move funds on
the same wallet from elsewhere while the app runs.

**If it was not yours:** treat the key as leaked and follow [section 1](#1-a-key-may-have-leaked).
If the Session Key was not the source (for example, the trades came from the owner's
own login), secure the owner account on polymarket.com as well.

**Unexpected fills on your own orders** are not foreign activity: an order you left
resting can fill later, including a stale order after a price jump. Check Orders and the
fills list, and the audit log for the order's `client_order_id`.

A **combo trade** the app did not make has its own section:
[section 7](#7-a-combo-trade-the-app-did-not-make).

## 3. The venue is in cancel-only or post-only mode

Polymarket sometimes restricts trading. The app shows the reason in reject toasts and, for
cancel-only, a status pill. The app never retries an order on its own.

| What you see | Code | What the venue allows | What to do |
| - | - | - | - |
| `Not placed · venue restarting (425)` | `venue_restarting` | Nothing, briefly. After a restart it is post-only for about two minutes | Wait at least `retry_after_ms`, then place again by hand |
| `Rejected · post-only mode` | `venue_post_only` | Post-only orders and cancels | Wait for `retry_after_ms`. Orders that would rest (not cross) can still be placed as post-only; Exit @ join is post-only |
| `Not placed · venue in cancel-only mode (503)` | `venue_cancel_only` | Cancels only | Cancel what you need to. New orders wait until the mode ends |
| A reject naming trading disabled on the venue (503) | `venue_disabled` | Nothing | Wait. Check Polymarket's status channels |
| `Not placed · rate limited (429)` | `venue_rate_limited` | Nothing until the limit resets | Wait at least `retry_after_ms`. Repeated 429s mean something is sending too many requests; check the log |

Cancels, Cancel All and the Exit @ join order keep working in post-only mode. In
cancel-only mode cancels still work. How long each mode lasts is up to the venue; the app
has no setting that changes it.

## 4. An order stuck in `unknown`

An order is `unknown` when the venue did not answer in time: it may or may not be on the
book. Before sending, the app recorded the signed order's id (its order hash) in the
audit log ([ADR 0011](../adr/0011-audit-first-order-path.md)). It never re-sends the
order; it reconciles by that id: the open orders holding it, a fill of it on the user
feed, or a lookup of it at the venue. The chip shows `?` (`Checking…`) until
reconciliation settles it.

**Unresolved placements.** `GET /api/trading` `unresolved_placements` counts the
placements still `unknown` (security finding P3R-07). The app looks each one up 2 to 30
seconds after the unknown outcome; a count still above 0 after that means the venue has
not answered the lookup, and the app asks again every 60 seconds until it does. While a
buy is unresolved, its notional stays reserved against `max_position_notional`, so new
buys in that market can be refused. An id the venue answers as not found is settled as
not placed. The count also survives a restart: at start, the app looks up every placement
the audit log left without a final outcome.

**Now:**

1. **Do not click again.** A new click is a new order.
2. Wait for the chip to become `placed`, a fill, or `Not placed`. Reconciliation normally
   settles it within 30 seconds.

**If it does not settle** (`unresolved_placements` stays above 0):

1. Check polymarket.com for the order at that market, side and price.
2. If it is on the book and you do not want it, cancel it in the app (cancel-market
   covers it even when the app has no order id for it) or on polymarket.com.
3. Check the user feed: Settings or `GET /api/status` should show `pm.user` connected.
   While it is down, order updates pause and the orders views are greyed.
4. Restart the app. At startup it reads the open orders from the venue and looks up each
   unresolved placement by its order id. It starts in SAFE.
5. Look up the `client_order_id` in the audit log: it has the intent, decision and
   `signed` rows (the signed row holds the order id), a response row recording `unknown`
   and, once settled, a `reconcile` row with what was found. Report a case that never
   settles, with the `client_order_id`, the order id and the log lines (`order outcome
   unknown`), to the tech lead.

## 5. Other problems

| Symptom | Likely cause | What to do |
| - | - | - |
| The switch drops to SAFE by itself | Idle timeout, logout, geoblock, credential loss or foreign activity | `GET /api/trading` `reasons` says which. For `foreign_activity`, see [section 2](#2-fills-or-orders-you-did-not-make); arming again asks for the password |
| `unresolved_placements` above 0 for more than a minute | The venue has not answered the lookup of an `unknown` order | See [section 4](#4-an-order-stuck-in-unknown) |
| Orders refused with `clock_skew` | The host clock drifted from the venue's | Resynchronise the clock (Windows: `w32tm /resync`; Linux: check `timedatectl`). The skew is re-measured periodically; the interval is defined by implementation |
| Orders refused with `geoblocked` | The geoblock check reports blocked or failed | Trade only from a permitted location. The app has no override ([ADR 0005](../adr/0005-no-geoblock-workarounds.md)) |
| Credentials `failed` with `wallet_mismatch` | The Session Key is not listed as a signer of `POLYMARKET_WALLET_ADDRESS`, or the address is wrong | Check the wallet address and that the key is authorised ([Session Key setup](session-key-setup.md#7-troubleshooting)) |
| Credentials `failed` with `auth_rejected` | The key was revoked, expired, or the configured CLOB credentials belong to another key | [Session Key setup §7](session-key-setup.md#7-troubleshooting) |
| Orders vanished after the app stopped | `cancel_on_shutdown` or the heartbeat cancelled them | Expected. See [Safety switches](../guides/user/settings.md#safety-switches) |
| All orders vanished while the app was running, with the heartbeat on | The venue missed heartbeats for about 10 s (network, a stalled host) | Check the log for heartbeat failures and the network. Re-place what you need |
| Resting orders on a game vanished at kick-off | Polymarket clears sports books at game start | Expected. Use **Re-post** ([user guide](../guides/user/order-rules.md#sports-markets)) |
| `task audit-verify` fails | A row in `order_audit` was changed or removed, or the database is damaged | Stop trading (SAFE, `LIVE_TRADING=false`). Keep a copy of the database file; do not edit it. Report to the tech lead |
| A combo slip stays on `Checking…` | The answer to an Accept was lost | See [section 6](#6-a-combo-accept-stuck-in-checking) |
| `Combos unavailable` | The Session Key lacks the combos scope, or the app runs without a Session Key | See [section 8](#8-combos-unavailable) |

## 6. A combo Accept stuck in `Checking…`

An Accept is unknown when Polymarket's answer did not arrive or could not be read: a
timeout, a lost connection, a 5xx (503 included) or a 409 `INVALID_RFQ_STATE`. The combo may
or may not have been bought. Before sending, the app recorded the signed order's hash, the
`rfq_id` and the `quote_id` in the audit log. It **never sends the Accept again**; it reads
the quote request's status at Polymarket by its `rfq_id`
([ADR 0015](../adr/0015-combo-accept-path.md)). The slip reads `Checking…`, then
`Checking combo · no response after 10s`.

The app reads the status every 500 ms, every 5 s once the Accept has run for a minute, and
backs off up to a minute between failed reads. It settles the Accept as:

* **filled** (`CONFIRMED` or `FILLED` with a transaction hash): `Combo filled`;
* **not filled** (`FAILED`, `CANCELED`, `EXPIRED`, `REJECTED` or a status the app does not
  know): `Combo rejected` or `Combo failed`;
* **never recorded** (Polymarket answers that it holds no Accept for the quote):
  `Rejected · never reached the venue` (`venue_not_found`).

While it is being checked, a buy's cost with fees stays reserved against
`max_combo_exposure` (a sell's shares against the position), the leg toggles and Exit are
disabled (`Combo being accepted`), and `GET /api/combos/rfqs` lists the quote request with
`status` `unknown` or `executing`.

**Now:**

1. **Do not get a new quote and accept it.** The first Accept may still fill, and a second
   one would buy a second combo.
2. Wait for the slip to read `Filled`, `Rejected · …` or the `Combo filled`, `Combo
   rejected` or `Combo failed` notification. Cancel all does not affect an Accept already
   sent.

**If it does not settle** within a few minutes:

1. On polymarket.com, check whether a combo position or combo trade appeared at the time
   of the Accept.
2. In the log, look for `combo accept outcome unknown` with its `rfqId`, and repeated
   `combo status read failed` lines: the status reads are failing, not the Accept.
3. Restart the app. At start it resumes the status reads of every Accept the audit log left
   without a final outcome, with its reservation held, and logs
   `reconciling combo accepts` with the count. It starts in Safe.
4. Look up the Accept's `client_order_id` in the audit log: the intent, decision and
   `signed` rows (`rfq_id`, `quote_id`, `order_hash`), a `response` row with `unknown` or
   `executing`, and, once settled, a `reconcile` row with the status and `tx_hash`. Report a
   case that never settles, with the `client_order_id`, the `rfq_id` and those log lines, to
   the tech lead.

## 7. A combo trade the app did not make

The app explains a combo trade in the account's activity (a `/v2/activity` trade marked
combo) only by one of its own filled Accepts with the same transaction hash (PLAN §3.7).
Any other combo trade is foreign activity, handled as in [section 2](#2-fills-or-orders-you-did-not-make):
an alert, Safe, and `foreign_activity` in `GET /api/trading` `reasons`. The cash it moved
is not explained either, since combo trade cash is counted only from the app's filled
Accepts, so a cash alert can follow. While one of the app's own Accepts is still being
checked (section 6), an unmatched combo trade waits for the next check instead of being
reported at once.

**Now:**

1. Cancel all. The app is already Safe; keep it there.
2. Ask whether it was yours: a combo bought or sold on polymarket.com or with another tool
   counts as foreign, like any trade made outside the app. Compare the trade's time and
   amount in polymarket.com's history with what you did.

**If it was yours:** arm again with the UI password, as in section 2.

**If it was not yours:** treat the key as leaked and follow [section 1](#1-a-key-may-have-leaked).
The combos scope lets the key request and accept quotes, so a leaked key can buy combos at
prices an attacker's quotes set.

**If it matches an Accept the app settled as not filled** (`venue_not_found`, `Combo
rejected` or `Combo failed`, with a trade at the same time on the same legs): the status
read and the trade disagree. Security finding P8-01 describes one way this happens: a
first status read that answers 409 before Polymarket has recorded the Accept. Stay Safe and
report it to the tech lead with the Accept's `client_order_id`, its `rfq_id`, the trade's
transaction hash and the log lines.

**If it is a combo the app filled** (`Combo filled` with the same transaction hash) and the
alert still came, or it came from the order manager rather than the activity check: the
combo fill reached the app through the CLOB user feed or the fills backfill (security
finding P8-04). Stay Safe and report it the same way.

**While an Accept of the app stays in `Checking…`,** unmatched combo trades wait for it to
settle and are not reported (security finding P8-03). If one stays in `Checking…` for more
than a few minutes, check polymarket.com's history for combo trades you did not make
([section 6](#6-a-combo-accept-stuck-in-checking)).

## 8. Combos unavailable

`combos_unavailable` has two causes.

**The Session Key lacks the combos scope.** The `Combo` toggle and `Get quote` read
`Combos unavailable · Session Key lacks the combo scope`, `GET /api/trading`
`combos_reasons` lists `combos_unavailable`, and a quote request is refused with it
before anything is sent. `GET /api/account` shows `credentials.scopes` without `combos`.
Predictions trading and the combo positions are not affected.

1. Check the start log's `session key confirmed` line: its `scopes` lists what Polymarket
   reports for the key.
2. Issue a key with `CLOB` and `COMBOSRFQ` (or `ALL`), switch to it and revoke the old one
   ([Session Key setup §8.2](session-key-setup.md#82-issue-a-key-with-the-combos-scope)).
   The scopes are read when the credentials start, so the change needs a restart.

While the credentials are not `ready`, combos report `no_credentials` instead; see the
credential state in Settings ([Session Key setup §7](session-key-setup.md#7-troubleshooting)).

**No Combos venue.** The combo endpoints answer HTTP 503 `combos_unavailable`:
`POST /api/combos/quote`, `/api/combos/accept` and `GET /api/combos/rfqs` when the app runs
without `POLYMARKET_SESSION_KEY`, and the combo positions and activity endpoints when it
also runs without `POLYMARKET_WALLET_ADDRESS`. Set the variables
([Session Key setup §3](session-key-setup.md#3-configure-the-app)) and restart.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.