Broker Token Refresh and Connection Failures

Automated trading systems often report connected when an HTTP health check succeeds with a new token. That can be dangerously incomplete. An older WebSocket may remain closed or authenticated under obsolete state, subscriptions may be missing, and order updates may have occurred during the gap. Local working orders and positions can therefore disagree with the broker even though the next REST request succeeds.

Use a recovery state machine. On an authentication or connection fault, stop accepting new order mutations, keep durable signals queued or rejected according to policy, and elect one session owner to refresh. Recreate streams through the documented flow, resubscribe, retrieve current orders, executions, and positions, resolve uncertain submissions, and only then mark the deployment ready.

A 401-like authorization response, an HTTP 429, a transport timeout, a clean WebSocket close, a missed heartbeat, and an order rejection demand different actions. Refreshing on every error can create a login storm and obscure the original cause. Parse the documented status and response body, preserve broker error fields, and attach the failure to the active credential generation and connection ID.

Invalid or revoked credentials require operator action; repeated automated refresh is unlikely to help. Rate limiting requires documented backoff. Network loss requires reconnect and state recovery. An application schema error requires a code or configuration fix. A provider policy or account-stage disablement must not be retried at all. Unknown errors should fail closed with bounded attempts and alerting.

Without coordination, every strategy that receives an authorization error can start its own renewal. That multiplies calls, races token generations, and can trigger strict authentication limits. Tradovate's rate-limit documentation, for example, identifies specific limits for access-token and renewal endpoints and warns about penalty behavior. The architecture should centralize authentication rather than let each order worker log in independently.

Assign a monotonically increasing credential generation. Requests read the current generation and, on expiry, join the same refresh promise. When renewal succeeds, atomically publish the new token and generation, invalidate old connection objects, and prevent late refresh results from overwriting newer credentials. When renewal fails definitively, open a circuit breaker, disarm execution, and notify an operator without printing secrets.

Tradovate documents access-token operations and rate-limit behavior, including penalty tickets for some endpoint limits. Topstep's ProjectX troubleshooting notes that session tokens expire and tells clients to refresh rather than reuse an old token, while ProjectX documents its own authenticated API limits. NinjaTrader's Trader API, Rithmic interfaces, and CQG products each have their own authorization and connection contracts. Use their current references and approved SDK behavior.

Differentiate a long-lived application secret or API key from the short-lived session material derived from it. Do not regenerate or rotate a user's root key every time an access token expires unless the official process says so. Conversely, do not reuse expired session tokens indefinitely. Store root secrets narrowly, hold access tokens in protected memory where practical, and log only fingerprints and generation numbers.

A socket open event proves transport, not continuity. Record the last processed event sequence or broker identifier when the interface provides one. After reconnect, use supported snapshots or queries to obtain working orders, recent executions, and positions. Apply the snapshot and subsequent stream events through an ordering strategy appropriate to the API so late messages cannot roll state backward.

If the broker does not expose a replay cursor, use a conservative overlap window and deduplicate by broker-native event identifiers. Keep the strategy disarmed until snapshot and stream agree. Replace the old socket rather than retaining two active consumers that can produce duplicate updates. Heartbeats should detect liveness, while a separate freshness metric proves that account state is advancing.

The client can lose connectivity after the broker accepts a request but before its acknowledgement arrives. Marking that request failed and replaying it can create a second order. Persist intent and a unique client correlation before submission. On recovery, search the supported order and execution state for that correlation or otherwise reconcile the account and contract carefully.

If the interface cannot prove the result, escalate the event as unresolved and block a conflicting strategy action. A human can inspect the broker-native platform and decide whether to cancel, flatten, or leave the position. Never automatically flatten solely because local state is uncertain; an extra flatten can reverse exposure if the assumed position was wrong. Establish actual broker state first.

Useful metrics include time until known expiry where the API exposes it, refresh attempts and outcomes, requests waiting on refresh, active connection count, reconnect reason, last account event, state age, unresolved intent count, and whether the deployment is armed. Alert on repeated refresh, two concurrent session owners, stale account state, and any transition that remains in reconciling beyond the system's tested expectation.

Run drills in the permitted test environment: expire or invalidate a session, interrupt the stream, restart during a working order, inject a duplicate event, trigger a documented rate limit carefully, and rotate the root credential. Verify that the system disarms, refreshes once, reconstructs state, and requires policy review where appropriate. Document a manual recovery path for times when authorization cannot be restored.