What breaks, and what does not
Triage
Separate an endpoint problem from a network problem before acting. They look identical from one client.
If a second provider works, the fault is yours and failover fixes it. If no provider advances, the network itself is degraded and there is nothing to fail over to. In that case the correct action is to wait; the protocol’s state is intact and no user is exposed by the delay.
Failover
Each deployment reads its endpoint from configuration rather than a hardcoded value, so failover is a configuration change and a restart, not a code change.1
Confirm the fault is the endpoint
Query a second provider for the current ledger. If it advances, your endpoint is the problem.
2
Repoint the affected services
Update the RPC URL for the application, the keeper, and the exporter or indexer. The network passphrase does not change — you are switching endpoints, not networks.
3
Restart and confirm reads
Confirm each service reads current ledger state before you consider it recovered.
4
Confirm submissions land
A successful read does not prove submissions work. Submit one low-value operation and confirm it lands.
Prolonged outages
The one component with a real deadline is the storage rent keeper. Soroban ledger entries expire if their time-to-live (TTL) is not extended. An expired entry archives, and it must be restored before the contract can use it again. The keeper exists to stay ahead of that. On mainnet it wakes every 6 hours and extends anything due to expire within the next 21 days (ttl_safety_margin_days: 21 in services/keeper/config/mainnet.yaml; testnet uses 14). It also restores archived entries it finds.
That margin is the outage budget. A few hours of downtime costs you a tick or two and nothing expires. Losing the keeper for three weeks is what actually risks expiry.
So if RPC access looks like it will be out for days, restore keeper connectivity before the application. Delayed user transactions cost users time. Expired storage costs a restore transaction per entry, and the contract cannot run until it is back.
Do not
- Do not point production at an endpoint you do not trust. RPC responses drive what your application shows users and what the keeper decides to submit. A hostile endpoint can lie about state. Prefer a known provider over the first reachable one.
- Do not assume the chain is down because your endpoint is. Check a second provider before declaring a network incident.
- Do not pause the protocol for an RPC outage. Pausing is an on-chain action addressing an on-chain problem. It does not repair connectivity, and it requires a governance action to reverse.