Impact
Triage
Compare independent providers before attributing the failure to the network.
Several failed providers do not establish a consensus failure. Check network
status and provider independence. During a network outage, monitor recovery and
pending submissions; delayed protective transactions can still increase risk.
For simulation failures, match the error code and emitting contract from the
same raw diagnostic event. A handled nested failure can precede the final
propagated error. Preserve raw diagnostics before using display helpers that
omit metadata or success flags.
Failover
The keeper tries configured fallback endpoints for idempotent reads and simulations. Submissions use the active endpoint without blind retry. Readrpc.urls in the keeper’s active configuration, using
services/keeper/config/<network>.yaml as the repository reference. Other
services may require an explicit switch.
1
Select a healthy provider
Query an independent, trusted provider for the current ledger and confirm
it advances. RPC responses influence displayed state and submitted actions.
2
Restore affected services
Check keeper fallback health. Update unavailable endpoints and switch the
application, exporter, and indexer according to their failover support.
Preserve the network passphrase.
3
Verify reads
Confirm each service reads current ledger state. Restore indexer ingestion
from its durable checkpoint and backfill the gap.
4
Resolve uncertain submissions
Follow the original-hash recovery procedure below. Confirm submission
recovery with an authorized test transaction and its final ledger result;
successful reads alone do not establish submission recovery.
Resolve an unknown transaction
Before the first send, persist the network, signed envelope XDR, original hash, source account, sequence, and validity bounds. Preserve these records across endpoint switches and process restarts.getTransaction returns SUCCESS, FAILED, or NOT_FOUND; it has no PENDING
status. While the original envelope remains valid, retry only that identical
signed envelope. A new simulation, signature, sequence, or validity bound can
produce a replacement transaction and must wait for outcome reconciliation.
Use ledger close time to determine whether time bounds have expired. Expiry
prevents future inclusion; it does not prove the original was never included.
Check the provider’s oldest and latest retained ledgers and close times. History
must cover every ledger in which the original could have been included, and the
provider must have caught up. Search complete history for the original hash and
reconcile the source sequence and affected contract state before replacement.
A current sequence alone does not identify which transaction consumed it.
If retained history cannot establish the original outcome, stop replacement
submission and recover complete history. A second endpoint with the same
retention gap does not resolve the uncertainty.
Storage-renewal deadline
Persistent Soroban entries archive when their time-to-live (TTL) expires and require restoration before use. On Protocol 23 and later, a normal simulated and assembled invocation can include inline restoration. Honor the RPC simulation and assembly result; use a separate restore transaction when that workflow requires one. Re-simulate after a separate restore before signing the invocation. The keeper also extends configured entries before expiry and submits dedicated restore operations for archived entries it finds. Readschedule.ttl_tick_seconds and schedule.ttl_safety_margin_days in the
active keeper configuration. The safety margin is a renewal policy, not a
guaranteed outage budget. Measure the minimum remaining TTL across required
entries and check failed extensions, omitted keys, and added dependencies.
Prioritize recovery using that deadline together with repayment and liquidation
coverage. Pausing the controller does not restore connectivity, and unpausing
requires governance.