Troubleshooting - XOXNO Docs

mxnode emits structured 3-line errors for everything operator-actionable: a one-line summary, the underlying cause, and the next step.

Error shape

error: <one-line summary>
  cause: <underlying cause>
   try: <next step>

JSON form (with --json):

{
  "error": {
    "summary": "...",
    "cause": "...",
    "try": "..."
  }
}

Stale `inflight.toml` (auto-clears)

When a crashed mxnode upgrade leaves an inflight.toml behind, the next mxnode upgrade invocation classifies the recorded pid. If the pid is dead, the file is cleared and the upgrade proceeds; the operator only sees a one-line stderr notice:

→ clearing stale inflight.toml from previous run (pid 12345 step BinaryReplaced); proceeding

The recovery procedure is therefore “rerun the upgrade” — every per-node step is idempotent, so steps that already ran become no-ops. The only refusal is when the recorded pid is still alive:

error: another mxnode upgrade is running (pid 12345)
  cause: inflight.toml records a live process
   try: wait for that invocation to finish, or kill it before retrying

Go toolchain too old

mx-chain-go’s go.mod floor is currently 1.22. mxnode auto-upgrades when /usr/local/go/ was put there by mxnode itself (marker file /usr/local/go/.mxnode-managed). When the existing Go came from another channel, mxnode refuses to clobber it:

warn: cannot auto-upgrade Go: /usr/local/go already exists but mxnode did not install it
error: go 1.20.7 is older than required (1.22.12)

Upgrade through the same channel you used to install Go:

Source	How
apt	`sudo apt install --only-upgrade golang-go` (often outdated; consider switching)
Homebrew	`brew upgrade go`
asdf	`asdf install golang 1.22.12 && asdf global golang 1.22.12`
Manual go.dev	`sudo rm -rf /usr/local/go && sudo tar -xzf go1.22.12.linux-amd64.tar.gz -C /usr/local`

After upgrading, re-run mxnode install or mxnode upgrade.

Missing validator keys

$ mxnode keys check
key dir: /home/<you>/VALIDATOR_KEYS
  ✓ node-0: node-0.zip (present)
  ✗ node-1: node-1.zip (missing)

1 node(s) have no zip yet — drop them in /home/<you>/VALIDATOR_KEYS before running `mxnode start`.

Drop the missing zips and re-run mxnode start --node 1. For observer / multikey installs, keys check is a no-op.

Multikey install can’t find `allValidatorsKeys.pem`

error: multikey install requires allValidatorsKeys.pem
  cause: neither --keys-file given nor /home/<you>/VALIDATOR_KEYS/allValidatorsKeys.pem found
   try: pass --keys-file <path-to-allValidatorsKeys.pem>, or drop the bundle into the node_keys dir and re-run

Either drop the file at the auto-detected path, or pass --keys-file explicitly.

P2P port collision

mx-chain-go uses TCP 37373..38383. If doctor reports:

! [p2p ports] could not bind tcp 37373: Address already in use

something else is holding the port. Find the offender:

sudo ss -tlnp 'sport = :37373'

Most common cause: another mxnode install on the same host without cleanup first. Firewall block (port not yet open):

Distro	Command
Ubuntu (ufw)	`sudo ufw allow 37373:38383/tcp`
firewalld	`sudo firewall-cmd --permanent --add-port=37373-38383/tcp && sudo firewall-cmd --reload`
iptables	`sudo iptables -A INPUT -p tcp --dport 37373:38383 -j ACCEPT`

Auto-init failed to detect user / home

error: could not determine where to write config.toml
  cause: ...
   try: set $XDG_CONFIG_HOME or $HOME so mxnode can place the file under <home>/.config/mxnode/

Happens under systemd-run or other contexts where $HOME is unset:

HOME=/home/operator mxnode install --role multikey
# or
mxnode --config /etc/mxnode/config.toml install --role multikey

Node fails to start after upgrade

sudo systemctl status elrond-node-0.service
sudo journalctl -u elrond-node-0.service --since '5 min ago' --no-pager

Symptom	Cause	Fix
`panic: open /home/.../config/...: no such file or directory`	New tag’s config repo has different filenames	Pass `--config-tag <T>` matching the binary tag
`cannot start node: shard X not assigned`	Squad mapping lost after a manual prefs edit	`mxnode reapply-config --restart`
`keys file is empty`	`allValidatorsKeys.pem` got truncated during a copy	Restore from your source-of-truth bundle, then re-install
Node exits immediately, no log	Wrong CPU flags (rare on ARM hosts)	mxnode auto-clears `[HardwareRequirements] CPUFlags` on non-x86 — re-run `mxnode reapply-config`

Out-of-sync after long downtime

If a node fell far behind (days, not minutes), reseeding is faster than catching up via P2P:

mxnode db reseed --node 0 --yes

GitHub rate limit

Tag resolution hits the GitHub API. Unauthenticated, 60 requests/hour per IP.

error: could not look up the latest release of multiversx/mx-chain-go
  cause: HTTP 403 — rate limit exceeded
   try: pass --binary-tag explicitly, or set MXNODE_GITHUB_TOKEN

Set a personal access token (no scopes needed):

export MXNODE_GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxx

The token isn’t logged or written to disk — it lives only in the env.

Diagnostic dump for support

When opening an issue:

mxnode --json doctor > doctor.json
mxnode --json status > status.json
mxnode --json config show > config.json

These are sandbox-safe (no keys, no tokens) and give a full picture of the host’s state.

​Error shape

​Stale inflight.toml (auto-clears)

​Go toolchain too old

​Missing validator keys

​Multikey install can’t find allValidatorsKeys.pem

​P2P port collision

​Auto-init failed to detect user / home

​Node fails to start after upgrade

​Out-of-sync after long downtime

​GitHub rate limit

​Diagnostic dump for support