> ## Documentation Index
> Fetch the complete documentation index at: https://xoxno.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Stale upgrade locks, Go toolchain mismatch, missing keys, port collisions, and reading mxnode's structured errors.

mxnode emits structured 3-line errors for everything operator-actionable: a one-line summary, the underlying cause, and the next step.

## Error shape

```
error: <one-line summary>
  cause: <underlying cause>
   try: <next step>
```

JSON form (with `--json`):

```json theme={"system"}
{
  "error": {
    "summary": "...",
    "cause": "...",
    "try": "..."
  }
}
```

## Stale `inflight.toml` (auto-clears)

When a crashed `mxnode upgrade` leaves an `inflight.toml` behind, the next `mxnode upgrade` invocation classifies the recorded pid. If the pid is dead, the file is cleared and the upgrade proceeds; the operator only sees a one-line stderr notice:

```
→ clearing stale inflight.toml from previous run (pid 12345 step BinaryReplaced); proceeding
```

The recovery procedure is therefore "rerun the upgrade" — every per-node step is idempotent, so steps that already ran become no-ops.

The only refusal is when the recorded pid is **still alive**:

```
error: another mxnode upgrade is running (pid 12345)
  cause: inflight.toml records a live process
   try: wait for that invocation to finish, or kill it before retrying
```

## Go toolchain too old

mx-chain-go's `go.mod` floor is currently 1.22. mxnode auto-upgrades when `/usr/local/go/` was put there by mxnode itself (marker file `/usr/local/go/.mxnode-managed`). When the existing Go came from another channel, mxnode refuses to clobber it:

```
warn: cannot auto-upgrade Go: /usr/local/go already exists but mxnode did not install it
error: go 1.20.7 is older than required (1.22.12)
```

Upgrade through the same channel you used to install Go:

| Source        | How                                                                                     |
| ------------- | --------------------------------------------------------------------------------------- |
| apt           | `sudo apt install --only-upgrade golang-go` (often outdated; consider switching)        |
| Homebrew      | `brew upgrade go`                                                                       |
| asdf          | `asdf install golang 1.22.12 && asdf global golang 1.22.12`                             |
| Manual go.dev | `sudo rm -rf /usr/local/go && sudo tar -xzf go1.22.12.linux-amd64.tar.gz -C /usr/local` |

After upgrading, re-run `mxnode install` or `mxnode upgrade`.

## Missing validator keys

```bash theme={"system"}
$ mxnode keys check
key dir: /home/<you>/VALIDATOR_KEYS
  ✓ node-0: node-0.zip (present)
  ✗ node-1: node-1.zip (missing)

1 node(s) have no zip yet — drop them in /home/<you>/VALIDATOR_KEYS before running `mxnode start`.
```

Drop the missing zips and re-run `mxnode start --node 1`. For observer / multikey installs, `keys check` is a no-op.

## Multikey install can't find `allValidatorsKeys.pem`

```
error: multikey install requires allValidatorsKeys.pem
  cause: neither --keys-file given nor /home/<you>/VALIDATOR_KEYS/allValidatorsKeys.pem found
   try: pass --keys-file <path-to-allValidatorsKeys.pem>, or drop the bundle into the node_keys dir and re-run
```

Either drop the file at the auto-detected path, or pass `--keys-file` explicitly.

## P2P port collision

mx-chain-go uses TCP `37373..38383`. If `doctor` reports:

```
! [p2p ports] could not bind tcp 37373: Address already in use
```

something else is holding the port. Find the offender:

```bash theme={"system"}
sudo ss -tlnp 'sport = :37373'
```

Most common cause: another mxnode install on the same host without `cleanup` first.

Firewall block (port not yet open):

| Distro       | Command                                                                                  |
| ------------ | ---------------------------------------------------------------------------------------- |
| Ubuntu (ufw) | `sudo ufw allow 37373:38383/tcp`                                                         |
| firewalld    | `sudo firewall-cmd --permanent --add-port=37373-38383/tcp && sudo firewall-cmd --reload` |
| iptables     | `sudo iptables -A INPUT -p tcp --dport 37373:38383 -j ACCEPT`                            |

## Auto-init failed to detect user / home

```
error: could not determine where to write config.toml
  cause: ...
   try: set $XDG_CONFIG_HOME or $HOME so mxnode can place the file under <home>/.config/mxnode/
```

Happens under `systemd-run` or other contexts where `$HOME` is unset:

```bash theme={"system"}
HOME=/home/operator mxnode install --role multikey
# or
mxnode --config /etc/mxnode/config.toml install --role multikey
```

## Node fails to start after upgrade

```bash theme={"system"}
sudo systemctl status elrond-node-0.service
sudo journalctl -u elrond-node-0.service --since '5 min ago' --no-pager
```

| Symptom                                                       | Cause                                               | Fix                                                                                              |
| ------------------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `panic: open /home/.../config/...: no such file or directory` | New tag's config repo has different filenames       | Pass `--config-tag <T>` matching the binary tag                                                  |
| `cannot start node: shard X not assigned`                     | Squad mapping lost after a manual prefs edit        | `mxnode reapply-config --restart`                                                                |
| `keys file is empty`                                          | `allValidatorsKeys.pem` got truncated during a copy | Restore from your source-of-truth bundle, then re-install                                        |
| Node exits immediately, no log                                | Wrong CPU flags (rare on ARM hosts)                 | mxnode auto-clears `[HardwareRequirements] CPUFlags` on non-x86 — re-run `mxnode reapply-config` |

## Out-of-sync after long downtime

If a node fell far behind (days, not minutes), reseeding is faster than catching up via P2P:

```bash theme={"system"}
mxnode db reseed --node 0 --yes
```

## GitHub rate limit

Tag resolution hits the GitHub API. Unauthenticated, 60 requests/hour per IP.

```
error: could not look up the latest release of multiversx/mx-chain-go
  cause: HTTP 403 — rate limit exceeded
   try: pass --binary-tag explicitly, or set MXNODE_GITHUB_TOKEN
```

Set a personal access token (no scopes needed):

```bash theme={"system"}
export MXNODE_GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxx
```

The token isn't logged or written to disk — it lives only in the env.

## Diagnostic dump for support

When opening an issue:

```bash theme={"system"}
mxnode --json doctor > doctor.json
mxnode --json status > status.json
mxnode --json config show > config.json
```

These are sandbox-safe (no keys, no tokens) and give a full picture of the host's state.