> ## Documentation Index
> Fetch the complete documentation index at: https://xoxno.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Multikey and redundancy

> How allValidatorsKeys.pem is distributed, and how --backup turns one host into a backup for another.

Multikey is the modern way to run validators on MultiversX. One host runs four observer-style nodes (one per shard) that jointly sign for **every** key in a shared `allValidatorsKeys.pem` bundle. mxnode automates the file distribution and per-node config edits.

## Primary install

```bash theme={"system"}
mxnode install --role multikey
```

The keys file is auto-detected at `~/VALIDATOR_KEYS/allValidatorsKeys.pem`. Drop it there before installing.

What mxnode does:

1. Builds the node binary once.
2. For each of the four nodes (shards 0/1/2/metachain):
   * Copies `allValidatorsKeys.pem` to `~/elrond-nodes/node-{i}/config/` with mode `0600`.
   * Stamps `Preferences.RedundancyLevel = 0`.
   * Stamps `Preferences.DestinationShardAsObserver = "<shard>"`.
   * Enables `[DbLookupExtensions] Enabled = true`.
3. Writes 4 systemd units and records everything in `state.toml`.

## Backup install

Same command, one extra flag:

```bash theme={"system"}
mxnode install --role multikey --backup
```

That stamps `RedundancyLevel = 1` on every node. The backup observes the network alongside the primary and only signs when mx-chain-go's redundancy logic detects the primary has gone silent.

For a backup-of-backup chain, increase the level:

```bash theme={"system"}
mxnode install --role multikey --backup 2
mxnode install --role multikey --backup 3
```

`--backup` with no value defaults to `1` — the common case.

<Warning>
  Every redundancy machine must hold the **byte-identical** `allValidatorsKeys.pem`. If the bundles diverge, the backup signs for a different key set and the network sees a fork. Verify with `sha256sum` before installing the backup.
</Warning>

## Concepts

| Term                    | Meaning                                                                                                                                                  |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `allValidatorsKeys.pem` | A single PEM containing every BLS private key the operator owns.                                                                                         |
| Multikey node           | An observer that detects `allValidatorsKeys.pem` in its `config/` and signs blocks for the keys inside, behind whichever observer is in the right shard. |
| Primary                 | A multikey host with `RedundancyLevel = 0`. Signs unconditionally.                                                                                       |
| Backup (`--backup`)     | A multikey host with `RedundancyLevel >= 1`. Takes over signing when the primary stops responding.                                                       |

## Explicit keys file path

If your bundle isn't at the default location (e.g. mounted from a vault):

```bash theme={"system"}
mxnode install --role multikey --keys-file /run/secrets/allValidatorsKeys.pem
```

The file must exist at install time. mxnode does not read it after install — the per-node copies are the source of truth at runtime.

## File permissions

Per-node copies are written with mode `0600` (owner read/write only). The owner is the `custom_user` from `config.toml` (auto-detected to your login user during init). No other user on the box can read the keys.

## Verifying

After install, sanity-check each node:

```bash theme={"system"}
for i in 0 1 2 3; do
  echo "=== node-$i ==="
  ls -la ~/elrond-nodes/node-$i/config/allValidatorsKeys.pem
  grep -E "^   (RedundancyLevel|DestinationShardAsObserver)" \
    ~/elrond-nodes/node-$i/config/prefs.toml
done
```

Expected output per node (primary):

```
=== node-0 ===
-rw------- 1 you you 1234 ...  /home/you/elrond-nodes/node-0/config/allValidatorsKeys.pem
   DestinationShardAsObserver = "0"
   RedundancyLevel = 0
```

For a `--backup` install, `RedundancyLevel` reads `1` (or your chosen N) instead.

## Adding nodes later

`mxnode add-nodes` reuses the existing `allValidatorsKeys.pem` from `node-0` automatically — no `--keys-file` needed:

```bash theme={"system"}
mxnode add-nodes --count 2 --role multikey
```

The new nodes inherit the bundle and the install's existing redundancy level. Rotating the bundle across an entire host isn't a built-in command yet — for now, run `mxnode cleanup --yes --execute` and reinstall.

## Observer keys

Plain observer installs ship no `validatorKey.pem`. The mx-chain-go binary generates one on first start when `config/` has none, so the observer comes up with a throwaway key without any operator action.

For a stable observer identity (e.g. to populate `Preferences.Identity`), generate a key pair manually with `mxnode keygen` and drop the resulting `validatorKey.pem` into the workdir's `config/` before starting the node. See [operations](/mxnode/operations#keygen).

## Configuration knobs

The relevant per-node fields, all stamped into `prefs.toml` automatically:

| Field                                             | mxnode behaviour                                                                 |
| ------------------------------------------------- | -------------------------------------------------------------------------------- |
| `Preferences.NodeDisplayName`                     | Stamped from `[node].name_template`                                              |
| `Preferences.DestinationShardAsObserver`          | Stamped to `0/1/2/metachain` for `--squad` (and for `--role multikey`)           |
| `Preferences.RedundancyLevel`                     | Stamped for `--role multikey` to `--backup <N>` (default `0` without `--backup`) |
| `[DbLookupExtensions] Enabled` (in `config.toml`) | Set to `true` for observer/multikey roles                                        |

Operators who need other prefs can layer overrides via `[overrides.prefs]` in `config.toml` — see [configuration](/mxnode/configuration).