LogoLogo
  • WELCOME TO ALEPH ZERO
  • EXPLORE
    • About Aleph Zero
    • AlephBFT Consensus
    • The Economy of Aleph Zero
    • Where to Buy AZERO
    • Decentralized Governance on Aleph Zero
    • Ecosystem
    • Aleph Zero Foundation Treasury Management
    • Community
    • Glossary
    • Audit & Research Papers
  • USE
    • Wallets
    • Explorer
    • Ledger
    • Telegram Notifications
    • Aleph Zero Signer
      • General introduction
      • What does Signer do?
      • What are Sub-accounts and Sub-account paths?
      • Why is it critical to store your Secret Phrase in a safe place?
      • How to forget and restore accounts?
      • What are Networks?
      • What are Trusted apps?
    • Dashboard
      • Dashboard basics
      • Overview
    • Stake
      • Staking Basics
      • Staking Menu Overview
      • How to Start Staking with the Aleph Zero Dashboard
      • How to Start Staking With the Developer Wallet
      • How to start staking using Ledger hardware wallet
      • How to Change Nominations
      • How to Stop Staking
      • Staking Rewards
      • Validators
      • Commission and Foundation Nodes
      • Proxy Accounts
    • Validate
      • Validating Overview
      • Hardware requirements
      • Running an Aleph Node on Testnet
        • Downloading and running the node
        • Verifying your setup
        • Customizing your setup
        • Building and running from source [advanced]
          • Building from source
          • Set environment variables
          • Download DB snapshot
          • Running the binary
        • Appendix: Ports, addresses, validators, and archivists
      • Running an Aleph Node on Mainnet
        • Running the node
        • Building and running from source [advanced]
      • Setting your identity
      • Making the node validate
      • Securing your validator
      • Troubleshooting
      • Elections and Rewards Math
      • Testnet Validator Airdrop
      • Foundation Nomination Program
    • Using the EVM-layer
    • Governance
      • Token
      • Multisig Accounts
  • BUILD
    • Aleph Zero smart contracts basics
      • Setting up a Testnet account
      • Installing required tools
      • Creating your first contract
      • Deploying your contract to Aleph Zero Testnet
      • Extending your contract
    • Cross contract calls
      • Using references
      • Using dynamic calls
    • Migrating from Solidity
    • Writing e2e tests with ink-wrapper
    • Aleph Zero Signer integration
    • Front-end app: smart contract interaction
    • Security Course by Kudelski Security
      • ink! Developers Security Guideline
      • Lesson 1 - Getting started with ink!
      • Lesson 2 - Threat Assessment
      • Lesson 3 - Integer Overflow
      • Lesson 4 - Signed-integer
      • Lesson 5 - Role-Based Access Control
      • Lesson 6 - Address Validation
      • Lesson 7 - Smart Contract Control
    • Development on EVM-layer
  • PROTOCOL DETAILS
    • Shielder
      • Overview
      • Design against Bad Actors
      • Preliminaries - ZK-relations
      • Notes and Accounts
      • ZK-ID and Registrars
      • Anonymity Revokers
      • PoW Anonymity Revoking
      • Relayers
      • Deterministic Secret Management
      • SNARK-friendly Symmetric Encryption
      • SNARK-friendly Asymmetric Encryption
      • Cryptography
      • Token shortlist
      • User Wallet
      • Versioning
      • PoC
      • Version 0.1.0
      • Version 0.2.0
    • Common DEX
      • Common Whitepaper - Differences
      • Dutch Auctions
  • FAQ
  • Tutorials
    • Withdrawing coins from exchanges
      • How to withdraw your AZERO coins from KuCoin
      • How to withdraw your AZERO coins from MEXC Global
      • How to withdraw your AZERO coins from HTX
  • Setting up or restoring a wallet
    • How to set up or recover your AZERO account using Aleph Zero Signer
    • How to set up or recover your AZERO account using the official mainnet web wallet
    • How to set up or recover your AZERO account using Nova Wallet
    • How to set up or recover your AZERO account using SubWallet
    • How to set up or recover your AZERO account using Talisman
  • Staking
    • How to stake via a direct nomination using the Aleph Zero Dashboard
    • How to stake via a nomination pool using the Aleph Zero Dashboard
    • How to destroy a nomination pool via the Aleph Zero Dashboard
Powered by GitBook
On this page
  • Suspension
  • Session keys
  • Is my node running with session keys?
  • Generating your session keys
  • Accepting inbound connections
  • Accepting consensus connections

Was this helpful?

  1. USE
  2. Validate

Troubleshooting

This page contains a handful of tips for solving problems with your validator node

PreviousSecuring your validatorNextElections and Rewards Math

Last updated 2 months ago

Was this helpful?

Suspension

If your node was underperforming, chances are you were suspended. There are two kinds of suspension that might happen to a validator, if they fail to meet the criteria for each of the following metrics:

  • Block production performance - more than 48 sessions in which a validator produced less than 80% of expected blocks,

  • ABFT performance - more than 24 sessions in which a validator was behind more than 11 units.

Both counters get reset every 10 eras, ie 960 sessions.

First of all, to check whether you are suspended, simply go to the (note that the link is for testnet, for mainnet please use the analogous tab on azero.dev) and use the search bar to see if your validator is there:

If you confirm you have been suspended, there are two things you should do:

  1. Locate and fix the source of your issues. Please feel free to contact the Aleph Zero team on Discord if you have trouble locating the issue.

Session keys

Having session keys in your keystore is crucial to being a successful validator. Without them your node won't be producing blocks and won't be taking part in the consensus. Read on to find out how to check if you actually have your session keys and how to restore them in case something went wrong.

If you are using the Aleph Node Runner, this will be checked automatically on each subsequent run of the node if you provide your stash account address to the --stash_account flag. The scripts will also help you generate new keys and set them for your account. That said, you might still consider following the procedure below useful, as it can give you a deeper understanding of the matter.

Is my node running with session keys?

This particular query is submitted using the green 'plus' sign to the right of the text fields.

The output should look like this:

{
  aura: 0x32...
  aleph: 0x51...
}

Now, you need to construct your public session key by gluing the two together: the aura key needs to be pasted in its entirety (including 0x) and the aleph key without the leading 0x. Copy the result, as we'll need it in the next step.

Example For the following output: {

aura: 0x11111111111111111111111111111111 aleph: 0x22222222222222222222222222222222 } The resulting public session key would be: 0x1111111111111111111111111111111122222222222222222222222222222222

Using the wallet

You need to go to Developer -> RPC Calls and choose the author::hasSessionKeys call. In the sessionKeys argument, you need to paste the previously constructed public key, like below:

If the output you get is true, you are all set. If you get false, you'll need to follow the next section on generating your session keys again.

Using the command line

You will need to run this command on your machine:

curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "author_hasSessionKeys", "params":["<YOUR NODE'S SESSION KEY>"]}' localhost:9944

If you see true in this output, everything is okay. If you see false, you'll need to follow the next section.

Generating your session keys

  1. You need to go to Developer -> RPC Calls section, call (on your local node!) the author::rotateKeys method and copy the result.

  2. You need to go the Network -> Staking -> Accounts, click on the "Set session keys" button and, in the modal, paste the key you got from rotateKeys.

And, just like that, you have your session keys again!

Accepting inbound connections

In order to successfully take part in the consensus, your node not only needs to be able to make outgoing connections to its peers but also accept incoming ones. As this process is a little more cumbersome, we created a handy Python script to do it for you. You will need Python >= 3.2 and the requests package (pip3 install requests).

import requests
​
LOCAL_RPC = "http://localhost:9944"
​
​
def analyze_peers(status):
    out_peers = 0
    in_peers = 0
    unknown = 0
    connected = status['result']['connectedPeers']
​
    for peer_id, peer in connected.items():
        try:
            endpoint = peer['endpoint']
            if 'dialing' in endpoint:
                out_peers += 1
            elif 'listening' in endpoint:
                in_peers += 1
            else:
                unknown += 1
        except Exception as e:
            unknown += 1
    print(f"out_peers: {out_peers}, in_peers: {in_peers}, unknown: {unknown}")
​
​
def network_status(addr):
    r = requests.post(addr, json={"jsonrpc":"2.0","id":"1","method":"system_unstable_networkState","params":[]})
    res_json = r.json()
    return res_json
​

if __name__ == '__main__':​
    analyze_peers(network_status(LOCAL_RPC))

You can copy it, save it to a file like check_peers.py and run it:

python3 check_peers.py

If in the output you see in_peers with a value greater than zero, you're good to go.

Otherwise, you can try the following:

  1. Make sure your ports 30333 and 30343 are open: this means making sure that it's not blocked by any firewall and also making sure that it's not hidden behind NAT (which probably comes down to getting a public, fixed IP).

  2. If you are sure you've done the above and the in_peers value is still 0, don't panic!. It may be the inherent non-determinism of peer-to-peer communication. Waiting some time (say, 24h) for the network state to stabilize might help.

Accepting consensus connections

Another reason your node is not taking part in consensus might be it is not accepting connections from other Validator nodes on port 30343. This might be caused by providing a wrong IP/DNS argument to the Node.

Healthy node, not participating in the current session

When your node does not take part in the current session of consensus, this status log will report the following message:

Clique Network status: not maintaining any connections;

This is fine and you should not worry about that.

Healthy node, participating in the current session

In case your Node is going to be in a session, this status log will change and start reporting your connections with other Validators. Start of session can be found in logs by searching for the initialization message. Example of such log is:

AlephBFT-member: NodeIndex(0) Starting a new session.    

where NodeIndex is your node's index in the committee, so might be different from the one in example. This log indicates initialization of AlephBFT and after that your node will start taking part in consensus.

Now after you have found a time at which node is taking part in consensus, a typical healthy Clique Network status log for a session with four Validators looks like this:

Clique Network status: expecting 2 incoming connections; have - 2 [5FrU…6yKJGgEt, 5GYJ…NgNcvREH]; attempting 1 outgoing connections; have - 1 [5DMy…7Y4teh9M];

The total number of incoming and outgoing connections should correspond to the number of Validators in the current session minus one. For every peer Validator, the connection type is randomly set either to incoming or outgoing with a 50/50 chance. Therefore, the exact amount of expected incoming connections is unknown, and may vary between sessions.

In case your node is healthy but there are some minor problems with the network, your node can start reporting other logs like:

Clique Network status: expecting 2 incoming connections; have - 1 [5FrU…6yKJGgEt]; missing - 1 [5GYJ…NgNcvREH]; attempting 1 outgoing connections; have - 1 [5DMy…7Y4teh9M];

This reports other types of connections, or missing connections. As long as only single connections are missing, you probably don't have to worry about that, as this may be the result of peer's misconfiguration.

Misconfigured node

A common problem that might appear if you do not have 30343 port open is that no one can connect to you. Then the status log will look like the following:

Clique Network status: expecting 2 incoming connections; WARNING! No incoming peers even though we expected them, maybe connecting to us is impossible; missing - 2 [5FrU…6yKJGgEt, 5GYJ…NgNcvREH]; attempting 1 outgoing connections; have - 1 [5DMy…7Y4teh9M];     

This can mean two things:

  • your public validator address is not available for other Validators to connect to,

  • mapping of your ports is incorrect and your node does not listen on port 30343.

Please, note that even though your node might have established all outgoing connections, this still means that it is misconfigured!

On the other hand, having no connections at all probably indicates a general problem with your Internet connection.

In order for your node to function correctly this needs to be fixed. To do that you can make sure that your validator address and port (--ip/--dns argument or VALIDATOR_PUBLIC_ADDRESS environment variable) is not blocked by any firewall or that it is not hidden behind NAT (which probably comes down to getting a public, fixed IP). You can also verify that your node is setup to listen on a correct port (VALIDATOR_PORT environment variable, by default set to 30343).

After your suspension period is over, click the "Validate" button again, as described in the section.

First, you need to go to and run the session::nextKeys command to find your keys (for Aura and for AlephBFT, which are used for block production and finalization, respectively).

Now we need to issue an RPC call to check if there exists a private session key matching this public key. Important: this needs to be run against your local node. If you are using the wallet, you need to make sure you switch to your local node as described in the section about .

Remember to replace <YOUR NODE'S SESSION KEY> with the value you copied in the previous step

As this is described in the section, we will only provide a brief reminder:

This can be diagnosed by searching Node logs. You can access your logs just like described in the section . Good idea would be to investigate logs from the last 24 hours of your node running. When looking at logs, you are interested in searching for status logs of the Clique Network, which is used to connect the Validators directly with each other. Those are the logs that start with: Clique Network status.

🙂
https://test.azero.dev/#/chainstate
"Making your node validate"
generating your keys
Suspensions tab
Validate
Logs