Recovery of a Peer Node in Hyperledger Fabric

Overview

This article is to show the recovery process of a peer node. For simulation, we will first bring down a running peer node, and resume it to a fully functioning node. Through this process, we can understand more how various parts in Hyperledger Fabric are working, including channel, ledger, chaincode etc.

Test Setup

We only use a fabric node, with proper fabric container images, fabric binary tools and fabric samples.

We are using the last Lab provided where we have added new peers in Test-Network. It is done by running the setup script.

Demonstration

Step 1: Bring up Test Network

We are using the last Lab provided where we have added new peers in Test-Network. It is done by running the setup script.

We use script network.sh to bring up the Test Network and mychannel.

cd fabric-samples/fabcar
./startFabric.sh

A total of six containers are running:

• 3 CAs, one for each organization

• 1 orderer

• 3 peers, two for Org1 and one for Org2

Upon completion, we have a fabric network (Test Network) with Fabcar chaincode deployed. Here is a summary of this setup.

• One orderer

• Two organizations (Org1 and Org2)

• Each organization comes with a Certificate Authority and two Peer Nodes (designated as peer0 and peer1). Therefore total of three peers nodes are running.

• Each peer node comes with a couchdb as the world state portion of the ledger.

• A channel mychannel is created and joint by all peers.

• Fabcar chaincode is installed in all three peer nodes.

• Fabcar chaincode is instantiated in mychannel. The endorsing poilcy requires endorsement from one member from each organization.

• The chaincode function initLedger() is invoked, and there are 10 sets of car record in the ledger

PreCheck:

To verify things are working well, we will do the checking on ALL peer nodes. We will get the same result.

Let's check from Org1 Peer0:

Now, let's source the script.

source ./scripts/setOrgPeerContext.sh 1
peer channel getinfo -c mychannel
peer chaincode query -C mychannel -n fabcar -c '{"Args":["queryCar","CAR0"]}'

Let's check from Org1 Peer1:

Before running the below script, let's update envVar.sh script for newly added Peer.

Now, let's source the script.

source ./scripts/setOrgPeerContext.sh 3
peer channel getinfo -c mychannel
peer chaincode query -C mychannel -n fabcar -c '{"Args":["queryCar","CAR0"]}'

Let's check from Org2 Peer0:

Now, let's source the script.

source ./scripts/setOrgPeerContext.sh 2
peer channel getinfo -c mychannel
peer chaincode query -C mychannel -n fabcar -c '{"Args":["queryCar","CAR0"]}'

And we can take a look at the world state in each node. The CouchDB can be observed through a browser (the port is 5984, 6984, 7984 corresponding to peer0.org1, peer1.org1, peer0.org2 respectively)

Shutdown a Peer Node

To shutdown peer0.org2.example.com completely, we will use the following docker commands.

docker kill peer0.org2.example.com
docker rm peer0.org2.example.com
docker volume rm docker_peer0.org2.example.com 
docker kill couchdb1
docker rm couchdb1

We can check docker ps and see those containers are gone.

The network is still functioning, in a sense that chaincode invoke and query can still be done across other peer nodes. We can perform query on the running peer nodes and we still get back the data stored in the ledger.

Recover this Peer Node into Full Service

To recover this node back to a functioning peer, here are three steps.

1. Bring up the peer node (container) and the world state database (couchdb)

2. Join it back to the channel

3. Install chaincode

Let’s observe what happens in each step.

1. Bring Up the Peer Node and CouchDB

In this setup, the bringing up node and the CouchDB through docker compose files.

export FABRIC_CFG_PATH=${PWD}/configtx
export CHANNEL_NAME=mychannel
export PATH=${PWD}/../bin:${PWD}:$PATH

IMAGE_TAG=latest docker-compose -f docker/docker-compose-test-net.yaml -f docker/docker-compose-couch.yaml up -d

docker ps

After we see them back as running containers, we can reach peer0.org2.example.com through the terminal. Let’s first see if the channel is already there.

source ./scripts/setOrgPeerContext.sh 2
peer channel list

We see peer0.org2.example.com is not part of mychannel yet. The next move is to join it back to mychannel. We can also see that the CouchDB is empty on this peer.

2. Join the Peer Node to mychannel

To make the peer node hold the same ledger as other peer nodes in the same channel, we do not need to copy the whole blockchain and world state. Instead, we can simply join the peer node to the channel and the ledger will be “sync-ed” to the node.

All we need is just the channel genesis block for that channel. In our environment, this file mychannel.block is already in the directory. In another case, you may get it directly from existing nodes.

This is the command to join the channel with the channel genesis block.

peer channel join -b ./channel-artifacts/mychannel.block
peer channel list

We see now peer0.org2.example.com now joined mychannel.

If we take a look on its blockchain information, we see the full blockchain is already there. That means the remaining blocks are obtained from other peers.

peer channel getinfo -c mychannel

We can compare the result with other peer node: for example, it is taken from peer1.org1.example.com.

source ./scripts/setOrgPeerContext.sh 3

peer channel getinfo -c mychannel

If we quickly check the world state, we see the state is also updated.

3. Install Chaincode

If we check the chaincode information on this node, we can see the chaincode is not installed yet, though the chaincode is already instantiated in mychannel, in which this peer is part of it.

source ./scripts/setOrgPeerContext.sh 2 

peer chaincode list --installed

peer chaincode list --instantiated -C mychannel

Now we install the chaincode back to peer0.org2.example.com.

peer lifecycle chaincode install fabcar.tar.gz

And we can query the chaincode from this node now.

peer chaincode query -C $CHANNEL_NAME -n fabcar -c '{"Args":["queryAllCars"]}'

Back to Normal

At this moment, we have completely brought up the peer0.org2.example.com node back to normal, serving as a full-function peer node in our network. We can try it again by invoking a chaincode function and pointing to peer0.org2.example.com as an endorsing peer.

peer chaincode invoke -o localhost:7050 --ordererTLSHostnameOverride orderer.example.com --tls --cafile $ORDERER_CA -C $CHANNEL_NAME -n fabcar $PEER_CONN_PARAMS -c '{"function":"changeCarOwner","Args":["CAR1","John"]}'

Clean Up

As good practice, always clean up everything after the demonstration.

./network.sh down