Topics

RAFT Orderer Issue #fabric #fabric-orderer


soumya nayak
 

Hi All,

Fabric - v1.4.3
I am running a cluster of three orderers out of which two are running without any issues. When the third orderer was started i was getting the below *panic *error --

Even i deleted the Ordererledger , wal folders and rstarted the orderer still the below issue - . PFA the logs.

[35m2019-12-10 13:06:22.537 UTC [orderer.consensus.etcdraft] commitTo -> PANI 020 tocommit(8) is out of range [lastIndex(3)]. Was the raft log corrupted, truncated, or lost? channel=ordererchannel node=3 panic: tocommit(8) is out of range [lastIndex(3)]. Was the raft log corrupted, truncated, or lost? goroutine 38 [running]: github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0001773f0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:229 +0x515 github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc00000e0f8, 0x4, 0x105c6a2, 0x5d, 0xc0003a9800, 0x2, 0x2, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0xf6 github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).Panicf(0xc00000e0f8, 0x105c6a2, 0x5d, 0xc0003a9800, 0x2, 0x2) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159 +0x79 github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(0xc00000e100, 0x105c6a2, 0x5d, 0xc0003a9800, 0x2, 0x2) /opt/gopath/src/github.com/hyperledger/fabric/common/flogging/zap.go:74 +0x60 github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.(*raftLog).commitTo(0xc00014dab0, 0x8) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/log.go:203 +0x14d github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.(*raft).handleHeartbeat(0xc0002ac140, 0x8, 0x3, 0x2, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/raft.go:1324 +0x54 github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.stepFollower(0xc0002ac140, 0x8, 0x3, 0x2, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/raft.go:1269 +0x450 github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.(*raft).Step(0xc0002ac140, 0x8, 0x3, 0x2, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/raft.go:971 +0x12db github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.(*node).run(0xc0002200c0, 0xc0002ac140) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/node.go:357 +0x1101 created by github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.RestartNode /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/node.go:246 +0x31b


Regards,
Soumya


Jason Yellick
 

The error message on this one can be taken at face value:

> PANI 020[0m tocommit(8) is out of range [lastIndex(3)]. Was the raft log corrupted, truncated, or lost?

And, as your post indicates, you have deleted the WAL.  This orderer cannot safely resume consenting, because you have deleted data required for its safety guarantees to hold true.

You should, perform a configuration update which shrinks the consenter set to your 2 working nodes (in each and every channel), then perform a second configuration update, which adds a new node.  This new node should be bootstrapped with the latest config block of the orderer system channel.  The new node will appropriately (and safely) initialize a new WAL, and replicate all of the chains.  Your old corrupt node can be safely deleted at any point.

Although it is probably painfully clear at this point, you should _never_ delete the WAL data for a node or you will find yourself in this situation.

Thanks,
~Jason
 

----- Original message -----
From: "soumya nayak" <soumyarjnnayak@...>
Sent by: fabric@...
To: fabric@...
Cc:
Subject: [EXTERNAL] [Hyperledger Fabric] RAFT Orderer Issue #fabric #fabric-orderer
Date: Wed, Dec 11, 2019 9:56 PM
 
Hi All,

Fabric - v1.4.3
I am running a cluster of three orderers out of which two are running without any issues. When the third orderer was started i was getting the below *panic *error --

Even i deleted the Ordererledger , wal folders and rstarted the orderer still the below issue - . PFA the logs.

[35m2019-12-10 13:06:22.537 UTC [orderer.consensus.etcdraft] commitTo -> PANI 020[0m tocommit(8) is out of range [lastIndex(3)]. Was the raft log corrupted, truncated, or lost? channel=ordererchannel node=3 panic: tocommit(8) is out of range [lastIndex(3)]. Was the raft log corrupted, truncated, or lost? goroutine 38 [running]: github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0001773f0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:229 +0x515 github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc00000e0f8, 0x4, 0x105c6a2, 0x5d, 0xc0003a9800, 0x2, 0x2, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0xf6 github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).Panicf(0xc00000e0f8, 0x105c6a2, 0x5d, 0xc0003a9800, 0x2, 0x2) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159 +0x79 github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(0xc00000e100, 0x105c6a2, 0x5d, 0xc0003a9800, 0x2, 0x2) /opt/gopath/src/github.com/hyperledger/fabric/common/flogging/zap.go:74 +0x60 github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.(*raftLog).commitTo(0xc00014dab0, 0x8) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/log.go:203 +0x14d github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.(*raft).handleHeartbeat(0xc0002ac140, 0x8, 0x3, 0x2, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/raft.go:1324 +0x54 github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.stepFollower(0xc0002ac140, 0x8, 0x3, 0x2, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/raft.go:1269 +0x450 github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.(*raft).Step(0xc0002ac140, 0x8, 0x3, 0x2, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/raft.go:971 +0x12db github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.(*node).run(0xc0002200c0, 0xc0002ac140) /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/node.go:357 +0x1101 created by github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft.RestartNode /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.etcd.io/etcd/raft/node.go:246 +0x31b


Regards,
Soumya