Channel snapshot failed on conclusion


Carlos Eduardo Matos Ellery
 

Hello all,

After almost 10 hours of waiting for a snapshot completion, there is this error on the peer log and nothing in 'completed' neither in 'temp' dirs of 'snapshots' folder. During the time it was taking the snapshot, the listpending always returned "Successfully got pending requests: [247663]". Does anyone have an idea of what occurred?

*Additional information: I know that CouchDB was running an automatic compaction task on the DB... could this be the cause? Because of the size of the stateDB, my CouchDB is configured for 16 shards (q=16).

[kvledger] func2 -> ERRO b8efb Failed to generate snapshot channelID=mychannel lastCommittedBlockNumber=247663 error="failed to retrieve the next entry from scanner associated with namespace consortium: error reading response body: unexpected EOF" errorVerbose="unexpected EOF error reading response body github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*couchDatabase).readDocRange /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/couchdb.go:904 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.rangeScanFilterCouchInternalDocs /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:611 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).getNextStateRangeScanResults /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:588 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1084 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*dbsScanner).Next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1191 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate.(*DB).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate/snapshot.go:44 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*LockBasedTxMgr).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr/lockbased_txmgr.go:603 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).generateSnapshot /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot.go:126 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).processSnapshotMgmtEvents.func2 /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot_mgmt.go:199 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1374 failed to retrieve the next entry from scanner associated with namespace mynetwork" Thanks,
-- 
Carlos Eduardo Matos Ellery


Manish
 



---------- Forwarded message ---------
From: Manish Sethi <manish.sethi@...>
Date: Wed, Sep 8, 2021 at 4:27 PM
Subject: Re: [Hyperledger Fabric] Channel snapshot failed on conclusion
To: Carlos Eduardo Matos Ellery <carlos.ellery@...>


Any error during reading from statedb (couchdb in your case) would abort the snapshot operation. You don't see anything in the 'snapshots' folder because only after successful completion of the snapshot, the snapshot is moved into this folder. Also, the partial snapshot data is discarded from the 'temp' folder.

As you mentioned that your couchdb was doing background compaction, that may likely have caused this error. Ensure that you have given enough resources to couchdb.

And yes, during the snapshot generation, the listpending is expected to keep showing the pending request!

Thanks,
Manish


On Wed, Sep 8, 2021 at 3:44 PM Carlos Eduardo Matos Ellery <carlos.ellery@...> wrote:
Hello all,

After almost 10 hours of waiting for a snapshot completion, there is this error on the peer log and nothing in 'completed' neither in 'temp' dirs of 'snapshots' folder. During the time it was taking the snapshot, the listpending always returned "Successfully got pending requests: [247663]". Does anyone have an idea of what occurred?

*Additional information: I know that CouchDB was running an automatic compaction task on the DB... could this be the cause? Because of the size of the stateDB, my CouchDB is configured for 16 shards (q=16).

[kvledger] func2 -> ERRO b8efb Failed to generate snapshot channelID=mychannel lastCommittedBlockNumber=247663 error="failed to retrieve the next entry from scanner associated with namespace consortium: error reading response body: unexpected EOF" errorVerbose="unexpected EOF error reading response body github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*couchDatabase).readDocRange /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/couchdb.go:904 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.rangeScanFilterCouchInternalDocs /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:611 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).getNextStateRangeScanResults /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:588 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1084 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*dbsScanner).Next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1191 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate.(*DB).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate/snapshot.go:44 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*LockBasedTxMgr).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr/lockbased_txmgr.go:603 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).generateSnapshot /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot.go:126 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).processSnapshotMgmtEvents.func2 /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot_mgmt.go:199 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1374 failed to retrieve the next entry from scanner associated with namespace mynetwork" Thanks,
-- 
Carlos Eduardo Matos Ellery


Carlos Eduardo Matos Ellery
 

Thanks, Manish,

The virtual server has 4 vCPUs, 8 GB of RAM and plenty of disk space (at 30% usage). It is running just 3 containers: peer, couchdb and cadvisor. Would that be undersized for a committing peer (there is no chaincode running)? Or do you mean "enough resources" in terms of IOPS?

Anyway, I started another snapshot process and this time it finished well (it took 16 hours). However, we noted that always some transactions occur on the channel, CouchDB starts the compaction process. On big statedbs the compaction can take a long time to complete. So, in the end, it looks like the CouchDB will always be doing compaction work if the channel is somewhat active. Is this behaviour expected? Is there any tunning we can make on CouchDB to avoid compaction always running in this case?

Thanks again,
Carlos Eduardo Matos Ellery
Em 09/09/2021 09:34, Manish escreveu:



---------- Forwarded message ---------
From: Manish Sethi <manish.sethi@...>
Date: Wed, Sep 8, 2021 at 4:27 PM
Subject: Re: [Hyperledger Fabric] Channel snapshot failed on conclusion
To: Carlos Eduardo Matos Ellery <carlos.ellery@...>


Any error during reading from statedb (couchdb in your case) would abort the snapshot operation. You don't see anything in the 'snapshots' folder because only after successful completion of the snapshot, the snapshot is moved into this folder. Also, the partial snapshot data is discarded from the 'temp' folder.

As you mentioned that your couchdb was doing background compaction, that may likely have caused this error. Ensure that you have given enough resources to couchdb.

And yes, during the snapshot generation, the listpending is expected to keep showing the pending request!

Thanks,
Manish

On Wed, Sep 8, 2021 at 3:44 PM Carlos Eduardo Matos Ellery <carlos.ellery@...> wrote:
Hello all,

After almost 10 hours of waiting for a snapshot completion, there is this error on the peer log and nothing in 'completed' neither in 'temp' dirs of 'snapshots' folder. During the time it was taking the snapshot, the listpending always returned "Successfully got pending requests: [247663]". Does anyone have an idea of what occurred?

*Additional information: I know that CouchDB was running an automatic compaction task on the DB... could this be the cause? Because of the size of the stateDB, my CouchDB is configured for 16 shards (q=16).

[kvledger] func2 -> ERRO b8efb Failed to generate snapshot channelID=mychannel lastCommittedBlockNumber=247663 error="failed to retrieve the next entry from scanner associated with namespace consortium: error reading response body: unexpected EOF" errorVerbose="unexpected EOF error reading response body github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*couchDatabase).readDocRange /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/couchdb.go:904 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.rangeScanFilterCouchInternalDocs /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:611 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).getNextStateRangeScanResults /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:588 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1084 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*dbsScanner).Next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1191 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate.(*DB).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate/snapshot.go:44 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*LockBasedTxMgr).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr/lockbased_txmgr.go:603 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).generateSnapshot /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot.go:126 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).processSnapshotMgmtEvents.func2 /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot_mgmt.go:199 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1374 failed to retrieve the next entry from scanner associated with namespace mynetwork" Thanks,
-- 
Carlos Eduardo Matos Ellery


Manish
 

Hi Carlos,

You will have to experiment a bit with couchdb compaction configurations in your set up and evaluate what works best for your workload. 

As far as I recall, the default config is automatic compaction on but you can experiment with other configs e.g., restricting to some time window or triggering it manually etc.

Thanks,
Manish

On Thu, Sep 9, 2021 at 12:09 PM Carlos Eduardo Matos Ellery <carlos.ellery@...> wrote:
Thanks, Manish,

The virtual server has 4 vCPUs, 8 GB of RAM and plenty of disk space (at 30% usage). It is running just 3 containers: peer, couchdb and cadvisor. Would that be undersized for a committing peer (there is no chaincode running)? Or do you mean "enough resources" in terms of IOPS?

Anyway, I started another snapshot process and this time it finished well (it took 16 hours). However, we noted that always some transactions occur on the channel, CouchDB starts the compaction process. On big statedbs the compaction can take a long time to complete. So, in the end, it looks like the CouchDB will always be doing compaction work if the channel is somewhat active. Is this behaviour expected? Is there any tunning we can make on CouchDB to avoid compaction always running in this case?

Thanks again,
Carlos Eduardo Matos Ellery
Em 09/09/2021 09:34, Manish escreveu:


---------- Forwarded message ---------
From: Manish Sethi <manish.sethi@...>
Date: Wed, Sep 8, 2021 at 4:27 PM
Subject: Re: [Hyperledger Fabric] Channel snapshot failed on conclusion
To: Carlos Eduardo Matos Ellery <carlos.ellery@...>


Any error during reading from statedb (couchdb in your case) would abort the snapshot operation. You don't see anything in the 'snapshots' folder because only after successful completion of the snapshot, the snapshot is moved into this folder. Also, the partial snapshot data is discarded from the 'temp' folder.

As you mentioned that your couchdb was doing background compaction, that may likely have caused this error. Ensure that you have given enough resources to couchdb.

And yes, during the snapshot generation, the listpending is expected to keep showing the pending request!

Thanks,
Manish

On Wed, Sep 8, 2021 at 3:44 PM Carlos Eduardo Matos Ellery <carlos.ellery@...> wrote:
Hello all,

After almost 10 hours of waiting for a snapshot completion, there is this error on the peer log and nothing in 'completed' neither in 'temp' dirs of 'snapshots' folder. During the time it was taking the snapshot, the listpending always returned "Successfully got pending requests: [247663]". Does anyone have an idea of what occurred?

*Additional information: I know that CouchDB was running an automatic compaction task on the DB... could this be the cause? Because of the size of the stateDB, my CouchDB is configured for 16 shards (q=16).

[kvledger] func2 -> ERRO b8efb Failed to generate snapshot channelID=mychannel lastCommittedBlockNumber=247663 error="failed to retrieve the next entry from scanner associated with namespace consortium: error reading response body: unexpected EOF" errorVerbose="unexpected EOF error reading response body github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*couchDatabase).readDocRange /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/couchdb.go:904 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.rangeScanFilterCouchInternalDocs /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:611 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).getNextStateRangeScanResults /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:588 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1084 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*dbsScanner).Next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1191 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate.(*DB).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate/snapshot.go:44 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*LockBasedTxMgr).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr/lockbased_txmgr.go:603 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).generateSnapshot /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot.go:126 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).processSnapshotMgmtEvents.func2 /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot_mgmt.go:199 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1374 failed to retrieve the next entry from scanner associated with namespace mynetwork" Thanks,
-- 
Carlos Eduardo Matos Ellery


David F. D. Reis
 

Hi Carlos.

You may try to do it putting your network in STATE_MAINTENANCE mode.

Please, let me know when you correct this problem. What was your workaround.


Best regards.

David

David Faulstich Diniz Reis
Certified Hyperledger Fabric Administrator
Certificate ID: LF-wfszprnvxq

Em qua., 8 de set. de 2021 às 16:44, Carlos Eduardo Matos Ellery <carlos.ellery@...> escreveu:

Hello all,

After almost 10 hours of waiting for a snapshot completion, there is this error on the peer log and nothing in 'completed' neither in 'temp' dirs of 'snapshots' folder. During the time it was taking the snapshot, the listpending always returned "Successfully got pending requests: [247663]". Does anyone have an idea of what occurred?

*Additional information: I know that CouchDB was running an automatic compaction task on the DB... could this be the cause? Because of the size of the stateDB, my CouchDB is configured for 16 shards (q=16).

[kvledger] func2 -> ERRO b8efb Failed to generate snapshot channelID=mychannel lastCommittedBlockNumber=247663 error="failed to retrieve the next entry from scanner associated with namespace consortium: error reading response body: unexpected EOF" errorVerbose="unexpected EOF error reading response body github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*couchDatabase).readDocRange /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/couchdb.go:904 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.rangeScanFilterCouchInternalDocs /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:611 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).getNextStateRangeScanResults /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:588 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1084 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*dbsScanner).Next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1191 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate.(*DB).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate/snapshot.go:44 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*LockBasedTxMgr).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr/lockbased_txmgr.go:603 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).generateSnapshot /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot.go:126 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).processSnapshotMgmtEvents.func2 /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot_mgmt.go:199 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1374 failed to retrieve the next entry from scanner associated with namespace mynetwork" Thanks,
-- 
Carlos Eduardo Matos Ellery


Carlos Eduardo Matos Ellery
 

Hi David,

Didn't know about this STATE_MAINTENANCE mode, thanks, I will keep this trick up my sleeve. By now, at weekends the network doesn't have any transactions, so we will use this period to take snapshots when needed. In parallell, we will try some configs on CouchDB to keep compaction under control, if possible.

Thanks,
Carlos Eduardo Matos Ellery
Em 10/09/2021 16:54, David Faulstich Diniz Reis escreveu:

Hi Carlos.

You may try to do it putting your network in STATE_MAINTENANCE mode.

Please, let me know when you correct this problem. What was your workaround.


Best regards.

David

David Faulstich Diniz Reis
Certified Hyperledger Fabric Administrator
Certificate ID: LF-wfszprnvxq

Em qua., 8 de set. de 2021 às 16:44, Carlos Eduardo Matos Ellery <carlos.ellery@...> escreveu:
Hello all,

After almost 10 hours of waiting for a snapshot completion, there is this error on the peer log and nothing in 'completed' neither in 'temp' dirs of 'snapshots' folder. During the time it was taking the snapshot, the listpending always returned "Successfully got pending requests: [247663]". Does anyone have an idea of what occurred?

*Additional information: I know that CouchDB was running an automatic compaction task on the DB... could this be the cause? Because of the size of the stateDB, my CouchDB is configured for 16 shards (q=16).

[kvledger] func2 -> ERRO b8efb Failed to generate snapshot channelID=mychannel lastCommittedBlockNumber=247663 error="failed to retrieve the next entry from scanner associated with namespace consortium: error reading response body: unexpected EOF" errorVerbose="unexpected EOF error reading response body github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*couchDatabase).readDocRange /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/couchdb.go:904 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.rangeScanFilterCouchInternalDocs /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:611 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).getNextStateRangeScanResults /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:588 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1084 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*dbsScanner).Next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1191 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate.(*DB).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate/snapshot.go:44 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*LockBasedTxMgr).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr/lockbased_txmgr.go:603 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).generateSnapshot /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot.go:126 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).processSnapshotMgmtEvents.func2 /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot_mgmt.go:199 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1374 failed to retrieve the next entry from scanner associated with namespace mynetwork" Thanks,
-- 
Carlos Eduardo Matos Ellery


David F. D. Reis
 


Good luck.

Carlos, I notice that you work @SERPRO GOV BR. I am from Brazil. Brasília.

Let's exchange some experiences regarding hlf.

Best regards.

David

Em sex., 10 de set. de 2021 às 17:40, Carlos Eduardo Matos Ellery <carlos.ellery@...> escreveu:

Hi David,

Didn't know about this STATE_MAINTENANCE mode, thanks, I will keep this trick up my sleeve. By now, at weekends the network doesn't have any transactions, so we will use this period to take snapshots when needed. In parallell, we will try some configs on CouchDB to keep compaction under control, if possible.

Thanks,
Carlos Eduardo Matos Ellery
Em 10/09/2021 16:54, David Faulstich Diniz Reis escreveu:
Hi Carlos.

You may try to do it putting your network in STATE_MAINTENANCE mode.

Please, let me know when you correct this problem. What was your workaround.


Best regards.

David

David Faulstich Diniz Reis
Certified Hyperledger Fabric Administrator
Certificate ID: LF-wfszprnvxq

Em qua., 8 de set. de 2021 às 16:44, Carlos Eduardo Matos Ellery <carlos.ellery@...> escreveu:
Hello all,

After almost 10 hours of waiting for a snapshot completion, there is this error on the peer log and nothing in 'completed' neither in 'temp' dirs of 'snapshots' folder. During the time it was taking the snapshot, the listpending always returned "Successfully got pending requests: [247663]". Does anyone have an idea of what occurred?

*Additional information: I know that CouchDB was running an automatic compaction task on the DB... could this be the cause? Because of the size of the stateDB, my CouchDB is configured for 16 shards (q=16).

[kvledger] func2 -> ERRO b8efb Failed to generate snapshot channelID=mychannel lastCommittedBlockNumber=247663 error="failed to retrieve the next entry from scanner associated with namespace consortium: error reading response body: unexpected EOF" errorVerbose="unexpected EOF error reading response body github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*couchDatabase).readDocRange /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/couchdb.go:904 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.rangeScanFilterCouchInternalDocs /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:611 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).getNextStateRangeScanResults /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:588 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*queryScanner).next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1084 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb.(*dbsScanner).Next /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go:1191 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate.(*DB).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/privacyenabledstate/snapshot.go:44 github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*LockBasedTxMgr).ExportPubStateAndPvtStateHashes /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr/lockbased_txmgr.go:603 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).generateSnapshot /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot.go:126 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).processSnapshotMgmtEvents.func2 /go/src/github.com/hyperledger/fabric/core/ledger/kvledger/snapshot_mgmt.go:199 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1374 failed to retrieve the next entry from scanner associated with namespace mynetwork" Thanks,
-- 
Carlos Eduardo Matos Ellery