Fabric Peer ledger Crashed when load Increase #fabric-sdk-node #fabric-chaincode #docker #fabric-kubernetes #network


inzamam ansari
 

Hi Team,

I have a kubernetes setup of HLF2.2 with Golang Chaincode and Nodejs SDK.
I am performing load test on the network, when i try with 250 Concurrent thread, it shows unexpectedly chaincode installation Error. (Error: Chaincode definiation exist but not found, make sure it's Installed properly)

Node SDK not able to find any endorsement peer for provided chaincode and throw Exception.

After inspecting I found that

  • 3 out 4 Peer's ledger were crashed.
  • Peer disconnects from channel.
  • It was not showing installed Chain-code. 
  • unable to Query or Invoke from Cli.

I tried restarting the cluster but no luck.

Then I had to re-Install new version of chaincode on Peer's, and it's working fine now.

I am bit worried about the robustness of HLF network and which is not giving me enough confidence to use it in production.

And also there is no mechanism to monitor container of Chaincode in kubernetes, as it does not show-up in the pods list. And even i am not sure how much of load is on Chain-code when performing load test.

I have allocated 4GB RAM and 2-Core CPU to each Peer. is that enough for production level setup.
How can we ensure that Chain-code's container gets enough resource allocation. Should I increase my peers RAM further.

 

I have already raise the same issue on jira on below link

https://jira.hyperledger.org/browse/FABCG-15

Join fabric@lists.hyperledger.org to automatically receive all group messages.