connection is in TRANSIENT_FAILURE #fabric


I am maintaining 1000 concurrent connections (SubmitTransactions) which invokes the smart-contract.

The smart contract does the following operations:
1. "EvaluateTransaction" - on another chaincode
2. "EvaluateTransaction" - on the same chaincode as the "SubmitTransaction" is called
3. "PutState" - on the same chaincode as the "SubmitTransaction" is called

I am processing around 10,000 records while maintaining a concurrency of 1000 transactions (1000 SubmitTransactions which is 1000 connections). All these transactions are independent of each other, that is none of these transactions are reading or writing the same key.

It is working for few seconds, and after that one or more peer goes down! and sometimes all the peer goes down as well and the processing stops altogether

Error: dynamicdiscovery.(*ChannelService).evaluate -> WARN error from discovery request [grpcs://localhost:7051]: From target: grpcs://localhost:7051: could not connect to grpcs://localhost:7051: dialing connection on target [localhost:7051]: connection is in TRANSIENT_FAILURE

My fabric network v2.3 consists of:
- 2 orderer nodes
- 3 peer nodes (couchDB)

I reduced the concurrency up to 100, and i still get the same error.

Join to automatically receive all group messages.