High latency during performance testing for orderer #fabric #fabric-sdk-node #grpc #fabric-orderer


susheeldighade@...
 

Hi everyone,
 
We are having 2 org, 3 orderer setup on fabric v2.2 and we are running performance tests on this network. The client which interacts with peers is written in nodejs and fabric-network library is used for endorsement and committing requests. All the timeouts are default as provided by library.

We have noticed high latency(~ 6-8 seconds) for endorsement and commit stage. Since we are trying to find out bottlenecks and improve the performance, we have observed these debug logs in orderer container. As you can notice there are multiple instances where the broadcast stream breaks.
Also you can notice the streaming call takes time like 4 seconds which is huge.
 
 
2020-11-04 13:13:10.169 UTC [orderer.common.broadcast] Handle -> DEBU 47c3d Received EOF from 10.244.x.xxx:57604, hangup
2020-11-04 13:13:10.169 UTC [orderer.common.broadcast] Handle -> DEBU 47c3f Received EOF from 10.244.x.xxx:57604, hangup
2020-11-04 13:13:10.169 UTC [orderer.common.server] func1 -> DEBU 47c40 Closing Broadcast stream
2020-11-04 13:13:10.169 UTC [orderer.common.server] func1 -> DEBU 47c41 Closing Broadcast stream
2020-11-04 13:13:10.169 UTC [orderer.common.server] func1 -> DEBU 47c42 Closing Broadcast stream
2020-11-04 13:13:10.169 UTC [orderer.common.broadcast] Handle -> DEBU 47c44 Received EOF from 10.244.x.xxx:57604, hangup
2020-11-04 13:13:10.169 UTC [comm.grpc.server] 1 -> INFO 47c45 streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Broadcast grpc.peer_address=10.244.x.xxx:57604 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=1.198834113s
2020-11-04 13:13:10.168 UTC [comm.grpc.server] 1 -> INFO 47bfc streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Broadcast grpc.peer_address=10.244.x.xxx:57604 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=4.280608734s
 
 
Can anyone please throw some light on what could be the possible reason of connection breaking, or suggest any parameters which can be used for tuning, reducing latency?
 
For orderer, we have kept this config:
    block BatchTimeout = 1s
    block max message counts = 300
 
The send rate is high enough to fill up the block before BatchTimeout. If anyone need more logs, we can provide it.

Join fabric@lists.hyperledger.org to automatically receive all group messages.