Peer Failed connecting to Orderer: Channel created but Peer is not connection to Orderer and unable to retrieve blocks from ordering service. #fabric-sdk-node #docker #raft #consensus #cal-notice


Akshay Soni
 

We are not using CLI. We are creating channel using Node SDK and creating infrastructure using shell script.
We are using Raft consensus with 5 orderer and Raft cluster is working fine.
Orderers elected the Leader and able to communicate with each other.
When we created new channel and peer joined, Peer created the Ledger with genesis block, joined gossip network of channel, created state database  and started delivering service in the channel.
At the time of Peer tried to retrieve blocks from ordering service, It gave following error 

2019-12-03 09:03:39.478 UTC [deliveryClient] StartDeliverForChannel -> INFO a88 This peer will retrieve blocks from ordering service and disseminate to other peers in the organization for channel swapchannel
2019-12-03 09:03:42.479 UTC [ConnProducer] NewConnection -> ERRO a89 Failed connecting to {localhost:51015 []} , error: context deadline exceeded
2019-12-03 09:03:45.482 UTC [ConnProducer] NewConnection -> ERRO a8a Failed connecting to {localhost:51014 []} , error: context deadline exceeded
2019-12-03 09:03:48.483 UTC [ConnProducer] NewConnection -> ERRO a8b Failed connecting to {oodjaeuen108-orderer4.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:03:51.485 UTC [ConnProducer] NewConnection -> ERRO a8c Failed connecting to {oodjaeuen108-orderer1.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:03:54.487 UTC [ConnProducer] NewConnection -> ERRO a8d Failed connecting to {localhost:51012 []} , error: context deadline exceeded
2019-12-03 09:03:57.488 UTC [ConnProducer] NewConnection -> ERRO a8e Failed connecting to {oodjaeuen108-orderer2.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:04:00.490 UTC [ConnProducer] NewConnection -> ERRO a8f Failed connecting to {oodjaeuen108-orderer3.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:04:03.492 UTC [ConnProducer] NewConnection -> ERRO a90 Failed connecting to {oodjaeuen108-orderer0.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:04:06.493 UTC [ConnProducer] NewConnection -> ERRO a91 Failed connecting to {localhost:51011 []} , error: context deadline exceeded
2019-12-03 09:04:09.495 UTC [ConnProducer] NewConnection -> ERRO a92 Failed connecting to {localhost:51013 []} , error: context deadline exceeded
2019-12-03 09:04:09.495 UTC [ConnProducer] NewConnection -> ERRO a93 Could not connect to any of the endpoints: [{localhost:51015 []} {localhost:51014 []} {oodjaeuen108-orderer4.orderer.com:7050 []} {oodjaeuen108-orderer1.orderer.com:7050 []} {localhost:51012 []} {oodjaeuen108-orderer2.orderer.com:7050 []} {oodjaeuen108-orderer3.orderer.com:7050 []} {oodjaeuen108-orderer0.orderer.com:7050 []} {localhost:51011 []} {localhost:51013 []}]
2019-12-03 09:04:09.495 UTC [deliveryClient] connect -> ERRO a94 Failed obtaining connection: could not connect to any of the endpoints: [{localhost:51015 []} {localhost:51014 []} {oodjaeuen108-orderer4.orderer.com:7050 []} {oodjaeuen108-orderer1.orderer.com:7050 []} {localhost:51012 []} {oodjaeuen108-orderer2.orderer.com:7050 []} {oodjaeuen108-orderer3.orderer.com:7050 []} {oodjaeuen108-orderer0.orderer.com:7050 []} {localhost:51011 []} {localhost:51013 []}]
2019-12-03 09:04:09.495 UTC [deliveryClient] try -> WARN a95 Got error: could not connect to any of the endpoints: [{localhost:51015 []} {localhost:51014 []} {oodjaeuen108-orderer4.orderer.com:7050 []} {oodjaeuen108-orderer1.orderer.com:7050 []} {localhost:51012 []} {oodjaeuen108-orderer2.orderer.com:7050 []} {oodjaeuen108-orderer3.orderer.com:7050 []} {oodjaeuen108-orderer0.orderer.com:7050 []} {localhost:51011 []} {localhost:51013 []}] , at 1 attempt. Retrying in 1s
2019-12-03 09:04:13.497 UTC [ConnProducer] NewConnection -> ERRO a96 Failed connecting to {localhost:51015 []} , error: context deadline exceeded
2019-12-03 09:04:16.498 UTC [ConnProducer] NewConnection -> ERRO a97 Failed connecting to {localhost:51014 []} , error: context deadline exceeded
2019-12-03 09:04:19.500 UTC [ConnProducer] NewConnection -> ERRO a98 Failed connecting to {oodjaeuen108-orderer4.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:04:22.501 UTC [ConnProducer] NewConnection -> ERRO a99 Failed connecting to {oodjaeuen108-orderer1.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:04:25.503 UTC [ConnProducer] NewConnection -> ERRO a9a Failed connecting to {localhost:51012 []} , error: context deadline exceeded
2019-12-03 09:04:28.505 UTC [ConnProducer] NewConnection -> ERRO a9b Failed connecting to {oodjaeuen108-orderer2.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:04:31.507 UTC [ConnProducer] NewConnection -> ERRO a9c Failed connecting to {oodjaeuen108-orderer3.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:04:34.508 UTC [ConnProducer] NewConnection -> ERRO a9d Failed connecting to {oodjaeuen108-orderer0.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:04:37.509 UTC [ConnProducer] NewConnection -> ERRO a9e Failed connecting to {localhost:51011 []} , error: context deadline exceeded
2019-12-03 09:04:40.511 UTC [ConnProducer] NewConnection -> ERRO a9f Failed connecting to {localhost:51013 []} , error: context deadline exceeded
2019-12-03 09:04:40.511 UTC [ConnProducer] NewConnection -> ERRO aa0 Could not connect to any of the endpoints: [{localhost:51015 []} {localhost:51014 []} {oodjaeuen108-orderer4.orderer.com:7050 []} {oodjaeuen108-orderer1.orderer.com:7050 []} {localhost:51012 []} {oodjaeuen108-orderer2.orderer.com:7050 []} {oodjaeuen108-orderer3.orderer.com:7050 []} {oodjaeuen108-orderer0.orderer.com:7050 []} {localhost:51011 []} {localhost:51013 []}]
2019-12-03 09:04:40.511 UTC [deliveryClient] connect -> ERRO aa1 Failed obtaining connection: could not connect to any of the endpoints: [{localhost:51015 []} {localhost:51014 []} {oodjaeuen108-orderer4.orderer.com:7050 []} {oodjaeuen108-orderer1.orderer.com:7050 []} {localhost:51012 []} {oodjaeuen108-orderer2.orderer.com:7050 []} {oodjaeuen108-orderer3.orderer.com:7050 []} {oodjaeuen108-orderer0.orderer.com:7050 []} {localhost:51011 []} {localhost:51013 []}]
2019-12-03 09:04:40.511 UTC [deliveryClient] try -> WARN aa2 Got error: could not connect to any of the endpoints: [{localhost:51015 []} {localhost:51014 []} {oodjaeuen108-orderer4.orderer.com:7050 []} {oodjaeuen108-orderer1.orderer.com:7050 []} {localhost:51012 []} {oodjaeuen108-orderer2.orderer.com:7050 []} {oodjaeuen108-orderer3.orderer.com:7050 []} {oodjaeuen108-orderer0.orderer.com:7050 []} {localhost:51011 []} {localhost:51013 []}] , at 2 attempt. Retrying in 2s
2019-12-03 09:04:45.512 UTC [ConnProducer] NewConnection -> ERRO aa3 Failed connecting to {localhost:51015 []} , error: context deadline exceeded
2019-12-03 09:04:48.514 UTC [ConnProducer] NewConnection -> ERRO aa4 Failed connecting to {localhost:51014 []} , error: context deadline exceeded
2019-12-03 09:04:51.515 UTC [ConnProducer] NewConnection -> ERRO aa5 Failed connecting to {oodjaeuen108-orderer4.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:04:54.517 UTC [ConnProducer] NewConnection -> ERRO aa6 Failed connecting to {oodjaeuen108-orderer1.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:04:57.518 UTC [ConnProducer] NewConnection -> ERRO aa7 Failed connecting to {localhost:51012 []} , error: context deadline exceeded
2019-12-03 09:05:00.520 UTC [ConnProducer] NewConnection -> ERRO aa8 Failed connecting to {oodjaeuen108-orderer2.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:05:03.522 UTC [ConnProducer] NewConnection -> ERRO aa9 Failed connecting to {oodjaeuen108-orderer3.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:05:06.523 UTC [ConnProducer] NewConnection -> ERRO aaa Failed connecting to {oodjaeuen108-orderer0.orderer.com:7050 []} , error: context deadlineexceeded
2019-12-03 09:05:09.525 UTC [ConnProducer] NewConnection -> ERRO aab Failed connecting to {localhost:51011 []} , error: context deadline exceeded
2019-12-03 09:05:12.526 UTC [ConnProducer] NewConnection -> ERRO aac Failed connecting to {localhost:51013 []} , error: context deadline exceeded
2019-12-03 09:05:12.526 UTC [ConnProducer] NewConnection -> ERRO aad Could not connect to any of the endpoints: [{localhost:51015 []} {localhost:51014 []} {oodjaeuen108-orderer4.orderer.com:7050 []} {oodjaeuen108-orderer1.orderer.com:7050 []} {localhost:51012 []} {oodjaeuen108-orderer2.orderer.com:7050 []} {oodjaeuen108-orderer3.orderer.com:7050 []} {oodjaeuen108-orderer0.orderer.com:7050 []} {localhost:51011 []} {localhost:51013 []}]
2019-12-03 09:05:12.526 UTC [deliveryClient] connect -> ERRO aae Failed obtaining connection: could not connect to any of the endpoints: [{localhost:51015 []} {localhost:51014 []} {oodjaeuen108-orderer4.orderer.com:7050 []} {oodjaeuen108-orderer1.orderer.com:7050 []} {localhost:51012 []} {oodjaeuen108-orderer2.orderer.com:7050 []} {oodjaeuen108-orderer3.orderer.com:7050 []} {oodjaeuen108-orderer0.orderer.com:7050 []} {localhost:51011 []} {localhost:51013 []}]
2019-12-03 09:05:12.526 UTC [deliveryClient] try -> WARN aaf Got error: could not connect to any of the endpoints: [{localhost:51015 []} {localhost:51014 []} {oodjaeuen108-orderer4.orderer.com:7050 []} {oodjaeuen108-orderer1.orderer.com:7050 []} {localhost:51012 []} {oodjaeuen108-orderer2.orderer.com:7050 []} {oodjaeuen108-orderer3.orderer.com:7050 []} {oodjaeuen108-orderer0.orderer.com:7050 []} {localhost:51011 []} {localhost:51013 []}] , at 3 attempt. Retrying in 4s



We are using following version of tools

Fabric Version :- 1.4.4
NodeJS version :- v8.16.0
NPM version :- 6.4.1
Docker Version :- 18.09.7
docker-compose version :- 1.24.1

Please help us regarding this. Its is a critical issue for us.