Newly added peer getting online/offline messages along with old non-anchor peers #hyperledger-fabric #grpc #tls


I am encountering the following scenario: 
  1. I am running a network on multiple hosts.
  2. The network has one Orderer Organization (3 orderers in RAFT mode) and 1 Peer organization (2 Peers). Both having different Fabric CA instances.
  3. The application channel is created using all the orderers. Peer organization is added to consortium and the application channel. 
  4. Both the peers join the channel. I have configured Peer1.PeerOrg to be an anchor peer. 
  5. I install and instantiate chaincode. Invoke and query them multiple times.
  6. Create a new peer node MSP with PeerOrg CA.
  7. Start the new peer node and join the channel.
  8. Upon successful sync with the channel, the new peer node keeps getting logs that state "Membership view changed. Peers went online/offline". 
The following warning messages are accompanied by ERR message: "TLS handshake failed with error remote error: tls: bad certificate server=PeerServer".
The same set of ERR and WARN messages flood the logs of this new peer as well as of Peer2.PeerOrg (the non-anchor peer). 

So, now I switched anchor peer endpoint in application channel to Peer2.PeerOrg endpoint. Now again the new peer and Peer1.PeerOrg logs are flooded by above messages. 

At this stage, I remove anchor peers completely. Restart the peer nodes: Peer3.PeerOrg and Peer1.PeerOrg. No ERR or WARN messages occur now.

NOTE: All peers can communicate through one another by external endpoints. TLS certificates of all the peer nodes have been validated and are configured with correct DNS Alt Names. 

Can anyone please tell why this issue occurs?

Join to automatically receive all group messages.