1. #fabric-questions
  2. Why does only the first peer report TLS handshake errors?

Why does only the first peer report TLS handshake errors? #tls #docker-compose #fabric-questions

chin-kai.ong@...
 

I am trying to deploy a three-orderer two-peer network in Docker swarm, but somehow I keep seeing TLS handshake errors logged in only the first peer.

I'm working with Fabric 1.4.9, and these are the commands I run:
1. cryptogen generate --config=./crypto-config.yaml
2. docker stack deploy --compose-file=docker-compose-orderer.yaml fabric
3. docker stack deploy --compose-file=docker-compose-peer.yaml fabric

My log is as follows:
 
[nodeCmd] serve -> INFO 001 Starting peer:
 Version: 1.4.9
 Commit SHA: da55272a7
 Go version: go1.13.12
 OS/Arch: linux/amd64
 Chaincode:
  Base Image Version: 0.4.21
  Base Docker Namespace: hyperledger
  Base Docker Label: org.hyperledger.fabric
  Docker Namespace: hyperledger
[ledgermgmt] initialize -> INFO 002 Initializing ledger mgmt
[kvledger] NewProvider -> INFO 003 Initializing ledger provider
[kvledger] NewProvider -> INFO 004 ledger provider Initialized
[couchdb] CreateDatabaseIfNotExist -> INFO 00c Created state database _users
[couchdb] CreateDatabaseIfNotExist -> INFO 00d Created state database _replicator
[ledgermgmt] initialize -> INFO 00e ledger mgmt initialized
[peer] func1 -> INFO 00f Auto-detected peer address: 10.0.2.46:7051
[peer] func1 -> INFO 010 Returning peer0.isprint.dev.accessreal.com:7051
[peer] func1 -> INFO 011 Auto-detected peer address: 10.0.2.46:7051
[peer] func1 -> INFO 012 Returning peer0.isprint.dev.accessreal.com:7051
[nodeCmd] serve -> INFO 013 Starting peer with TLS enabled
[nodeCmd] computeChaincodeEndpoint -> INFO 014 Entering computeChaincodeEndpoint with peerHostname: peer0.isprint.dev.accessreal.com
[nodeCmd] computeChaincodeEndpoint -> INFO 015 Exit with ccEndpoint: peer0.isprint.dev.accessreal.com:7052
[sccapi] registerSysCC -> INFO 016 system chaincode lscc(github.com/hyperledger/fabric/core/scc/lscc) registered
[sccapi] registerSysCC -> INFO 017 system chaincode cscc(github.com/hyperledger/fabric/core/scc/cscc) registered
[sccapi] registerSysCC -> INFO 018 system chaincode qscc(github.com/hyperledger/fabric/core/scc/qscc) registered
[sccapi] registerSysCC -> INFO 019 system chaincode (+lifecycle,github.com/hyperledger/fabric/core/chaincode/lifecycle,true) disabled
[certmonitor] trackCertExpiration -> INFO 01a The enrollment certificate will expire on 2030-12-16 19:25:00 +0000 UTC
[certmonitor] trackCertExpiration -> INFO 01b The server TLS certificate will expire on 2030-12-16 19:25:00 +0000 UTC
[peer1.isprint.dev.accessreal.com:8051]
[gossip.gossip] NewGossipService -> INFO 01d Creating gossip service with self membership of Endpoint: peer0.isprint.dev.accessreal.com:7051, InternalEndpoint: peer0.isprint.dev.accessreal.com:7051, PKI-ID: 3f45d9ec65a3844d4ba7ae9393cc94fc95ad037c1bbcdeb982c98b42d0db9bce, Metadata:
[gossip.gossip] start -> INFO 01e Gossip instance peer0.isprint.dev.accessreal.com:7051 started
[sccapi] deploySysCC -> INFO 01f system chaincode lscc/(github.com/hyperledger/fabric/core/scc/lscc) deployed
[cscc] Init -> INFO 020 Init CSCC
[sccapi] deploySysCC -> INFO 021 system chaincode cscc/(github.com/hyperledger/fabric/core/scc/cscc) deployed
[qscc] Init -> INFO 022 Init QSCC
[sccapi] deploySysCC -> INFO 023 system chaincode qscc/(github.com/hyperledger/fabric/core/scc/qscc) deployed
[sccapi] deploySysCC -> INFO 024 system chaincode (+lifecycle,github.com/hyperledger/fabric/core/chaincode/lifecycle) disabled
[nodeCmd] serve -> INFO 025 Deployed system chaincodes
[discovery] NewService -> INFO 026 Created with config TLS: true, authCacheMaxSize: 1000, authCachePurgeRatio: 0.750000
[nodeCmd] registerDiscoveryService -> INFO 027 Discovery service activated
[peer0.isprint.dev.accessreal.com:7051]
[peer0.isprint.dev.accessreal.com:7051]
[nodeCmd] func7 -> INFO 02a Starting profiling server with listenAddress = 0.0.0.0:6060
[/var/hyperledger/production/ledgersData/chains]
[fsblkstorage] LoadPreResetHeight -> INFO 02c Loading Pre-reset heights
[/var/hyperledger/production/ledgersData/chains/chains] missing... exiting
[fsblkstorage] LoadPreResetHeight -> INFO 02e Pre-reset heights loaded
[core.comm] ServerHandshake -> ERRO 02f TLS handshake failed with error remote error: tls: internal error server=PeerServer remoteaddress=10.11.65.1:39393
[core.comm] ServerHandshake -> ERRO 030 TLS handshake failed with error remote error: tls: internal error server=PeerServer remoteaddress=10.11.65.1:41613
[core.comm] ServerHandshake -> ERRO 031 TLS handshake failed with error remote error: tls: internal error server=PeerServer remoteaddress=10.11.65.1:39394
...(and then it's just this error message)

My crypto-config.yaml is as follows:

OrdererOrgs:
  - Name: Orderer
    Domain: dev.accessreal.com
    EnableNodeOUs: true
    Specs:
      - Hostname: orderer1
        SANS:
           - 192.168.50.65
           - isprintdev
      - Hostname: orderer2
        SANS:
           - 192.168.50.65
           - isprintdev
      - Hostname: orderer3
        SANS:
           - 192.168.50.65
           - isprintdev
 
PeerOrgs:
  - Name: isprint
    Domain: isprint.dev.accessreal.com
    EnableNodeOUs: true
    Specs:
      - Hostname: peer0
        SANS: 
           - 192.168.50.65
           - isprintdev
      - Hostname: peer1
        SANS: 
           - 192.168.50.65
           - isprintdev
    Template:
      Count: 2
    Users:
      Count: 1

This is my Docker compose file for the orderers:

version: '3.4'
 
volumes:
  orderer1.dev.accessreal.com:
  orderer2.dev.accessreal.com:
  orderer3.dev.accessreal.com:
  
networks:
  isprint:
    external:
      name: fabric
 
services:
  dev_orderer1:
    image: hyperledger/fabric-orderer
    environment:
      - ORDERER_GENERAL_LOGLEVEL=INFO
      - ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
      - ORDERER_GENERAL_GENESISMETHOD=file
      - ORDERER_GENERAL_GENESISFILE=/var/hyperledger/orderer/orderer.genesis.block
      - ORDERER_GENERAL_LOCALMSPID=OrdererMSP
      - ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
      - ORDERER_GENERAL_TLS_ENABLED=true
      - ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
      - ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
      - ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
      - ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
      - ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
      - ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric
    command: orderer
    volumes:
    - ./channel-artifacts/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
    - ./crypto-config/ordererOrganizations/dev.accessreal.com/orderers/orderer1.dev.accessreal.com/msp:/var/hyperledger/orderer/msp
    - ./crypto-config/ordererOrganizations/dev.accessreal.com/orderers/orderer1.dev.accessreal.com/tls/:/var/hyperledger/orderer/tls
    - orderer1.dev.accessreal.com:/var/hyperledger/production/orderer
    deploy:
            mode: replicated
            replicas: 1
            restart_policy:
              condition: on-failure
            placement:
                constraints:
                    - node.hostname == isprintdev
    ports:
      - published: 7050
        target: 7050
        mode: host
    networks:
      isprint:
        aliases:
          - orderer1.dev.accessreal.com
 
  dev_orderer2:
    image: hyperledger/fabric-orderer:latest
    environment:
      - ORDERER_GENERAL_LOGLEVEL=INFO
      - ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
      - ORDERER_GENERAL_GENESISMETHOD=file
      - ORDERER_GENERAL_GENESISFILE=/var/hyperledger/orderer/orderer.genesis.block
      - ORDERER_GENERAL_LOCALMSPID=OrdererMSP
      - ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
      - ORDERER_GENERAL_TLS_ENABLED=true
      - ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
      - ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
      - ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
      - ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
      - ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
      - ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric
    command: orderer
    volumes:
    - ./channel-artifacts/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
    - ./crypto-config/ordererOrganizations/dev.accessreal.com/orderers/orderer2.dev.accessreal.com/msp:/var/hyperledger/orderer/msp
    - ./crypto-config/ordererOrganizations/dev.accessreal.com/orderers/orderer2.dev.accessreal.com/tls/:/var/hyperledger/orderer/tls
    - orderer2.dev.accessreal.com:/var/hyperledger/production/orderer
    deploy:
            mode: replicated
            replicas: 1
            restart_policy:
              condition: on-failure
            placement:
                constraints:
                    - node.hostname == isprintdev
    ports:
      - published: 8050
        target: 7050
        mode: host
    networks:
      isprint:
        aliases:
          - orderer2.dev.accessreal.com
 
  dev_orderer3:
    image: hyperledger/fabric-orderer:latest
    environment:
      - ORDERER_GENERAL_LOGLEVEL=INFO
      - ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
      - ORDERER_GENERAL_GENESISMETHOD=file
      - ORDERER_GENERAL_GENESISFILE=/var/hyperledger/orderer/orderer.genesis.block
      - ORDERER_GENERAL_LOCALMSPID=OrdererMSP
      - ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
      - ORDERER_GENERAL_TLS_ENABLED=true
      - ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
      - ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
      - ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
      - ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
      - ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
      - ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric
    command: orderer
    volumes:
    - ./channel-artifacts/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
    - ./crypto-config/ordererOrganizations/dev.accessreal.com/orderers/orderer3.dev.accessreal.com/msp:/var/hyperledger/orderer/msp
    - ./crypto-config/ordererOrganizations/dev.accessreal.com/orderers/orderer3.dev.accessreal.com/tls/:/var/hyperledger/orderer/tls
    - orderer3.dev.accessreal.com:/var/hyperledger/production/orderer
    deploy:
            mode: replicated
            replicas: 1
            restart_policy:
              condition: on-failure
            placement:
                constraints:
                    - node.hostname == isprintdev
    ports:
      - published: 9050
        target: 7050
        mode: host
    networks:
      isprint:
        aliases:
          - orderer3.dev.accessreal.com

Then this is the Docker compose file that I use to raise the peers:

version: '3.4'
 
volumes:
  peer0.isprint.dev.accessreal.com:
  peer1.isprint.dev.accessreal.com:
  couchdb1.isprint.dev.accessreal.com:
  couchdb2.isprint.dev.accessreal.com:
  
networks:
  isprint:
    external:
      name: fabric
 
services:
  dev_couchdb1:
    image: hyperledger/fabric-couchdb
    environment:
      - COUCHDB_USER= couchdb
      - COUCHDB_PASSWORD=couchdb123
    volumes:
        - couchdb1.isprint.dev.accessreal.com:/opt/couchdb/data
    deploy:
            mode: replicated
            replicas: 1
            restart_policy:
              condition: on-failure
            placement:
                constraints:
                    - node.hostname == isprintdev
    ports:
     - published: 5984
       target: 5984
       mode: host
    networks:
      isprint:
        aliases:
          - couchdb1.isprint.dev.accessreal.com
 
  dev_couchdb2:
    image: hyperledger/fabric-couchdb
    environment:
      - COUCHDB_USER= couchdb
      - COUCHDB_PASSWORD=couchdb123
    volumes:
        - couchdb2.isprint.dev.accessreal.com:/opt/couchdb/data
    deploy:
            mode: replicated
            replicas: 1
            restart_policy:
              condition: on-failure
            placement:
                constraints:
                    - node.hostname == isprintdev
    ports:
     - published: 6984
       target: 5984
       mode: host
    networks:
      isprint:
        aliases:
          - couchdb2.isprint.dev.accessreal.com
 
  dev_peer0:
    image: hyperledger/fabric-peer:latest
    environment:
      - CORE_LEDGER_STATE_STATEDATABASE=CouchDB
      - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb1.isprint.dev.accessreal.com:5984
      - CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=couchdb
      - CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=couchdb123
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=fabric
      - CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052
      - FABRIC_LOGGING_SPEC=INFO
      - CORE_PEER_TLS_ENABLED=true
      - CORE_PEER_GOSSIP_USELEADERELECTION=true
      - CORE_PEER_GOSSIP_ORGLEADER=false
      - CORE_PEER_PROFILE_ENABLED=true
      - CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt
      - CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key
      - CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt
      - CORE_PEER_ID=peer0.isprint.dev.accessreal.com
      - CORE_PEER_ADDRESS=peer0.isprint.dev.accessreal.com:7051
      - CORE_PEER_GOSSIP_BOOTSTRAP=peer1.isprint.dev.accessreal.com:8051
      - CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.isprint.dev.accessreal.com:7051
      - CORE_PEER_LOCALMSPID=isprintMSP
      - CORE_VM_DOCKER_ATTACHSTDOUT=true
      - CORE_CHAINCODE_STARTUPTIMEOUT=1200s
      - CORE_CHAINCODE_EXECUTETIMEOUT=800s
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric/peer
    command: peer node start
    volumes:
        - /var/run/:/host/var/run/
        - ./crypto-config/peerOrganizations/isprint.dev.accessreal.com/peers/peer0.isprint.dev.accessreal.com/msp:/etc/hyperledger/fabric/msp
        - ./crypto-config/peerOrganizations/isprint.dev.accessreal.com/peers/peer0.isprint.dev.accessreal.com/tls:/etc/hyperledger/fabric/tls
        - peer0.isprint.dev.accessreal.com:/var/hyperledger/production
    deploy:
            mode: replicated
            replicas: 1
            restart_policy:
              condition: on-failure
            placement:
                constraints:
                    - node.hostname == isprintdev
    ports:
            - published: 7051
              target: 7051
              mode: host
            - published: 7053
              target: 7053
              mode: host
    networks:
      isprint:
        aliases:
          - peer0.isprint.dev.accessreal.com
 
 
  dev_peer1:
    image: hyperledger/fabric-peer:latest
    environment:
      - CORE_LEDGER_STATE_STATEDATABASE=CouchDB
      - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb2.isprint.dev.accessreal.com:5984
      - CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=couchdb
      - CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=couchdb123
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=fabric
      - CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052
      - FABRIC_LOGGING_SPEC=INFO
      - CORE_PEER_TLS_ENABLED=true
      - CORE_PEER_GOSSIP_USELEADERELECTION=true
      - CORE_PEER_GOSSIP_ORGLEADER=false
      - CORE_PEER_PROFILE_ENABLED=true
      - CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt
      - CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key
      - CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt
      - CORE_PEER_ID=peer1.isprint.dev.accessreal.com
      - CORE_PEER_ADDRESS=peer1.isprint.dev.accessreal.com:8051
      - CORE_PEER_GOSSIP_BOOTSTRAP=peer0.isprint.dev.accessreal.com:7051
      - CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer1.isprint.dev.accessreal.com:8051
      - CORE_PEER_LOCALMSPID=isprintMSP
      - CORE_VM_DOCKER_ATTACHSTDOUT=true
      - CORE_CHAINCODE_STARTUPTIMEOUT=1200s
      - CORE_CHAINCODE_EXECUTETIMEOUT=800s
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric/peer
    command: peer node start
 
    volumes:
        - /var/run/:/host/var/run/
        - ./crypto-config/peerOrganizations/isprint.dev.accessreal.com/peers/peer1.isprint.dev.accessreal.com/msp:/etc/hyperledger/fabric/msp
        - ./crypto-config/peerOrganizations/isprint.dev.accessreal.com/peers/peer1.isprint.dev.accessreal.com/tls:/etc/hyperledger/fabric/tls
        - peer1.isprint.dev.accessreal.com:/var/hyperledger/production
    deploy:
            mode: replicated
            replicas: 1
            restart_policy:
              condition: on-failure
            placement:
                constraints:
                    - node.hostname == isprintdev
    ports:
            - published: 8051
              target: 7051
              mode: host
            - published: 8053
              target: 7053
              mode: host
    networks:
      isprint:
        aliases:
          - peer1.isprint.dev.accessreal.com

Please let me know if there's any other information that I should add to this post. I'm really not that familiar working with Fabric, and I've spent days trying to resolve this but I don't know where to look.
Join fabric@lists.hyperledger.org to automatically receive all group messages.