HLF Explorer fails to connect to the peer #fabric-kubernetes #fabric-questions #blockchain-explorer


anton.zaslavskiy@...
 

Hi Community,
 
We're trying to run hlf explorer instance in k8s. Explorer looses a connection to a peer sometimes and outputs an error.
[2020-11-26T12:55:34.394] [ERROR] FabricGateway - Failed to get discovery result from channel defaultchannel : { Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
at checkState (/opt/explorer/node_modules/@grpc/grpc-js/build/src/client.js:69:26)
at process._tickCallback (internal/process/next_tick.js:63:19) connectFailed: true }
[2020-11-26T12:55:34.397] [ERROR] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Synchronizer Error >>>>>>>>>>>>>>>>>>>>>
[2020-11-26T12:55:34.397] [ERROR] Sync - TypeError: Cannot use 'in' operator to search for 'peers_by_org' in null
at FabricClient.<anonymous> (/opt/explorer/app/platform/fabric/FabricClient.js:175:32)
at Generator.next (<anonymous>)
at fulfilled (/opt/explorer/app/platform/fabric/FabricClient.js:8:58)
at process._tickCallback (internal/process/next_tick.js:68:7)
 

After that it usually stops syncing new blocks and transactions. In order to overcome this we restart hlf explorer whenever there is a "Failed to connect before the deadline" string in the logs, as we couldn't find any other way to check the syncing process.
 
The problem is that when there are several organization added to a channel and thus there are a few blocks (more than 10-20) hlf explorer tries to perform discovery for every synced block and outputs a lot of connection errors during start up. Consequently, our healthcheck gives false negative result.
 
Tried on 1.1.2 and 1.1.3.

Maybe anyone can help with the following questions:
1. Does anybody know the correct way of checking the liveness of the syncing process?
2. What can we do with these "Failed to connect" errors? We tried to increase timeouts with "grpc-wait-for-ready-timeout" and "request-timeout" options, after that we got a different error: "Failed to get discovery result from channel defaultchannel :  { Error: 14 UNAVAILABLE: Connection dropped" because TLS handshake timeout exceeded on peer's side.
 
Info about configuration of the explorer:
  • DISCOVERY_AS_LOCALHOST env is set to false
  • Connection-profile is as follows
    {
    "name": "nw-767-370-1",
    "version": "1.0.0",
    "client": {
    "tlsEnable": false,
    "adminCredential": {
    "id": "***",
    "password": "***"
    },
    "enableAuthentication": false,
    "organization": "RG-488-419-MSP",
    "connection": {
    "timeout": {
    "peer": {
    "endorser": "300"
    },
    "orderer": "300"
    }
    }
    },
    "channels": {
    "defaultchannel": {
    "peers": {
    "nd-980-845-068.***.com": {}
    },
    "connection": {
    "timeout": {
    "peer": {
    "endorser": "6000",
    "eventHub": "6000",
    "eventReg": "6000"
    }
    }
    }
    }
    },
    "organizations": {
    "RG-488-419-MSP": {
    "mspid": "RG-488-419-MSP",
    "fullpath": true,
    "adminPrivateKey": {
    "path": "/var/explorer/msp/adminkey/key.pem"
    },
    "peers": ["nd-980-845-068.***.com"],
    "signedCert": {
    "path": "/var/explorer/msp/admincert/cert.pem"
    }
    }
    },
    "peers": {
    "nd-980-845-068.***.com": {
    "tlsCACerts": {
    "path": "/var/explorer/peer/tls/ca.crt"
    },
    "url": "grpcs://nd-980-845-068.***.com:7051",
    "eventUrl": "grpcs://nd-980-845-068.***.com:7053"
    }
    }
    }
     
 
 
Start up logs of the explorer:


> hyperledger-explorer@1.1.3 app-start /opt/explorer
> ./start.sh

************************************************************************************
**************************** Hyperledger Explorer **********************************
************************************************************************************
[2020-11-26T12:54:41.301] [INFO] Synchronizer - initialize : 60
[2020-11-26T12:54:41.301] [INFO] SyncPlatform - Sync process is started for the network = [network] and client = [nw-155-715-7]
[2020-11-26T12:54:41.302] [INFO] FabricConfig - config.client.tlsEnable false
[2020-11-26T12:54:41.303] [INFO] FabricConfig - FabricConfig, this.config.channels defaultchannel
[2020-11-26T12:54:46.712] [INFO] FabricUtils - generateBlockHash 0
[2020-11-26T12:54:46.721] [INFO] SyncServices - SyncServices.synchNetworkConfigToDB client network channel_name defaultchannel
[2020-11-26T12:54:46.812] [INFO] FabricUtils - generateBlockHash 0
[2020-11-26T12:54:47.640] [INFO] FabricClient - Discovered RG-958-375-MSP [ { mspid: 'RG-958-375-MSP',
endpoint: 'nd-113-392-727.****.com:7051',
name: 'nd-113-392-727.****.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:54:47.641] [INFO] FabricClient - Discovered RG-785-558-MSP [ { mspid: 'RG-785-558-MSP',
endpoint: 'nd-679-110-235.****.com:7051',
name: 'nd-679-110-235.****.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:54:47.642] [INFO] FabricClient - Discovered RG-907-954-MSP [ { mspid: 'RG-907-954-MSP',
endpoint: 'nd-603-674-180.****.com:7051',
name: 'nd-603-674-180.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:54:47.642] [INFO] FabricClient - Discovered RG-837-380-MSP [ { mspid: 'RG-837-380-MSP',
endpoint: 'nd-739-228-632.***.com:7051',
name: 'nd-739-228-632.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:54:47.654] [INFO] FabricClient - Discovered RG-239-893-MSP [ { mspid: 'RG-239-893-MSP',
endpoint: 'nd-423-261-488.***.com:7051',
name: 'nd-423-261-488.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:54:47.654] [INFO] FabricClient - Discovered RG-922-625-MSP [ { mspid: 'RG-922-625-MSP',
endpoint: 'nd-186-074-086.***.com:7051',
name: 'nd-186-074-086.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:54:47.654] [INFO] FabricClient - Discovered RG-954-235-MSP [ { mspid: 'RG-954-235-MSP',
endpoint: 'nd-226-796-011.***.com:7051',
name: 'nd-226-796-011.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:54:47.697] [INFO] FabricGateway - queryInstantiatedChaincodes defaultchannel
[2020-11-26T12:54:47.751] [INFO] FabricGateway - 1: InstalledChaincode {
references: { defaultchannel: References { chaincodes: [Array] } },
package_id:
'freedomDividendContract1:7f3f8622516757155cb0bfd22913c670664812d3b0528d3725684001d174c9d4',
label: 'freedomDividendContract1' }
[2020-11-26T12:54:47.751] [INFO] FabricGateway - 2: References {
chaincodes:
[ Chaincode { name: 'freedomDividendContract', version: '1' } ] }
[2020-11-26T12:54:47.820] [INFO] Sync - Synchronizer pid is 32
[2020-11-26T12:54:47.947] [INFO] FabricEvent - Successfully created channel event hub for [defaultchannel]
[2020-11-26T12:54:47.951] [INFO] FabricUtils - generateBlockHash 1
[2020-11-26T12:54:47.960] [INFO] SyncServices - block_row.blocknum 1
[2020-11-26T12:54:48.003] [INFO] FabricUtils - generateBlockHash 2
[2020-11-26T12:54:48.014] [INFO] SyncServices - block_row.blocknum 2
[2020-11-26T12:54:48.096] [INFO] FabricUtils - generateBlockHash 3
[2020-11-26T12:54:48.110] [INFO] SyncServices - block_row.blocknum 3
[2020-11-26T12:54:48.199] [INFO] FabricUtils - generateBlockHash 4
[2020-11-26T12:54:48.211] [INFO] SyncServices - block_row.blocknum 4
[2020-11-26T12:54:48.215] [INFO] FabricUtils - generateBlockHash 5
[2020-11-26T12:54:48.219] [INFO] SyncServices - block_row.blocknum 5
[2020-11-26T12:54:48.300] [INFO] FabricUtils - generateBlockHash 6
[2020-11-26T12:54:48.399] [INFO] SyncServices - block_row.blocknum 6
[2020-11-26T12:54:48.403] [INFO] FabricUtils - generateBlockHash 7
[2020-11-26T12:54:48.421] [INFO] SyncServices - block_row.blocknum 7
[2020-11-26T12:54:48.424] [INFO] FabricUtils - generateBlockHash 8
[2020-11-26T12:54:48.428] [INFO] SyncServices - block_row.blocknum 8
[2020-11-26T12:54:48.515] [INFO] FabricUtils - generateBlockHash 9
[2020-11-26T12:54:48.628] [INFO] SyncServices - block_row.blocknum 9
[2020-11-26T12:54:48.657] [INFO] FabricUtils - generateBlockHash 10

... up to 125 block

[2020-11-26T12:54:51.195] [INFO] FabricUtils - generateBlockHash 125
[2020-11-26T12:54:51.198] [INFO] SyncServices - block_row.blocknum 125
[2020-11-26T12:55:34.201] [INFO] SyncPlatform - Updating the client network and other details to DB
2020-11-26T12:55:34.208Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.209Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.210Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.210Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.210Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.210Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.210Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.210Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
2020-11-26T12:55:34.211Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true

... a lot of duplications of the same line

[2020-11-26T12:55:34.296] [ERROR] FabricGateway - Failed to get discovery result from channel defaultchannel : { Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
at checkState (/opt/explorer/node_modules/@grpc/grpc-js/build/src/client.js:69:26)
at process._tickCallback (internal/process/next_tick.js:63:19) connectFailed: true }
[2020-11-26T12:55:34.296] [ERROR] FabricGateway - Failed to get discovery result from channel defaultchannel : { Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
at checkState (/opt/explorer/node_modules/@grpc/grpc-js/build/src/client.js:69:26)
at process._tickCallback (internal/process/next_tick.js:63:19) connectFailed: true }
[2020-11-26T12:55:34.296] [ERROR] FabricGateway - Failed to get discovery result from channel defaultchannel : { Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
at checkState (/opt/explorer/node_modules/@grpc/grpc-js/build/src/client.js:69:26)
at process._tickCallback (internal/process/next_tick.js:63:19) connectFailed: true }
[2020-11-26T12:55:34.297] [ERROR] FabricGateway - Failed to get discovery result from channel defaultchannel : { Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
at checkState (/opt/explorer/node_modules/@grpc/grpc-js/build/src/client.js:69:26)
at process._tickCallback (internal/process/next_tick.js:63:19) connectFailed: true }

... a lot of duplications of the same line


[2020-11-26T12:55:34.397] [ERROR] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Synchronizer Error >>>>>>>>>>>>>>>>>>>>>
[2020-11-26T12:55:34.397] [ERROR] Sync - TypeError: Cannot use 'in' operator to search for 'peers_by_org' in null
at FabricClient.<anonymous> (/opt/explorer/app/platform/fabric/FabricClient.js:175:32)
at Generator.next (<anonymous>)
at fulfilled (/opt/explorer/app/platform/fabric/FabricClient.js:8:58)
at process._tickCallback (internal/process/next_tick.js:68:7)
[2020-11-26T12:55:34.398] [INFO] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Closing client processor >>>>>>>>>>>>>>>>>>>>>
[2020-11-26T12:55:34.398] [ERROR] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Synchronizer Error >>>>>>>>>>>>>>>>>>>>>
[2020-11-26T12:55:34.398] [ERROR] Sync - TypeError: Cannot use 'in' operator to search for 'peers_by_org' in null
at FabricClient.<anonymous> (/opt/explorer/app/platform/fabric/FabricClient.js:175:32)
at Generator.next (<anonymous>)
at fulfilled (/opt/explorer/app/platform/fabric/FabricClient.js:8:58)
at process._tickCallback (internal/process/next_tick.js:68:7)
[2020-11-26T12:55:34.399] [INFO] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Closing client processor >>>>>>>>>>>>>>>>>>>>>
[2020-11-26T12:55:34.399] [ERROR] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Synchronizer Error >>>>>>>>>>>>>>>>>>>>>
[2020-11-26T12:55:34.399] [ERROR] Sync - TypeError: Cannot use 'in' operator to search for 'peers_by_org' in null
at FabricClient.<anonymous> (/opt/explorer/app/platform/fabric/FabricClient.js:175:32)
at Generator.next (<anonymous>)
at fulfilled (/opt/explorer/app/platform/fabric/FabricClient.js:8:58)
at process._tickCallback (internal/process/next_tick.js:68:7)
[2020-11-26T12:55:34.399] [INFO] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Closing client processor >>>>>>>>>>>>>>>>>>>>>
[2020-11-26T12:55:34.399] [ERROR] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Synchronizer Error >>>>>>>>>>>>>>>>>>>>>
[2020-11-26T12:55:34.399] [ERROR] Sync - TypeError: Cannot use 'in' operator to search for 'peers_by_org' in null
at FabricClient.<anonymous> (/opt/explorer/app/platform/fabric/FabricClient.js:175:32)
at Generator.next (<anonymous>)
at fulfilled (/opt/explorer/app/platform/fabric/FabricClient.js:8:58)
at process._tickCallback (internal/process/next_tick.js:68:7)


... a lot of duplications of the same line


[2020-11-26T12:55:34.698] [INFO] SyncServices - SyncServices.synchNetworkConfigToDB client network channel_name defaultchannel
[2020-11-26T12:55:34.796] [INFO] FabricUtils - generateBlockHash 0
[2020-11-26T12:55:34.797] [ERROR] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Synchronizer Error >>>>>>>>>>>>>>>>>>>>>
[2020-11-26T12:55:34.797] [ERROR] Sync - Error: Client was closed and is not queryable
at process.nextTick (/opt/explorer/node_modules/pg/lib/client.js:535:25)
at process._tickCallback (internal/process/next_tick.js:61:11)
[2020-11-26T12:55:34.797] [INFO] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Closing client processor >>>>>>>>>>>>>>>>>>>>>
2020-11-26T12:55:34.807Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
2020-11-26T12:55:34.807Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server be discoverer nd-113-392-727.***.com url:grpcs://nd-113-392-727.***.com:7051 timeout:3000
[2020-11-26T12:55:34.807] [ERROR] FabricGateway - Failed to get discovery result from channel defaultchannel : { Error: Failed to connect before the deadline on Discoverer- name: be discoverer nd-113-392-727.***.com, url:grpcs://nd-113-392-727.***.com:7051, connected:false, connectAttempted:true
at checkState (/opt/explorer/node_modules/@grpc/grpc-js/build/src/client.js:69:26)
at Timeout.watcherObject.timer.setTimeout [as _onTimeout] (/opt/explorer/node_modules/@grpc/grpc-js/build/src/channel.js:292:17)
at ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at listOnTimeout (timers.js:263:5)
at Timer.processTimers (timers.js:223:10) connectFailed: true }
[2020-11-26T12:55:34.807] [ERROR] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Synchronizer Error >>>>>>>>>>>>>>>>>>>>>
[2020-11-26T12:55:34.808] [ERROR] Sync - TypeError: Cannot use 'in' operator to search for 'peers_by_org' in null
at FabricClient.<anonymous> (/opt/explorer/app/platform/fabric/FabricClient.js:175:32)
at Generator.next (<anonymous>)
at fulfilled (/opt/explorer/app/platform/fabric/FabricClient.js:8:58)
at process._tickCallback (internal/process/next_tick.js:68:7)
[2020-11-26T12:55:34.808] [INFO] Sync - <<<<<<<<<<<<<<<<<<<<<<<<<< Closing client processor >>>>>>>>>>>>>>>>>>>>>

... a lot of duplications of the same line

[2020-11-26T12:55:36.102] [INFO] FabricClient - Discovered RG-837-380-MSP [ { mspid: 'RG-837-380-MSP',
endpoint: 'nd-739-228-632.***.com:7051',
name: 'nd-739-228-632.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.103] [INFO] FabricClient - Discovered RG-239-893-MSP [ { mspid: 'RG-239-893-MSP',
endpoint: 'nd-423-261-488.***.com:7051',
name: 'nd-423-261-488.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:55:36.103] [INFO] FabricClient - Discovered RG-922-625-MSP [ { mspid: 'RG-922-625-MSP',
endpoint: 'nd-186-074-086.***.com:7051',
name: 'nd-186-074-086.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.103] [INFO] FabricClient - Discovered RG-954-235-MSP [ { mspid: 'RG-954-235-MSP',
endpoint: 'nd-226-796-011.***.com:7051',
name: 'nd-226-796-011.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.103] [INFO] FabricClient - Discovered RG-958-375-MSP [ { mspid: 'RG-958-375-MSP',
endpoint: 'nd-113-392-727.***.com:7051',
name: 'nd-113-392-727.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.103] [INFO] FabricClient - Discovered RG-785-558-MSP [ { mspid: 'RG-785-558-MSP',
endpoint: 'nd-679-110-235.***.com:7051',
name: 'nd-679-110-235.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:55:36.104] [INFO] FabricClient - Discovered RG-907-954-MSP [ { mspid: 'RG-907-954-MSP',
endpoint: 'nd-603-674-180.***.com:7051',
name: 'nd-603-674-180.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:55:36.514] [INFO] FabricClient - Discovered RG-954-235-MSP [ { mspid: 'RG-954-235-MSP',
endpoint: 'nd-226-796-011.***.com:7051',
name: 'nd-226-796-011.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.514] [INFO] FabricClient - Discovered RG-958-375-MSP [ { mspid: 'RG-958-375-MSP',
endpoint: 'nd-113-392-727.***.com:7051',
name: 'nd-113-392-727.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.514] [INFO] FabricClient - Discovered RG-785-558-MSP [ { mspid: 'RG-785-558-MSP',
endpoint: 'nd-679-110-235.***.com:7051',
name: 'nd-679-110-235.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:55:36.514] [INFO] FabricClient - Discovered RG-907-954-MSP [ { mspid: 'RG-907-954-MSP',
endpoint: 'nd-603-674-180.***.com:7051',
name: 'nd-603-674-180.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:55:36.514] [INFO] FabricClient - Discovered RG-837-380-MSP [ { mspid: 'RG-837-380-MSP',
endpoint: 'nd-739-228-632.***.com:7051',
name: 'nd-739-228-632.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.515] [INFO] FabricClient - Discovered RG-239-893-MSP [ { mspid: 'RG-239-893-MSP',
endpoint: 'nd-423-261-488.***.com:7051',
name: 'nd-423-261-488.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:55:36.515] [INFO] FabricClient - Discovered RG-922-625-MSP [ { mspid: 'RG-922-625-MSP',
endpoint: 'nd-186-074-086.***.com:7051',
name: 'nd-186-074-086.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.901] [INFO] FabricClient - Discovered RG-954-235-MSP [ { mspid: 'RG-954-235-MSP',
endpoint: 'nd-226-796-011.***.com:7051',
name: 'nd-226-796-011.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.901] [INFO] FabricClient - Discovered RG-958-375-MSP [ { mspid: 'RG-958-375-MSP',
endpoint: 'nd-113-392-727.***.com:7051',
name: 'nd-113-392-727.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.901] [INFO] FabricClient - Discovered RG-785-558-MSP [ { mspid: 'RG-785-558-MSP',
endpoint: 'nd-679-110-235.***.com:7051',
name: 'nd-679-110-235.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:55:36.901] [INFO] FabricClient - Discovered RG-907-954-MSP [ { mspid: 'RG-907-954-MSP',
endpoint: 'nd-603-674-180.***.com:7051',
name: 'nd-603-674-180.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:55:36.901] [INFO] FabricClient - Discovered RG-837-380-MSP [ { mspid: 'RG-837-380-MSP',
endpoint: 'nd-739-228-632.***.com:7051',
name: 'nd-739-228-632.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]
[2020-11-26T12:55:36.901] [INFO] FabricClient - Discovered RG-239-893-MSP [ { mspid: 'RG-239-893-MSP',
endpoint: 'nd-423-261-488.***.com:7051',
name: 'nd-423-261-488.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object] ] } ]
[2020-11-26T12:55:36.902] [INFO] FabricClient - Discovered RG-922-625-MSP [ { mspid: 'RG-922-625-MSP',
endpoint: 'nd-186-074-086.***.com:7051',
name: 'nd-186-074-086.***.com:7051',
ledgerHeight: Long { low: 126, high: 0, unsigned: true },
chaincodes: [ [Object], [Object] ] } ]


 


nekia
 

Hi,

Thank you for reporting your issue. Hyperledger Explorer Group is right place where sharing this issue.

This 'connection failure' issue has been reported in hyperledger explorer channel. I'm not sure this issue also has same root cause, but it's worthwhile to try the following step to update Explorer image on your local.

There were the following concurrent tx issue (FABN-1646) on fabric-sdk-node v2.2.2 which Explorer is using. It's been resolved in the latest package v2.2.3 but this revision has introduced another issue (FABN-1663) which is critical for Explorer. The fix for this issue has not been released yet, but you can still use the latest snapshot package (2.2.4-snapshot.7).
 
https://jira.hyperledger.org/browse/FABN-1646
https://jira.hyperledger.org/browse/FABN-1663

cd /some/where/blockchain-explorer
# update rev of fabric-network package
npm install fabric-network@2.2.4-snapshot.7
# rebuild Explorer container image
./build_docker_image.sh -e
# apply this image into your runtime

Thanks,
Atsushi