Hyperledger Fabric Gateway fails with certificate signed by unknown authority but certificates pass an openssl test #hyperledger-fabric #tls #fabric-sdk-go


afrancoc2000@...
 

Hi Team, 

I'm having trouble making the "Hyperledger Fabric Gateway for Go" execute a smart contract, although I am using it successfully for listening to events.

This is the error I get in the client:

Submit error with gRPC status DeadlineExceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded

This is the error I get in the orderer:

Server TLS handshake failed in 2.628549ms with error remote error: tls: bad certificate server=Orderer remoteaddress=10.250.37.115:40858

This is the error I get in the peer:

Client TLS handshake failed after 6.156747ms with error: x509: certificate signed by unknown authority remoteaddress=10.250.37.195:443

The IP 10.250.37.195 corresponds to the ingress service external IP and the 10.250.37.115 corresponds to the ingress controller. The ingress controller has ssl-passtrough configured so the certificates are not being overridden by it.

I'm using a chain certificate that begins with an ica cert and then on the same file the root certificate, this chain.crt file is stored in the orderers in this folder: "/var/hyperledger/orderer/tls/chain.crt" both ORDERER_GENERAL_TLS_ROOTCAS and ORDERER_GENERAL_TLS_CLIENTROOTCAS point to it.

I tested my tls certificate against the chain.crt using openssl and the answer is ok.

I'm not sure where else to look to make it work.

I tested my code against the test network and it works, also using the old node SDK the connection works too.

This is my code (deleting err handling for readability):

func NewGrpcConnection(fabricConfig config.FabricConfig) (*grpc.ClientConn, error) {
    caCertificate, err := loadCertificate(fabricConfig.TlsCaPath)
    keypair, err := loadCertificateKeyPair(fabricConfig.TlsCertPath, fabricConfig.TlsKeyPath)

    certPool := x509.NewCertPool()
    certPool.AddCert(caCertificate)

    transportCredentials := credentials.NewTLS(&tls.Config{
        ServerName:   fabricConfig.HostName,
        RootCAs:      certPool,
        ClientCAs:    certPool,
        Certificates: []tls.Certificate{*keypair},
    })

    connection, err := grpc.Dial(fabricConfig.PeerEndpoint, grpc.WithTransportCredentials(transportCredentials))
    return connection, nil
}

func NewGateway(clientConnection *grpc.ClientConn, fabricConfig config.FabricConfig) (*client.Gateway, error) {
    id, err := newIdentity(fabricConfig.CertPath, fabricConfig.OrgName)
    sign, err := newSign(fabricConfig.KeyPath)

    gateway, err := client.Connect(
        id,
        client.WithSign(sign),
        client.WithClientConnection(clientConnection),
        client.WithEvaluateTimeout(5*time.Second),
        client.WithEndorseTimeout(15*time.Second),
        client.WithSubmitTimeout(5*time.Second),
        client.WithCommitStatusTimeout(1*time.Minute),
    )

    return gateway, err
}

func main() {
    grpcConnection, err := fabric.NewGrpcConnection(*clientConfig.FabricConfig)
    defer grpcConnection.Close()
   
    gateway, err := fabric.NewGateway(grpcConnection, *clientConfig.FabricConfig)
    defer gateway.Close()

    network := gateway.GetNetwork(clientConfig.FabricConfig.ChannelName)
    contract := network.GetContract(clientConfig.FabricConfig.ContractName)

result, err = contract.SubmitTransaction(request.Fcn, request.Args...)
}

I tried also overiding the pod's ssl certificates by adding the ca certificates to /usr/local/share/ca-certificates/ and using the update-ca-certificates command with no luck.

What am I doing wrong? Any ideas where else to look? do I need to add the tls-root and ica certificates to the msp folder under msp/
tlscacerts and msp/tlsintermediatecerts too? I have FABRIC_LOGGING_SPEC="grpc=debug:info" is there a way to see what certificate is comming in for validation?

Thanks,

Ana Franco
Tech Leader


Yacov
 

you have an interediate CA but you use AddCert which receives a single certificate object (not a PEM but literally the x509 certificate).
you need to add both root CA and intermediate CA. Easiest is to use AppendCertsFromPEM on the content of the concatenated file.

From: fabric@... <fabric@...> on behalf of afrancoc2000@... <afrancoc2000@...>
Sent: Thursday, February 10, 2022 2:38 PM
To: fabric@... <fabric@...>
Subject: [EXTERNAL] [Hyperledger Fabric] Hyperledger Fabric Gateway fails with certificate signed by unknown authority but certificates pass an openssl test #hyperledger-fabric #tls #fabric-sdk-go
 
Hi Team, I'm having trouble making the "Hyperledger Fabric Gateway for Go" execute a smart contract, although I am using it successfully for listening to events. This is the error I get in the client: Submit error with gRPC status DeadlineExceeded: ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Team, 

I'm having trouble making the "Hyperledger Fabric Gateway for Go" execute a smart contract, although I am using it successfully for listening to events.

This is the error I get in the client:

Submit error with gRPC status DeadlineExceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded

This is the error I get in the orderer:

Server TLS handshake failed in 2.628549ms with error remote error: tls: bad certificate server=Orderer remoteaddress=10.250.37.115:40858

This is the error I get in the peer:

Client TLS handshake failed after 6.156747ms with error: x509: certificate signed by unknown authority remoteaddress=10.250.37.195:443

The IP 10.250.37.195 corresponds to the ingress service external IP and the 10.250.37.115 corresponds to the ingress controller. The ingress controller has ssl-passtrough configured so the certificates are not being overridden by it.

I'm using a chain certificate that begins with an ica cert and then on the same file the root certificate, this chain.crt file is stored in the orderers in this folder: "/var/hyperledger/orderer/tls/chain.crt" both ORDERER_GENERAL_TLS_ROOTCAS and ORDERER_GENERAL_TLS_CLIENTROOTCAS point to it.

I tested my tls certificate against the chain.crt using openssl and the answer is ok.

I'm not sure where else to look to make it work.

I tested my code against the test network and it works, also using the old node SDK the connection works too.

This is my code (deleting err handling for readability):

func NewGrpcConnection(fabricConfig config.FabricConfig) (*grpc.ClientConn, error) {
    caCertificate, err := loadCertificate(fabricConfig.TlsCaPath)
    keypair, err := loadCertificateKeyPair(fabricConfig.TlsCertPath, fabricConfig.TlsKeyPath)

    certPool := x509.NewCertPool()
    certPool.AddCert(caCertificate)

    transportCredentials := credentials.NewTLS(&tls.Config{
        ServerName:   fabricConfig.HostName,
        RootCAs:      certPool,
        ClientCAs:    certPool,
        Certificates: []tls.Certificate{*keypair},
    })

    connection, err := grpc.Dial(fabricConfig.PeerEndpoint, grpc.WithTransportCredentials(transportCredentials))
    return connection, nil
}

func NewGateway(clientConnection *grpc.ClientConn, fabricConfig config.FabricConfig) (*client.Gateway, error) {
    id, err := newIdentity(fabricConfig.CertPath, fabricConfig.OrgName)
    sign, err := newSign(fabricConfig.KeyPath)

    gateway, err := client.Connect(
        id,
        client.WithSign(sign),
        client.WithClientConnection(clientConnection),
        client.WithEvaluateTimeout(5*time.Second),
        client.WithEndorseTimeout(15*time.Second),
        client.WithSubmitTimeout(5*time.Second),
        client.WithCommitStatusTimeout(1*time.Minute),
    )

    return gateway, err
}

func main() {
    grpcConnection, err := fabric.NewGrpcConnection(*clientConfig.FabricConfig)
    defer grpcConnection.Close()
   
    gateway, err := fabric.NewGateway(grpcConnection, *clientConfig.FabricConfig)
    defer gateway.Close()

    network := gateway.GetNetwork(clientConfig.FabricConfig.ChannelName)
    contract := network.GetContract(clientConfig.FabricConfig.ContractName)

result, err = contract.SubmitTransaction(request.Fcn, request.Args...)
}

I tried also overiding the pod's ssl certificates by adding the ca certificates to /usr/local/share/ca-certificates/ and using the update-ca-certificates command with no luck.

What am I doing wrong? Any ideas where else to look? do I need to add the tls-root and ica certificates to the msp folder under msp/
tlscacerts and msp/tlsintermediatecerts too? I have FABRIC_LOGGING_SPEC="grpc=debug:info" is there a way to see what certificate is comming in for validation?

Thanks,

Ana Franco
Tech Leader


Yacov
 

On second thought, you said that code works? How come when you only add a single certificate and you have two in the chain?

From: Yacov Manevich <YACOVM@...>
Sent: Thursday, February 10, 2022 4:42 PM
To: afrancoc2000@... <afrancoc2000@...>; fabric@... <fabric@...>
Subject: Re: [EXTERNAL] [Hyperledger Fabric] Hyperledger Fabric Gateway fails with certificate signed by unknown authority but certificates pass an openssl test #hyperledger-fabric #tls #fabric-sdk-go
 
you have an interediate CA but you use AddCert which receives a single certificate object (not a PEM but literally the x509 certificate).
you need to add both root CA and intermediate CA. Easiest is to use AppendCertsFromPEM on the content of the concatenated file.

From: fabric@... <fabric@...> on behalf of afrancoc2000@... <afrancoc2000@...>
Sent: Thursday, February 10, 2022 2:38 PM
To: fabric@... <fabric@...>
Subject: [EXTERNAL] [Hyperledger Fabric] Hyperledger Fabric Gateway fails with certificate signed by unknown authority but certificates pass an openssl test #hyperledger-fabric #tls #fabric-sdk-go
 
Hi Team, I'm having trouble making the "Hyperledger Fabric Gateway for Go" execute a smart contract, although I am using it successfully for listening to events. This is the error I get in the client: Submit error with gRPC status DeadlineExceeded: ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Team, 

I'm having trouble making the "Hyperledger Fabric Gateway for Go" execute a smart contract, although I am using it successfully for listening to events.

This is the error I get in the client:

Submit error with gRPC status DeadlineExceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded

This is the error I get in the orderer:

Server TLS handshake failed in 2.628549ms with error remote error: tls: bad certificate server=Orderer remoteaddress=10.250.37.115:40858

This is the error I get in the peer:

Client TLS handshake failed after 6.156747ms with error: x509: certificate signed by unknown authority remoteaddress=10.250.37.195:443

The IP 10.250.37.195 corresponds to the ingress service external IP and the 10.250.37.115 corresponds to the ingress controller. The ingress controller has ssl-passtrough configured so the certificates are not being overridden by it.

I'm using a chain certificate that begins with an ica cert and then on the same file the root certificate, this chain.crt file is stored in the orderers in this folder: "/var/hyperledger/orderer/tls/chain.crt" both ORDERER_GENERAL_TLS_ROOTCAS and ORDERER_GENERAL_TLS_CLIENTROOTCAS point to it.

I tested my tls certificate against the chain.crt using openssl and the answer is ok.

I'm not sure where else to look to make it work.

I tested my code against the test network and it works, also using the old node SDK the connection works too.

This is my code (deleting err handling for readability):

func NewGrpcConnection(fabricConfig config.FabricConfig) (*grpc.ClientConn, error) {
    caCertificate, err := loadCertificate(fabricConfig.TlsCaPath)
    keypair, err := loadCertificateKeyPair(fabricConfig.TlsCertPath, fabricConfig.TlsKeyPath)

    certPool := x509.NewCertPool()
    certPool.AddCert(caCertificate)

    transportCredentials := credentials.NewTLS(&tls.Config{
        ServerName:   fabricConfig.HostName,
        RootCAs:      certPool,
        ClientCAs:    certPool,
        Certificates: []tls.Certificate{*keypair},
    })

    connection, err := grpc.Dial(fabricConfig.PeerEndpoint, grpc.WithTransportCredentials(transportCredentials))
    return connection, nil
}

func NewGateway(clientConnection *grpc.ClientConn, fabricConfig config.FabricConfig) (*client.Gateway, error) {
    id, err := newIdentity(fabricConfig.CertPath, fabricConfig.OrgName)
    sign, err := newSign(fabricConfig.KeyPath)

    gateway, err := client.Connect(
        id,
        client.WithSign(sign),
        client.WithClientConnection(clientConnection),
        client.WithEvaluateTimeout(5*time.Second),
        client.WithEndorseTimeout(15*time.Second),
        client.WithSubmitTimeout(5*time.Second),
        client.WithCommitStatusTimeout(1*time.Minute),
    )

    return gateway, err
}

func main() {
    grpcConnection, err := fabric.NewGrpcConnection(*clientConfig.FabricConfig)
    defer grpcConnection.Close()
   
    gateway, err := fabric.NewGateway(grpcConnection, *clientConfig.FabricConfig)
    defer gateway.Close()

    network := gateway.GetNetwork(clientConfig.FabricConfig.ChannelName)
    contract := network.GetContract(clientConfig.FabricConfig.ContractName)

result, err = contract.SubmitTransaction(request.Fcn, request.Args...)
}

I tried also overiding the pod's ssl certificates by adding the ca certificates to /usr/local/share/ca-certificates/ and using the update-ca-certificates command with no luck.

What am I doing wrong? Any ideas where else to look? do I need to add the tls-root and ica certificates to the msp folder under msp/
tlscacerts and msp/tlsintermediatecerts too? I have FABRIC_LOGGING_SPEC="grpc=debug:info" is there a way to see what certificate is comming in for validation?

Thanks,

Ana Franco
Tech Leader


afrancoc2000@...
 

Hi Yacov, well it works on the testnetwork where there's no intermediate CA file. you are right maybe that's the problem let me test it.
Thanks!


afrancoc2000@...
 

This is the code I'm using to create the CA certificate: 

func loadCertificate(filename string) (*x509.Certificate, error) {
    certificatePEM, err := ioutil.ReadFile(filename)
    if err != nil {
        return nil, fmt.Errorf(ErrorReadingFile, err)
    }
    return identity.CertificateFromPEM(certificatePEM)
}

 


afrancoc2000@...
 

Hi Yacov, 

The change didn't work I'm getting the same errors, "certificate signed by unknown authority" on the peer, "tls: bad certificate" on the orderer. what I find weird is that it looks like the error is in the comunication between peers and orderers and this works using the node client.


afrancoc2000@...
 

I'm going to try to upload them individually


afrancoc2000@...
 

No, still the same, I verified that both certificates go into the certpool but I'm still getting the same error


afrancoc2000@...
 

I'm also getting this message:
[gateway] orderers -> Failed to connect to orderer address=orderer2.blockchainnp.company.com.co:443 err="failed to create new connection: context deadline exceeded"
2022-02-10 17:31:30.461 UTC 0b95 INFO [comm.grpc.server] 1 -> unary call completed grpc.service=gateway.Gateway grpc.method=Submit grpc.request_deadline=2022-02-10T17:30:35.43Z grpc.peer_address=10.250.37.48:44406 grpc.peer_subject="CN=admin-tls@...,OU=admin,O=company,L=MEDELLIN,ST=ANTIOQUIA,C=CO" error="rpc error: code = Unavailable desc = no orderer nodes available" grpc.code=Unavailable grpc.call_duration=1m0.030407308s.

That's why I think the problem is between peer and orderer, the weird thing is that it only happens using the new gateway.


Yacov
 

  1. Can you send transactions to the orderer from a client without the gateway?
  1. Do peers get blocks from the orderers?



From: fabric@... <fabric@...> on behalf of afrancoc2000@... <afrancoc2000@...>
Sent: Thursday, February 10, 2022 7:36 PM
To: fabric@... <fabric@...>
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Hyperledger Fabric Gateway fails with certificate signed by unknown authority but certificates pass an openssl test #hyperledger-fabric #tls #fabric-sdk-go
 
I'm also getting this message: [gateway] orderers -> Failed to connect to orderer address=orderer2.blockchainnp.company.com.co:443 err="failed to create new connection: context deadline exceeded" 2022-02-10 17:31:30.461 UTC 0b95 INFO [comm.grpc.server] ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I'm also getting this message:
[gateway] orderers -> Failed to connect to orderer address=orderer2.blockchainnp.company.com.co:443 err="failed to create new connection: context deadline exceeded"
2022-02-10 17:31:30.461 UTC 0b95 INFO [comm.grpc.server] 1 -> unary call completed grpc.service=gateway.Gateway grpc.method=Submit grpc.request_deadline=2022-02-10T17:30:35.43Z grpc.peer_address=10.250.37.48:44406 grpc.peer_subject="CN=admin-tls@...,OU=admin,O=company,L=MEDELLIN,ST=ANTIOQUIA,C=CO" error="rpc error: code = Unavailable desc = no orderer nodes available" grpc.code=Unavailable grpc.call_duration=1m0.030407308s.

That's why I think the problem is between peer and orderer, the weird thing is that it only happens using the new gateway.


afrancoc2000@...
 

1. Yes I can, I'm able to execute the contract directly from the console using "peer chaincode invoke" and using a client with the old node js SDK. Both use mutual TLS.
2. Well I'm not completely sure what you mean, isn't that the way it's supposed to work? do you want me to check a property from the Core.yaml or the Orderer.yaml files? I was looking at this post because one of my peers is getting behind from time to time, I haven't changed anything yet, but I think it that the peers do get blocks from the orderer, and that's why it's getting behind. What I can tell you for sure is that inside both the peers and orderers all the blocks exist.


Yacov
 

I am simply asking to understand if the peers can connect to the orderers to pull blocks, to understand whether the problem is only in the gateway or in all communications between the peer and the orderers

From: fabric@... <fabric@...> on behalf of afrancoc2000@... <afrancoc2000@...>
Sent: Thursday, February 10, 2022 8:41 PM
To: fabric@... <fabric@...>
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Hyperledger Fabric Gateway fails with certificate signed by unknown authority but certificates pass an openssl test #hyperledger-fabric #tls #fabric-sdk-go
 
1. Yes I can, I'm able to execute the contract directly from the console using "peer chaincode invoke" and using a client with the old node js SDK. Both use mutual TLS. 2. Well I'm not completely sure what you mean, isn't that the way it's supposed ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
1. Yes I can, I'm able to execute the contract directly from the console using "peer chaincode invoke" and using a client with the old node js SDK. Both use mutual TLS.
2. Well I'm not completely sure what you mean, isn't that the way it's supposed to work? do you want me to check a property from the Core.yaml or the Orderer.yaml files? I was looking at this post because one of my peers is getting behind from time to time, I haven't changed anything yet, but I think it that the peers do get blocks from the orderer, and that's why it's getting behind. What I can tell you for sure is that inside both the peers and orderers all the blocks exist.


afrancoc2000@...
 

So the issue was resolved with the help of the Hyperledger team here
Thank you, all