Topics

Adding a new organisation to the system channel fails after FAB-17733 #fabric #raft #fabric-orderer


@chbtt
 

Hey,

we are using an architecture where there is a TLS root CA and each organisation has its own intermediate TLS CA which is an immediate child to the TLS root CA (realized with fabric-ca v1.4.7). We only want to specify the TLS root CA certificate for any connection within the network (e.g. for "peer channel update --cafile") and have had some trouble achieving that in the first place. Our workaround is to append their respective intermediate TLS CA certificates to all peer/orderer TLS certificates building the proper chain of trust up to the TLS root CA certificate.
Now, when checking if we could upgrade to Hyperledger Fabric v2.1.1 (from 2.1.0), updating the system channel with a new organisation (including a new orderer) fails with the error "x509: certificate signed by unknown authority". We made sure that "tls_root_certs" and "tls_intermediate_certs" within "channel_group.groups.Orderer.groups.newOrg.values.MSP.value" both contain the correct certificates, but still faced the same error. We believe to have tracked down the problem to FAB-17733 which adds a certificate check when adding a new consenter to the raft configuration. Our configuration update for the system channel contains both the "channel_group.groups.Orderer.groups" definition for the new organisation as well as the new orderer in "channel_group.groups.Orderer.values.ConsensusType.consenters". Could this be a problem? Should we first update the orderer group definition and then add the new consenter?
Any pointers towards a fix are greatly appreciated. We would also love to learn more about how peers/orderers handle TLS connections - especially how they handle multiple CAs in their chain of trust - to be able to move away from appending intermediate TLS CA certificates to build the chain of trust manually.

Thanks,
Christoph


Yacov
 

Hi.
  1. Yes, with FAB-17733 implemented you need to do it in 2 stages- first expand the organizations and then add the new consenter.
  2. I guess it's possible to address your problem by speculatively looking at how the root TLS CA will look like after applying the config and not before it. Please open a new JIRA? :-)
  3. I'm not sure why you can't specify the intermediate certificates in the channel config for each organization?  



From:        "christoph.buttler via lists.hyperledger.org" <christoph.buttler=ruhr-uni-bochum.de@...>
To:        fabric@...
Date:        06/16/2020 03:57 AM
Subject:        [EXTERNAL] [Hyperledger Fabric] Adding a new organisation to the system channel fails after FAB-17733 #fabric #fabric-orderer #raft
Sent by:        fabric@...




Hey,

we are using an architecture where there is a TLS root CA and each organisation has its own intermediate TLS CA which is an immediate child to the TLS root CA (realized with fabric-ca v1.4.7). We only want to specify the TLS root CA certificate for any connection within the network (e.g. for "peer channel update --cafile") and have had some trouble achieving that in the first place. Our workaround is to append their respective intermediate TLS CA certificates to all peer/orderer TLS certificates building the proper chain of trust up to the TLS root CA certificate.
Now, when checking if we could upgrade to Hyperledger Fabric v2.1.1 (from 2.1.0), updating the system channel with a new organisation (including a new orderer) fails with the error "x509: certificate signed by unknown authority". We made sure that "tls_root_certs" and "tls_intermediate_certs" within "channel_group.groups.Orderer.groups.newOrg.values.MSP.value" both contain the correct certificates, but still faced the same error. We believe to have tracked down the problem to FAB-17733 which adds a certificate check when adding a new consenter to the raft configuration. Our configuration update for the system channel contains both the "channel_group.groups.Orderer.groups" definition for the new organisation as well as the new orderer in "channel_group.groups.Orderer.values.ConsensusType.consenters". Could this be a problem? Should we first update the orderer group definition and then add the new consenter?
Any pointers towards a fix are greatly appreciated. We would also love to learn more about how peers/orderers handle TLS connections - especially how they handle multiple CAs in their chain of trust - to be able to move away from appending intermediate TLS CA certificates to build the chain of trust manually.

Thanks,
Christoph





@chbtt
 

Hey Yacov,

thanks for your quick reply. As suggested, I have opened a new JIRA (https://jira.hyperledger.org/browse/FAB-17998).
Regarding (3.), we have already added the intermediate certificates to the channel configuration. Let me try to give a proper explanation on what we want to achieve:
Suppose we have an orderer with a TLS chain "orderer.crt -> intermediate.crt -> root.crt" and want to create a new channel with "peer channel create". Now, if we are contacting the organisations orderer, we could simply specify "--cafile intermediate.crt" and the TLS handshake would succeed. However, if we are contacting another organisations orderer, we do not want look up its "intermediate.crt" within the channel config. We would like to be able to specify "--cafile root.crt" no matter which orderer we are contacting. For some reason, this always results in a failed TLS handshake. We would expect the orderer to supply its full chain of certificates (or at least "orderer.crt -> intermediate.crt") in the handshake, but can not figure out how to achieve this. As mentioned, our workaround is to build the TLS chain through manually appending "intermediate.crt" to "orderer.crt". Is there a way for the orderer/peer to do this automatically? Maybe through the fabric configuration files orderer.yaml/core.yaml?

Thanks,
Christoph


Yacov
 

I don't see why you don't want to specify the intermediate certificate, but - your workaround looks fine to me assuming it works.

To do what you want (just specify the root cert and not the intermediate) would require the TLS server handshake to send the full validation chain which happens only if we specify the validation chain itself in the TLS config,
which we do not do: https://github.com/hyperledger/fabric/blob/master/internal/pkg/comm/server.go#L70-L95

therefore I find it hard to believe that the workaround you mention indeed works (unless I am misinterpreting your workaround - i assume you mean you do it in the server side and not in the client side)



From:        "christoph.buttler via lists.hyperledger.org" <christoph.buttler=ruhr-uni-bochum.de@...>
To:        fabric@...
Date:        06/16/2020 04:05 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Adding a new organisation to the system channel fails after FAB-17733 #fabric #fabric-orderer #raft
Sent by:        fabric@...




Hey Yacov,

thanks for your quick reply. As suggested, I have opened a new JIRA (https://jira.hyperledger.org/browse/FAB-17998).
Regarding (3.), we have already added the intermediate certificates to the channel configuration. Let me try to give a proper explanation on what we want to achieve:
Suppose we have an orderer with a TLS chain "orderer.crt -> intermediate.crt -> root.crt" and want to create a new channel with "peer channel create". Now, if we are contacting the organisations orderer, we could simply specify "--cafile intermediate.crt" and the TLS handshake would succeed. However, if we are contacting another organisations orderer, we do not want look up its "intermediate.crt" within the channel config. We would like to be able to specify "--cafile root.crt" no matter which orderer we are contacting. For some reason, this always results in a failed TLS handshake. We would expect the orderer to supply its full chain of certificates (or at least "orderer.crt -> intermediate.crt") in the handshake, but can not figure out how to achieve this. As mentioned, our workaround is to build the TLS chain through manually appending "intermediate.crt" to "orderer.crt". Is there a way for the orderer/peer to do this automatically? Maybe through the fabric configuration files orderer.yaml/core.yaml?

Thanks,
Christoph





@chbtt
 

We are indeed doing this on the server side (see attached certificates). If you ever wonder how it can work at all, I have put together a small example environment of what this would look like.

Thank you very much for your insights!


Yacov
 

oh I see now why it works, because tls.X509KeyPair    https://github.com/hyperledger/fabric/blob/master/internal/pkg/comm/server.go#L74
iterates over the PEM and just put multiple certificates into the tls.Certificate object https://golang.org/src/crypto/tls/tls.go

So, basically this seems fine to me....



From:        "christoph.buttler via lists.hyperledger.org" <christoph.buttler=ruhr-uni-bochum.de@...>
To:        fabric@...
Date:        06/16/2020 11:45 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Adding a new organisation to the system channel fails after FAB-17733 #fabric #fabric-orderer #raft
Sent by:        fabric@...




One or more of the following files ( example_env.zip ) violates IBM policy and all attachment(s) have been removed from the message.

We are indeed doing this on the server side (see attached certificates). If you ever wonder how it can work at all, I have put together a small example environment of what this would look like.

Thank you very much for your insights!