Permission denied from "peer channel create" command using certificate generated by third-party certificate authority (HyperLedger Fabric 2.2.2) #tls #signcerts


Robert Broeckelmann
 

Hello. Thanks in advance.

For a while now (at least two years), we've had the following channel create command working (originally with HLF 1.4.x and more recently with HLF 2.2.2):

docker exec -e CORE_PEER_LOCALMSPID=Org1MSP -e CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin@.../msp peer0.org1.example.org peer channel create -o orderer0.example.org:7050 -c example-channel -f /etc/hyperledger/configtx/channel.tx --tls --cafile /var/hyperledger/orderer/tls/ca.crt --certfile /etc/hyperledger/fabric/tls/server.crt --keyfile /etc/hyperledger/fabric/tls/server.key --clientauth

I've changed the names of certain configuration elements to avoid references to the client.

Recently, we replaced our crypto material with certificate/keys that were generated by a third-party certificate authority product--Hashicorp Vault. The certificate Subject DN naming conventions are preserved from the standpoint of OU--well, it's close. I am attempting to create a new network from scratch. So, configtxgen is run to create a new genesis block. The peer & orderer successfully start, but when we go to run this first command in our configuration script, we get the following error:

2021-05-10 17:03:55.867 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Error: got unexpected status: FORBIDDEN -- implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Writers' sub-policies to be satisfied: permission denied

The PEER_MSPCONFIGPATH value above points at a directory structure that matches what is generated by cryptogen, but the CA, key, and certificate files have been updated with information for our new private CA and certificates.The TLS client certificate and key files mentioned in the command arguments above are for the peer. The peer is in a separate organization from the orderer cluster.

The orderer logs contains the following (I had to sanitize the log, but I replaced the actual names with Org1, Org2, etc, and any domain names with example.org):

2021-05-10 23:09:14.397 UTC [msp] newBccspMsp -> DEBU 1d8 Creating BCCSP-based MSP instance

2021-05-10 23:09:14.398 UTC [msp] New -> DEBU 1d9 Creating Cache-MSP instance

2021-05-10 23:09:14.398 UTC [msp] Setup -> DEBU 1da Setting up MSP instance OrdererMSP

2021-05-10 23:09:14.398 UTC [msp.identity] newIdentity -> DEBU 1db Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIIChDCCAiugAw...

-----END CERTIFICATE-----

2021-05-10 23:09:14.398 UTC [msp.identity] newIdentity -> DEBU 1dc Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIIDzzCCA3WgAwIBA...

-----END CERTIFICATE-----

2021-05-10 23:09:14.398 UTC [msp] hasOURole -> DEBU 1dd MSP OrdererMSP checking if the identity is a client

2021-05-10 23:09:14.398 UTC [msp] getCertificationChain -> DEBU 1de MSP OrdererMSP getting certification chain

2021-05-10 23:09:14.399 UTC [msp] hasOURole -> DEBU 1df MSP OrdererMSP checking if the identity is a client

2021-05-10 23:09:14.399 UTC [msp] newBccspMsp -> DEBU 1e0 Creating BCCSP-based MSP instance

2021-05-10 23:09:14.399 UTC [msp] New -> DEBU 1e1 Creating Cache-MSP instance

2021-05-10 23:09:14.399 UTC [msp] Setup -> DEBU 1e2 Setting up MSP instance Org1MSP

2021-05-10 23:09:14.399 UTC [msp.identity] newIdentity -> DEBU 1e3 Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIICkTCCAjegAw…

-----END CERTIFICATE-----

2021-05-10 23:09:14.400 UTC [msp.identity] newIdentity -> DEBU 1e4 Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIIEBjCCA6ygAwIBAgIUN...

-----END CERTIFICATE-----

2021-05-10 23:09:14.401 UTC [msp] hasOURole -> DEBU 1e5 MSP Org1MSP checking if the identity is a client

2021-05-10 23:09:14.401 UTC [msp] getCertificationChain -> DEBU 1e6 MSP Org1MSP getting certification chain

2021-05-10 23:09:14.401 UTC [msp] hasOURole -> DEBU 1e7 MSP Org1MSP checking if the identity is a client

2021-05-10 23:09:14.401 UTC [msp] newBccspMsp -> DEBU 1e8 Creating BCCSP-based MSP instance

2021-05-10 23:09:14.401 UTC [msp] New -> DEBU 1e9 Creating Cache-MSP instance

2021-05-10 23:09:14.401 UTC [msp] Setup -> DEBU 1ea Setting up MSP instance Org2MSP

2021-05-10 23:09:14.401 UTC [msp.identity] newIdentity -> DEBU 1eb Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIICojCCAkmgAwIBAgIUQnfd…

-----END CERTIFICATE-----

2021-05-10 23:09:14.402 UTC [msp.identity] newIdentity -> DEBU 1ec Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIIEWzCCBAGgAwIBAgIU…

-----END CERTIFICATE-----

2021-05-10 23:09:14.402 UTC [msp] hasOURole -> DEBU 1ed MSP Org2MSP checking if the identity is a client

2021-05-10 23:09:14.402 UTC [msp] getCertificationChain -> DEBU 1ee MSP Org2MSP getting certification chain

2021-05-10 23:09:14.402 UTC [msp] hasOURole -> DEBU 1ef MSP Org2MSP checking if the identity is a client

2021-05-10 23:09:14.402 UTC [msp] newBccspMsp -> DEBU 1f0 Creating BCCSP-based MSP instance

2021-05-10 23:09:14.402 UTC [msp] New -> DEBU 1f1 Creating Cache-MSP instance

2021-05-10 23:09:14.402 UTC [msp] Setup -> DEBU 1f2 Setting up MSP instance Org3MSP

2021-05-10 23:09:14.403 UTC [msp.identity] newIdentity -> DEBU 1f3 Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIIClDCCAjmgAwIBAgIUd…

-----END CERTIFICATE-----

2021-05-10 23:09:14.404 UTC [msp.identity] newIdentity -> DEBU 1f4 Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIIEDjCCA7WgAwIBAg...

-----END CERTIFICATE-----

2021-05-10 23:09:14.404 UTC [msp] hasOURole -> DEBU 1f5 MSP Org3MSP checking if the identity is a client

2021-05-10 23:09:14.404 UTC [msp] getCertificationChain -> DEBU 1f6 MSP Org3MSP getting certification chain

2021-05-10 23:09:14.404 UTC [msp] hasOURole -> DEBU 1f7 MSP Org3MSP checking if the identity is a client

2021-05-10 23:09:14.404 UTC [msp] newBccspMsp -> DEBU 1f8 Creating BCCSP-based MSP instance

2021-05-10 23:09:14.404 UTC [msp] New -> DEBU 1f9 Creating Cache-MSP instance

2021-05-10 23:09:14.404 UTC [msp] Setup -> DEBU 1fa Setting up MSP instance Org4MSP

2021-05-10 23:09:14.404 UTC [msp.identity] newIdentity -> DEBU 1fb Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIICkTCCAjegAwIBAgIUcckxGcnYwCQ...

-----END CERTIFICATE-----

2021-05-10 23:09:14.406 UTC [msp.identity] newIdentity -> DEBU 1fc Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIIEBTCCA6ugAwIBAgI...

-----END CERTIFICATE-----

2021-05-10 23:09:14.406 UTC [msp] hasOURole -> DEBU 1fd MSP Org4MSP checking if the identity is a client

2021-05-10 23:09:14.406 UTC [msp] getCertificationChain -> DEBU 1fe MSP Org4MSP getting certification chain

2021-05-10 23:09:14.406 UTC [msp] hasOURole -> DEBU 1ff MSP Org4MSP checking if the identity is a client

2021-05-10 23:09:14.406 UTC [msp] Setup -> DEBU 200 Setting up the MSP manager (5 msps)

2021-05-10 23:09:14.406 UTC [msp] Setup -> DEBU 201 MSP manager setup complete, setup 5 msps

2021-05-10 23:09:14.407 UTC [msp] DeserializeIdentity -> DEBU 202 Obtaining identity

2021-05-10 23:09:14.407 UTC [msp.identity] newIdentity -> DEBU 203 Creating identity instance for cert -----BEGIN CERTIFICATE-----

MIIEBTCCA6ygAwIBAgIUN/TW0leZER...

-----END CERTIFICATE-----

2021-05-10 23:09:14.407 UTC [msp.identity] Verify -> DEBU 204 Verify: digest = 00000000  98 a6 d0 82 56 95 eb eb  02 1b 73 d3 f1 22 02 50  |....V.....s..".P|

00000010  58 e1 b8 2c 21 bd 23 1a  75 1a 7a d4 fa f4 91 fc  |X..,!.#.u.z.....|

2021-05-10 23:09:14.407 UTC [msp.identity] Verify -> DEBU 205 Verify: sig = 00000000  30 44 02 20 58 c6 c5 27  40 65 3c d3 45 90 f9 03  |0D. X..'@e<.E...|

00000010  44 09 8a e9 5f 1c a3 69  64 47 27 e4 d5 bd 85 16  |D..._..idG'.....|

00000020  d0 73 33 6a 02 20 36 de  d3 54 58 df 20 b2 bf ec  |.s3j. 6..TX. ...|

00000030  83 89 98 2c 52 60 a3 ce  72 29 5d dc 19 7e 62 03  |...,R`..r)]..~b.|

00000040  3b b4 12 69 b6 8c                                 |;..i..|

2021-05-10 23:09:14.407 UTC [msp.identity] Verify -> DEBU 206 Verify: digest = 00000000  98 a6 d0 82 56 95 eb eb  02 1b 73 d3 f1 22 02 50  |....V.....s..".P|

00000010  58 e1 b8 2c 21 bd 23 1a  75 1a 7a d4 fa f4 91 fc  |X..,!.#.u.z.....|

2021-05-10 23:09:14.407 UTC [msp.identity] Verify -> DEBU 207 Verify: sig = 00000000  30 44 02 20 58 c6 c5 27  40 65 3c d3 45 90 f9 03  |0D. X..'@e<.E...|

00000010  44 09 8a e9 5f 1c a3 69  64 47 27 e4 d5 bd 85 16  |D..._..idG'.....|

00000020  d0 73 33 6a 02 20 36 de  d3 54 58 df 20 b2 bf ec  |.s3j. 6..TX. ...|

00000030  83 89 98 2c 52 60 a3 ce  72 29 5d dc 19 7e 62 03  |...,R`..r)]..~b.|

00000040  3b b4 12 69 b6 8c                                 |;..i..|

2021-05-10 23:09:14.407 UTC [msp] satisfiesPrincipalInternalV142 -> DEBU 208 Checking if identity has been named explicitly as an admin for Org1MSP

2021-05-10 23:09:14.408 UTC [msp.identity] Sign -> DEBU 209 Sign: plaintext: ...

2021-05-10 23:09:14.408 UTC [msp.identity] Sign -> DEBU 20a Sign: digest: 98343EBCEE7E1709A1D974F702D0018881D4FFF173062DA05D5E09AECBE5A235

2021-05-10 23:09:14.408 UTC [msp.identity] Sign -> DEBU 20b Sign: plaintext: ...

2021-05-10 23:09:14.408 UTC [msp.identity] Sign -> DEBU 20c Sign: digest: C45BC19C1D6C873C1300A363C2ED9BFB0734C118EB97149289DF8F55143FE76F

2021-05-10 23:09:14.409 UTC [msp.identity] Verify -> DEBU 20d Verify: digest = 00000000  c4 5b c1 9c 1d 6c 87 3c  13 00 a3 63 c2 ed 9b fb  |.[...l.<...c....|

00000010  07 34 c1 18 eb 97 14 92  89 df 8f 55 14 3f e7 6f  |.4.........U.?.o|

2021-05-10 23:09:14.409 UTC [msp.identity] Verify -> DEBU 20e Verify: sig = 00000000  30 44 02 20 31 55 c3 76  31 ef a6 93 f6 cf 73 31  |0D. 1U.v1.....s1|

00000010  d8 86 3a 6f 7d 3a 97 d6  a4 91 82 6b 84 2c a5 fc  |..:o}:.....k.,..|

00000020  44 4b 67 9a 02 20 47 69  6a 31 f0 8b db fd 33 7a  |DKg.. Gij1....3z|

00000030  28 89 e7 f9 d3 fc b3 5e  b6 18 33 ed c3 c8 3a 24  |(......^..3...:$|

00000040  23 80 91 90 00 7c                                 |#....||

2021-05-10 23:09:14.409 UTC [orderer.common.broadcast] ProcessMessage -> WARN 20f [channel: Org1-channel] Rejecting broadcast of config message from 172.28.0.24:38316 because of error: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Writers' sub-policies to be satisfied: permission denied

2021-05-10 23:09:14.409 UTC [comm.grpc.server] 1 -> INFO 210 streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Broadcast grpc.peer_address=172.28.0.24:38316 grpc.peer_subject="CN=Admin@...,OU=admin,O=Example Org,L=Somewhere,ST=WA,C=US" grpc.code=OK grpc.call_duration=11.582222ms

2021-05-10 23:09:14.411 UTC [common.deliver] Handle -> WARN 211 Error reading from 172.28.0.24:38314: rpc error: code = Canceled desc = context canceled

2021-05-10 23:09:14.411 UTC [comm.grpc.server] 1 -> INFO 212 streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=172.28.0.24:38314 grpc.peer_subject="CN=Admin@...,OU=admin,O=Example Org,L=Somewhere,ST=WA,C=US" error="rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.call_duration=17.109561ms

In our original cryptogen setup, we did not have a config.yaml file on the orderer. I tried adding:

/etc/hyperledger/msp/orderer/msp/config.yaml
NodeOUs:
  Enable: true
  ClientOUIdentifier:
    Certificate: cacerts/ca.example.com-cert.pem
    OrganizationalUnitIdentifier: client
  PeerOUIdentifier:
    Certificate: cacerts/ca.example.com-cert.pem
    OrganizationalUnitIdentifier: peer
  AdminOUIdentifier:
    Certificate: cacerts/ca.example.com-cert.pem
    OrganizationalUnitIdentifier: admin

The result is the same.

Any idea what I might be doing wrong?

Thank you for your time in advance.

Join fabric@lists.hyperledger.org to automatically receive all group messages.