Consenter Certificate Updates When Using a Third-Party CA


Robert Broeckelmann
 

I have a blockchain network with four organizations working with crypto material generated with cryptogen. I also have it working with crypto material generated from a third-party certificate authority that's introduced during initial creation of the network. Using the same third-party CA, I'm now trying to update issuing CAs and regenerate all the certificates (out of the new issuing CAs) on an existing blockchain network. I have created a script that updates the TLS & Digital Signature Issuing CA certificates for the OrdererOrg in an existing blockchain network. The Issuing CA certificates are updated in the system-channel and the application channel. This appears to be successful. I generate new certificates (TLS, Digital Signature, Admin) and update the channel configuration for each channel. THat also appears to be successful. Then, I have to update the consenter certificates in each channel. I update the consenter certificates one orderer at a time, in its own transaction. To be more specific, the following updates are made in each transaction for the consenter certificates:

system-channel:
  cat ${TMP_DIR}/config.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].client_tls_cert = "'${ENCODED_CERT}'" |
        .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].server_tls_cert = "'${ENCODED_CERT}'"' \
       > ${TMP_DIR}/modified-${neworg}-root.json

app-channel:
  cat ${TMP_DIR}/config.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].client_tls_cert = "'${ENCODED_CERT}'" |
        .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].server_tls_cert = "'${ENCODED_CERT}'"' \
        > ${TMP_DIR}/modified-${neworg}-root.json

There are five orderers in my OrdererOrg using the RAFT Orderer. For app-channel, Block 12 contains the configuration update transaction to update the consenter certificate for orderer0, Block 13 contains the transaction for updating the consenter certificate info for orderer1, etc,etc to Block 16 for consenter certificate of orderer4.

The system-channel transactions seem to be validated without issue. For the app-channel, all transactions that update the consenter certificates produce the following error on each peer:

[[34m2022-01-31 20:29:09.642 UTC [gossip.privdata] StoreBlock -> INFO 032^[[0m Received block [12] from buffer channel=app-channel^M
^[[31m2022-01-31 20:29:09.644 UTC [protoutils] ValidateTransaction -> ERRO 033^[[0m checkSignatureFromCreator returns err creator certificate is not valid: could not validate identity's OUs: certifiersIdentifier does not match: [orderer(6BF380B427FE7D15)], MSP: [OrdererMSP]^M
^[[31m2022-01-31 20:29:09.644 UTC [committer.txvalidator] validateTx -> ERRO 034^[[0m Invalid transaction with index 0^M
^[[34m2022-01-31 20:29:09.644 UTC [committer.txvalidator] Validate -> INFO 035^[[0m [app-channel] Validated block [12] in 2ms^M
^[[33m2022-01-31 20:29:09.644 UTC [validation] preprocessProtoBlock -> WARN 036^[[0m Channel [app-channel]: Block [12] Transaction index [0] TxId [] marked as invalid by committer. Reason code [BAD_CREATOR_SIGNATURE]^M
^[[34m2022-01-31 20:29:09.669 UTC [kvledger] CommitLegacy -> INFO 037^[[0m [app-channel] Committed block [12] with 1 transaction(s) in 24ms (state_validation=0ms block_and_pvtdata_commit=7ms state_commit=15ms) commitHash=[64eccde911898a82e6d767b4ca13099c3d5594bde5d418bb1cb883d339b8cd94]^M

What does the hex # in "orderer(6BF380B427FE7D15)" represent?

My config.yaml on the peers and orderers contains:

NodeOUs:
  Enable: true
  ClientOUIdentifier:
    Certificate: cacerts/ca.org1.example.org-cert.pem
    OrganizationalUnitIdentifier: client
  PeerOUIdentifier:
    Certificate: cacerts/ca.org2.example.org-cert.pem
    OrganizationalUnitIdentifier: peer
  AdminOUIdentifier:
    Certificate: cacerts/ca.org3.example.org-cert.pem
    OrganizationalUnitIdentifier: admin
  OrdererOUIdentifier:
    Certificate: cacerts/ca.org4.org4.example.org-cert.pem
    OrganizationalUnitIdentifier: orderer

For the consenter (TLS) certificate for the orderers, I've tried with OU=orderer and no OU in the Subject DN. The results were the same.

I'm not sure if there is something wrong with the certificates I'm generating, the order I'm updating the channel configuration for the consenter certificates, or the config changes I'm making. Any ideas?


Yacov
 

Maybe on the application channel, the orderer MSP has an OU configured and your identity you used to submit the transaction was issued by a CA with a different OU?

From: fabric@... <fabric@...> on behalf of Robert Broeckelmann <robert@...>
Sent: Tuesday, February 1, 2022 8:45 PM
To: fabric@... <fabric@...>
Subject: [EXTERNAL] [Hyperledger Fabric] Consenter Certificate Updates When Using a Third-Party CA
 
I have a blockchain network with four organizations working with crypto material generated with cryptogen. I also have it working with crypto material generated from a third-party certificate authority that's introduced during initial creation ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I have a blockchain network with four organizations working with crypto material generated with cryptogen. I also have it working with crypto material generated from a third-party certificate authority that's introduced during initial creation of the network. Using the same third-party CA, I'm now trying to update issuing CAs and regenerate all the certificates (out of the new issuing CAs) on an existing blockchain network. I have created a script that updates the TLS & Digital Signature Issuing CA certificates for the OrdererOrg in an existing blockchain network. The Issuing CA certificates are updated in the system-channel and the application channel. This appears to be successful. I generate new certificates (TLS, Digital Signature, Admin) and update the channel configuration for each channel. THat also appears to be successful. Then, I have to update the consenter certificates in each channel. I update the consenter certificates one orderer at a time, in its own transaction. To be more specific, the following updates are made in each transaction for the consenter certificates:

system-channel:
  cat ${TMP_DIR}/config.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].client_tls_cert = "'${ENCODED_CERT}'" |
        .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].server_tls_cert = "'${ENCODED_CERT}'"' \
       > ${TMP_DIR}/modified-${neworg}-root.json

app-channel:
  cat ${TMP_DIR}/config.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].client_tls_cert = "'${ENCODED_CERT}'" |
        .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].server_tls_cert = "'${ENCODED_CERT}'"' \
        > ${TMP_DIR}/modified-${neworg}-root.json

There are five orderers in my OrdererOrg using the RAFT Orderer. For app-channel, Block 12 contains the configuration update transaction to update the consenter certificate for orderer0, Block 13 contains the transaction for updating the consenter certificate info for orderer1, etc,etc to Block 16 for consenter certificate of orderer4.

The system-channel transactions seem to be validated without issue. For the app-channel, all transactions that update the consenter certificates produce the following error on each peer:

[[34m2022-01-31 20:29:09.642 UTC [gossip.privdata] StoreBlock -> INFO 032^[[0m Received block [12] from buffer channel=app-channel^M
^[[31m2022-01-31 20:29:09.644 UTC [protoutils] ValidateTransaction -> ERRO 033^[[0m checkSignatureFromCreator returns err creator certificate is not valid: could not validate identity's OUs: certifiersIdentifier does not match: [orderer(6BF380B427FE7D15)], MSP: [OrdererMSP]^M
^[[31m2022-01-31 20:29:09.644 UTC [committer.txvalidator] validateTx -> ERRO 034^[[0m Invalid transaction with index 0^M
^[[34m2022-01-31 20:29:09.644 UTC [committer.txvalidator] Validate -> INFO 035^[[0m [app-channel] Validated block [12] in 2ms^M
^[[33m2022-01-31 20:29:09.644 UTC [validation] preprocessProtoBlock -> WARN 036^[[0m Channel [app-channel]: Block [12] Transaction index [0] TxId [] marked as invalid by committer. Reason code [BAD_CREATOR_SIGNATURE]^M
^[[34m2022-01-31 20:29:09.669 UTC [kvledger] CommitLegacy -> INFO 037^[[0m [app-channel] Committed block [12] with 1 transaction(s) in 24ms (state_validation=0ms block_and_pvtdata_commit=7ms state_commit=15ms) commitHash=[64eccde911898a82e6d767b4ca13099c3d5594bde5d418bb1cb883d339b8cd94]^M

What does the hex # in "orderer(6BF380B427FE7D15)" represent?

My config.yaml on the peers and orderers contains:

NodeOUs:
  Enable: true
  ClientOUIdentifier:
    Certificate: cacerts/ca.org1.example.org-cert.pem
    OrganizationalUnitIdentifier: client
  PeerOUIdentifier:
    Certificate: cacerts/ca.org2.example.org-cert.pem
    OrganizationalUnitIdentifier: peer
  AdminOUIdentifier:
    Certificate: cacerts/ca.org3.example.org-cert.pem
    OrganizationalUnitIdentifier: admin
  OrdererOUIdentifier:
    Certificate: cacerts/ca.org4.org4.example.org-cert.pem
    OrganizationalUnitIdentifier: orderer

For the consenter (TLS) certificate for the orderers, I've tried with OU=orderer and no OU in the Subject DN. The results were the same.

I'm not sure if there is something wrong with the certificates I'm generating, the order I'm updating the channel configuration for the consenter certificates, or the config changes I'm making. Any ideas?


Robert Broeckelmann
 

Good news. I was able to succesfully update the consenter certificates for all five of my RAFT orderer nodes. I got this working a couple of days ago.

I wanted to share what I found. Mainly because I'm very curious if there is a more straightforward way of doing this.

Before I attempt to update the consenter certificates in the application channel, I do the following:
I generate a new Digital Signature CA certificate/key and TLS CA certificate/key. For testing purposes, I'm using openssl to do this.
I generate all the new certificates (signed by the corresponding CA certs from the last step) needed for the orderers. This includes a dsig cert, tls cert, and admin cert for each orderer. The private keys are regenerated as well.
All of these certs/keys are placed in the appropriate places in the crypto-config directory for each orderer.
The system-channel configuration is updated successfully without issue.
Then, I update the application channel configuration with the Issuing CAs and certs I generated. I originally, was updating the application channel at this step with the following JQ expression:

'.channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.root_certs += ["'${DSIG_CA_ENCODED}'"] |
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.admins += ["'${ADMIN_CERT_ENCODED}'"] |
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.fabric_node_ous.client_ou_identifier.certificate ="'${DSIG_CA_ENCODED}'" |
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.fabric_node_ous.orderer_ou_identifier.certificate ="'${DSIG_CA_ENCODED}'" |       
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.fabric_node_ous.peer_ou_identifier.certificate ="'${DSIG_CA_ENCODED}'" |       
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.fabric_node_ous.admin_ou_identifier.certificate ="'${DSIG_CA_ENCODED}'" |       
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.tls_root_certs += ["'${TLS_CA_ENCODED}'"]'

After enabling debug on the protoutils module, I got the following log entries (note, I have trimmed out the full certificate and channel name):

2022-02-16 09:44:06.431 UTC [gossip.privdata] StoreBlock -> INFO 05e Received block [13] from buffer channel=app-channel
2022-02-16 09:44:06.431 UTC [protoutils] ValidateTransaction -> DEBU 05f ValidateTransactionEnvelope starts for envelope 0xc004d30410
2022-02-16 09:44:06.431 UTC [protoutils] ValidateTransaction -> DEBU 060 Header is channel_header:"\010...\"\r..." signature_header:"\n\321\n\n\nOrdererMSP\022\302\n-----BEGIN CERTIFICATE-----\nMIIDt...-----END CERTIFICATE-----\n\022...365"
2022-02-16 09:44:06.431 UTC [protoutils] validateChannelHeader -> DEBU 061 validateChannelHeader info: header type 1
2022-02-16 09:44:06.431 UTC [protoutils] checkSignatureFromCreator -> DEBU 062 begin
2022-02-16 09:44:06.432 UTC [protoutils] checkSignatureFromCreator -> DEBU 063 creator is &{OrdererMSP 956707729a2593b00cc66ca1ca8c9978c9304a2c3219c6c21cad9fce656c6c98}
2022-02-16 09:44:06.432 UTC [protoutils] ValidateTransaction -> ERRO 064 checkSignatureFromCreator returns err creator certificate is not valid: could not validate identity's OUs: certifiersIdentifier does not match: [orderer(65EE5DB1234EEE0E)], MSP: [OrdererMSP]
2022-02-16 09:44:06.432 UTC [committer.txvalidator] validateTx -> ERRO 065 Invalid transaction with index 0

So, the "creator certificate" in this case is the original digital signature certificate for our orderer0 node, which makes sense since the original certificate is still being used and this is the transaction that is supposed to introduce the new certificates to the channel configuration. Among the configuration elements being updated in this transaction is "fabric_node_ous.orderer_ou_identifier.certificate". So, I'm introducing the new digital signature CA certificate there. If I remove the update to this configuration element from the transaction and wait until the last orderer consenter certificate update to add it, then things start working as desired. I found this by temperarily removing each field update being made to the channel configuration that references the digital signature CA for the OrdererOrg (one at a time).

So, the application channel configuration update now looks like:

'.channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.root_certs += ["'${DSIG_CA_ENCODED}'"] |
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.admins += ["'${ADMIN_CERT_ENCODED}'"] |
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.fabric_node_ous.client_ou_identifier.certificate ="'${DSIG_CA_ENCODED}'" |
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.fabric_node_ous.peer_ou_identifier.certificate ="'${DSIG_CA_ENCODED}'" |
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.fabric_node_ous.admin_ou_identifier.certificate ="'${DSIG_CA_ENCODED}'" |
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.tls_root_certs += ["'${TLS_CA_ENCODED}'"]'

So, now the original ordererOrg digital signature CA cert is still trusted when the consenter certificates are being updated. So, the original error about BAD_CREATOR_SIGNATURE goes away and the new consenter certificate is accepted by each peer.

In the final consenter certificate update transaction, I add the orderer OU CA certificate update as well with:

  if [ ${ELEMENT} -eq 4 ];
  then
    cat ${TMP_DIR}/config.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].client_tls_cert = "'${ENCODED_CERT}'" |
        .channel_group.groups.Orderer.groups.OrdererOrg.values.MSP.value.config.fabric_node_ous.orderer_ou_identifier.certificate ="'${DSIG_CA_ENCODED}'" |
        .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].server_tls_cert = "'${ENCODED_CERT}'"' \
        > ${TMP_DIR}/modified-${neworg}-root.json
  else
    cat ${TMP_DIR}/config.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].client_tls_cert = "'${ENCODED_CERT}'" |
        .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].server_tls_cert = "'${ENCODED_CERT}'"' \
        > ${TMP_DIR}/modified-${neworg}-root.json
  fi

This completes the updates required to change out the Issuing CAs for the OrdererOrg.

Is there an easier way to do this? There are several non-obvious steps here.

------- Original Message -------
On Tuesday, February 1st, 2022 at 10:45 AM, Robert Broeckelmann <robert@...> wrote:

I have a blockchain network with four organizations working with crypto material generated with cryptogen. I also have it working with crypto material generated from a third-party certificate authority that's introduced during initial creation of the network. Using the same third-party CA, I'm now trying to update issuing CAs and regenerate all the certificates (out of the new issuing CAs) on an existing blockchain network. I have created a script that updates the TLS & Digital Signature Issuing CA certificates for the OrdererOrg in an existing blockchain network. The Issuing CA certificates are updated in the system-channel and the application channel. This appears to be successful. I generate new certificates (TLS, Digital Signature, Admin) and update the channel configuration for each channel. THat also appears to be successful. Then, I have to update the consenter certificates in each channel. I update the consenter certificates one orderer at a time, in its own transaction. To be more specific, the following updates are made in each transaction for the consenter certificates:

system-channel:
  cat ${TMP_DIR}/config.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].client_tls_cert = "'${ENCODED_CERT}'" |
        .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].server_tls_cert = "'${ENCODED_CERT}'"' \
       > ${TMP_DIR}/modified-${neworg}-root.json

app-channel:
  cat ${TMP_DIR}/config.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].client_tls_cert = "'${ENCODED_CERT}'" |
        .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].server_tls_cert = "'${ENCODED_CERT}'"' \
        > ${TMP_DIR}/modified-${neworg}-root.json

There are five orderers in my OrdererOrg using the RAFT Orderer. For app-channel, Block 12 contains the configuration update transaction to update the consenter certificate for orderer0, Block 13 contains the transaction for updating the consenter certificate info for orderer1, etc,etc to Block 16 for consenter certificate of orderer4.

The system-channel transactions seem to be validated without issue. For the app-channel, all transactions that update the consenter certificates produce the following error on each peer:

[[34m2022-01-31 20:29:09.642 UTC [gossip.privdata] StoreBlock -> INFO 032^[[0m Received block [12] from buffer channel=app-channel^M
^[[31m2022-01-31 20:29:09.644 UTC [protoutils] ValidateTransaction -> ERRO 033^[[0m checkSignatureFromCreator returns err creator certificate is not valid: could not validate identity's OUs: certifiersIdentifier does not match: [orderer(6BF380B427FE7D15)], MSP: [OrdererMSP]^M
^[[31m2022-01-31 20:29:09.644 UTC [committer.txvalidator] validateTx -> ERRO 034^[[0m Invalid transaction with index 0^M
^[[34m2022-01-31 20:29:09.644 UTC [committer.txvalidator] Validate -> INFO 035^[[0m [app-channel] Validated block [12] in 2ms^M
^[[33m2022-01-31 20:29:09.644 UTC [validation] preprocessProtoBlock -> WARN 036^[[0m Channel [app-channel]: Block [12] Transaction index [0] TxId [] marked as invalid by committer. Reason code [BAD_CREATOR_SIGNATURE]^M
^[[34m2022-01-31 20:29:09.669 UTC [kvledger] CommitLegacy -> INFO 037^[[0m [app-channel] Committed block [12] with 1 transaction(s) in 24ms (state_validation=0ms block_and_pvtdata_commit=7ms state_commit=15ms) commitHash=[64eccde911898a82e6d767b4ca13099c3d5594bde5d418bb1cb883d339b8cd94]^M

What does the hex # in "orderer(6BF380B427FE7D15)" represent?

My config.yaml on the peers and orderers contains:

NodeOUs:
  Enable: true
  ClientOUIdentifier:
    Certificate: cacerts/ca.org1.example.org-cert.pem
    OrganizationalUnitIdentifier: client
  PeerOUIdentifier:
    Certificate: cacerts/ca.org2.example.org-cert.pem
    OrganizationalUnitIdentifier: peer
  AdminOUIdentifier:
    Certificate: cacerts/ca.org3.example.org-cert.pem
    OrganizationalUnitIdentifier: admin
  OrdererOUIdentifier:
    Certificate: cacerts/ca.org4.org4.example.org-cert.pem
    OrganizationalUnitIdentifier: orderer

For the consenter (TLS) certificate for the orderers, I've tried with OU=orderer and no OU in the Subject DN. The results were the same.

I'm not sure if there is something wrong with the certificates I'm generating, the order I'm updating the channel configuration for the consenter certificates, or the config changes I'm making. Any ideas?


Robert Broeckelmann
 

Yacov,

Thanks for responding. As I noted in my last post, I was able to fix my problem.

I saw your response. I tried updating my Issuing CA certificates for the OrdererOrg to have OU=orderer. THere was no difference. I also tried no OU and a completely random OU on the Issuing CA certificate. It didn't seem to matter. I tried this on both the TLS Issuing CA and the digital signature Issuing CA certificates.

All Issuing CA certificates (both original and new) were from a third-party CA and had no OU defined in my original solution. What I'm planning to move forward with has no OU in the Issuing CA DNs for any org.



------- Original Message -------
On Tuesday, February 1st, 2022 at 11:47 AM, Yacov Manevich <YACOVM@...> wrote:
Maybe on the application channel, the orderer MSP has an OU configured and your identity you used to submit the transaction was issued by a CA with a different OU?

From: fabric@... <fabric@...> on behalf of Robert Broeckelmann <robert@...>
Sent: Tuesday, February 1, 2022 8:45 PM
To: fabric@... <fabric@...>
Subject: [EXTERNAL] [Hyperledger Fabric] Consenter Certificate Updates When Using a Third-Party CA
 
I have a blockchain network with four organizations working with crypto material generated with cryptogen. I also have it working with crypto material generated from a third-party certificate authority that's introduced during initial creation ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I have a blockchain network with four organizations working with crypto material generated with cryptogen. I also have it working with crypto material generated from a third-party certificate authority that's introduced during initial creation of the network. Using the same third-party CA, I'm now trying to update issuing CAs and regenerate all the certificates (out of the new issuing CAs) on an existing blockchain network. I have created a script that updates the TLS & Digital Signature Issuing CA certificates for the OrdererOrg in an existing blockchain network. The Issuing CA certificates are updated in the system-channel and the application channel. This appears to be successful. I generate new certificates (TLS, Digital Signature, Admin) and update the channel configuration for each channel. THat also appears to be successful. Then, I have to update the consenter certificates in each channel. I update the consenter certificates one orderer at a time, in its own transaction. To be more specific, the following updates are made in each transaction for the consenter certificates:

system-channel:
  cat ${TMP_DIR}/config.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].client_tls_cert = "'${ENCODED_CERT}'" |
        .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].server_tls_cert = "'${ENCODED_CERT}'"' \
       > ${TMP_DIR}/modified-${neworg}-root.json

app-channel:
  cat ${TMP_DIR}/config.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].client_tls_cert = "'${ENCODED_CERT}'" |
        .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters['${ELEMENT}'].server_tls_cert = "'${ENCODED_CERT}'"' \
        > ${TMP_DIR}/modified-${neworg}-root.json

There are five orderers in my OrdererOrg using the RAFT Orderer. For app-channel, Block 12 contains the configuration update transaction to update the consenter certificate for orderer0, Block 13 contains the transaction for updating the consenter certificate info for orderer1, etc,etc to Block 16 for consenter certificate of orderer4.

The system-channel transactions seem to be validated without issue. For the app-channel, all transactions that update the consenter certificates produce the following error on each peer:

[[34m2022-01-31 20:29:09.642 UTC [gossip.privdata] StoreBlock -> INFO 032^[[0m Received block [12] from buffer channel=app-channel^M
^[[31m2022-01-31 20:29:09.644 UTC [protoutils] ValidateTransaction -> ERRO 033^[[0m checkSignatureFromCreator returns err creator certificate is not valid: could not validate identity's OUs: certifiersIdentifier does not match: [orderer(6BF380B427FE7D15)], MSP: [OrdererMSP]^M
^[[31m2022-01-31 20:29:09.644 UTC [committer.txvalidator] validateTx -> ERRO 034^[[0m Invalid transaction with index 0^M
^[[34m2022-01-31 20:29:09.644 UTC [committer.txvalidator] Validate -> INFO 035^[[0m [app-channel] Validated block [12] in 2ms^M
^[[33m2022-01-31 20:29:09.644 UTC [validation] preprocessProtoBlock -> WARN 036^[[0m Channel [app-channel]: Block [12] Transaction index [0] TxId [] marked as invalid by committer. Reason code [BAD_CREATOR_SIGNATURE]^M
^[[34m2022-01-31 20:29:09.669 UTC [kvledger] CommitLegacy -> INFO 037^[[0m [app-channel] Committed block [12] with 1 transaction(s) in 24ms (state_validation=0ms block_and_pvtdata_commit=7ms state_commit=15ms) commitHash=[64eccde911898a82e6d767b4ca13099c3d5594bde5d418bb1cb883d339b8cd94]^M

What does the hex # in "orderer(6BF380B427FE7D15)" represent?

My config.yaml on the peers and orderers contains:

NodeOUs:
  Enable: true
  ClientOUIdentifier:
    Certificate: cacerts/ca.org1.example.org-cert.pem
    OrganizationalUnitIdentifier: client
  PeerOUIdentifier:
    Certificate: cacerts/ca.org2.example.org-cert.pem
    OrganizationalUnitIdentifier: peer
  AdminOUIdentifier:
    Certificate: cacerts/ca.org3.example.org-cert.pem
    OrganizationalUnitIdentifier: admin
  OrdererOUIdentifier:
    Certificate: cacerts/ca.org4.org4.example.org-cert.pem
    OrganizationalUnitIdentifier: orderer

For the consenter (TLS) certificate for the orderers, I've tried with OU=orderer and no OU in the Subject DN. The results were the same.

I'm not sure if there is something wrong with the certificates I'm generating, the order I'm updating the channel configuration for the consenter certificates, or the config changes I'm making. Any ideas?