Update expired orderer org admin certificate and orderer certs #fabric #fabric-questions #fabric-orderer #signcerts


Mattia Bolzonella
 

Hi, i need to update the orderer org admin of the system channel, it's expired but i managed to start the orderer and peer following FAB-18384. Now i need to update the channel configuration but I have no clue on what to modify and how, can someone help me?


Chris Gabriel
 

Hi Mattia,

Happy to help you.  Can you please provide a bit more detail about your environment?  For example, are you on Fabric 1.4 with a solo orderer, raft, etc.?
Beyond the above question, here are the relevant sections from the Fabric documentation to follow https://hyperledger-fabric.readthedocs.io/en/release-2.3/config_update.html#editing-a-config

To figure out what you have to edit, in the docs at the link above, pay close attention to the ‘Channel parameters that can be updated’ section as this contains a sample de-coded configuration with a section for each org msp and the orderer (click the dropdown where it says ‘click here to see the config’).  In the example, you will note that all certs are converted to base64 from pem so you will have to do this when you paste the new certs into the configuration.  Also, the example only shows a single raft node consenter, so it you have more you will have to update for each.

At a high level, the steps are to:
Set the environment variables for the org you are acting as
Pull and decode the configuration (to human readable form in the form of a json file. If you copy and paste the commands from the docs you will end up with a file called ‘modified_config.json’ which you will edit in a text editor or VSCode, etc.)
Modify (edit) the configuration
Re-encode the configuration reflecting your updated certs
Submit the update using the ‘peer channel update’ command
Get the necessary signatures according to your policies (note you will have to sign using your old certs before the update can happen)

I hope this helps,
Chris



On May 5, 2021, at 6:02 AM, Mattia Bolzonella <mattia.bolzonella@...> wrote:

Hi, i need to update the orderer org admin of the system channel, it's expired but i managed to start the orderer and peer following FAB-18384. Now i need to update the channel configuration but I have no clue on what to modify and how, can someone help me?


Mattia Bolzonella
 

Hi Chris,
Thank you for the quick reply. I will illustrate my situation, it's the worst possible scenario (production environment). So I'm on Fabric 2.2.2 with all my components except Fabric CAs which is 1.4.6 version.
I generated all my certs Orderer org and peer org with 2 separate CA but now i found out that *all* the certs are expired (except fo root CAs certs). So my situation is:
  • all admin certs (Orderer org and peer org) expired;
  • all MSP certs (singcerts I think it's how they are called) of both orgs of every components (I have 3 orderers and 2 peers) expired;
  • all TLS Certs expired;
I followed the Jira Issue which helped to start the orderers, even with the flag of ignoring TLS cert expiration date they were not starting. To resolve I had to generate new certs for the orderers MSP, i kept the expired TLS.
I generated new certs for the peer organization and started them pointing at the new certs. Plus to that peer cli has visibility on the expired MSP (and TLS) of orderer Org so in this way I managed to fetch the system-channel config block.

So now what I have to do (I think): 
  • Update orderer admin certs in the *system-channel* but with section of the block i have to modify? Channel, Application, Consortium or Orderer?
  • Update cosenter certs with updated certs for orderer org, in this case i know that i have to do one update at a time but same as before: which section?
  • Repeat previous steps for all "normal" channels
Is this the right procedure? 
I really appreciate your help, thank you!


Il giorno mer 5 mag 2021 alle ore 15:06 Chris Gabriel <alaskadd@...> ha scritto:
Hi Mattia,

Happy to help you.  Can you please provide a bit more detail about your environment?  For example, are you on Fabric 1.4 with a solo orderer, raft, etc.?
Beyond the above question, here are the relevant sections from the Fabric documentation to follow https://hyperledger-fabric.readthedocs.io/en/release-2.3/config_update.html#editing-a-config

To figure out what you have to edit, in the docs at the link above, pay close attention to the ‘Channel parameters that can be updated’ section as this contains a sample de-coded configuration with a section for each org msp and the orderer (click the dropdown where it says ‘click here to see the config’).  In the example, you will note that all certs are converted to base64 from pem so you will have to do this when you paste the new certs into the configuration.  Also, the example only shows a single raft node consenter, so it you have more you will have to update for each.

At a high level, the steps are to:
Set the environment variables for the org you are acting as
Pull and decode the configuration (to human readable form in the form of a json file. If you copy and paste the commands from the docs you will end up with a file called ‘modified_config.json’ which you will edit in a text editor or VSCode, etc.)
Modify (edit) the configuration
Re-encode the configuration reflecting your updated certs
Submit the update using the ‘peer channel update’ command
Get the necessary signatures according to your policies (note you will have to sign using your old certs before the update can happen)

I hope this helps,
Chris



On May 5, 2021, at 6:02 AM, Mattia Bolzonella <mattia.bolzonella@...> wrote:

Hi, i need to update the orderer org admin of the system channel, it's expired but i managed to start the orderer and peer following FAB-18384. Now i need to update the channel configuration but I have no clue on what to modify and how, can someone help me?



--

Mattia Bolzonella

Software Developer

mattia.bolzonella@...

Via G. Medici 9/a - 35138 Padova (PD)

+39 049 500 1 500  www.ifin.it
      



IFIN SISTEMI S.R.L. a socio unico

Via G. Medici 9/a
35138 Padova 


Le informazioni, i dati e le notizie contenute nella presente comunicazione e i relativi allegati sono di natura privata e come tali possono essere riservate e sono, comunque, destinate esclusivamente ai destinatari sopra indicati. La diffusione, la distribuzione e/o la copiatura del documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario è proibita, sia ai sensi dell’art. 616 c.p., sia ai sensi del D.Lgs. 196/2003 e Regolamento UE 2016/679. Se avete ricevuto questo messaggio per errore, vi preghiamo di distruggerlo e di darcene immediata comunicazione anche inviando un messaggio di ritorno all'indirizzo e-mail del mittente. L’interessato può esercitare tutti i diritti previsti ai sensi degli articoli degli articoli 13, comma 2, lettere (b) e (d), 15-21 del Regolamento UE 2016/679, inviando un messaggio all'indirizzo e-mail del mittente o telefonando allo 049 500 1 500. Si prega di leggere Privacy Policy presente in https://ifin.it/privacy-policy-informativa-sui-cookies.


Chris Gabriel
 

Hi Mattia,

***IMPORTANT**
If you are in production and will have adverse customer impact, I recommend taking a look at doing this until you get it fixed (this is for extreme measures, perform this step at your own risk):
Just remember to undo this after updating your certs!!
*********************

You are on the right track!  Take a look below:
So now what I have to do (I think): 
  • Update orderer admin certs in the *system-channel* but with section of the block i have to modify? Channel, Application, Consortium or Orderer?
In the system channel config, scroll down to the first OrgMSP section and note the admin cert followed by the node OU section for the admin, client, 
orderer, and peer.  You will need to update these certs with your newly generated ones. Then scroll down to the Orderer MSP section and update those certs.  Then, scroll down to the ConsensusType section where you will see your consenter certs in the “consenters subsection.  Note that each consenter has a “client_tls_cert" and a “server_tls_cert".  In my case, these are the same on a per consenter basis.  You can double-check yours to be sure. 
  • Update cosenter certs with updated certs for orderer org, in this case i know that i have to do one update at a time but same as before: which section? 
Yes, see above.
  • Repeat previous steps for all "normal" channels
Yes, but start with the system channel, then the application channel.

I have a few tips for examining existing certs:

Base64 to PEM
export FLAG=$(if [ "$(uname -s)" == "Linux" ]; then echo "-w 0"; else echo "-b 0"; fi) 
echo <insert your base64 string here> | base64 --decode $FLAG 

PEM to Base64
export FLAG=$(if [ "$(uname -s)" == "Linux" ]; then echo "-w 0"; else echo "-b 0"; fi)
cat <insert path to .pem here> | base64 $FLAG 


I use certlogik.com to decode PEM certs and examine them.

Hope this helps,
Chris

On May 5, 2021, at 8:33 AM, Mattia Bolzonella <mattia.bolzonella@...> wrote:

Hi Chris,
Thank you for the quick reply. I will illustrate my situation, it's the worst possible scenario (production environment). So I'm on Fabric 2.2.2 with all my components except Fabric CAs which is 1.4.6 version.
I generated all my certs Orderer org and peer org with 2 separate CA but now i found out that *all* the certs are expired (except fo root CAs certs). So my situation is:
  • all admin certs (Orderer org and peer org) expired;
  • all MSP certs (singcerts I think it's how they are called) of both orgs of every components (I have 3 orderers and 2 peers) expired;
  • all TLS Certs expired;
I followed the Jira Issue which helped to start the orderers, even with the flag of ignoring TLS cert expiration date they were not starting. To resolve I had to generate new certs for the orderers MSP, i kept the expired TLS.
I generated new certs for the peer organization and started them pointing at the new certs. Plus to that peer cli has visibility on the expired MSP (and TLS) of orderer Org so in this way I managed to fetch the system-channel config block.

So now what I have to do (I think): 
  • Update orderer admin certs in the *system-channel* but with section of the block i have to modify? Channel, Application, Consortium or Orderer?
  • Update cosenter certs with updated certs for orderer org, in this case i know that i have to do one update at a time but same as before: which section?
  • Repeat previous steps for all "normal" channels
Is this the right procedure? 
I really appreciate your help, thank you!


Il giorno mer 5 mag 2021 alle ore 15:06 Chris Gabriel <alaskadd@...> ha scritto:
Hi Mattia,

Happy to help you.  Can you please provide a bit more detail about your environment?  For example, are you on Fabric 1.4 with a solo orderer, raft, etc.?
Beyond the above question, here are the relevant sections from the Fabric documentation to follow https://hyperledger-fabric.readthedocs.io/en/release-2.3/config_update.html#editing-a-config

To figure out what you have to edit, in the docs at the link above, pay close attention to the ‘Channel parameters that can be updated’ section as this contains a sample de-coded configuration with a section for each org msp and the orderer (click the dropdown where it says ‘click here to see the config’).  In the example, you will note that all certs are converted to base64 from pem so you will have to do this when you paste the new certs into the configuration.  Also, the example only shows a single raft node consenter, so it you have more you will have to update for each.

At a high level, the steps are to:
Set the environment variables for the org you are acting as
Pull and decode the configuration (to human readable form in the form of a json file. If you copy and paste the commands from the docs you will end up with a file called ‘modified_config.json’ which you will edit in a text editor or VSCode, etc.)
Modify (edit) the configuration
Re-encode the configuration reflecting your updated certs
Submit the update using the ‘peer channel update’ command
Get the necessary signatures according to your policies (note you will have to sign using your old certs before the update can happen)

I hope this helps,
Chris



On May 5, 2021, at 6:02 AM, Mattia Bolzonella <mattia.bolzonella@...> wrote:

Hi, i need to update the orderer org admin of the system channel, it's expired but i managed to start the orderer and peer following FAB-18384. Now i need to update the channel configuration but I have no clue on what to modify and how, can someone help me?



--

Mattia Bolzonella
Software Developer
Via G. Medici 9/a - 35138 Padova (PD)
+39 049 500 1 500  www.ifin.it
      


IFIN SISTEMI S.R.L. a socio unico
Via G. Medici 9/a
35138 Padova 

Le informazioni, i dati e le notizie contenute nella presente comunicazione e i relativi allegati sono di natura privata e come tali possono essere riservate e sono, comunque, destinate esclusivamente ai destinatari sopra indicati. La diffusione, la distribuzione e/o la copiatura del documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario è proibita, sia ai sensi dell’art. 616 c.p., sia ai sensi del D.Lgs. 196/2003 e Regolamento UE 2016/679. Se avete ricevuto questo messaggio per errore, vi preghiamo di distruggerlo e di darcene immediata comunicazione anche inviando un messaggio di ritorno all'indirizzo e-mail del mittente. L’interessato può esercitare tutti i diritti previsti ai sensi degli articoli degli articoli 13, comma 2, lettere (b) e (d), 15-21 del Regolamento UE 2016/679, inviando un messaggio all'indirizzo e-mail del mittente o telefonando allo 049 500 1 500. Si prega di leggere Privacy Policy presente in https://ifin.it/privacy-policy-informativa-sui-cookies.


Mattia Bolzonella
 

Hi Chris,
Thank you again for the reply, I'm going to try what you suggested, for the replacement of the admin certs in the various sections of the config, I do that in one whole update, right?

After doing that (the update configuration for the admin cert), I need to replace the `CORE_PEER_MSPCONFIGPATH` variable of the cli in order to proceed with the cosenters config, am I right?

Now I'm facing a minor problem with my TLS CA, since the tlsca-admin cert is also expired, I will disable temporally the TLS on the CA in order to generate new TLS materials for the cosenters (but on this i dunno how to renew the certs of the tlsca admin, if you have any suggestions these are more than welcome)

I'm going to perform these operation tomorrow, I'll keep you posted, thank you again

Mattia


Chris Gabriel
 

Hi Mattia,

Yes, you need to export the environment variables to act as orderer MSP admin, then as Org MSP admin when doing that step.  I would update Orderer Org first, then Org in separate updates but I welcome anyone else’s perspective.  When looking through the config, note the policy as that is the clue to which/how many signatures you need to get to sign the config update transaction.  

Chris

On May 5, 2021, at 10:27 AM, Mattia Bolzonella <mattia.bolzonella@...> wrote:

Hi Chris,
Thank you again for the reply, I'm going to try what you suggested, for the replacement of the admin certs in the various sections of the config, I do that in one whole update, right?

After doing that (the update configuration for the admin cert), I need to replace the `CORE_PEER_MSPCONFIGPATH` variable of the cli in order to proceed with the cosenters config, am I right?

Now I'm facing a minor problem with my TLS CA, since the tlsca-admin cert is also expired, I will disable temporally the TLS on the CA in order to generate new TLS materials for the cosenters (but on this i dunno how to renew the certs of the tlsca admin, if you have any suggestions these are more than welcome)

I'm going to perform these operation tomorrow, I'll keep you posted, thank you again

Mattia



Mattia Bolzonella
 

Just to be clear about what you told me to replace in my system channel configuration, sorry but I want to be 100% sure about what I'm doing:
My json looks like this:

"channel_group": {
    "groups": {
      "Application": { <- MAIN SECTION APPLICATION
        "groups": {
          "OrderersMSP": {
            "groups": {},
            "mod_policy""Admins",
            "policies": {
              "Admins": {
                "mod_policy""Admins",
               // other sections
              "Readers": {
              },
              "Writers": {
            },
            "values": { 
              "MSP": {
                "mod_policy""Admins",
                "value": {
                  "config": {
                    "admins": [ 
                      "EXPIRED CERT BASE64" <--- TO REPLACERIGHT?
                     ],
                    "crypto_config": {
                      "identity_identifier_hash_function""SHA256",
                      "signature_hash_family""SHA2"
                    },
                    "fabric_node_ous": {
                      "admin_ou_identifier": {
                        "certificate""NOT EXPIRED",
                        "organizational_unit_identifier""admin"
                      },
                      "client_ou_identifier": {
                        "certificate""SAME as admin_ou_identifier",
                        "organizational_unit_identifier""client"
                      },
                      "enable"true,
                      "orderer_ou_identifier": {
                        "certificate""SAME as previous",
                        "organizational_unit_identifier""orderer"
                      },
                      "peer_ou_identifier": {
                        "certificate""same as previuos",
                        "organizational_unit_identifier""peer"
                      }
                    },
                    .... other sections
                 } 
                }
            }
        }
    }, // End of Application section
    "Consortiums": { <- MAIN SECTION CONSORTIUM
        "groups": {
          "SampleConsortium": {
            "groups": {
              "IfinPeerMSP": {
                "groups": {},
                "mod_policy""Admins",
                "policies": {
                }, // other sections
                "values": {
                  "MSP": {
                    "mod_policy""Admins",
                    "value": {
                      "config": {
                        "admins": [
                          "EXPIRED" <- TO REPLACE?
                        ],
                        "crypto_config": {
                          "identity_identifier_hash_function""SHA256",
                          "signature_hash_family""SHA2"
                        },
                        "fabric_node_ous": {
                          "admin_ou_identifier": {
                            "certificate""NOT Expired",
                            "organizational_unit_identifier""admin"
                          },
                          "client_ou_identifier"
                            "certificate": "NOT Expired",
                            "organizational_unit_identifier""client"
                          },
                          "enable"true,
                          "orderer_ou_identifier": {
                            "certificate""NOT Expired",
                            "organizational_unit_identifier""orderer"
                          },
                          "peer_ou_identifier": {
                            "certificate""NOT Expired",
                            "organizational_unit_identifier""peer"
                          }
                        },
                       // others
                      },
                      "type"0
                    },
                  }
                },
               
              }
            },
        },
      
      },
    // ORDERER SECTION WILL BE MODIFIED IN ANOTHER TRANSACTION


Thank you again,

Mattia
 


Chris Gabriel
 

Yes, that is it.  You got it!

On May 5, 2021, at 10:50 AM, Mattia Bolzonella <mattia.bolzonella@...> wrote:

Just to be clear about what you told me to replace in my system channel configuration, sorry but I want to be 100% sure about what I'm doing:
My json looks like this:

"channel_group": {
    "groups": {
      "Application": { <- MAIN SECTION APPLICATION
        "groups": {
          "OrderersMSP": {
            "groups": {},
            "mod_policy""Admins",
            "policies": {
              "Admins": {
                "mod_policy""Admins",
               // other sections
              "Readers": {
              },
              "Writers": {
            },
            "values": { 
              "MSP": {
                "mod_policy""Admins",
                "value": {
                  "config": {
                    "admins": [ 
                      "EXPIRED CERT BASE64" <--- TO REPLACERIGHT?
                     ],
                    "crypto_config": {
                      "identity_identifier_hash_function""SHA256",
                      "signature_hash_family""SHA2"
                    },
                    "fabric_node_ous": {
                      "admin_ou_identifier": {
                        "certificate""NOT EXPIRED",
                        "organizational_unit_identifier""admin"
                      },
                      "client_ou_identifier": {
                        "certificate""SAME as admin_ou_identifier",
                        "organizational_unit_identifier""client"
                      },
                      "enable"true,
                      "orderer_ou_identifier": {
                        "certificate""SAME as previous",
                        "organizational_unit_identifier""orderer"
                      },
                      "peer_ou_identifier": {
                        "certificate""same as previuos",
                        "organizational_unit_identifier""peer"
                      }
                    },
                    .... other sections
                 } 
                }
            }
        }
    }, // End of Application section
    "Consortiums": { <- MAIN SECTION CONSORTIUM
        "groups": {
          "SampleConsortium": {
            "groups": {
              "IfinPeerMSP": {
                "groups": {},
                "mod_policy""Admins",
                "policies": {
                }, // other sections
                "values": {
                  "MSP": {
                    "mod_policy""Admins",
                    "value": {
                      "config": {
                        "admins": [
                          "EXPIRED" <- TO REPLACE?
                        ],
                        "crypto_config": {
                          "identity_identifier_hash_function""SHA256",
                          "signature_hash_family""SHA2"
                        },
                        "fabric_node_ous": {
                          "admin_ou_identifier": {
                            "certificate""NOT Expired",
                            "organizational_unit_identifier""admin"
                          },
                          "client_ou_identifier"
                            "certificate": "NOT Expired",
                            "organizational_unit_identifier""client"
                          },
                          "enable"true,
                          "orderer_ou_identifier": {
                            "certificate""NOT Expired",
                            "organizational_unit_identifier""orderer"
                          },
                          "peer_ou_identifier": {
                            "certificate""NOT Expired",
                            "organizational_unit_identifier""peer"
                          }
                        },
                       // others
                      },
                      "type"0
                    },
                  }
                },
               
              }
            },
        },
      
      },
    // ORDERER SECTION WILL BE MODIFIED IN ANOTHER TRANSACTION


Thank you again,

Mattia
 


Mattia Bolzonella
 

Hi, I'm working on the solution proposed, now I'm using a cli peer to do my channel update operations, before doing that I set the following variables in the peer cli:

CORE_PEER_LOCALMSPID=MyOrderersMSP

CORE_PEER_TLS_CERT_FILE=Expired TSL Orderer cert

CORE_PEER_TLS_KEY_FILE=Expired TSL Orderer key

CORE_PEER_TLS_ROOTCERT_FILE=Not expired TLS CA root cert for MyOrdererMSP

CORE_PEER_MSPCONFIGPATH=Expired msp MyOrdererMSP (all expired certs of orderers and admin org)

 

Now when i run the following command: 

peer channel fetch config crypto/sys-channel-with-tls-expired-tesst.pb -o orderer0.obscureddomain.com:7050  -c sys-channel --tls --cafile  $ORDERER_TLS_CA_CERT --tlsHandshakeTimeShift 1240h

(the certs expired 100h ago)

I get the error:

ERRO 001 Cannot run peer because error when setting up MSP of type bccsp from directory path/to/orderMSP_admin/msp: signing identity expired 107h4m58.724186994s ago

 


So, if I cannot do a fetch of the channel config (I managed to do it with a new admin cert), ho can i do the channel update? Because i think I need the expired orderer admin MSP in order to sign the channel update configuration, or am I missing something?

 

 


Mattia Bolzonella
 

Hi Chris,
I updated one orderer TLS certs in the system channel but it's not included in the raft consensus. If i try to fecth the channel confing block from the updated orderer i get SERVICE UNAVAILABLE.
On the updated orderer i get:
2021-05-06 18:21:23.035 CEST [orderer.consensus.etcdraft] campaign -> INFO 272 2 [logterm: 364, index: 371] sent MsgPreVote request to 1 at term 364 channel=sys-channel node=2 2021-05-06 18:21:23.036 CEST [orderer.consensus.etcdraft] campaign -> INFO 273 2 [logterm: 364, index: 371] sent MsgPreVote request to 3 at term 364 channel=sys-channel node=2 2021-05-06 18:21:27.220 CEST [comm.tls] ClientHandshake -> ERRO 274 Client TLS handshake failed after 1.087456ms with error: public key of server certificate presented by orderer0.obsucureddomain:7050 doesn't match the expected public key remoteaddress=xxxxxx:7050


So I'm no confident to update one more orderer in the system channel. Plus to that, if I updated all my orderers in the system channel, then replace the tls certs, and restart the orderer then i would find 2 orderers updatet in the system channel and not in the application channel which will lose the quorum.

 

I don't understand what's wrong, I cannot lose my production channels and data.



Chris Gabriel
 

Hi Mattia,
You have a cert mismatch. Make sure to double check where you placed the newly created certs and make sure they match.
Chris 


On May 6, 2021, at 11:27 AM, Mattia Bolzonella <mattia.bolzonella@...> wrote:



Hi Chris,
I updated one orderer TLS certs in the system channel but it's not included in the raft consensus. If i try to fecth the channel confing block from the updated orderer i get SERVICE UNAVAILABLE.
On the updated orderer i get:
2021-05-06 18:21:23.035 CEST [orderer.consensus.etcdraft] campaign -> INFO 272 2 [logterm: 364, index: 371] sent MsgPreVote request to 1 at term 364 channel=sys-channel node=2 2021-05-06 18:21:23.036 CEST [orderer.consensus.etcdraft] campaign -> INFO 273 2 [logterm: 364, index: 371] sent MsgPreVote request to 3 at term 364 channel=sys-channel node=2 2021-05-06 18:21:27.220 CEST [comm.tls] ClientHandshake -> ERRO 274 Client TLS handshake failed after 1.087456ms with error: public key of server certificate presented by orderer0.obsucureddomain:7050 doesn't match the expected public key remoteaddress=xxxxxx:7050


So I'm no confident to update one more orderer in the system channel. Plus to that, if I updated all my orderers in the system channel, then replace the tls certs, and restart the orderer then i would find 2 orderers updatet in the system channel and not in the application channel which will lose the quorum.

 

I don't understand what's wrong, I cannot lose my production channels and data.



Mattia Bolzonella
 

Hi Chris,
Thanks for the advice, the certs are correct. I will explain better the situation:

Currently I have 2 channels, 3 orderers with the TLS certs expired. 

  • the system channel
  • an application channel, called 'mychannel'
I've updated the admins of orderer and peer organization (2 separate orgs) in the system channel, and after that I executed a channel config update to change the certs of the first orderer (called ord0).

After that i restart the orderer and now I have:
  • In mychannel ord0 cannot partecipate in the consensus, as expected since the channel config is not updated
  • in the system channel i continuosly have TLS handshake error with the others orderers (ord1 & ord2)


The configuration of the orderer containers are the following:

ORD0
ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true

ORD1 & ORD2

ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true
ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT=200h
ORDERER_GENERAL_CLUSTER_TLSHANDSHAKETIMESHIFT=200h

Now, since ord1 & ord2 have expired certs i think i need the TimeShift, in fact they are able to reach consensus (I can pull the latest config block from the system channel). 
But with this settings, certs of ord0 result not yet valid since ord1 & ord2 are in the past. 

So I tried a different configuration just to see what happens:
 
ORD0
ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT=200h
ORD1 -> no timeshift set
ORD2 -> Stopped to not flood the logs of the others orderer

In this way on the system channel I can get the quorum using ord0 (new certs) and ord1 (expired). I think I got the procedure. But there is a thing that I'm not sure of:
Now I'm in a situation in which I'm able to get 2 orderers out of 3 to communicate thanks to the time shift parameter. If update all the orderers in the system channel with new certs i would have the system channel correctly configured and all the certs on the FS of the orderers updated.
 
Now the problem comes with the application channel  "mychannel"
 
After udpating the system channel I have new certs in the FS of the orderers, but the update wouldn't be possible on mychannel because I have a mismatch between the certs on the FS and the certs on the config block of the channel, am I right?
 
So I should update a orderer at a time in every channel, that's what I'm considering


ramesh.bobbala1990@...
 

Hi Mattia Bolzonella,


Is this issue resolved?
Even I am facing similar issue related to my certs got expired. I have created new certs and replaced in the same folder as before and my application is working fine. Currently we are trying to add new org, so we need to update channel configuration. But we are getting below error while doing peer channel update with new certs.

Error: got unexpected status: BAD_REQUEST -- error authorizing update: error validating DeltaSet: policy for [Group] /Channel/Application not satisfied: Failed to reach implicit threshold of 3 sub-policies, required 3 remaining.

I want to know the procedure after certs got expired and update channel configuration to add new org.

1. Shall I do peer channel update after signing with old expired certs or new certs?
2. Or before doing peer channel update shall I replace new certs in latest config block and genesis.block? 

Please correct me if I am doing anything wrong.

 


Mattia Bolzonella
 

Hi Ramesh,
I've managed to restore my production network but without adding new org. I think it's better if you first restore the network and then add the new organization (having an extra org involves more unwanted complexity). 
So my procedure was, i just replaced orderers MSP certs (*not TLS folder*) with new certs (had a problem of msp cert expiration), and peers certs:
  1. update orderer admin certs first in system channel then in every other application channel
  2. update peer admin certs first in system channle then in every other application channel
  3. Update orderers
Update Orderers
To update the orderer I make sure of the following:
  • At every moment in all my channels the quorum must be reached (with 3 orderers, 2 of them must communicate to reache consensum)
  • When updating orderer X, the orderer X must be on. After updating all the channels I replaced the TLS certs and restarded the node.

Update Orderer0
All the orderers had  the following parameters set:
ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true
ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT=200h
ORDERER_GENERAL_CLUSTER_TLSHANDSHAKETIMESHIFT=200h

I updated all the channels config (consenter section) with the updated TLS cert of orderer0. Then restarded the node.

Update Orderer1
Orderer parameters : 
- Orderer0 with timeshift on (can communicate with orderer2)
- Orderer1 with timeshift
- Orderer2 without timeshift (so it can communicate with orderer0 and send update to Orderer1)

As for Orderer0, this orderer was up and running while updating all the channel configurations. Then replaced the new TLS certs and restart the orderer.
Note: when changing the orderer paramenter you need to execute docker-compose up of the orderer.

Update Orderer2
Orderer parameters : 
- Orderer0 without timeshift on 
- Orderer1 without timeshift
- Orderer2 without timeshift 


I'm not sure 100% about the timeshift combination, in any case, if the combination is wrong you will get an error on the update saying that you will lose consensum. Try other comibination and you are good to go. 

P.S. About your error:
I think you need to sign the update block with the necessary admin orgs in order to update. 



Chris Gabriel
 

Ramesh,
When you sign the channel config update transaction, it has to be with the old certs.


Mattia,
I am glad you were able to recover.

Chris

On May 14, 2021, at 5:51 AM, Mattia Bolzonella <mattia.bolzonella@...> wrote:

Hi Ramesh,
I've managed to restore my production network but without adding new org. I think it's better if you first restore the network and then add the new organization (having an extra org involves more unwanted complexity). 
So my procedure was, i just replaced orderers MSP certs (*not TLS folder*) with new certs (had a problem of msp cert expiration), and peers certs:
  1. update orderer admin certs first in system channel then in every other application channel
  2. update peer admin certs first in system channle then in every other application channel
  3. Update orderers
Update Orderers
To update the orderer I make sure of the following:
  • At every moment in all my channels the quorum must be reached (with 3 orderers, 2 of them must communicate to reache consensum)
  • When updating orderer X, the orderer X must be on. After updating all the channels I replaced the TLS certs and restarded the node.

Update Orderer0
All the orderers had  the following parameters set:
ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true
ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT=200h
ORDERER_GENERAL_CLUSTER_TLSHANDSHAKETIMESHIFT=200h

I updated all the channels config (consenter section) with the updated TLS cert of orderer0. Then restarded the node.

Update Orderer1
Orderer parameters : 
- Orderer0 with timeshift on (can communicate with orderer2)
- Orderer1 with timeshift
- Orderer2 without timeshift (so it can communicate with orderer0 and send update to Orderer1)

As for Orderer0, this orderer was up and running while updating all the channel configurations. Then replaced the new TLS certs and restart the orderer.
Note: when changing the orderer paramenter you need to execute docker-compose up of the orderer.

Update Orderer2
Orderer parameters : 
- Orderer0 without timeshift on 
- Orderer1 without timeshift
- Orderer2 without timeshift 


I'm not sure 100% about the timeshift combination, in any case, if the combination is wrong you will get an error on the update saying that you will lose consensum. Try other comibination and you are good to go. 

P.S. About your error:
I think you need to sign the update block with the necessary admin orgs in order to update. 




ramesh.bobbala1990@...
 

Hi Mattia and Chris,

Thank you for your suggestions. 
  1. update orderer admin certs first in system channel then in every other application channel
  2. update peer admin certs first in system channle then in every other application channel
  3. Update orderers
Even I will try to follow the same steps before adding new org in the network.
I have some doubts while updating both system channel update and application channel update.

Doubt 1: After changing orderer admin certs in system channel, shall I directly convert to pb format and run peer channel update with ordererMSP or shall I do all the update process like comparing delta between actual config and modified config, decoding to editable JSON format, Wrap in an envelope message and Convert it into the fully fledged protobuf format and then run peer channel update?

Doubt 2: Shall I update all the orderer admin certs and peer certs in system channel separately? Or can I do the update peer channel update at a time for both orderer admin certs and peer certs in system channel at once?
That means
1. change orderer admin certs and update system channel
2. change orderer admin certs and update application channel
3. change peer admin certs and update system channel
4. change peer admin certs and update application channel


As I need to do this on production environment, I want to do this process smoothly without any issues. 


Regards,
Ramesh.





Mattia Bolzonella
 

Hi Ramesh,
Doubt 1: If i have understood correctly, for every update I followed the usual procces for channel update which is:
  1. fetch the latest config block
  2. decode to json
  3. copy the decoded json
  4. modified the copied json
  5. encoded the copied json and the modified json
  6. computed  the diff between the two and encoded in a .pb 
  7. converted the .pb in json
  8. added information about the channel to update (envelope)
  9. encoded the envelope
  10. updated channel configuration using the envelope pb

Doubt 2: I updated everything in atomic operations (just to be sure and have smallest changes possible every time), I think that updating orderer admin and peer admin certs will not be a problem but I didn't tested it, so do at your own risk

Mattia


email4tong@gmail.com
 

Mattia,Ramesh,
   If you have not seen Minifabric, you probably can take a few minutes looking into that. It allows you to deploy fabric in literally few minutes and allows you to update channel configuration very easily. The steps you described in your email largely match how Minifabric does channel update, the good thing is that Minifabric records all the steps to do this task in vars/run/channelupdate.sh script so that you can inspect how it does it.

On Tuesday, May 18, 2021, 10:32:42 AM EDT, Mattia Bolzonella <mattia.bolzonella@...> wrote:


Hi Ramesh,
Doubt 1: If i have understood correctly, for every update I followed the usual procces for channel update which is:
  1. fetch the latest config block
  2. decode to json
  3. copy the decoded json
  4. modified the copied json
  5. encoded the copied json and the modified json
  6. computed  the diff between the two and encoded in a .pb 
  7. converted the .pb in json
  8. added information about the channel to update (envelope)
  9. encoded the envelope
  10. updated channel configuration using the envelope pb

Doubt 2: I updated everything in atomic operations (just to be sure and have smallest changes possible every time), I think that updating orderer admin and peer admin certs will not be a problem but I didn't tested it, so do at your own risk

Mattia


ramesh.bobbala1990@...
 

Hi Mattia,

Doubt1:. Does your both Orderer admin certs and Peer admin certs got expired or only Orderer admin certs got expired? 
In my case both Orderer admin certs and Peer admin certs got expired.

Doubt2:. I have replaced all the Orderer admin certs and Peer admin certs in system channel configuration and trying to run peer channel update.
Then I got the error asking for PeerMSP admin signature, so when I tried to sign with old certificates getting this error, service-peer1/msp/user/msp: signing identity expired 1210h2m17.90829443s ago

Is there a way to sign with expired certificates from peer?

Doubt3:

I did'nt understood the timeshift combination logic exactly
- ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true
- ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT=200h
ORDERER_GENERAL_CLUSTER_TLSHANDSHAKETIMESHIFT=200h

I mean if I set these env variables in ord0, ord1 and or2 containers and after restarting orderer container where should I set these values for all orderers again?
- Orderer0 with timeshift on (can communicate with orderer2)
- Orderer1 with timeshift
- Orderer2 without timeshift




Regards,
Ramesh.


Mattia Bolzonella
 

Hi Ramesh, 
1: In my situation I had all the certs expired, both admin (peer and orderers), TLS and MSP certs.
2: You need to sign with old certs in orderer tu run the update, if I recall correctly the ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true (in the docker compose file of the orderer) should get the job done --tlsHandshakeTimeShift x where x is an amount of time greater than the expired time, in your case should be > 1211h. Try with this and the update should run smoothly.
3: I wasn't sure about the cluster parameter, in any case they are well documented here: official documentation.
I set the parameteres in the docker compose yaml file for every orderer commenting out to disable the paramenter and restart with docker compose up