Date
1 - 20 of 20
Update expired orderer org admin certificate and orderer certs #fabric-questions #fabric-orderer #signcerts #fabric
|
Mattia Bolzonella
Hi, i need to update the orderer org admin of the system channel, it's expired but i managed to start the orderer and peer following FAB-18384. Now i need to update the channel configuration but I have no clue on what to modify and how, can someone help me?
|
||
|
|
||
|
Chris Gabriel <alaskadd@...>
Hi Mattia,
toggle quoted messageShow quoted text
Happy to help you. Can you please provide a bit more detail about your environment? For example, are you on Fabric 1.4 with a solo orderer, raft, etc.? Beyond the above question, here are the relevant sections from the Fabric documentation to follow https://hyperledger-fabric.readthedocs.io/en/release-2.3/config_update.html#editing-a-config To figure out what you have to edit, in the docs at the link above, pay close attention to the ‘Channel parameters that can be updated’ section as this contains a sample de-coded configuration with a section for each org msp and the orderer (click the dropdown where it says ‘click here to see the config’). In the example, you will note that all certs are converted to base64 from pem so you will have to do this when you paste the new certs into the configuration. Also, the example only shows a single raft node consenter, so it you have more you will have to update for each. At a high level, the steps are to: Set the environment variables for the org you are acting as Pull and decode the configuration (to human readable form in the form of a json file. If you copy and paste the commands from the docs you will end up with a file called ‘modified_config.json’ which you will edit in a text editor or VSCode, etc.) Modify (edit) the configuration Re-encode the configuration reflecting your updated certs Submit the update using the ‘peer channel update’ command Get the necessary signatures according to your policies (note you will have to sign using your old certs before the update can happen) I hope this helps, Chris
|
||
|
|
||
|
Mattia Bolzonella
Hi Chris, Thank you for the quick reply. I will illustrate my situation, it's the worst possible scenario (production environment). So I'm on Fabric 2.2.2 with all my components except Fabric CAs which is 1.4.6 version. I generated all my certs Orderer org and peer org with 2 separate CA but now i found out that *all* the certs are expired (except fo root CAs certs). So my situation is:
I followed the Jira Issue which helped to start the orderers, even with the flag of ignoring TLS cert expiration date they were not starting. To resolve I had to generate new certs for the orderers MSP, i kept the expired TLS. I generated new certs for the peer organization and started them pointing at the new certs. Plus to that peer cli has visibility on the expired MSP (and TLS) of orderer Org so in this way I managed to fetch the system-channel config block. So now what I have to do (I think):
Is this the right procedure? I really appreciate your help, thank you!
Il giorno mer 5 mag 2021 alle ore 15:06 Chris Gabriel <alaskadd@...> ha scritto:
--
IFIN SISTEMI S.R.L. a socio unico Via G. Medici 9/a Le informazioni, i dati e le notizie contenute nella presente comunicazione e i relativi allegati sono di natura privata e come tali possono essere riservate e sono, comunque, destinate esclusivamente ai destinatari sopra indicati. La diffusione, la distribuzione e/o la copiatura del documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario è proibita, sia ai sensi dell’art. 616 c.p., sia ai sensi del D.Lgs. 196/2003 e Regolamento UE 2016/679. Se avete ricevuto questo messaggio per errore, vi preghiamo di distruggerlo e di darcene immediata comunicazione anche inviando un messaggio di ritorno all'indirizzo e-mail del mittente. L’interessato può esercitare tutti i diritti previsti ai sensi degli articoli degli articoli 13, comma 2, lettere (b) e (d), 15-21 del Regolamento UE 2016/679, inviando un messaggio all'indirizzo e-mail del mittente o telefonando allo 049 500 1 500. Si prega di leggere Privacy Policy presente in https://ifin.it/privacy-
|
||
|
|
||
|
Chris Gabriel <alaskadd@...>
Hi Mattia,
***IMPORTANT** If you are in production and will have adverse customer impact, I recommend taking a look at doing this until you get it fixed (this is for extreme measures, perform this step at your own risk): Just remember to undo this after updating your certs!! ********************* You are on the right track! Take a look below: In the system channel config, scroll down to the first OrgMSP section and note the admin cert followed by the node OU section for the admin, client, orderer, and peer. You will need to update these certs with your newly generated ones. Then scroll down to the Orderer MSP section and update those certs. Then, scroll down to the ConsensusType section where you will see your consenter certs in the “consenters” subsection. Note that each consenter has a “client_tls_cert" and a “server_tls_cert". In my case, these are the same on a per consenter basis. You can double-check yours to be sure. Yes, see above.
Yes, but start with the system channel, then the application channel. I have a few tips for examining existing certs: Base64 to PEM export FLAG=$(if [ "$(uname -s)" == "Linux" ]; then echo "-w 0"; else echo "-b 0"; fi) echo <insert your base64 string here> | base64 --decode $FLAG PEM to Base64 export FLAG=$(if [ "$(uname -s)" == "Linux" ]; then echo "-w 0"; else echo "-b 0"; fi) cat <insert path to .pem here> | base64 $FLAG I use certlogik.com to decode PEM certs and examine them. Hope this helps, Chris
|
||
|
|
||
|
Mattia Bolzonella
Hi Chris, After doing that (the update configuration for the admin cert), I need to replace the `CORE_PEER_MSPCONFIGPATH` variable of the cli in order to proceed with the cosenters config, am I right?
|
||
|
|
||
|
Chris Gabriel <alaskadd@...>
Hi Mattia,
toggle quoted messageShow quoted text
Yes, you need to export the environment variables to act as orderer MSP admin, then as Org MSP admin when doing that step. I would update Orderer Org first, then Org in separate updates but I welcome anyone else’s perspective. When looking through the config, note the policy as that is the clue to which/how many signatures you need to get to sign the config update transaction. Chris
|
||
|
|
||
|
Mattia Bolzonella
Just to be clear about what you told me to replace in my system channel configuration, sorry but I want to be 100% sure about what I'm doing:
My json looks like this: "channel_group": {
"groups": {
"Application": { <- MAIN SECTION APPLICATION
"groups": {
"OrderersMSP": {
"groups": {},
"mod_policy": "Admins",
"policies": {
"Admins": {
"mod_policy": "Admins",
// other sections
"Readers": {
},
"Writers": {
},
"values": {
"MSP": {
"mod_policy": "Admins",
"value": {
"config": {
"admins": [
"EXPIRED CERT BASE64" <--- TO REPLACE, RIGHT?
],
"crypto_config": {
"identity_identifier_hash_function": "SHA256",
"signature_hash_family": "SHA2"
},
"fabric_node_ous": {
"admin_ou_identifier": {
"certificate": "NOT EXPIRED",
"organizational_unit_identifier": "admin"
},
"client_ou_identifier": {
"certificate": "SAME as admin_ou_identifier",
"organizational_unit_identifier": "client"
},
"enable": true,
"orderer_ou_identifier": {
"certificate": "SAME as previous",
"organizational_unit_identifier": "orderer"
},
"peer_ou_identifier": {
"certificate": "same as previuos",
"organizational_unit_identifier": "peer"
}
},
.... other sections
}
}
}
}
}, // End of Application section
"Consortiums": { <- MAIN SECTION CONSORTIUM
"groups": {
"SampleConsortium": {
"groups": {
"IfinPeerMSP": {
"groups": {},
"mod_policy": "Admins",
"policies": {
}, // other sections
"values": {
"MSP": {
"mod_policy": "Admins",
"value": {
"config": {
"admins": [
"EXPIRED" <- TO REPLACE?
],
"crypto_config": {
"identity_identifier_hash_function": "SHA256",
"signature_hash_family": "SHA2"
},
"fabric_node_ous": {
"admin_ou_identifier": {
"certificate": "NOT Expired",
"organizational_unit_identifier": "admin"
},
"client_ou_identifier":
"certificate": "NOT Expired",
"organizational_unit_identifier": "client"
},
"enable": true,
"orderer_ou_identifier": {
"certificate": "NOT Expired",
"organizational_unit_identifier": "orderer"
},
"peer_ou_identifier": {
"certificate": "NOT Expired",
"organizational_unit_identifier": "peer"
}
},
// others
},
"type": 0
},
}
},
}
},
},
},
// ORDERER SECTION WILL BE MODIFIED IN ANOTHER TRANSACTION
Thank you again, Mattia
|
||
|
|
||
|
Chris Gabriel <alaskadd@...>
Yes, that is it. You got it!
toggle quoted messageShow quoted text
|
||
|
|
||
|
Mattia Bolzonella
Hi, I'm working on the solution proposed, now I'm using a cli peer to do my channel update operations, before doing that I set the following variables in the peer cli: CORE_PEER_LOCALMSPID=MyOrderersMSP CORE_PEER_TLS_CERT_FILE=Expired TSL Orderer cert CORE_PEER_TLS_KEY_FILE=Expired TSL Orderer key CORE_PEER_TLS_ROOTCERT_FILE=Not expired TLS CA root cert for MyOrdererMSP CORE_PEER_MSPCONFIGPATH=Expired msp MyOrdererMSP (all expired certs of orderers and admin org)
Now when i run the following command: peer channel fetch config crypto/sys-channel-with-tls-expired-tesst.pb -o orderer0.obscureddomain.com:7050 -c sys-channel --tls --cafile $ORDERER_TLS_CA_CERT --tlsHandshakeTimeShift 1240h (the certs expired 100h ago) I get the error: ERRO 001 Cannot run peer because error when setting up MSP of type bccsp from directory path/to/orderMSP_admin/msp: signing identity expired 107h4m58.724186994s ago
|
||
|
|
||
|
Mattia Bolzonella
Hi Chris,
I don't understand what's wrong, I cannot lose my production channels and data.
|
||
|
|
||
|
Chris Gabriel <alaskadd@...>
Hi Mattia,
toggle quoted messageShow quoted text
You have a cert mismatch. Make sure to double check where you placed the newly created certs and make sure they match. Chris
On May 6, 2021, at 11:27 AM, Mattia Bolzonella <mattia.bolzonella@...> wrote:
|
||
|
|
||
|
Mattia Bolzonella
Hi Chris, Currently I have 2 channels, 3 orderers with the TLS certs expired.
After that i restart the orderer and now I have:
ORD0 - ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true
- ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT=200h
- ORDERER_GENERAL_CLUSTER_TLSHANDSHAKETIMESHIFT=200h Now, since ord1 & ord2 have expired certs i think i need the TimeShift, in fact they are able to reach consensus (I can pull the latest config block from the system channel). But with this settings, certs of ord0 result not yet valid since ord1 & ord2 are in the past. So I tried a different configuration just to see what happens: ORD0
- ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT=200h
ORD1 -> no timeshift set ORD2 -> Stopped to not flood the logs of the others orderer In this way on the system channel I can get the quorum using ord0 (new certs) and ord1 (expired). I think I got the procedure. But there is a thing that I'm not sure of: Now I'm in a situation in which I'm able to get 2 orderers out of 3 to communicate thanks to the time shift parameter. If update all the orderers in the system channel with new certs i would have the system channel correctly configured and all the certs on the FS of the orderers updated. Now the problem comes with the application channel "mychannel"
After udpating the system channel I have new certs in the FS of the orderers, but the update wouldn't be possible on mychannel because I have a mismatch between the certs on the FS and the certs on the config block of the channel, am I right?
So I should update a orderer at a time in every channel, that's what I'm considering
|
||
|
|
||
|
ramesh.bobbala1990@...
Hi Mattia Bolzonella,
Is this issue resolved? Even I am facing similar issue related to my certs got expired. I have created new certs and replaced in the same folder as before and my application is working fine. Currently we are trying to add new org, so we need to update channel configuration. But we are getting below error while doing peer channel update with new certs. Error: got unexpected status: BAD_REQUEST -- error authorizing update: error validating DeltaSet: policy for [Group] /Channel/Application not satisfied: Failed to reach implicit threshold of 3 sub-policies, required 3 remaining. I want to know the procedure after certs got expired and update channel configuration to add new org. 1. Shall I do peer channel update after signing with old expired certs or new certs? 2. Or before doing peer channel update shall I replace new certs in latest config block and genesis.block? Please correct me if I am doing anything wrong.
|
||
|
|
||
|
Mattia Bolzonella
Hi Ramesh,
I've managed to restore my production network but without adding new org. I think it's better if you first restore the network and then add the new organization (having an extra org involves more unwanted complexity). So my procedure was, i just replaced orderers MSP certs (*not TLS folder*) with new certs (had a problem of msp cert expiration), and peers certs:
To update the orderer I make sure of the following:
Update Orderer0 All the orderers had the following parameters set: - ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true - ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT=200h - ORDERER_GENERAL_CLUSTER_TLSHANDSHAKETIMESHIFT=200h I updated all the channels config (consenter section) with the updated TLS cert of orderer0. Then restarded the node. Update Orderer1 Orderer parameters : - Orderer0 with timeshift on (can communicate with orderer2) - Orderer1 with timeshift - Orderer2 without timeshift (so it can communicate with orderer0 and send update to Orderer1) As for Orderer0, this orderer was up and running while updating all the channel configurations. Then replaced the new TLS certs and restart the orderer. Note: when changing the orderer paramenter you need to execute docker-compose up of the orderer. Update Orderer2 Orderer parameters : - Orderer0 without timeshift on - Orderer1 without timeshift - Orderer2 without timeshift I'm not sure 100% about the timeshift combination, in any case, if the combination is wrong you will get an error on the update saying that you will lose consensum. Try other comibination and you are good to go. P.S. About your error: I think you need to sign the update block with the necessary admin orgs in order to update.
|
||
|
|
||
|
Chris Gabriel <alaskadd@...>
Ramesh,
toggle quoted messageShow quoted text
When you sign the channel config update transaction, it has to be with the old certs. Mattia, I am glad you were able to recover. Chris
|
||
|
|
||
|
ramesh.bobbala1990@...
Hi Mattia and Chris,
Thank you for your suggestions.
I have some doubts while updating both system channel update and application channel update. Doubt 1: After changing orderer admin certs in system channel, shall I directly convert to pb format and run peer channel update with ordererMSP or shall I do all the update process like comparing delta between actual config and modified config, decoding to editable JSON format, Wrap in an envelope message and Convert it into the fully fledged protobuf format and then run peer channel update? Doubt 2: Shall I update all the orderer admin certs and peer certs in system channel separately? Or can I do the update peer channel update at a time for both orderer admin certs and peer certs in system channel at once? That means 1. change orderer admin certs and update system channel 2. change orderer admin certs and update application channel 3. change peer admin certs and update system channel 4. change peer admin certs and update application channel As I need to do this on production environment, I want to do this process smoothly without any issues. Regards, Ramesh.
|
||
|
|
||
|
Mattia Bolzonella
Hi Ramesh,
Doubt 1: If i have understood correctly, for every update I followed the usual procces for channel update which is:
Doubt 2: I updated everything in atomic operations (just to be sure and have smallest changes possible every time), I think that updating orderer admin and peer admin certs will not be a problem but I didn't tested it, so do at your own risk Mattia
|
||
|
|
||
|
email4tong@gmail.com
Mattia,Ramesh, If you have not seen Minifabric, you probably can take a few minutes looking into that. It allows you to deploy fabric in literally few minutes and allows you to update channel configuration very easily. The steps you described in your email largely match how Minifabric does channel update, the good thing is that Minifabric records all the steps to do this task in vars/run/channelupdate.sh script so that you can inspect how it does it.
On Tuesday, May 18, 2021, 10:32:42 AM EDT, Mattia Bolzonella <mattia.bolzonella@...> wrote:
Hi Ramesh, Doubt 1: If i have understood correctly, for every update I followed the usual procces for channel update which is:
Doubt 2: I updated everything in atomic operations (just to be sure and have smallest changes possible every time), I think that updating orderer admin and peer admin certs will not be a problem but I didn't tested it, so do at your own risk Mattia
|
||
|
|
||
|
ramesh.bobbala1990@...
Hi Mattia,
Doubt1:. Does your both Orderer admin certs and Peer admin certs got expired or only Orderer admin certs got expired? In my case both Orderer admin certs and Peer admin certs got expired. Doubt2:. I have replaced all the Orderer admin certs and Peer admin certs in system channel configuration and trying to run peer channel update. Then I got the error asking for PeerMSP admin signature, so when I tried to sign with old certificates getting this error, service-peer1/msp/user/msp: signing identity expired 1210h2m17.90829443s ago Is there a way to sign with expired certificates from peer? Doubt3:. I did'nt understood the timeshift combination logic exactly - ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true - ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT=200h - ORDERER_GENERAL_CLUSTER_TLSHANDSHAKETIMESHIFT=200h I mean if I set these env variables in ord0, ord1 and or2 containers and after restarting orderer container where should I set these values for all orderers again? - Orderer0 with timeshift on (can communicate with orderer2) - Orderer1 with timeshift - Orderer2 without timeshift Regards, Ramesh.
|
||
|
|
||
|
Mattia Bolzonella
Hi Ramesh,
1: In my situation I had all the certs expired, both admin (peer and orderers), TLS and MSP certs. 2: You need to sign with old certs in orderer tu run the update, if I recall correctly the ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true (in the docker compose file of the orderer) should get the job done --tlsHandshakeTimeShift x where x is an amount of time greater than the expired time, in your case should be > 1211h. Try with this and the update should run smoothly. 3: I wasn't sure about the cluster parameter, in any case they are well documented here: official documentation. I set the parameteres in the docker compose yaml file for every orderer commenting out to disable the paramenter and restart with docker compose up
|
||
|
|
