peer node unjoin command question
|
Brett Tiller
I've been able to get the 'peer node unjoin' command to work by following the steps Josh provided. The command line requires the records provided by docker inspect <container id>; specifically the data from Config/Env and HostConfig/Binds. I've provided a command line example below that works.
docker run --rm --env FABRIC_CFG_PATH=/etc/hyperledger/peercfg --env FABRIC_LOGGING_SPEC=INFO --env CORE_PEER_TLS_ENABLED=true --env CORE_PEER_PROFILE_ENABLED=false --env CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt --env CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key --env CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt --env CORE_PEER_ID=peer0.org2.example.com --env CORE_PEER_ADDRESS=peer0.org2.example.com:9051 --env CORE_PEER_LISTENADDRESS=0.0.0.0:9051 --env CORE_PEER_CHAINCODEADDRESS=peer0.org2.example.com:9052 --env CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:9052 --env CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org2.example.com:9051 --env CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:9051 --env CORE_PEER_LOCALMSPID=Org2MSP --env CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/fabric/msp --env CORE_OPERATIONS_LISTENADDRESS=peer0.org2.example.com:9445 --env CORE_METRICS_PROVIDER=prometheus --env CHAINCODE_AS_A_SERVICE_BUILDER_CONFIG={\peername\:\peer0org2\} --env CORE_CHAINCODE_EXECUTETIMEOUT=300s --env CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock --env CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=fabric_test --env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -v /var/run/docker.sock:/host/var/run/docker.sock:rw -v compose_peer0.org2.example.com:/var/hyperledger/production:rw -v /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/a5b28420ffb3e2319edddf9935165174833b3fa392fb430a0ab1ff127fd2b0df:/etc/hyperledger/peercfg:rw -v /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/43060a93a20773973c4b6f32d9fd587016a97684b66f6fbc63b0fdea76ceebe2:/etc/hyperledger/fabric:rw --rm hyperledger/fabric-peer:latest peer node unjoin -c mychannel After the modified peer has been restarted in the fabric-samples test-network to rejoin the channel environment variables must be set. I've provided an example below. export CORE_PEER_TLS_ENABLED=true export CORE_PEER_LOCALMSPID="Org2MSP" export CORE_PEER_TLS_ROOTCERT_FILE=${PWD}/organizations/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt export CORE_PEER_MSPCONFIGPATH=${PWD}/organizations/peerOrganizations/org2.example.com/users/Admin@.../msp export CORE_PEER_ADDRESS=localhost:9051 Regarding having the set/reset the environment variables to add a peer to the channel, is this the only way to do this process in the test-network? I was hoping to be able to join a peer to a network without need for setting environment variables each time. Thanks, Brett |
||||
|
|
||||
|
Brett Tiller
Thanks Josh,
I've run 'docker inspect' on the peer and there's a lot of data there so I'm unclear of what to use and how to pass it in 'docker run'. Regarding using 'docker run' to run the peer node unjoin command on the shutdown peer - step 3 that you provided - would you point me to/provide a detailed example of the command line with the volume mount points and ENV settings passed in? Thanks, Brett |
||||
|
|
||||
|
jkneubuh@...
Hi Brett -
Unfortunately, no; There is no way to keep the peer process running in an "offline" state. There are a few file-level atomic locks in the code that ensure that admin (e.g. "peer CLI" - client) commands do not interfere with a running peer (server node) process, but other than at this granularity there's no logic or concept of an "admin mode" for a running peer in Fabric. The peer process includes a small section of the bootstrap that enters an "init" phase before the peer will accept traffic, but unfortunately there's no way to revert into this "init" phase for a running peer. The peer is either running, or it's not. This presents a challenge for Fabric networks based on containers (kube, swarm, compose, minikube, etc....), all of which will terminate the container as soon as the peer binary terminates. When implementing the node unjoin feature, we really tried to find a solution that would allow for a channel to be unjoined while the node was up and running. But unfortunately, the way the data structures and functions in the peer code are organized, it is virtually impossible to unjoin a channel from a running peer without performing substantial reorganization of the code and concurrency controls. We did, however, try to achieve an OK compromise that would allow for a peer to be unjoined, despite the additional complexity in the operational burdens. Really what I recommend in this case is to try to emulate, as closely as possible, the natural state that is set up in the peer container by Minikube, and re-launch the peer container with the same or similar setup. Effectively: 1. run a "docker inspect" on the target peer, and mine the ENV and volume mounts from the running container. 2. shut down the peer container. 3. manually run "docker run --rm ... fabric-peer node unjoin ...", passing the volume mount points and ENV settings from the inspection in #1. 4. restart the peer. While the peer command can be run on your local machine, the important bits to replicate are the location and config of core.yaml, the ENV settings, and the volume mounts with access to the ledger persistent data. Once those are aligned the peer node unjoin commands will scrub the ledger from the peer file system (Docker volume mount), and gone from the channel for good. --josh |
||||
|
|
||||
|
Brett Tiller
Thanks for the info Josh. As you know the 'peer node unjoin' command requires that the peer running in the docker container is in an offline state. You've pointed out that if the peer process is shutdown in a docker container, effectively making it offline, that the container will be shutdown as well which poses a problem.
Is there a way to keep both the peer process and container alive, but with the peer in an offline state? Given that the peer node cli requires the peer to be offline, it seems to me that there should also be another cli call to change the state of the peer to offline, but I haven't found it yet. -Brett |
||||
|
|
||||
|
Brett Tiller
Thanks for the tips. As I previously mentioned I’m using the minifab to create my network. In addition to starting the peers in docker containers, among other things minifab creates a var/keyfiles directory that contains the peer, certs, msps and orderers. In order to get the peer cli working I’ve download the Hyperledger-fabric-2.4.5, and took the steps below. There seems to be a setup issue I’m seeing now when running the ‘peer node unjoin’ which has something to do with the msp, though the issue is not clear to me.
Brett Tiller Sr. Software Engineer 984-349-4239 (mobile)
From: Samuel Venzi <samuel.venzi@...>
Sent: Wednesday, July 13, 2022 6:43 PM To: Tong Li <litong01@...>; Brett Tiller <btiller@...> Cc: fabric@... Subject: Re: [Hyperledger Fabric] peer node unjoin command question
Hi Brett, On 13 Jul 2022 18:30 -0300, Brett Tiller <btiller@...>, wrote:
Message Sender is EXTERNAL to Securboration. Carefully examine this message before you open any links or attachments. |
||||
|
|
||||
|
jkneubuh@...
Hi Sam and Tong and Brett,
To unjoin a peer from a channel, the peer server must be shut down at the time the command is issued. For bare metal environments, this is straightforward, as it only requires stopping the peer process, issuing the unjoin command, and restarting the peer node after the unjoin has removed the channel ledger data. But when running the peer node in a docker container, this is not simple as the container is executing the peer binary as PID=0. When the peer process terminates, so will the container! I.e.. it's impossible to unjoin the peer from within a container running the peer. Sam is correct - you will need to issue the "peer unjoin" command somewhere with inputs (env, core.yaml, volume mounts, etc.) similar to how the peer would be running in its normal state. What you might try doing is to inspect the peer docker image, extract the env and file system mounts, and use these inputs to issue the peer unjoin command from another container. The key here is that the process running the peer node command must be set up with a volume mount and env matching what is normally specified in the peer container. Something like : docker run --rm -e FABRIC_CFG_PATH -e CORE_PEER_XYZ_OVERRIDE -v /some/local/path:/var/hyperledger/fabric --rm hyperledger/fabric-peer:2.4.4 peer node unjoin -c mychannel When run in this fashion, the peer node command running in docker just needs access to the same volume mount, core.yaml config, and environment overrides. The node command will scrub the ledger data from the persistent volume mount, and at the next startup of the peer, the channel metadata will be removed at init time. -josh |
||||
|
|
||||
|
Sam Venzi
Hi Brett,
I want to clarify that the peer command is nothing more than a binary. You don’t need access to the peer container’s shell to run that command. The peer binary is available in the official Fabric repository, under Releases. You can download it and run it from your own machine. That means that you can shut down the node, and run an administrative command from outside with the binary. What you need is access to the correct peer or admin certificates in a MSP folder structure, and pointing to it via the FABRIC_CFG_PATH environment variable (which the peer binary uses to get signing certs and private keys). Also, make sure that your orderer is reachable if your command requires it. Hope it clarifies! Best, On 13 Jul 2022 18:30 -0300, Brett Tiller <btiller@...>, wrote:
|
||||
|
|
||||
|
Brett Tiller
Thanks for your response. I’ve tried doing that step, however, there’s two issues that occur:
i. # Run the chaincode install script on cli container *********** non-zero return code Error: failed to retrieve endorser client for install: endorser client failed to connect to 172.26.126.138:7405: failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 172.26.126.138:7405: connect: connection refused"
I’m assuming that the ‘peer node’ cli commands work in the proper conditions. So there must be a way to set the peer to be offline, then access its shell and unjoin the peer from the channel?
Thanks,
Brett Tiller Sr. Software Engineer 984-349-4239 (mobile)
From: Tong Li <litong01@...>
Sent: Wednesday, July 13, 2022 5:13 PM To: Brett Tiller <btiller@...> Cc: fabric@... Subject: Re: [Hyperledger Fabric] peer node unjoin command question
You can shutdown the docker node.
Message Sender is EXTERNAL to Securboration. Carefully examine this message before you open any links or attachments. |
||||
|
|
||||
|
Tong Li
You can shutdown the docker node.
toggle quoted message
Show quoted text
On Jul 13, 2022, at 5:06 PM, Brett Tiller <btiller@...> wrote:
|
||||
|
|
||||
|
Brett Tiller
I’m using minifab and I’ve accessed the shell of a peer running in docker and want to unjoin it from the channel. When I enter the command ‘peer node unjoin -c aoc1channel’ I get the message below:
Error: as another peer node command is executing, wait for that command to complete its execution or terminate it before retrying.
From the peer node api I understand that the peer must be offline for the ‘peer node unjoin’ command to work. This requirement applies to most of the ‘peer node’ subcommands such as: pause, reset, rollback.
What I haven’t found is how to make the peer offline so that I can successfully run my command?
Thanks,
Brett Tiller Sr. Software Engineer 984-349-4239 (mobile)
https://www.linkedin.com/company/securboration
|
||||
|
|
