peer node unjoin command question
I’m using minifab and I’ve accessed the shell of a peer running in docker and want to unjoin it from the channel. When I enter the command ‘peer node unjoin -c aoc1channel’ I get the message below:
Error: as another peer node command is executing, wait for that command to complete its execution or terminate it before retrying.
From the peer node api I understand that the peer must be offline for the ‘peer node unjoin’ command to work. This requirement applies to most of the ‘peer node’ subcommands such as: pause, reset, rollback.
What I haven’t found is how to make the peer offline so that I can successfully run my command?
Thanks,
Brett Tiller
Sr. Software Engineer
984-349-4239 (mobile)
https://www.linkedin.com/company/securboration
On Jul 13, 2022, at 5:06 PM, Brett Tiller <btiller@...> wrote:
This Message Is From an External SenderThis message came from outside your organization.I’m using minifab and I’ve accessed the shell of a peer running in docker and want to unjoin it from the channel. When I enter the command ‘peer node unjoin -c aoc1channel’ I get the message below:
Error: as another peer node command is executing, wait for that command to complete its execution or terminate it before retrying.
From the peer node api I understand that the peer must be offline for the ‘peer node unjoin’ command to work. This requirement applies to most of the ‘peer node’ subcommands such as: pause, reset, rollback.
What I haven’t found is how to make the peer offline so that I can successfully run my command?
Thanks,
Brett Tiller
Sr. Software Engineer
984-349-4239 (mobile)
https://www.linkedin.com/company/securboration
Thanks for your response. I’ve tried doing that step, however, there’s two issues that occur:
- With the node shutdown I cannot access the shell to unjoin the channel.
- With the node shutdown when I try to push chaincode I get the error below
- ./minifab install,approve,commit,initialize -n simple -l go
i. # Run the chaincode install script on cli container ***********
non-zero return code
Error: failed to retrieve endorser client for install: endorser client failed to connect to 172.26.126.138:7405: failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 172.26.126.138:7405: connect: connection refused"
I’m assuming that the ‘peer node’ cli commands work in the proper conditions. So there must be a way to set the peer to be offline, then access its shell and unjoin the peer from the channel?
Thanks,
Brett Tiller
Sr. Software Engineer
984-349-4239 (mobile)
Sent: Wednesday, July 13, 2022 5:13 PM
To: Brett Tiller <btiller@...>
Cc: fabric@...
Subject: Re: [Hyperledger Fabric] peer node unjoin command question
You can shutdown the docker node.
On Jul 13, 2022, at 5:06 PM, Brett Tiller <btiller@...> wrote:
I’m using minifab and I’ve accessed the shell of a peer running in docker and want to unjoin it from the channel. When I enter the command ‘peer node unjoin -c aoc1channel’ I get the message below:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I’m using minifab and I’ve accessed the shell of a peer running in docker and want to unjoin it from the channel. When I enter the command ‘peer node unjoin -c aoc1channel’ I get the message below:
Error: as another peer node command is executing, wait for that command to complete its execution or terminate it before retrying.
From the peer node api I understand that the peer must be offline for the ‘peer node unjoin’ command to work. This requirement applies to most of the ‘peer node’ subcommands such as: pause, reset, rollback.
What I haven’t found is how to make the peer offline so that I can successfully run my command?
Thanks,
Brett Tiller
Sr. Software Engineer
984-349-4239 (mobile)
https://www.linkedin.com/company/securboration
Message Sender is EXTERNAL to Securboration. Carefully examine this message before you open any links or attachments.
I want to clarify that the peer command is nothing more than a binary. You don’t need access to the peer container’s shell to run that command. The peer binary is available in the official Fabric repository, under Releases. You can download it and run it from your own machine. That means that you can shut down the node, and run an administrative command from outside with the binary.
What you need is access to the correct peer or admin certificates in a MSP folder structure, and pointing to it via the FABRIC_CFG_PATH environment variable (which the peer binary uses to get signing certs and private keys). Also, make sure that your orderer is reachable if your command requires it.
Hope it clarifies!
Best,
Thanks for your response. I’ve tried doing that step, however, there’s two issues that occur:
- With the node shutdown I cannot access the shell to unjoin the channel.
- With the node shutdown when I try to push chaincode I get the error below
- ./minifab install,approve,commit,initialize -n simple -l go
# Run the chaincode install script on cli container ***********
non-zero return code
Error: failed to retrieve endorser client for install: endorser client failed to connect to 172.26.126.138:7405: failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 172.26.126.138:7405: connect: connection refused"
I’m assuming that the ‘peer node’ cli commands work in the proper conditions. So there must be a way to set the peer to be offline, then access its shell and unjoin the peer from the channel?
Thanks,
Brett Tiller
Sr. Software Engineer
984-349-4239 (mobile)
<image001.png>
From: Tong Li <litong01@...>
Sent: Wednesday, July 13, 2022 5:13 PM
To: Brett Tiller <btiller@...>
Cc: fabric@...
Subject: Re: [Hyperledger Fabric] peer node unjoin command question
You can shutdown the docker node.
On Jul 13, 2022, at 5:06 PM, Brett Tiller <btiller@...> wrote:
I’m using minifab and I’ve accessed the shell of a peer running in docker and want to unjoin it from the channel. When I enter the command ‘peer node unjoin -c aoc1channel’ I get the message below:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I’m using minifab and I’ve accessed the shell of a peer running in docker and want to unjoin it from the channel. When I enter the command ‘peer node unjoin -c aoc1channel’ I get the message below:
Error: as another peer node command is executing, wait for that command to complete its execution or terminate it before retrying.
From the peer node api I understand that the peer must be offline for the ‘peer node unjoin’ command to work. This requirement applies to most of the ‘peer node’ subcommands such as: pause, reset, rollback.
What I haven’t found is how to make the peer offline so that I can successfully run my command?
Thanks,
Brett Tiller
Sr. Software Engineer
984-349-4239 (mobile)
<image001.png>
https://www.linkedin.com/company/securboration
Message Sender is EXTERNAL to Securboration. Carefully examine this message before you open any links or attachments.
To unjoin a peer from a channel, the peer server must be shut down at the time the command is issued. For bare metal environments, this is straightforward, as it only requires stopping the peer process, issuing the unjoin command, and restarting the peer node after the unjoin has removed the channel ledger data. But when running the peer node in a docker container, this is not simple as the container is executing the peer binary as PID=0. When the peer process terminates, so will the container! I.e.. it's impossible to unjoin the peer from within a container running the peer. Sam is correct - you will need to issue the "peer unjoin" command somewhere with inputs (env, core.yaml, volume mounts, etc.) similar to how the peer would be running in its normal state.
What you might try doing is to inspect the peer docker image, extract the env and file system mounts, and use these inputs to issue the peer unjoin command from another container. The key here is that the process running the peer node command must be set up with a volume mount and env matching what is normally specified in the peer container. Something like :
docker run --rm -e FABRIC_CFG_PATH -e CORE_PEER_XYZ_OVERRIDE -v /some/local/path:/var/hyperledger/fabric --rm hyperledger/fabric-peer:2.4.4 peer node unjoin -c mychannel
When run in this fashion, the peer node command running in docker just needs access to the same volume mount, core.yaml config, and environment overrides. The node command will scrub the ledger data from the persistent volume mount, and at the next startup of the peer, the channel metadata will be removed at init time.
-josh
Thanks for the tips. As I previously mentioned I’m using the minifab to create my network. In addition to starting the peers in docker containers, among other things minifab creates a var/keyfiles directory that contains the peer, certs, msps and orderers. In order to get the peer cli working I’ve download the Hyperledger-fabric-2.4.5, and took the steps below. There seems to be a setup issue I’m seeing now when running the ‘peer node unjoin’ which has something to do with the msp, though the issue is not clear to me.
- Added to PATH environment variable so peer commands can be run from anywhere
- export PATH=$PATH:~/hyperledger-fabric-2.4.5/bin
- Set environment variable FABRIC_CFG_PATH to be the location of the core.yaml file.
- In ~/.bashrc added
- export FABRIC_CFG_PATH=~/hyperledger-fabric-2.4.5/config
- Set environment variable CORE_PEER_MSPCONFIGPATH
- Source
- export CORE_PEER_MSPCONFIGPATH=$PWD/msp
- I expect to run peer commands from the directory of the peer and the msp is a subdirectory there.
- Example
- /mywork/primii/vars/keyfiles/peerOrganizations/wing11.aoc1.af.mil/peers/squadron111.wing11.aoc1.af.mil
- msp is a subdirectory beneath that.
- Issue with this approach is that I need to source in the .bashrc file everytime I move to a new correct directory so that the environment variable gets reset in my environment.
- Getting error
- peer node unjoin
- 2022-07-14 16:05:25.881 EDT 0001 ERRO [main] InitCmd -> Cannot run peer because error when setting up MSP of type bccsp from directory /home/btiller/mywork/primii/vars/keyfiles/peerOrganizations/wing11.aoc1.af.mil/peers/squadron111.wing11.aoc1.af.mil/msp: KeyMaterial not found in SigningIdentityInfo
- Attempted solutions
- In file ~/hyperledger-fabric-2.4.5/config.core.yaml changed:
- localMsId: wing11.aoc1-af-mil (FAILED!!)
- Was SampleOrg
- localMsId: aoc1-af-mil (FAILED!!)
- Was SampleOrg
Brett Tiller
Sr. Software Engineer
984-349-4239 (mobile)
Sent: Wednesday, July 13, 2022 6:43 PM
To: Tong Li <litong01@...>; Brett Tiller <btiller@...>
Cc: fabric@...
Subject: Re: [Hyperledger Fabric] peer node unjoin command question
Hi Brett,
I want to clarify that the peer command is nothing more than a binary. You don’t need access to the peer container’s shell to run that command. The peer binary is available in the official Fabric repository, under Releases.
You can download it and run it from your own machine. That means that you can shut down the node, and run an administrative command from outside with the binary.
What you need is access to the correct peer or admin certificates in a MSP folder structure, and pointing to it via the FABRIC_CFG_PATH environment variable (which the peer binary uses to
get signing certs and private keys). Also, make sure that your orderer is reachable if your command requires it.
Hope it clarifies!
Best,
On 13 Jul 2022 18:30 -0300, Brett Tiller <btiller@...>, wrote:
Thanks for your response. I’ve tried doing that step, however, there’s two issues that occur:
- With the node shutdown I cannot access the shell to unjoin the channel.
- With the node shutdown when I try to push chaincode I get the error below
- ./minifab install,approve,commit,initialize -n simple -l go
# Run the chaincode install script on cli container ***********
non-zero return code
Error: failed to retrieve endorser client for install: endorser client failed to connect to 172.26.126.138:7405: failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 172.26.126.138:7405: connect: connection refused"
I’m assuming that the ‘peer node’ cli commands work in the proper conditions. So there must be a way to set the peer to be offline, then access its shell and unjoin the peer from the channel?
Thanks,
Brett Tiller
Sr. Software Engineer
984-349-4239 (mobile)
<image001.png>
From: Tong Li <litong01@...>
Sent: Wednesday, July 13, 2022 5:13 PM
To: Brett Tiller <btiller@...>
Cc: fabric@...
Subject: Re: [Hyperledger Fabric] peer node unjoin command question
You can shutdown the docker node.
On Jul 13, 2022, at 5:06 PM, Brett Tiller <btiller@...> wrote:
I’m using minifab and I’ve accessed the shell of a peer running in docker and want to unjoin it from the channel. When I enter the command ‘peer node unjoin -c aoc1channel’ I get the message below:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I’m using minifab and I’ve accessed the shell of a peer running in docker and want to unjoin it from the channel. When I enter the command ‘peer node unjoin -c aoc1channel’ I get the message below:
Error: as another peer node command is executing, wait for that command to complete its execution or terminate it before retrying.
From the peer node api I understand that the peer must be offline for the ‘peer node unjoin’ command to work. This requirement applies to most of the ‘peer node’ subcommands such as: pause, reset, rollback.
What I haven’t found is how to make the peer offline so that I can successfully run my command?
Thanks,
Brett Tiller
Sr. Software Engineer
984-349-4239 (mobile)
<image001.png>
https://www.linkedin.com/company/securboration
Message Sender is EXTERNAL to Securboration. Carefully examine this message before you open any links or attachments.
Message Sender is EXTERNAL to Securboration. Carefully examine this message before you open any links or attachments.
Is there a way to keep both the peer process and container alive, but with the peer in an offline state?
Given that the peer node cli requires the peer to be offline, it seems to me that there should also be another cli call to change the state of the peer to offline, but I haven't found it yet.
-Brett
Unfortunately, no; There is no way to keep the peer process running in an "offline" state. There are a few file-level atomic locks in the code that ensure that admin (e.g. "peer CLI" - client) commands do not interfere with a running peer (server node) process, but other than at this granularity there's no logic or concept of an "admin mode" for a running peer in Fabric. The peer process includes a small section of the bootstrap that enters an "init" phase before the peer will accept traffic, but unfortunately there's no way to revert into this "init" phase for a running peer. The peer is either running, or it's not. This presents a challenge for Fabric networks based on containers (kube, swarm, compose, minikube, etc....), all of which will terminate the container as soon as the peer binary terminates.
When implementing the node unjoin feature, we really tried to find a solution that would allow for a channel to be unjoined while the node was up and running. But unfortunately, the way the data structures and functions in the peer code are organized, it is virtually impossible to unjoin a channel from a running peer without performing substantial reorganization of the code and concurrency controls. We did, however, try to achieve an OK compromise that would allow for a peer to be unjoined, despite the additional complexity in the operational burdens.
Really what I recommend in this case is to try to emulate, as closely as possible, the natural state that is set up in the peer container by Minikube, and re-launch the peer container with the same or similar setup. Effectively:
1. run a "docker inspect" on the target peer, and mine the ENV and volume mounts from the running container.
2. shut down the peer container.
3. manually run "docker run --rm ... fabric-peer node unjoin ...", passing the volume mount points and ENV settings from the inspection in #1.
4. restart the peer.
While the peer command can be run on your local machine, the important bits to replicate are the location and config of core.yaml, the ENV settings, and the volume mounts with access to the ledger persistent data. Once those are aligned the peer node unjoin commands will scrub the ledger from the peer file system (Docker volume mount), and gone from the channel for good.
--josh
I've run 'docker inspect' on the peer and there's a lot of data there so I'm unclear of what to use and how to pass it in 'docker run'. Regarding using 'docker run' to run the peer node unjoin command on the shutdown peer - step 3 that you provided - would you point me to/provide a detailed example of the command line with the volume mount points and ENV settings passed in?
Thanks,
Brett
docker run --rm --env FABRIC_CFG_PATH=/etc/hyperledger/peercfg --env FABRIC_LOGGING_SPEC=INFO --env CORE_PEER_TLS_ENABLED=true --env CORE_PEER_PROFILE_ENABLED=false --env CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt --env CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key --env CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt --env CORE_PEER_ID=peer0.org2.example.com --env CORE_PEER_ADDRESS=peer0.org2.example.com:9051 --env CORE_PEER_LISTENADDRESS=0.0.0.0:9051 --env CORE_PEER_CHAINCODEADDRESS=peer0.org2.example.com:9052 --env CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:9052 --env CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org2.example.com:9051 --env CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:9051 --env CORE_PEER_LOCALMSPID=Org2MSP --env CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/fabric/msp --env CORE_OPERATIONS_LISTENADDRESS=peer0.org2.example.com:9445 --env CORE_METRICS_PROVIDER=prometheus --env CHAINCODE_AS_A_SERVICE_BUILDER_CONFIG={\peername\:\peer0org2\} --env CORE_CHAINCODE_EXECUTETIMEOUT=300s --env CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock --env CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=fabric_test --env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -v /var/run/docker.sock:/host/var/run/docker.sock:rw -v compose_peer0.org2.example.com:/var/hyperledger/production:rw -v /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/a5b28420ffb3e2319edddf9935165174833b3fa392fb430a0ab1ff127fd2b0df:/etc/hyperledger/peercfg:rw -v /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/43060a93a20773973c4b6f32d9fd587016a97684b66f6fbc63b0fdea76ceebe2:/etc/hyperledger/fabric:rw --rm hyperledger/fabric-peer:latest peer node unjoin -c mychannel
After the modified peer has been restarted in the fabric-samples test-network to rejoin the channel environment variables must be set. I've provided an example below.
export CORE_PEER_TLS_ENABLED=true
export CORE_PEER_LOCALMSPID="Org2MSP"
export CORE_PEER_TLS_ROOTCERT_FILE=${PWD}/organizations/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt
export CORE_PEER_MSPCONFIGPATH=${PWD}/organizations/peerOrganizations/org2.example.com/users/Admin@.../msp
export CORE_PEER_ADDRESS=localhost:9051
Regarding having the set/reset the environment variables to add a peer to the channel, is this the only way to do this process in the test-network? I was hoping to be able to join a peer to a network without need for setting environment variables each time.
Thanks,
Brett