1. Topics
  2. Endorsement Policy Failure while committing chaincode

Endorsement Policy Failure while committing chaincode #fabric-kubernetes #hyperledger-fabric #chaincode

Arsh <arshimag@...>
 

I'm very much new to blockchain and hyperledger. I'm trying to setup my first network with 8 orgs containing 3 peers each and 7 orderers. Everything is working fine using docker-compose. I'm trying to replicate the same on kubernetes following this tutorial https://github.com/hyfen-nl/PIVT. All the stages of chaincode right from packaging till checking for commit readiness are giving a positive and desired response. However, committing the chaincode fails with 


2022-06-20 12:35:38.641 UTC [chaincodeCmd] ClientWait -> INFO 0a0 txid [619d23f6fac89442c9f858749e6c2c809a7a1833373a634cd4cb56a2660bb84e] committed with status (ENDORSEMENT_POLICY_FAILURE) at peer0.ni.example.org:7051
2022-06-20 12:35:38.641 UTC [chaincodeCmd] ClientWait -> INFO 0a1 txid [619d23f6fac89442c9f858749e6c2c809a7a1833373a634cd4cb56a2660bb84e] committed with status (ENDORSEMENT_POLICY_FAILURE) at peer0.es.example.org:7051
2022-06-20 12:35:38.644 UTC [chaincodeCmd] ClientWait -> INFO 0a2 txid [619d23f6fac89442c9f858749e6c2c809a7a1833373a634cd4cb56a2660bb84e] committed with status (ENDORSEMENT_POLICY_FAILURE) at peer0.highereducation.example.org:7051
2022-06-20 12:35:38.644 UTC [chaincodeCmd] ClientWait -> INFO 0a3 txid [619d23f6fac89442c9f858749e6c2c809a7a1833373a634cd4cb56a2660bb84e] committed with status (ENDORSEMENT_POLICY_FAILURE) at peer0.schooleducation.example.org:7051
Error: transaction invalidated with status (ENDORSEMENT_POLICY_FAILURE) 


The following is a part of configtx.yaml which I'm using and is the same file used while setting up using docker-compose except for minor changes like using different port numbers.
 
 
        # Policies defines the set of policies at this level of the config tree
        # For organization policies, their canonical path is usually
        #   /Channel/<Application|Orderer>/<OrgName>/<PolicyName>
        Policies:
            Readers:
                Type: Signature
                Rule: "OR('OrdererMSP.member')"
            Writers:
                Type: Signature
                Rule: "OR('OrdererMSP.member')"
            Admins:
                Type: Signature
                Rule: "OR('OrdererMSP.admin')"
    - &Ni
        Name: NiMSP
        ID: NiMSP
        MSPDir: crypto-config/peerOrganizations/ni.example.org/msp
        Policies:
            Readers:
                Type: Signature
                Rule: "OR('NiMSP.admin', 'NiMSP.peer', 'NiMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('NiMSP.admin', 'NiMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('NiMSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('NiMSP.peer')"
        AnchorPeers:
            - Host: peer0.ni.example.org
              Port: 7051
 
    - &Schooleducation
        Name: SchooleducationMSP
        ID: SchooleducationMSP
        MSPDir: crypto-config/peerOrganizations/schooleducation.example.org/msp
        Policies:
            Readers:
                Type: Signature
                Rule: "OR('SchooleducationMSP.admin', 'SchooleducationMSP.peer', 'SchooleducationMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('SchooleducationMSP.admin', 'SchooleducationMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('SchooleducationMSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('SchooleducationMSP.peer')"
        AnchorPeers:
            - Host: peer0.schooleducation.example.org
              Port: 7051
 
    - &Highereducation
        Name: HighereducationMSP
        ID: HighereducationMSP
        MSPDir: crypto-config/peerOrganizations/highereducation.example.org/msp
        Policies:
            Readers:
                Type: Signature
                Rule: "OR('HighereducationMSP.admin', 'HighereducationMSP.peer', 'HighereducationMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('HighereducationMSP.admin', 'HighereducationMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('HighereducationMSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('HighereducationMSP.peer')"
        AnchorPeers:
            - Host: peer0.highereducation.example.org
              Port: 7051
 
    - &Es
        Name: EsMSP
        ID: EsMSP
        MSPDir: crypto-config/peerOrganizations/esevai.example.org/msp
        Policies:
            Readers:
                Type: Signature
                Rule: "OR('EsMSP.admin', 'EsMSP.peer', 'EsMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('EsMSP.admin', 'EsMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('EsMSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('EsMSP.peer')"
        AnchorPeers:
            - Host: peer0.es.example.org
              Port: 7051
 
    - &Igr
        Name: IgrMSP
        ID: IgrMSP
        MSPDir: crypto-config/peerOrganizations/igr.example.org/msp
        Policies:
            Readers:
                Type: Signature
                Rule: "OR('IgrMSP.admin', 'IgrMSP.peer', 'IgrMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('IgrMSP.admin', 'IgrMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('IgrMSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('IgrMSP.peer')"
        AnchorPeers:
            - Host: peer0.igr.example.org
              Port: 7051
 
    - &Forest
        Name: ForestMSP
        ID: ForestMSP
        MSPDir: crypto-config/peerOrganizations/forest.example.org/msp
        Policies:
            Readers:
                Type: Signature
                Rule: "OR('ForestMSP.admin', 'ForestMSP.peer', 'ForestMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('ForestMSP.admin', 'ForestMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('ForestMSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('ForestMSP.peer')"
        AnchorPeers:
            - Host: peer0.forest.example.org
              Port: 7051
 
    - &Handicrafts
        Name: HandicraftsMSP
        ID: HandicraftsMSP
        MSPDir: crypto-config/peerOrganizations/handicrafts.example.org/msp
        Policies:
            Readers:
                Type: Signature
                Rule: "OR('HandicraftsMSP.admin', 'HandicraftsMSP.peer', 'HandicraftsMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('HandicraftsMSP.admin', 'HandicraftsMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('HandicraftsMSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('HandicraftsMSP.peer')"
        AnchorPeers:
            - Host: peer0.handicrafts.example.org
              Port: 7051
 
    - &ITDepartment
        Name: ITDepartmentMSP
        ID: ITDepartmentMSP
        MSPDir: crypto-config/peerOrganizations/itdepartment.example.org/msp
        Policies:
            Readers:
                Type: Signature
                Rule: "OR('ITDepartmentMSP.admin', 'ITDepartmentMSP.peer', 'ITDepartmentMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('ITDepartmentMSP.admin', 'ITDepartmentMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('ITDepartmentMSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('ITDepartmentMSP.peer')"
        AnchorPeers:
            - Host: peer0.itdepartment.example.org
              Port: 7051

    #   /Channel/Application/<PolicyName>
    Policies:
        Readers:
            Type: ImplicitMeta
            Rule: "ANY Readers"
        Writers:
            Type: ImplicitMeta
            Rule: "ANY Writers"
        Admins:
            Type: ImplicitMeta
            Rule: "MAJORITY Admins"
        LifecycleEndorsement:
            Type: ImplicitMeta
            Rule: "MAJORITY Endorsement"
        Endorsement:
            Type: ImplicitMeta
            Rule: "MAJORITY Endorsement"
 
    Capabilities:
        <<: *ApplicationCapabilities
    Policies:
        Readers:
              Type: ImplicitMeta
              Rule: "ANY Readers"
        Writers:
            Type: ImplicitMeta
            Rule: "ANY Writers"
        Admins:
            Type: ImplicitMeta
            Rule: "MAJORITY Admins"
        BlockValidation:
            Type: ImplicitMeta
            Rule: "ANY Writers"
 
Below is the output for check commit readiness command:
 
{
        "approvals": {
                "EsMSP": true,
                "HighereducationMSP": true,
                "NiMSP": true,
                "SchooleducationMSP": true
        }
}


Command for approving the chaincode:
peer lifecycle chaincode approveformyorg --tls --cafile /hl_config/orderer-tlsca/tlscacert.pem  --channelID certificatechannel --name cc --version 1 --package-id  cc_1:1234567.....4965 --sequence 1 --waitForEvent
 
Command for committing the chaincode:
peer lifecycle chaincode commit -o orderer1.example.org:7050 --channelID certificatechannel --name cc --version 1 --sequence 1 --tls --cafile /hl_config/orde
rer-tlsca/tlscacert.pem  --peerAddresses peer0.ni.example.org:7051 --tlsRootCertFiles /etc/hyperledger/fabric/crypto-config/peerOrganizations/ni.tnega.org/peers/peer0.ni.
example.org/tls/ca.crt --peerAddresses peer0.schooleducation.example.org:7051 --tlsRootCertFiles /etc/hyperledger/fabric/crypto-config/peerOrganizations/schooleducation.example.org/peers/peer0.schooleducation.example.org/tls/ca.crt --peerAddresses peer0.highereducation.example.org:7051 --tlsRootCertFiles /etc/hyperledger/fabric/crypto-config/pe
erOrganizations/highereducation.example.org/peers/peer0.highereducation.example.org/tls/ca.crt --peerAddresses peer0.es.example.org:7051 --tlsRootCertFiles /etc/hyperledger/fabric/crypto-config/peerOrganizations/es.example.org/peers/peer0.es.example.org/tls/ca.crt

What I've tried:
I've tried running commit command passing one peer address of only one org at a time and received the same output. 
I've tried running commit command passing one peer address of all the orgs that have approved the chaincode and have recieved the same output.
I tried executing the command from inside one of the peers.

Note: 
-------
I'm making use of argo workflows along with helm templates and kubernetes to automate the chaincode process.

Any help is much appreciated. Thanks in advance!
jkneubuh@...
 

Hi Arsh,

The `peer` CLI also uses some context and input parameters from the core.yaml configuration file - make sure that these are setting the correct MSP context when running the commit, or have suitable overrides in the environment (e.g. CORE_PEER_ ...).  I wonder if your commit is being rejected as it's submitting a transaction with an incorrect policy.

Also you should be aware that last week we released a production-ready Kubernetes fabric-operator to the open source community.  The operator includes support for both chaincode-as-a-service and the new fabric-builder-k8s chaincode builders, both of which may help to reduce or eliminate the need for Argo workflows in your chaincode deployments.  You might have a good luck starting with the operator sample-network, then building up towards your multi-org blockchain from a stable base.

Would love to hear feedback on the new operator, and if it meets your needs for quickly and predictably spinning up blockchains with the declarative CRDs and Helm.  There is also a public Discord channel over at #fabric-kubernetes where the community can engage on topics related to deployment on a cluster.

-josh
David Enyeart
 

Your LifecycleEndorsement is set to a majority of orgs. So you will need to approve the chaincode on at least 5 of 8 orgs and then send the commit request to at least those 5 orgs. It looks like your commit request is only going to 4 of 8 orgs based on the peerAddresses properties.

 

From: fabric@... <fabric@...> on behalf of Arsh <arshimag@...>
Date: Tuesday, June 21, 2022 at 5:25 AM
To: fabric@... <fabric@...>
Subject: [EXTERNAL] [Hyperledger Fabric] Endorsement Policy Failure while committing chaincode #fabric-kubernetes #hyperledger-fabric #chaincode

I'm very much new to blockchain and hyperledger. I'm trying to setup my first network with 8 orgs containing 3 peers each and 7 orderers. Everything is working fine using docker-compose. I'm trying to replicate the same on kubernetes following

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

I'm very much new to blockchain and hyperledger. I'm trying to setup my first network with 8 orgs containing 3 peers each and 7 orderers. Everything is working fine using docker-compose. I'm trying to replicate the same on kubernetes following this tutorial https://github.com/hyfen-nl/PIVT. All the stages of chaincode right from packaging till checking for commit readiness are giving a positive and desired response. However, committing the chaincode fails with 

 

2022-06-20 12:35:38.641 UTC [chaincodeCmd] ClientWait -> INFO 0a0 txid [619d23f6fac89442c9f858749e6c2c809a7a1833373a634cd4cb56a2660bb84e] committed with status (ENDORSEMENT_POLICY_FAILURE) at peer0.ni.example.org:7051

2022-06-20 12:35:38.641 UTC [chaincodeCmd] ClientWait -> INFO 0a1 txid [619d23f6fac89442c9f858749e6c2c809a7a1833373a634cd4cb56a2660bb84e] committed with status (ENDORSEMENT_POLICY_FAILURE) at peer0.es.example.org:7051

2022-06-20 12:35:38.644 UTC [chaincodeCmd] ClientWait -> INFO 0a2 txid [619d23f6fac89442c9f858749e6c2c809a7a1833373a634cd4cb56a2660bb84e] committed with status (ENDORSEMENT_POLICY_FAILURE) at peer0.highereducation.example.org:7051

2022-06-20 12:35:38.644 UTC [chaincodeCmd] ClientWait -> INFO 0a3 txid [619d23f6fac89442c9f858749e6c2c809a7a1833373a634cd4cb56a2660bb84e] committed with status (ENDORSEMENT_POLICY_FAILURE) at peer0.schooleducation.example.org:7051

Error: transaction invalidated with status (ENDORSEMENT_POLICY_FAILURE) 

The following is a part of configtx.yaml which I'm using and is the same file used while setting up using docker-compose except for minor changes like using different port numbers.

 

 

        # Policies defines the set of policies at this level of the config tree

        # For organization policies, their canonical path is usually

        #   /Channel/<Application|Orderer>/<OrgName>/<PolicyName>

        Policies:

            Readers:

                Type: Signature

                Rule: "OR('OrdererMSP.member')"

            Writers:

                Type: Signature

                Rule: "OR('OrdererMSP.member')"

            Admins:

                Type: Signature

                Rule: "OR('OrdererMSP.admin')"

    - &Ni

        Name: NiMSP

        ID: NiMSP

        MSPDir: crypto-config/peerOrganizations/ni.example.org/msp

        Policies:

            Readers:

                Type: Signature

                Rule: "OR('NiMSP.admin', 'NiMSP.peer', 'NiMSP.client')"

            Writers:

                Type: Signature

                Rule: "OR('NiMSP.admin', 'NiMSP.client')"

            Admins:

                Type: Signature

                Rule: "OR('NiMSP.admin')"

            Endorsement:

                Type: Signature

                Rule: "OR('NiMSP.peer')"

        AnchorPeers:

            - Host: peer0.ni.example.org

              Port: 7051

 

    - &Schooleducation

        Name: SchooleducationMSP

        ID: SchooleducationMSP

        MSPDir: crypto-config/peerOrganizations/schooleducation.example.org/msp

        Policies:

            Readers:

                Type: Signature

                Rule: "OR('SchooleducationMSP.admin', 'SchooleducationMSP.peer', 'SchooleducationMSP.client')"

            Writers:

                Type: Signature

                Rule: "OR('SchooleducationMSP.admin', 'SchooleducationMSP.client')"

            Admins:

                Type: Signature

                Rule: "OR('SchooleducationMSP.admin')"

            Endorsement:

                Type: Signature

                Rule: "OR('SchooleducationMSP.peer')"

        AnchorPeers:

            - Host: peer0.schooleducation.example.org

              Port: 7051

 

    - &Highereducation

        Name: HighereducationMSP

        ID: HighereducationMSP

        MSPDir: crypto-config/peerOrganizations/highereducation.example.org/msp

        Policies:

            Readers:

                Type: Signature

                Rule: "OR('HighereducationMSP.admin', 'HighereducationMSP.peer', 'HighereducationMSP.client')"

            Writers:

                Type: Signature

                Rule: "OR('HighereducationMSP.admin', 'HighereducationMSP.client')"

            Admins:

                Type: Signature

                Rule: "OR('HighereducationMSP.admin')"

            Endorsement:

                Type: Signature

                Rule: "OR('HighereducationMSP.peer')"

        AnchorPeers:

            - Host: peer0.highereducation.example.org

              Port: 7051

 

    - &Es

        Name: EsMSP

        ID: EsMSP

        MSPDir: crypto-config/peerOrganizations/esevai.example.org/msp

        Policies:

            Readers:

                Type: Signature

                Rule: "OR('EsMSP.admin', 'EsMSP.peer', 'EsMSP.client')"

            Writers:

                Type: Signature

                Rule: "OR('EsMSP.admin', 'EsMSP.client')"

            Admins:

                Type: Signature

                Rule: "OR('EsMSP.admin')"

            Endorsement:

                Type: Signature

                Rule: "OR('EsMSP.peer')"

        AnchorPeers:

            - Host: peer0.es.example.org

              Port: 7051

 

    - &Igr

        Name: IgrMSP

        ID: IgrMSP

        MSPDir: crypto-config/peerOrganizations/igr.example.org/msp

        Policies:

            Readers:

                Type: Signature

                Rule: "OR('IgrMSP.admin', 'IgrMSP.peer', 'IgrMSP.client')"

            Writers:

                Type: Signature

                Rule: "OR('IgrMSP.admin', 'IgrMSP.client')"

            Admins:

                Type: Signature

                Rule: "OR('IgrMSP.admin')"

            Endorsement:

                Type: Signature

                Rule: "OR('IgrMSP.peer')"

        AnchorPeers:

            - Host: peer0.igr.example.org

              Port: 7051

 

    - &Forest

        Name: ForestMSP

        ID: ForestMSP

        MSPDir: crypto-config/peerOrganizations/forest.example.org/msp

        Policies:

            Readers:

                Type: Signature

                Rule: "OR('ForestMSP.admin', 'ForestMSP.peer', 'ForestMSP.client')"

            Writers:

                Type: Signature

                Rule: "OR('ForestMSP.admin', 'ForestMSP.client')"

            Admins:

                Type: Signature

                Rule: "OR('ForestMSP.admin')"

            Endorsement:

                Type: Signature

                Rule: "OR('ForestMSP.peer')"

        AnchorPeers:

            - Host: peer0.forest.example.org

              Port: 7051

 

    - &Handicrafts

        Name: HandicraftsMSP

        ID: HandicraftsMSP

        MSPDir: crypto-config/peerOrganizations/handicrafts.example.org/msp

        Policies:

            Readers:

                Type: Signature

                Rule: "OR('HandicraftsMSP.admin', 'HandicraftsMSP.peer', 'HandicraftsMSP.client')"

            Writers:

                Type: Signature

                Rule: "OR('HandicraftsMSP.admin', 'HandicraftsMSP.client')"

            Admins:

                Type: Signature

                Rule: "OR('HandicraftsMSP.admin')"

            Endorsement:

                Type: Signature

                Rule: "OR('HandicraftsMSP.peer')"

        AnchorPeers:

            - Host: peer0.handicrafts.example.org

              Port: 7051

 

    - &ITDepartment

        Name: ITDepartmentMSP

        ID: ITDepartmentMSP

        MSPDir: crypto-config/peerOrganizations/itdepartment.example.org/msp

        Policies:

            Readers:

                Type: Signature

                Rule: "OR('ITDepartmentMSP.admin', 'ITDepartmentMSP.peer', 'ITDepartmentMSP.client')"

            Writers:

                Type: Signature

                Rule: "OR('ITDepartmentMSP.admin', 'ITDepartmentMSP.client')"

            Admins:

                Type: Signature

                Rule: "OR('ITDepartmentMSP.admin')"

            Endorsement:

                Type: Signature

                Rule: "OR('ITDepartmentMSP.peer')"

        AnchorPeers:

            - Host: peer0.itdepartment.example.org

              Port: 7051


    #   /Channel/Application/<PolicyName>

    Policies:

        Readers:

            Type: ImplicitMeta

            Rule: "ANY Readers"

        Writers:

            Type: ImplicitMeta

            Rule: "ANY Writers"

        Admins:

            Type: ImplicitMeta

            Rule: "MAJORITY Admins"

        LifecycleEndorsement:

            Type: ImplicitMeta

            Rule: "MAJORITY Endorsement"

        Endorsement:

            Type: ImplicitMeta

            Rule: "MAJORITY Endorsement"

 

    Capabilities:

        <<: *ApplicationCapabilities

    Policies:

        Readers:

              Type: ImplicitMeta

              Rule: "ANY Readers"

        Writers:

            Type: ImplicitMeta

            Rule: "ANY Writers"

        Admins:

            Type: ImplicitMeta

            Rule: "MAJORITY Admins"

        BlockValidation:

            Type: ImplicitMeta

            Rule: "ANY Writers"

 
Below is the output for check commit readiness command:

 

{

        "approvals": {

                "EsMSP": true,

                "HighereducationMSP": true,

                "NiMSP": true,

                "SchooleducationMSP": true

        }

}

 

Command for approving the chaincode:

peer lifecycle chaincode approveformyorg --tls --cafile /hl_config/orderer-tlsca/tlscacert.pem  --channelID certificatechannel --name cc --version 1 --package-id  cc_1:1234567.....4965 --sequence 1 --waitForEvent

 

Command for committing the chaincode:

peer lifecycle chaincode commit -o orderer1.example.org:7050 --channelID certificatechannel --name cc --version 1 --sequence 1 --tls --cafile /hl_config/orde

rer-tlsca/tlscacert.pem  --peerAddresses peer0.ni.example.org:7051 --tlsRootCertFiles /etc/hyperledger/fabric/crypto-config/peerOrganizations/ni.tnega.org/peers/peer0.ni.

example.org/tls/ca.crt --peerAddresses peer0.schooleducation.example.org:7051 --tlsRootCertFiles /etc/hyperledger/fabric/crypto-config/peerOrganizations/schooleducation.example.org/peers/peer0.schooleducation.example.org/tls/ca.crt --peerAddresses peer0.highereducation.example.org:7051 --tlsRootCertFiles /etc/hyperledger/fabric/crypto-config/pe

erOrganizations/highereducation.example.org/peers/peer0.highereducation.example.org/tls/ca.crt --peerAddresses peer0.es.example.org:7051 --tlsRootCertFiles /etc/hyperledger/fabric/crypto-config/peerOrganizations/es.example.org/peers/peer0.es.example.org/tls/ca.crt


What I've tried:
I've tried running commit command passing one peer address of only one org at a time and received the same output. 
I've tried running commit command passing one peer address of all the orgs that have approved the chaincode and have recieved the same output.

I tried executing the command from inside one of the peers.

Note: 
-------
I'm making use of argo workflows along with helm templates and kubernetes to automate the chaincode process.

Any help is much appreciated. Thanks in advance!
Arsh <arshimag@...>
 

Hi Josh, thank you for your quick response, I'd love to explore the new fabric-operator. However, I'm still intrigued as to what might have caused the error above mentioned. It is working fine in my docker environment. I find it worth mentioning that I haven't set the CORE_PEER env variables in the argo workflow that executes the chaincode commands. Does it even matter? Like, will that impact while just committing the chaincode and nowhere before? Waiting to receive some clarity. . . 

Regards,
Arsh
Arsh <arshimag@...>
 

Thanks David, that was precisely what I was pondering over too but I would like to provide you a piece of information here. I'm using two channels and the network setup is as shown below. As per the setup, only 4 orgs are joining the certificate channel who will be using the chaincode on that channel and hence the reason behind the commit readiness command showing approval of only 4 orgs and not 8 orgs with 4 orgs set to false. Atleast that was what I was thinking as the reason behind it. So in that case, isn't the MAJORITY already achieved? 4/4 ?

channels:
    - name: certificatechannel
      # all peers in these organizations will join the channel
      orgs: [Ni, Schooleducation, Highereducation, Es]
    - name: registrationchannel
      # all peers in these organizations will join the channel
      orgs: [Ni, Igr]
chaincodes:
    - name: cc
      # if defined, this will override the global chaincode.version value
      version: #"1.0"
      # chaincode will be installed to all peers in these organizations
      orgs: [Ni, Igr, Schooleducation, Highereducation, Es]
      # at which channels are we instantiating/upgrading chaincode?
      channels:
      - name: certificatechannel
        # chaincode will be instantiated/upgraded using the first peer in the first organization
        # chaincode will be invoked on all peers in these organizations
        orgs: [Ni, Schooleducation, Highereducation, Es]
      - name: registrationchannel
        # chaincode will be instantiated/upgraded using the first peer in the first organization
        # chaincode will be invoked on all peers in these organizations
        orgs: [Ni, Igr]
Arsh <arshimag@...>
 

These are the logs from one of the peers of an org.

?
peer2.ni.example.com.txt

David Enyeart
 

Here is the key lines:

 

2022-06-22 07:16:04.421 UTC [policies] func1 -> DEBU 777c66 Evaluation Failed: Only 0 policies were satisfied, but needed 3 of [ SchooleducationMSP/Endorsement HighereducationMSP/Endorsement EsevaiMSP/Endorsement NiMSP/Endorsement ]

2022-06-22 07:16:04.421 UTC [policies] EvaluateSignedData -> DEBU 777c67 Signature set did not satisfy policy /Channel/Application/LifecycleEndorsement

 

It means of the 4 channel members, it needs a majority (3) to pass LifecycleEndorsement policy.  LifecycleEndorement policy itself references each of the org specific Endorsement policies. So double check the org Endorsement policies, e.g. here:

https://github.com/hyperledger/fabric/blob/main/sampleconfig/configtx.yaml#L59-L61

 

BTW What Fabric version is this? I’m not seeing the improved policy error logging that was added to v2.4.

 

From: fabric@... <fabric@...> on behalf of Arsh <arshimag@...>
Date: Thursday, June 23, 2022 at 3:19 AM
To: fabric@... <fabric@...>
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Endorsement Policy Failure while committing chaincode #fabric-kubernetes #hyperledger-fabric
jkneubuh@...
 

Arsh / Dave - a couple of questions: 

  • Arsh I don't recognize the yaml syntax where you are describing your network.  Is this a descriptive format that you built in-house, or an input grammar supported by one of the community / Fabric configuration systems?   (Not suggesting this is related to the issue you are encountering, but simply curious... it looks good!)

  • If `CORE_PEER_LOCALMSPID` is not specified in the env of the process running the `peer` binary, then it will use the value as defined in the core.yaml.  The log looks like the chaincode commit is being invoked for the `NiMSP` org, not for the higher order education context that is hinted at by the CLI call above.  There could be some other differences going on, but this still looks suspicious.  (Short answer is:  YES, the peer CLI env context is important - not all of the inputs to the peer binary are specified on the command line.)
Arsh <arshimag@...>
 

@Dave  As you can see from the configtx.yaml that I've previously shared, all the orgs have LifecycleEndorsement policy set to  

Endorsement:

                Type: Signature

                Rule: "OR('Org1MSP.peer')"

and from the example configtx file which you shared, I can see that the policy is set to "OR('OrgMSP.member')". Will that impact? 
The hyperledger version I'm using is 2.2

@Josh, thank you for the compliment, it's a template I took from the repo I'm using as a reference for this project. Coming to your second point, the CORE_PEER_LOCALMSPID along with all the other required peer env variables is specified while creating peer containers but not in the argo workflows which execute the chaincode process. Even though I'm not using cli containers in my cluster unlike docker, I can safely tell you that my argo workflows do the same job of the cli containers more or less. Argo templates are used to do nothing but create containers and run chaincode scripts on them. Also, I haven't specified any CORE_PEER env variables in my docker-compose file for cli containers either. They are specified for the peer containers. At this point, I'm having another doubt. Why didn't any of this affect the previous chaincode commands?
jkneubuh@...
 

Argo provides an incredible workflow engine for Kubernetes.  It is _more_ than enough to handle the workloads of running Fabric CLI binaries - you are on a solid course for workflow automation!

Regarding the context, it's possible / likely that PIVT is setting the core.yaml in a way that is different than the test network's use of Compose.  Perhaps there are some environmental differences that are getting flushed out by calling the peer CLI in a new context.  The ledger interaction with the final phase of the commit is different than the install / approve - I don't think it's actually attempting a consensus across orgs until the final commit step in the chaincode lifecycle.

Maybe - one last plug - try the new fabric-operator and fabric-k8s-builder lifecycle?  At least to see if it reduces the need for workflow automation, or what steps would be necessary to reach parity with the PIVT syntax?  BTW there are some efforts underway by the community to assemble a test and performance benchmarking environment, based in part on a workflow engine (Argo!) or scripted automation.  Would welcome your feedback on the general approach for CI and deployment automation with Argo - this is a great step forward.  Please keep the mailing list posted on your progress, or drop a note over in the #fabric-kubernetes channel at Discord when you have finally arm-wrestled Fabric into submission.

Cheers
-josh
David Enyeart
 

You can use either OrgMSP.member or OrgMSP.peer format, assuming you’ve configured the MSP for Node OU roles and provided the matching OU “peer” tag in the peer’s certificate.

More details at https://hyperledger-fabric.readthedocs.io/en/latest/membership/membership.html#node-ou-roles-and-msps

You could try OrgMSP.member in your channel configuration to verify that you don’t have a Node OU configuration issue.

 

The additional policy error logging was backported to v2.2 in the recent v2.2.6, it may provide additional hints in the peer log.

 

From: fabric@... <fabric@...> on behalf of Arsh <arshimag@...>
Date: Thursday, June 23, 2022 at 11:24 AM
To: fabric@... <fabric@...>
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Endorsement Policy Failure while committing chaincode #fabric-kubernetes #hyperledger-fabric

@Dave  As you can see from the configtx.yaml that I've previously shared, all the orgs have LifecycleEndorsement policy set to   Endorsement:                 Type: Signature                 Rule: "OR('Org1MSP.peer')" ‍ ‍ ‍ ‍

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

@Dave  As you can see from the configtx.yaml that I've previously shared, all the orgs have LifecycleEndorsement policy set to  

Endorsement:

                Type: Signature

                Rule: "OR('Org1MSP.peer')"

and from the example configtx file which you shared, I can see that the policy is set to "OR('OrgMSP.member')". Will that impact? 
The hyperledger version I'm using is 2.2

@Josh, thank you for the compliment, it's a template I took from the repo I'm using as a reference for this project. Coming to your second point, the CORE_PEER_LOCALMSPID along with all the other required peer env variables is specified while creating peer containers but not in the argo workflows which execute the chaincode process. Even though I'm not using cli containers in my cluster unlike docker, I can safely tell you that my argo workflows do the same job of the cli containers more or less. Argo templates are used to do nothing but create containers and run chaincode scripts on them. Also, I haven't specified any CORE_PEER env variables in my docker-compose file for cli containers either. They are specified for the peer containers. At this point, I'm having another doubt. Why didn't any of this affect the previous chaincode commands?
Arsh <arshimag@...>
 

@Josh, I would love to see if fabric-operator and k8s-builder lifecycle make my life any easier. Need to find time to test them. Also, will keep my investigation of finding what went wrong with the facsimile of configurations I'm using in my env which are successfully reproduced by others, on. Wish me luck!

 
Arsh <arshimag@...>
 

@Dave, Yes the NodeOUs are also configured. The logs state that the sub-policies are not being satisfied but there is evidently no such thing in the configtx.yaml which indicates/backs the claim. How is it possible? 
1 - 13 of 13