Inconsistent Private Data Collection Write operation


Hritik Gupta <hritikgupta9@...>
 

Hi all,

I have a fabric 2.0 network with 2 orgs having 1 peer each (similar to test-network). I implemented a private data collection that persists in the off-chain DB of the peer of one of the orgs (Org1, as shown below)
"name": "collectionMarblePrivateDetails", 
"policy": "OR('Org1MSP.member')", 
"requiredPeerCount": 0, 
"maxPeerCount": 3, 
"blockToLive": 0, 
"memberOnlyRead": true, 
"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member')" } 
}
I tried to change the value of the key in the private collection by invoking the relevant chaincode method from : (I'm using the submitTransaction method from the Contract class of fabric node-sdk) 
1) an authorized peer
2) an unauthorized peer after making the following change to the collection endorsement policy:
"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" } 

Although the above 2 operations work sometimes, but very inconsistently. It fails by giving the following endorsement policy error:

2020-06-19T10:01:19.660Z - warn: [TransactionEventHandler]: strategyFail: commit failure for transaction "8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d": Error: Commit of transaction 8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d failed on peer peer0.org1.example.com:7051 with status ENDORSEMENT_POLICY_FAILURE
Failed to submit transaction: Error: Commit of transaction 8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d failed on peer peer0.org1.example.com:7051 with status ENDORSEMENT_POLICY_FAILURE

The logs from the peer docker containers are as follows:
2020-06-19 10:01:14.584 UTC [committer.txvalidator] validateTx -> ERRO 0f2 Dispatch for transaction txId = 16c33a328f40e579577c1767b0b55054cf903c328cf1a236f1da0aafb1762099 returned error: validation of endorsement policy for chaincode fabcar in tx 15:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

Although, aforementioned methods are able to change the key-value in the private collection if executed successfully, it fails a lot many number of times too. I'm not sure why that happens.
Appreciate any help!

Thanks and regards.


--

Hritik Gupta

B.Tech, Computer Science and Engineering

Indian Institute of Technology

Mandi, Himachal Pradesh – 175005

+91-7838869482 | b16097@...



Senthil Nathan
 

1) an authorized peer
2) an unauthorized peer after making the following change to the collection endorsement policy:
"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

  1. As the memberOnlyWrite is set to false by default (not configured in the collection config as well), all peers are authorized to write if it can satisfy the endorsement policy.
  2. Is the endorsement policy syntax correct for your scenario? Shouldn't it be "OR('Org1MSP.member', 'Org2MSP.member')" if you want to allow either of the organization to write to the collection?
Regards,
Senthil


On Fri, Jun 19, 2020 at 3:54 PM Hritik Gupta <hritikgupta9@...> wrote:
Hi all,

I have a fabric 2.0 network with 2 orgs having 1 peer each (similar to test-network). I implemented a private data collection that persists in the off-chain DB of the peer of one of the orgs (Org1, as shown below)
"name": "collectionMarblePrivateDetails", 
"policy": "OR('Org1MSP.member')", 
"requiredPeerCount": 0, 
"maxPeerCount": 3, 
"blockToLive": 0, 
"memberOnlyRead": true, 
"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member')" } 
}
I tried to change the value of the key in the private collection by invoking the relevant chaincode method from : (I'm using the submitTransaction method from the Contract class of fabric node-sdk) 
1) an authorized peer
2) an unauthorized peer after making the following change to the collection endorsement policy:
"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" } 

Although the above 2 operations work sometimes, but very inconsistently. It fails by giving the following endorsement policy error:

2020-06-19T10:01:19.660Z - warn: [TransactionEventHandler]: strategyFail: commit failure for transaction "8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d": Error: Commit of transaction 8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d failed on peer peer0.org1.example.com:7051 with status ENDORSEMENT_POLICY_FAILURE
Failed to submit transaction: Error: Commit of transaction 8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d failed on peer peer0.org1.example.com:7051 with status ENDORSEMENT_POLICY_FAILURE

The logs from the peer docker containers are as follows:
2020-06-19 10:01:14.584 UTC [committer.txvalidator] validateTx -> ERRO 0f2 Dispatch for transaction txId = 16c33a328f40e579577c1767b0b55054cf903c328cf1a236f1da0aafb1762099 returned error: validation of endorsement policy for chaincode fabcar in tx 15:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

Although, aforementioned methods are able to change the key-value in the private collection if executed successfully, it fails a lot many number of times too. I'm not sure why that happens.
Appreciate any help!

Thanks and regards.


--

Hritik Gupta

B.Tech, Computer Science and Engineering

Indian Institute of Technology

Mandi, Himachal Pradesh – 175005

+91-7838869482 | b16097@...



Hritik Gupta <hritikgupta9@...>
 

Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

"endorsementPolicy": {
      "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
    }

Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 


Senthil Nathan
 

but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

the above error seems to match your old definition -- "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

Anyway, I do not have expertise in this. If Yacov looks at this message, he might clarify.

Regards,
Senthil

On Fri, Jun 19, 2020 at 5:21 PM Hritik Gupta <hritikgupta9@...> wrote:
Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

"endorsementPolicy": {
      "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
    }

Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 


Hritik Gupta <hritikgupta9@...>
 

I ran the network again, to double check. The inconsistency issue still exists. The issue occurs even when I pass OR("Org1MSP.Member") as the EP and try to update the key via Org1 client (which is an authorized peer).



On Fri, Jun 19, 2020 at 5:31 PM Cendhu U <cendhu@...> wrote:
but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

the above error seems to match your old definition -- "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

Anyway, I do not have expertise in this. If Yacov looks at this message, he might clarify.

Regards,
Senthil

On Fri, Jun 19, 2020 at 5:21 PM Hritik Gupta <hritikgupta9@...> wrote:
Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

"endorsementPolicy": {
      "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
    }

Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 



-- 


David Enyeart
 

The endorsement policy check only checks which peers have endorsed, not which clients have submitted the transaction. The peer validation logic is deterministic in this regard.
Could you share the Org1 peer log showing it working once, and not working once, with no other changes inbetween? Please highlight the log timestamp of endorsement and validation in both cases.


Thanks,

Dave Enyeart

"Hritik Gupta" ---06/19/2020 08:11:18 AM---I ran the network again, to double check. The inconsistency issue still exists. The issue occurs eve

From: "Hritik Gupta" <hritikgupta9@...>
To: Cendhu U <cendhu@...>
Cc: fabric <fabric@...>
Date: 06/19/2020 08:11 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Inconsistent Private Data Collection Write operation
Sent by: fabric@...





I ran the network again, to double check. The inconsistency issue still exists. The issue occurs even when I pass OR("Org1MSP.Member") as the EP and try to update the key via Org1 client (which is an authorized peer).



On Fri, Jun 19, 2020 at 5:31 PM Cendhu U <cendhu@...> wrote:
    but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

    the above error seems to match your old definition -- "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

    Anyway, I do not have expertise in this. If Yacov looks at this message, he might clarify.

    Regards,
    Senthil


On Fri, Jun 19, 2020 at 5:21 PM Hritik Gupta <hritikgupta9@...> wrote:
Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

"endorsementPolicy": {
      "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
    }

Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 



-- 





email4tong@gmail.com
 

Curious about this policy,

"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }


was it accepted as a valid policy? if it was accepted, should this be a bug? what does this policy exactly mean? Is it the same as this

"endorsementPolicy": {
      "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
    }

Thanks.

On Friday, June 19, 2020, 8:29:10 AM EDT, David Enyeart <enyeart@...> wrote:


The endorsement policy check only checks which peers have endorsed, not which clients have submitted the transaction. The peer validation logic is deterministic in this regard.
Could you share the Org1 peer log showing it working once, and not working once, with no other changes inbetween? Please highlight the log timestamp of endorsement and validation in both cases.


Thanks,

Dave Enyeart

"Hritik Gupta" ---06/19/2020 08:11:18 AM---I ran the network again, to double check. The inconsistency issue still exists. The issue occurs eve

From: "Hritik Gupta" <hritikgupta9@...>
To: Cendhu U <cendhu@...>
Cc: fabric <fabric@...>
Date: 06/19/2020 08:11 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Inconsistent Private Data Collection Write operation
Sent by: fabric@...





I ran the network again, to double check. The inconsistency issue still exists. The issue occurs even when I pass OR("Org1MSP.Member") as the EP and try to update the key via Org1 client (which is an authorized peer).



On Fri, Jun 19, 2020 at 5:31 PM Cendhu U <cendhu@...> wrote:
    but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

      the above error seems to match your old definition -- "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

      Anyway, I do not have expertise in this. If Yacov looks at this message, he might clarify.

      Regards,
      Senthil


      On Fri, Jun 19, 2020 at 5:21 PM Hritik Gupta <hritikgupta9@...> wrote:
      Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

      "endorsementPolicy": {
            "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
          }

      Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 



      -- 





      Hritik Gupta <hritikgupta9@...>
       

      I have attached the screenshots of the logs from Org1 peer container for your reference, for better readability. 

      Logs when it worked:
      2020-06-19 12:49:09.580 UTC [comm.grpc.server] 1 -> INFO 122 unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.112.1:57578 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=679.842µs
      2020-06-19 12:49:09.608 UTC [comm.grpc.server] 1 -> INFO 123 unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.112.1:57578 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=813.325µs
      2020-06-19 12:49:09.622 UTC [endorser] callChaincode -> INFO 124 finished chaincode: fabcar duration: 1ms channel=mychannel txID=de27f4ec
      2020-06-19 12:49:09.635 UTC [comm.grpc.server] 1 -> INFO 125 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=192.168.112.1:57586 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=14.715518ms
      2020-06-19 12:49:11.678 UTC [gossip.privdata] StoreBlock -> INFO 126 [mychannel] Received block [20] from buffer
      2020-06-19 12:49:11.680 UTC [committer.txvalidator] Validate -> INFO 127 [mychannel] Validated block [20] in 2ms
      2020-06-19 12:49:11.681 UTC [gossip.privdata] prepareBlockPvtdata -> INFO 128 Successfully fetched all eligible collection private write sets for block [20] channel=mychannel
      2020-06-19 12:49:11.712 UTC [kvledger] CommitLegacy -> INFO 129 [mychannel] Committed block [20] with 1 transaction(s) in 30ms (state_validation=0ms block_and_pvtdata_commit=18ms state_commit=5ms) commitHash=[6bfa36ced7abf414a278943a0efe5cc5bbbfeecb9ef5edce84b862e314b688c4]
      2020-06-19 12:49:11.723 UTC [comm.grpc.server] 1 -> INFO 12a streaming call completed grpc.service=protos.Deliver grpc.method=DeliverFiltered grpc.peer_address=192.168.112.1:57594 grpc.peer_subject="CN=fabric-common" error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=2.075316216s

      Logs when it failed:
      2020-06-19 12:45:31.198 UTC [comm.grpc.server] 1 -> INFO 0ea unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.112.1:57492 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=805.632µs
      2020-06-19 12:45:31.224 UTC [comm.grpc.server] 1 -> INFO 0eb unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.112.1:57492 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=757.737µs
      2020-06-19 12:45:31.238 UTC [endorser] callChaincode -> INFO 0ec finished chaincode: fabcar duration: 1ms channel=mychannel txID=ee3f5ebe
      2020-06-19 12:45:31.775 UTC [comm.grpc.server] 1 -> INFO 0ed unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=192.168.112.1:57500 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=539.382571ms
      2020-06-19 12:45:35.051 UTC [gossip.privdata] StoreBlock -> INFO 0ee [mychannel] Received block [16] from buffer
      2020-06-19 12:45:35.053 UTC [policies] SignatureSetToValidIdentities -> WARN 0ef signature for identity 0 is invalid: The signature is invalid
      2020-06-19 12:45:35.054 UTC [policies] SignatureSetToValidIdentities -> WARN 0f0 signature for identity 0 is invalid: The signature is invalid
      2020-06-19 12:45:35.054 UTC [vscc] Validate -> ERRO 0f1 VSCC error: stateBasedValidator.Validate failed, err validation of endorsement policy for chaincode fabcar in tx 16:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied
      2020-06-19 12:45:35.054 UTC [committer.txvalidator] validateTx -> ERRO 0f2 Dispatch for transaction txId = ee3f5ebe9aee21b245ef1ceb376e0e8facfd0b71965c2defae37d5a70b3558db returned error: validation of endorsement policy for chaincode fabcar in tx 16:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied
      2020-06-19 12:45:35.054 UTC [committer.txvalidator] Validate -> INFO 0f3 [mychannel] Validated block [16] in 3ms
      2020-06-19 12:45:35.054 UTC [gossip.privdata] prepareBlockPvtdata -> INFO 0f4 Successfully fetched all eligible collection private write sets for block [16] channel=mychannel
      2020-06-19 12:45:35.055 UTC [valimpl] preprocessProtoBlock -> WARN 0f5 Channel [mychannel]: Block [16] Transaction index [0] TxId [ee3f5ebe9aee21b245ef1ceb376e0e8facfd0b71965c2defae37d5a70b3558db] marked as invalid by committer. Reason code [ENDORSEMENT_POLICY_FAILURE]
      2020-06-19 12:45:36.879 UTC [kvledger] CommitLegacy -> INFO 0f6 [mychannel] Committed block [16] with 1 transaction(s) in 1824ms (state_validation=0ms block_and_pvtdata_commit=1075ms state_commit=363ms) commitHash=[0caec1920f414394bd085c351bf04c66b18ee0d67c9d7313687b9216e79261d5]
      2020-06-19 12:45:36.893 UTC [comm.grpc.server] 1 -> INFO 0f7 streaming call completed grpc.service=protos.Deliver grpc.method=DeliverFiltered grpc.peer_address=192.168.112.1:57508 grpc.peer_subject="CN=fabric-common" error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=5.099824363s

      @email4tong : I tried deploying the chaincode with this policy, it gave me an Error: invalid collection configuration: invalid endorsement policy. 

      Thanks.

      On Fri, Jun 19, 2020 at 6:05 PM email4tong@... <email4tong@...> wrote:
      Curious about this policy,

      "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }


      was it accepted as a valid policy? if it was accepted, should this be a bug? what does this policy exactly mean? Is it the same as this

      "endorsementPolicy": {
            "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
          }

      Thanks.

      On Friday, June 19, 2020, 8:29:10 AM EDT, David Enyeart <enyeart@...> wrote:


      The endorsement policy check only checks which peers have endorsed, not which clients have submitted the transaction. The peer validation logic is deterministic in this regard.
      Could you share the Org1 peer log showing it working once, and not working once, with no other changes inbetween? Please highlight the log timestamp of endorsement and validation in both cases.


      Thanks,

      Dave Enyeart

      "Hritik Gupta" ---06/19/2020 08:11:18 AM---I ran the network again, to double check. The inconsistency issue still exists. The issue occurs eve

      From: "Hritik Gupta" <hritikgupta9@...>
      To: Cendhu U <cendhu@...>
      Cc: fabric <fabric@...>
      Date: 06/19/2020 08:11 AM
      Subject: [EXTERNAL] Re: [Hyperledger Fabric] Inconsistent Private Data Collection Write operation
      Sent by: fabric@...





      I ran the network again, to double check. The inconsistency issue still exists. The issue occurs even when I pass OR("Org1MSP.Member") as the EP and try to update the key via Org1 client (which is an authorized peer).



      On Fri, Jun 19, 2020 at 5:31 PM Cendhu U <cendhu@...> wrote:
        but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

          the above error seems to match your old definition -- "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

          Anyway, I do not have expertise in this. If Yacov looks at this message, he might clarify.

          Regards,
          Senthil

          On Fri, Jun 19, 2020 at 5:21 PM Hritik Gupta <hritikgupta9@...> wrote:
          Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

          "endorsementPolicy": {
                "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
              }

          Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 



          -- 









          David Enyeart
           

          The second scenario failed due to:
          2020-06-19 12:45:35.053 UTC [policies] SignatureSetToValidIdentities -> WARN 0ef signature for identity 0 is invalid: The signature is invalid

          How many peers are endorsing? Are you always getting endorsement from the same exact peers?
          What is the endorsement policy for both the chaincode and the collection?
          Can you provide the uninterrupted log?


          Dave Enyeart

          "Hritik Gupta" ---06/19/2020 09:01:25 AM---I have attached the screenshots of the logs from Org1 peer container for your reference, for better

          From: "Hritik Gupta" <hritikgupta9@...>
          To: "email4tong@..." <email4tong@...>
          Cc: David Enyeart <enyeart@...>, Cendhu U <cendhu@...>, fabric <fabric@...>
          Date: 06/19/2020 09:01 AM
          Subject: [EXTERNAL] Re: [Hyperledger Fabric] Inconsistent Private Data Collection Write operation
          Sent by: fabric@...





          I have attached the screenshots of the logs from Org1 peer container for your reference, for better readability. 

          Logs when it worked:
          2020-06-19 12:49:09.580 UTC [comm.grpc.server] 1 -> INFO 122 unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.112.1:57578 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=679.842µs
          2020-06-19 12:49:09.608 UTC [comm.grpc.server] 1 -> INFO 123 unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=
          192.168.112.1:57578 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=813.325µs
          2020-06-19 12:49:09.622 UTC [endorser] callChaincode -> INFO 124 finished chaincode: fabcar duration: 1ms channel=mychannel txID=de27f4ec
          2020-06-19 12:49:09.635
          UTC [comm.grpc.server] 1 -> INFO 125 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=192.168.112.1:57586 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=14.715518ms
          2020-06-19 12:49:11.678
          UTC [gossip.privdata] StoreBlock -> INFO 126 [mychannel] Received block [20] from buffer
          2020-06-19 12:49:11.680
          UTC [committer.txvalidator] Validate -> INFO 127 [mychannel] Validated block [20] in 2ms
          2020-06-19 12:49:11.681
          UTC [gossip.privdata] prepareBlockPvtdata -> INFO 128 Successfully fetched all eligible collection private write sets for block [20] channel=mychannel
          2020-06-19 12:49:11.712
          UTC [kvledger] CommitLegacy -> INFO 129 [mychannel] Committed block [20] with 1 transaction(s) in 30ms (state_validation=0ms block_and_pvtdata_commit=18ms state_commit=5ms) commitHash=[6bfa36ced7abf414a278943a0efe5cc5bbbfeecb9ef5edce84b862e314b688c4]
          2020-06-19 12:49:11.723
          UTC [comm.grpc.server] 1 -> INFO 12a streaming call completed grpc.service=protos.Deliver grpc.method=DeliverFiltered grpc.peer_address=192.168.112.1:57594 grpc.peer_subject="CN=fabric-common" error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=2.075316216s

          Logs when it failed:
          2020-06-19 12:45:31.198 UTC [comm.grpc.server] 1 -> INFO 0ea unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.112.1:57492 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=805.632µs
          2020-06-19 12:45:31.224 UTC [comm.grpc.server] 1 -> INFO 0eb unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=
          192.168.112.1:57492 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=757.737µs
          2020-06-19 12:45:31.238 UTC [endorser] callChaincode -> INFO 0ec finished chaincode: fabcar duration: 1ms channel=mychannel txID=ee3f5ebe
          2020-06-19 12:45:31.775 UTC [comm.grpc.server] 1 -> INFO 0ed unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=
          192.168.112.1:57500 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=539.382571ms
          2020-06-19 12:45:35.051 UTC [gossip.privdata] StoreBlock -> INFO 0ee [mychannel] Received block [16] from buffer

          2020-06-19 12:45:35.053 UTC
          [policies] SignatureSetToValidIdentities -> WARN 0ef signature for identity 0 is invalid: The signature is invalid
          2020-06-19 12:45:35.054 UTC
          [policies] SignatureSetToValidIdentities -> WARN 0f0 signature for identity 0 is invalid: The signature is invalid
          2020-06-19 12:45:35.054 UTC
          [vscc] Validate -> ERRO 0f1 VSCC error: stateBasedValidator.Validate failed, err validation of endorsement policy for chaincode fabcar in tx 16:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied
          2020-06-19 12:45:35.054 UTC
          [committer.txvalidator] validateTx -> ERRO 0f2 Dispatch for transaction txId = ee3f5ebe9aee21b245ef1ceb376e0e8facfd0b71965c2defae37d5a70b3558db returned error: validation of endorsement policy for chaincode fabcar in tx 16:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied
          2020-06-19 12:45:35.054 UTC
          [committer.txvalidator] Validate -> INFO 0f3 [mychannel] Validated block [16] in 3ms
          2020-06-19 12:45:35.054 UTC
          [gossip.privdata] prepareBlockPvtdata -> INFO 0f4 Successfully fetched all eligible collection private write sets for block [16] channel=mychannel
          2020-06-19 12:45:35.055 UTC
          [valimpl] preprocessProtoBlock -> WARN 0f5 Channel [mychannel]: Block [16] Transaction index [0] TxId [ee3f5ebe9aee21b245ef1ceb376e0e8facfd0b71965c2defae37d5a70b3558db] marked as invalid by committer. Reason code [ENDORSEMENT_POLICY_FAILURE]
          2020-06-19 12:45:36.879 UTC [kvledger] CommitLegacy -> INFO 0f6 [mychannel] Committed block [16] with 1 transaction(s) in 1824ms (state_validation=0ms block_and_pvtdata_commit=1075ms state_commit=363ms) commitHash=[0caec1920f414394bd085c351bf04c66b18ee0d67c9d7313687b9216e79261d5]

          2020-06-19 12:45:36.893 UTC
          [comm.grpc.server] 1 -> INFO 0f7 streaming call completed grpc.service=protos.Deliver grpc.method=DeliverFiltered grpc.peer_address=192.168.112.1:57508 grpc.peer_subject="CN=fabric-common" error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=5.099824363s

          @email4tong : I tried deploying the chaincode with this policy, it gave me an Error: invalid collection configuration: invalid endorsement policy. 

          Thanks.

          On Fri, Jun 19, 2020 at 6:05 PM email4tong@... <email4tong@...> wrote:
            Curious about this policy,

            "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }


            was it accepted as a valid policy? if it was accepted, should this be a bug? what does this policy exactly mean? Is it the same as this

            "endorsementPolicy": {
                  "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
                }


            Thanks.

            On Friday, June 19, 2020, 8:29:10 AM EDT, David Enyeart <enyeart@...> wrote:

            The endorsement policy check only checks which peers have endorsed, not which clients have submitted the transaction. The peer validation logic is deterministic in this regard.
            Could you share the Org1 peer log showing it working once, and not working once, with no other changes inbetween? Please highlight the log timestamp of endorsement and validation in both cases.


            Thanks,

            Dave Enyeart

            "Hritik Gupta" ---06/19/2020 08:11:18 AM---I ran the network again, to double check. The inconsistency issue still exists. The issue occurs eve

            From:
            "Hritik Gupta" <hritikgupta9@...>
            To:
            Cendhu U <cendhu@...>
            Cc:
            fabric <fabric@...>
            Date:
            06/19/2020 08:11 AM
            Subject:
            [EXTERNAL] Re: [Hyperledger Fabric] Inconsistent Private Data Collection Write operation
            Sent by:
            fabric@...





            I ran the network again, to double check. The inconsistency issue still exists. The issue occurs even when I pass OR("Org1MSP.Member") as the EP and try to update the key via Org1 client (which is an authorized peer).




            On Fri, Jun 19, 2020 at 5:31 PM Cendhu U <
            cendhu@...> wrote:
            but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

            the above error seems to match your old definition --
            "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

            Anyway, I do not have expertise in this. If Yacov looks at this message, he might clarify
            .

            Regards,
            Senthil


          On Fri, Jun 19, 2020 at 5:21 PM Hritik Gupta <hritikgupta9@...> wrote:
          Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

          "endorsementPolicy": {
                "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
              }

          Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 


          -- 











          Hritik Gupta <hritikgupta9@...>
           

          I haven't mentioned any chaincode endorsement policy during deployment thinking that the collection level endorsement policy overrides the chaincode level policy.
          The collection level policy is: "OR('Org1MSP.member', 'Org2MSP.member')". 

          According to the documentation, "You can use collection-level endorsement policies to restrict which organization peers can write to the private data collection key namespace". Since the intent is allowing only these 2 org peers to allow writing to private data, hence the policy is so. 
          Since there is one peer per org, that makes 2 endorsing peers. I'm not sure what the second question means. Since there are the only 2 peers in the network, I'm thus getting endorsement from the same exact peers.

          PFA the peer0.org1 logs screenshot.
          On a side note, I'm trying to implement a variant of the bidding pattern you've suggested here: https://lists.hyperledger.org/g/fabric/message/7554; keeping Org1 analogous to the seller that persists on its private collection the bids of sellers (Org2).

          Thanks.


          On Fri, Jun 19, 2020 at 7:05 PM David Enyeart <enyeart@...> wrote:

          The second scenario failed due to:
          2020-06-19 12:45:35.053 UTC [policies] SignatureSetToValidIdentities -> WARN 0ef signature for identity 0 is invalid: The signature is invalid

          How many peers are endorsing? Are you always getting endorsement from the same exact peers?
          What is the endorsement policy for both the chaincode and the collection?
          Can you provide the uninterrupted log?


          Dave Enyeart

          "Hritik Gupta" ---06/19/2020 09:01:25 AM---I have attached the screenshots of the logs from Org1 peer container for your reference, for better

          From: "Hritik Gupta" <hritikgupta9@...>
          To: "email4tong@..." <email4tong@...>
          Cc: David Enyeart <enyeart@...>, Cendhu U <cendhu@...>, fabric <fabric@...>
          Date: 06/19/2020 09:01 AM
          Subject: [EXTERNAL] Re: [Hyperledger Fabric] Inconsistent Private Data Collection Write operation
          Sent by: fabric@...





          I have attached the screenshots of the logs from Org1 peer container for your reference, for better readability. 

          Logs when it worked:
          2020-06-19 12:49:09.580 UTC [comm.grpc.server] 1 -> INFO 122 unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.112.1:57578 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=679.842µs
          2020-06-19 12:49:09.608 UTC [comm.grpc.server] 1 -> INFO 123 unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=
          192.168.112.1:57578 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=813.325µs
          2020-06-19 12:49:09.622 UTC [endorser] callChaincode -> INFO 124 finished chaincode: fabcar duration: 1ms channel=mychannel txID=de27f4ec
          2020-06-19 12:49:09.635
          UTC [comm.grpc.server] 1 -> INFO 125 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=192.168.112.1:57586 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=14.715518ms
          2020-06-19 12:49:11.678
          UTC [gossip.privdata] StoreBlock -> INFO 126 [mychannel] Received block [20] from buffer
          2020-06-19 12:49:11.680
          UTC [committer.txvalidator] Validate -> INFO 127 [mychannel] Validated block [20] in 2ms
          2020-06-19 12:49:11.681
          UTC [gossip.privdata] prepareBlockPvtdata -> INFO 128 Successfully fetched all eligible collection private write sets for block [20] channel=mychannel
          2020-06-19 12:49:11.712
          UTC [kvledger] CommitLegacy -> INFO 129 [mychannel] Committed block [20] with 1 transaction(s) in 30ms (state_validation=0ms block_and_pvtdata_commit=18ms state_commit=5ms) commitHash=[6bfa36ced7abf414a278943a0efe5cc5bbbfeecb9ef5edce84b862e314b688c4]
          2020-06-19 12:49:11.723
          UTC [comm.grpc.server] 1 -> INFO 12a streaming call completed grpc.service=protos.Deliver grpc.method=DeliverFiltered grpc.peer_address=192.168.112.1:57594 grpc.peer_subject="CN=fabric-common" error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=2.075316216s

          Logs when it failed:
          2020-06-19 12:45:31.198 UTC [comm.grpc.server] 1 -> INFO 0ea unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.112.1:57492 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=805.632µs
          2020-06-19 12:45:31.224 UTC [comm.grpc.server] 1 -> INFO 0eb unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=
          192.168.112.1:57492 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=757.737µs
          2020-06-19 12:45:31.238 UTC [endorser] callChaincode -> INFO 0ec finished chaincode: fabcar duration: 1ms channel=mychannel txID=ee3f5ebe
          2020-06-19 12:45:31.775 UTC [comm.grpc.server] 1 -> INFO 0ed unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=
          192.168.112.1:57500 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=539.382571ms
          2020-06-19 12:45:35.051 UTC [gossip.privdata] StoreBlock -> INFO 0ee [mychannel] Received block [16] from buffer

          2020-06-19 12:45:35.053 UTC
          [policies] SignatureSetToValidIdentities -> WARN 0ef signature for identity 0 is invalid: The signature is invalid
          2020-06-19 12:45:35.054 UTC
          [policies] SignatureSetToValidIdentities -> WARN 0f0 signature for identity 0 is invalid: The signature is invalid
          2020-06-19 12:45:35.054 UTC
          [vscc] Validate -> ERRO 0f1 VSCC error: stateBasedValidator.Validate failed, err validation of endorsement policy for chaincode fabcar in tx 16:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied
          2020-06-19 12:45:35.054 UTC
          [committer.txvalidator] validateTx -> ERRO 0f2 Dispatch for transaction txId = ee3f5ebe9aee21b245ef1ceb376e0e8facfd0b71965c2defae37d5a70b3558db returned error: validation of endorsement policy for chaincode fabcar in tx 16:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied
          2020-06-19 12:45:35.054 UTC
          [committer.txvalidator] Validate -> INFO 0f3 [mychannel] Validated block [16] in 3ms
          2020-06-19 12:45:35.054 UTC
          [gossip.privdata] prepareBlockPvtdata -> INFO 0f4 Successfully fetched all eligible collection private write sets for block [16] channel=mychannel
          2020-06-19 12:45:35.055 UTC
          [valimpl] preprocessProtoBlock -> WARN 0f5 Channel [mychannel]: Block [16] Transaction index [0] TxId [ee3f5ebe9aee21b245ef1ceb376e0e8facfd0b71965c2defae37d5a70b3558db] marked as invalid by committer. Reason code [ENDORSEMENT_POLICY_FAILURE]
          2020-06-19 12:45:36.879 UTC [kvledger] CommitLegacy -> INFO 0f6 [mychannel] Committed block [16] with 1 transaction(s) in 1824ms (state_validation=0ms block_and_pvtdata_commit=1075ms state_commit=363ms) commitHash=[0caec1920f414394bd085c351bf04c66b18ee0d67c9d7313687b9216e79261d5]

          2020-06-19 12:45:36.893 UTC
          [comm.grpc.server] 1 -> INFO 0f7 streaming call completed grpc.service=protos.Deliver grpc.method=DeliverFiltered grpc.peer_address=192.168.112.1:57508 grpc.peer_subject="CN=fabric-common" error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=5.099824363s

          @email4tong : I tried deploying the chaincode with this policy, it gave me an Error: invalid collection configuration: invalid endorsement policy. 

          Thanks.

          On Fri, Jun 19, 2020 at 6:05 PM email4tong@... <email4tong@...> wrote:
            Curious about this policy,

            "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }


            was it accepted as a valid policy? if it was accepted, should this be a bug? what does this policy exactly mean? Is it the same as this

            "endorsementPolicy": {
                  "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
                }


            Thanks.

            On Friday, June 19, 2020, 8:29:10 AM EDT, David Enyeart <enyeart@...> wrote:

            The endorsement policy check only checks which peers have endorsed, not which clients have submitted the transaction. The peer validation logic is deterministic in this regard.
            Could you share the Org1 peer log showing it working once, and not working once, with no other changes inbetween? Please highlight the log timestamp of endorsement and validation in both cases.


            Thanks,

            Dave Enyeart

            "Hritik Gupta" ---06/19/2020 08:11:18 AM---I ran the network again, to double check. The inconsistency issue still exists. The issue occurs eve

            From:
            "Hritik Gupta" <hritikgupta9@...>
            To:
            Cendhu U <cendhu@...>
            Cc:
            fabric <fabric@...>
            Date:
            06/19/2020 08:11 AM
            Subject:
            [EXTERNAL] Re: [Hyperledger Fabric] Inconsistent Private Data Collection Write operation
            Sent by:
            fabric@...





            I ran the network again, to double check. The inconsistency issue still exists. The issue occurs even when I pass OR("Org1MSP.Member") as the EP and try to update the key via Org1 client (which is an authorized peer).




            On Fri, Jun 19, 2020 at 5:31 PM Cendhu U <
            cendhu@...> wrote:
            but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

            the above error seems to match your old definition --
            "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

            Anyway, I do not have expertise in this. If Yacov looks at this message, he might clarify
            .

            Regards,
            Senthil

            On Fri, Jun 19, 2020 at 5:21 PM Hritik Gupta <
            hritikgupta9@...> wrote:
            Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

            "endorsementPolicy": {
                  "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
                }

            Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 


            -- 












          --



          David Enyeart
           

          Right, if the chaincode only writes to the collection namespace, you only need to meet the collection endorsement policy.

          Turning on full debug may provide a hint as to why a signature may be invalid:
          FABRIC_LOGGING_SPEC=debug

          Compare the debug processing of the valid and the invalid transaction.

          Are you able to reproduce the problem when using the Test Network?
          https://hyperledger-fabric.readthedocs.io/en/master/test_network.html


          Dave Enyeart


          Adhav Pavan
           

          Hello Hritik,

          1) As you mentioned about PDC configuration:

          {
          "name": "collectionCarPrivateDetails",
          "policy": "OR('Org1MSP.member')",
          "requiredPeerCount": 0,
          "maxPeerCount": 3,
          "blockToLive": 0,
          "memberOnlyRead": true,
          "memberOnlyWrite": false,
          "endorsementPolicy": {
          "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
          }
          }

          By Default Chaincode policy is the Majority of Peer. But here you are overriding the endorsement Policy("OR('Org1MSP.member', 'Org2MSP.member')":any of the peers).

          As per our discussion, you are getting private data before update it. But for org2 users have no access to PDC.

          Whenever the client getting an endorsement response from the peer(org1), your transaction getting successful, and whenever the client getting an endorsement response from Peer(Org2), the transaction getting failed as peer form org 2 won't be able to read private data before putting private state.

          I hope this helps you.

          @David Enyeart ,  I have a couple of related doubts as below:

          2) "memberOnlyRead": true,

          Question 1: I tried this field with both (true/false) value, giving the same result and it is expected. If anyone is able to read PDC then it beat the purpose of having private data. What is the purpose of having this field- "MemberReadOnly"?
                              If we make it false, do other members of org can read data? Does this field relate to signature policy?

          Question2: If transactions require only one endorsement, does SDK(Fabric Network) send tx to all endorsing peers or only one?

          Question 3: If the client gets both endorsements (One is correct and one is wrong(peer from org2 could not read data)) from endorsing peer and client send tx(Having both endorsement response from both peer) to orderer and orderer to committing peer.                         Does Committing peer check only one endorsement or both?

          Thanks in advance.



          Heartfelt Regards,

          Pavan Adhav
          Blockchain Developer, Infinichains
          phone:  8390114357
          email:  pavan@...
          ------
          Please excuse my brevity.


          On Fri, Jun 19, 2020 at 9:12 PM David Enyeart <enyeart@...> wrote:

          Right, if the chaincode only writes to the collection namespace, you only need to meet the collection endorsement policy.

          Turning on full debug may provide a hint as to why a signature may be invalid:
          FABRIC_LOGGING_SPEC=debug

          Compare the debug processing of the valid and the invalid transaction.

          Are you able to reproduce the problem when using the Test Network?
          https://hyperledger-fabric.readthedocs.io/en/master/test_network.html


          Dave Enyeart


          David Enyeart
           

          Thanks Adhav, I think you've assessed correctly. If the client gets two endorsements back it may randomly pick one of the responses to submit and apply both peer endorser signatures. In this case one of endorsed results is different and therefore one of the signatures won't match the results. Upon validation on each peer, the non-matching signature gets thrown out with warning "signature for identity 1 is invalid: The signature is invalid". Half the time org1 endorsement results will be passed, and half the time org2 endorsement results will be passed, hence the inconsistency. A well behaving client should detect the different endorsed results before submitting the transaction to orderer, so that it fails early rather than failing at validation time. I *think* Fabric Network can do that comparison, at least, it should be an option if not already.

          As for the questions...

          Question 1: I tried this field with both (true/false) value, giving the same result and it is expected. If anyone is able to read PDC then it beat the purpose of having private data. What is the purpose of having this field- "MemberReadOnly"?
                              If we make it false, do other members of org can read data? Does this field relate to signature policy?


          Answer 1: PDC specifies which org PEERS are able to persist the private data, and request it from other peers if needed. The access control for CLIENTS may or may not align with that definition. Here's how the documentation explains MemberOnlyRead:
          "a value of true indicates that peers automatically enforce that only clients belonging to one of the collection member organizations are allowed read access to private data. If a client from a non-member org attempts to execute a chaincode function that performs a read of a private data key, the chaincode invocation is terminated with an error. Utilize a value of false if you would like to encode more granular access control within individual chaincode functions."


          Question2: If transactions require only one endorsement, does SDK(Fabric Network) send tx to all endorsing peers or only one?

          Answer 2: With service discovery you can pass in a collection name, and it will return the minimum set of peers required to get an endorsement (one of either org's peers in this scenario). For using Fabric Network Node.js SDK with service discovery, see https://hyperledger.github.io/fabric-sdk-node/master/tutorial-discovery-fabric-network.html. I suspect in this scenario the collection name was not passed correctly to service discovery.


          Question 3: If the client gets both endorsements (One is correct and one is wrong(peer from org2 could not read data)) from endorsing peer and client send tx(Having both endorsement response from both peer) to orderer and orderer to committing peer.                         Does Committing peer check only one endorsement or both?

          Answer 3: As stated at the top, only one result is submitted, and both signatures are applied. If the endorsing peers signed over different data, then the signature that doesn't match gets thrown out as invalid. This leaves only one endorsement to be evaluated against the endorsement policy. In this scenario, the 'right' endorsement was being provided half the time.


          Dave Enyeart