Re: Security Analysis of Private Data Collection of Hyperledger Fabric


David Enyeart
 

As far as opening up the original HackerOne reports, I'd be fine with that if the process allows since the reports were resolved with documentation clarifications. The same documentation references are also copy/pasted from the reports into the email thread below (March 23rd response) for all to see.

Although closed as not vulnerabilities, the reports were worthwhile as they helped to identify areas where users may be confused and where documentation updates were required.


Dave Enyeart

"Senthil Nathan" ---03/30/2021 01:08:47 PM---These are neither design flaws nor vulnerabilities. We have various configuration parameters for pri

From: "Senthil Nathan" <cendhu@...>
To: Todd Little <toddjlittle@...>
Cc: fabric <fabric@...>
Date: 03/30/2021 01:08 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Security Analysis of Private Data Collection of Hyperledger Fabric
Sent by: fabric@...





These are neither design flaws nor vulnerabilities. We have various configuration parameters for private data collection. For example, the following is a configuration of a private data collection. { "name": "collectionMarblePrivateDetails" ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
These are neither design flaws nor vulnerabilities. We have various configuration parameters for private data collection. For example, the following is a configuration of a private data collection.
 {
   "name": "collectionMarblePrivateDetails",
   "policy": "OR('Org1MSP.member')",
   "requiredPeerCount": 0,
   "maxPeerCount": 3,
   "blockToLive":3,
   "memberOnlyRead": true,
   "memberOnlyWrite": true,
   "endorsementPolicy": {
     "signaturePolicy": "OR('Org1MSP.member')"
   }
}

Depending on the use-case, one can set the
memberOnlyRead and memberOnlyWrite to false. As a result,
a non-member can read and write to a collection. If an use-case wants a very restrictive access to the private
data collection, these parameters must be set to true. I don't think we can call this a design flaw. This is by
design and the documentation explains these configuration in details
(
https://hyperledger-fabric.readthedocs.io/en/release-2.2/private-data-arch.html)
Similarly, the endorsement policy can be set at the collection level as seen in the above configuration. It can also be at the key-level for more fine-grained control. Again, these are configuration parameters for the collection and the use-case dictates the required configuration.

I can go on but there are no design flaws or vulnerability in the Private Data Collection.

If the application/use-case does not configure the private data collection as per the need/requirement, it is not the fault of Hyperledger Fabric. For example, one must configure the firewall correctly in the production network. Without doing that, we cannot claim the networking stack in Linux Kernel to have vulnerabilities that allow unrestricted access.

I agree with Dave that the paper needs to be revised to say how to configure private data collections for various use-cases and requirements. 

Regards,
Senthil

On Tue, Mar 30, 2021 at 10:01 PM Todd Little <toddjlittle@...> wrote:
    As it appears the "issues" described in this paper are not really security issues, can the paper be made available to a wider audience?  Hackerone has all the reports related to this marked as private and inaccessible. Or can the reports be made public?

    Regards,
    Todd

    On 3/29/2021 9:28 AM, David Enyeart wrote:
        I still think it is in your best interest to make some minor updates to the paper content to avoid using sensational language such as "design flaws" and "vulnerabilities". Researchers tend to get discredited when their assertions get shown to be misleading or untrue. The original sponsor for the private data feature specifically requested the existing chaincode-level endorsement policy design for their use cases, and the more constrained collection-level endorsement policies were added as a parallel approach for other use cases. Calling out the original broadly as a design flaw is misleading and untrue when you consider the sponsor use case or review the Fabric documentation.

        I would very much support moving forward with the paper if it were recast as guidance for securing private data collections under various use case considerations.


        Dave Enyeart
        IBM Blockchain
        enyeart@...

        "Fu, Xinwen" ---03/28/2021 11:17:37 PM---Dear All, We want to say Hyperledger Fabric is a great blockchain framework and did not find issues

        From:
        "Fu, Xinwen" <Xinwen_Fu@...>
        To:
        David Enyeart <enyeart@...>, Brian Behlendorf <bbehlendorf@...>, "fabric@..." <fabric@...>
        Cc:
        Manish Sethi1 <Manish.Sethi1@...>, "security@..." <security@...>, "Wang, Shan" <Shan_Wang@...>, Yue Zhang <zyueinfosec@...>
        Date:
        03/28/2021 11:17 PM
        Subject:
        [EXTERNAL] RE: Security Analysis of Private Data Collection of Hyperledger Fabric





        Dear All,


        We want to say Hyperledger Fabric is a great blockchain framework and did not find issues with other parts of Hyperledger Fabric although the students spent a year on security analysis of the entire Hyperledger Fabric. I also think the students’ work shall be recognized by ICDCS.


        We propose to put the following “Responsible Disclosure” statement into the paper and clarify your stance.


        “Responsible Disclosure: The authors have communicated and reported the findings of this paper to the Hyperledger Fabric team since August 2020. According to the Hyperledger Fabric team, some findings reported in the paper are the design features and they may not cause security threats as the users may choose not to use them. We nevertheless report these findings here to raise user awareness and avoid security pitfalls.


        Here is our response to some of the questions.


        We agree with Brian: “we should not assume that Fabric networks will always be small and all users will be trusted - if there are opportunities for abuse by peers or even orderers who have been compromised, those should be closed down by design (or at least by default).”.


        The problem is kind of analogous to the issue with strcpy(). strcpy() is a feature of C, but causes buffer overflow attacks. Our paper shows potential attacks against some of the designs and proposes defense measures to defeat potential attacks.


        Most of the related statements in the Hyperledger Fabric documents Dave provided in the email were added after we reported these issues at HackerOne below. We are wondering if we contributed to the refinement of the document.

        https://hackerone.com/bugs?report_id=962705&subject=hyperledger
        https://hackerone.com/bugs?report_id=926222&subject=hyperledger
        https://hackerone.com/bugs?report_id=951623&subject=hyperledger

        We also believe that some attacks on PDC read-only transactions cannot be avoided by only following the documented guidance.
                1. For the third “vulnerability” on the exposed ``Payload” field in the transaction proposal response, under current design, if users choose to submit the read-only transaction for the proof purpose, the PDC will be definitely revealed to all peers in the same channel. To defeat such PDC leakage issues, we have proposed a solution in Section IV-C3 in our paper.

                2. For the second “vulnerability” (PDC transactions are validated through the chaincode-level policy by default), under current design, the read-only PDC transactions are validated always by the chaincode-level policy according to our source code analysis. Consequently, when users submit the PDC read-only transaction for the proof purpose, users can not use collection-level endorsement policies to further restrict which organization peers can endorse such transactions.


        We also have disagreements on other arguments and have presented our reasoning in the paper.


        Thanks.


        Xinwen Fu
        Professor
        Department of Computer Science
        University of Massachusetts Lowell

        http://www.cs.uml.edu/~xinwenfu/



From: Fu, Xinwen
Sent:
Wednesday, March 24, 2021 11:20 PM
To:
David Enyeart <enyeart@...>; Brian Behlendorf <bbehlendorf@...>
Cc:
Manish Sethi1 <Manish.Sethi1@...>; security@...; Wang, Shan <Shan_Wang@...>; wangshan@...; Yue Zhang <zyueinfosec@...>
Subject:
RE: Security Analysis of Private Data Collection of Hyperledger Fabric

Hi Dave and Brian,


Here is the third thread of our report to HackerOne:
https://hackerone.com/bugs?report_id=951623&subject=hyperledger.

I’m currently tangled with other errands. We will post our followup to
fabric@... by this weekend.

Regards,


Xinwen Fu


From:
David Enyeart <enyeart@...>
Sent:
Wednesday, March 24, 2021 1:39 AM
To:
Brian Behlendorf <bbehlendorf@...>
Cc:
Manish Sethi1 <Manish.Sethi1@...>; security@...; Wang, Shan <Shan_Wang@...>; wangshan@...; Fu, Xinwen <Xinwen_Fu@...>; Yue Zhang <zyueinfosec@...>
Subject:
RE: Security Analysis of Private Data Collection of Hyperledger Fabric

This e-mail originated from outside the UMass Lowell network.


I have joined HackerOne now and added more details to the HackerOne resolutions.

I agree with Brian, since the reported vulnerabilities are features rather than vulnerabilities and already described in documentation with use cases and avoidance guidance, any further discussion would be appropriate on the Fabric mailing list.



Dave Enyeart
IBM Blockchain

enyeart@...

Brian Behlendorf ---03/24/2021 12:45:00 AM---First off, it's great news to see university research groups performing this kind of review of any

From:
Brian Behlendorf <bbehlendorf@...>
To:
"Fu, Xinwen" <Xinwen_Fu@...>, David Enyeart <enyeart@...>, "Wang, Shan" <Shan_Wang@...>
Cc:
"security@..." <security@...>, "wangshan@..." <wangshan@...>, Yue Zhang <zyueinfosec@...>, Manish Sethi1 <Manish.Sethi1@...>
Date:
03/24/2021 12:45 AM
Subject:
[EXTERNAL] Re: Security Analysis of Private Data Collection of Hyperledger Fabric





First off, it's great news to see university research groups performing this kind of review of any Hyperledger project, and I for one am appreciative of the interest. even if the end results are educational. Thank you to Mr. Fu and the rest

First off, it's great news to see university research groups performing this kind of review of any Hyperledger project, and I for one am appreciative of the interest. even if the end results are educational. Thank you to Mr. Fu and the rest of the team. And thank you Dave for such a rapid and thorough response.

Jumping in for context for others on security@: I found two threads from HackerOne on these issues:


https://hackerone.com/bugs?report_id=926222&subject=hyperledger
https://hackerone.com/bugs?report_id=962705&subject=hyperledger

In both cases there were responses from Gari Singh that mostly mirrored Dave's reply, though without the fuller detail. Both ended with some remaining questions from the researchers, but there wasn't a closing follow up. After a couple of months for the first and a couple of weeks for the second, Ry closed them as "As designed".

There isn't an obligation from teams to satisfy everyone with replies, but this feels like the kind of conversation that should have been moved to a more public forum after the first replies from Gari. It's terrific that the initial reports were made privately just in case they had been serious issues, as that's proper practice, but when the conversations happen privately they can't help inform the next wave of users who believe they've found the same holes, or to address what may be gaps in documentation or even design that could be addressed.

Xinwen, would you feel comfortable posting your followup to Dave's points to the Fabric developer mailing list, at
fabric@...? You can join at https://lists.hyperledger.org/g/fabric

Dave, would you or other Fabric maintainers engage on this topic there?

I suspect at the very least this conversation would demonstrate that understanding the security issues around PDC and endorsement in general is really important, so as to avoid people using it incorrectly, or inadvertently leaving open holes. A research paper on that may be less sexy than one that achieves a CERT but it might also help advance the field anyways. What would be even more helpful would be suggested fixes or additions to the Fabric docs on the topic.

I also think we should not assume that Fabric networks will always be small and all users will be trusted - if there are opportunities for abuse by peers or even orderers who have been compromised, those should be closed down by design (or at least by default).

Brian

On 3/23/21 9:07 PM, Fu, Xinwen wrote:
                  Hi Dave,

                  We have been reporting our research to Hyperledger Fabric through HackerOne since August 2020 while the reply at HackerOne was often super brief. We believe those designs in question are problematic and will provide more detailed explanation later.

                  Thanks.

                  Xinwen Fu
                  Professor
                  Department of Computer Science
                  University of Massachusetts Lowell

                  http://www.cs.uml.edu/~xinwenfu/



                  From:
                  David Enyeart <enyeart@...>
                  Sent:
                  Tuesday, March 23, 2021 11:37 PM
                  To:
                  Wang, Shan <Shan_Wang@...>
                  Cc:
                  security@...; wangshan@...; Fu, Xinwen <Xinwen_Fu@...>; Yue Zhang <zyueinfosec@...>; Manish Sethi1 <Manish.Sethi1@...>
                  Subject:
                  Re: Security Analysis of Private Data Collection of Hyperledger Fabric

                  Hello, thank you for the submission. The reported vulnerabilities are actually not vulnerabilities however, but are working as designed. In fact, they are a critical and necessary aspect for how private data is used in many production solutions. And for use cases where the behavior is not desirable, it can be disabled and avoided through documented guidance. I will include links to the documentation that describe the appropriate use.
                  Since the reported vulnerabilities are features of Fabric with appropriate use already documented, I would request that the paper be withdrawn.
                  In the future, feel free to reach out to me or any of the Fabric maintainers, so that we can explain the design before you invest time in writing a paper.

                  The first two reported vulnerabilities are related, therefore I will point out references in the Fabric documentation that describe the appropriate use of these aspects.

                  (i) PDC Non-member peers can endorse PDC transactions;
                  (ii) PDC transactions are validated through the same endorsement policy as public data transactions;


                  https://hyperledger-fabric.readthedocs.io/en/latest/private-data/private-data.html#sharing-private-data
                  "You don’t necessarily have to be a member of a collection to write to a key in a collection, as long as the endorsement policy is satisfied. Endorsement policy can be defined at the chaincode level, key level (using state-based endorsement), or collection level (starting in Fabric v2.0)."

                  "Sharing private data with other collections - You could ‘share’ the private data on-chain with chaincode that creates a matching key/value in the other organization’s private data collection. You’d pass the private data key/value to chaincode via transient field, and the chaincode could confirm a hash of the passed private data matches the on-chain hash from your collection using GetPrivateDataHash(), and then write the private data to the other organization’s private data collection."


                  https://hyperledger-fabric.readthedocs.io/en/latest/private-data-arch.html
                  "For write-only transactions, organizations that are not members of the collection distribution policy but are included in the chaincode level endorsement policy may endorse transactions that write to the private data collection. If this is not desirable, utilize a collection level
                  endorsementPolicy to restrict the set of allowed endorsers to the private data distribution policy members."

                  https://hyperledger-fabric.readthedocs.io/en/latest/endorsement-policies.html#setting-collection-level-endorsement-policies
                  "You can use collection-level endorsement policies to restrict which organization peers can write to the private data collection key namespace, for example to ensure that non-authorized organizations cannot write to a collection, and to have confidence that any state in a private data collection has been endorsed by the required collection organization(s)."

                  "If you do not specify a collection-level endorsement policy, the chaincode-level endorsement policy will be applied to protect writes to a private data collection key namespace. This may be desirable if a set of organizations meeting the chaincode-level endorsement policy are authorized to create data in other organization’s private data collection. For example if those organizations are trusted to process transactions but are not authorized to store and later query private data due to industry privacy regulations, or if the private data is being shared or transferred from one set of organizations to another through the use of private data collections. In other scenarios, the private data collection members may need to be in full control of writes to the private data collection, in which case a collection-level endorsement policy should be provided."


                  The third reported vulnerability is just something that chaincode authors need to be aware of, which has been added to the Fabric documentation:
                  (iii) Exposed ``Payload” field in transaction proposal response.


                  https://hyperledger-fabric.readthedocs.io/en/latest/private-data-arch.html#protecting-private-data-responses
                  "Chaincode can return any data to a client application in the proposal response payload field. For read-only chaincode functions that query private data and which will not get submitted as transactions to the ordering service, private data may be returned in the proposal response payload field to the requesting client. For chaincode functions that propose private data writes however, take care not to include private data in the proposal response payload field, since this field will get included in the transaction which all channel members can access."

                  Similarly, the reported attacks are not possible if the endorsement policies and proposal response payload are used per the documented guidance.


                  Thank you,


                  Dave Enyeart
                  IBM Blockchain

                  enyeart@...

                  "Wang, Shan" ---03/23/2021 11:17:16 AM---Dear Hyperledger Fabric, We report some of our security analysis of the private data collections (PD

                  From:
                  "Wang, Shan" <Shan_Wang@...>
                  To:
                  "security@..." <security@...>
                  Cc:
                  "Fu, Xinwen" <Xinwen_Fu@...>, Yue Zhang <zyueinfosec@...>, "wangshan@..." <wangshan@...>
                  Date:
                  03/23/2021 11:17 AM
                  Subject:
                  [EXTERNAL] Security Analysis of Private Data Collection of Hyperledger Fabric







                  Dear Hyperledger Fabric,

                  We report some of our security analysis of the private data collections (PDC) to you and attached is a paper systematically presenting a complete security analysis. The paper will be published in July 2021 at the IEEE International Conference on Distributed Computing Systems (ICDCS).

                  We believe we have discovered three design flaws on the private data collection in Hyperledger Fabric:
                  (i) PDC Non-member peers can endorse PDC transactions;
                  (ii) PDC transactions are validated through the same endorsement policy as public data transactions;
                  (iii) Exposed ``Payload” field in transaction proposal response.

                  We have discovered two classes of attacks against PDC transactions by exploiting the three design flaws:
                  (i) Fake PDC results injection attack: malicious peers or clients may disrupt the integrity of the ledger. They may inject a valid transaction with a fake value into the blockchain or write fake values into the PDC in the ledger's world state.
                  (ii) PDC leakage issues: the PDC can be revealed to PDC non-member peers and this violates the PDC design principles.

                  We have designed defense measures to fix the three design flaws so as to defeat the discovered attacks, and implement these defense measures by modifying the source code of Hyperledger Fabric. Our patches have minor or negligible impact on the system performance.

                  More details are presented in the attached paper. Please let us know if you have any questions.


                  Best Regards,

                  Shan Wang, Southeast University/University of Massachusetts Lowell
                  Yue Zhang, Jinan University
                  Xinwen Fu, University of Massachusetts Lowell



                  (See attached file: Security Analysis of Private Data Collection of Hyperledger Fabric_report.pdf)
--
Brian Behlendorf
General Manager for Blockchain, Healthcare and Identity

bbehlendorf@...
Twitter: @brianbehlendorf

[attachment "Hyperledger_PDC_ICDCS2021_final.pdf" deleted by David Enyeart/Durham/IBM]





Join fabric@lists.hyperledger.org to automatically receive all group messages.