PrivateData is marketed as a data privacy solution in Hyperledger Fabric. Unfortunately, this is just another serious security hole somehow went under the radar, and all projects using this function are at risk.  

It amazes me that nobody had mentioned this before so I guess I better point this out now before more damages are being done.  

The logic behind Privated data is simple, it put data in a local embedded data store and put a hash of that data on blockchain.  

The issue is that cryptographic hash is not an encryption mechanism, same data hashed by anyone using the same hashing algorithm will always get you the same hash! This is exactly what hash functions are designed for, and that’s why we use hash in digital signature to allow anyone to validate signed data.   However, this also means that anyone can “decrypt” the data behind the hash by launching dictionary attack.  

Hashing is cheap, the cost of each hash on a normal laptop cpu core is about 3 microseconds, basically I can create 1 billion candidate result hashes within one hour on a single laptop cpu, and check if they match to the hashes on hyperledger fabric DLT.   And I am just talking about using a single cpu on my laptop, not even 50% of its processing power  

Why is it dangerous? Because if an attacker is connected to a blockchain system, the attacker likely know the range of the data being hashed (for example, hashed data could be trade ID, item name, bank name, address, cell phone number), so you can easily create dictionary attack to get the true data behind the hash.  

How about adding salt to each data to be hashed? Well, that’s one thing Hyperledger Fabric didn’t do.   To their defense, hyperledger didn’t implement salt because it is difficult to pass salts to counter parties. You can’t use DLT to pass salt value to counter parties because attackers would see it, so you have to create another p2p connection with counter party and send it over.

If you already have p2p connection with all the counter parties, what’s the point of using blockchain in the first place? just send your data over! It’s just scary that so many people are using this security hole and put their data in de facto clear text.  

Sure, if the hashed data is so big then it would harder to perform dictionary attack, but you better be very careful before using this feature because any mis-use will result in data leak, it is sad so many people actually believe this is a problem solver

It amazes me that nobody had mentioned this before so I guess I better point this out now before more damages are being done.  

The logic behind Privated data is simple, it put data in a local embedded data store and put a hash of that data on blockchain.  

The issue is that cryptographic hash is not an encryption mechanism, same data hashed by anyone using the same hashing algorithm will always get you the same hash! This is exactly what hash functions are designed for, and that’s why we use hash in digital signature to allow anyone to validate signed data.   However, this also means that anyone can “decrypt” the data behind the hash by launching dictionary attack.  

Hashing is cheap, the cost of each hash on a normal laptop cpu core is about 3 microseconds, basically I can create 1 billion candidate result hashes within one hour on a single laptop cpu, and check if they match to the hashes on hyperledger fabric DLT.   And I am just talking about using a single cpu on my laptop, not even 50% of its processing power  

Why is it dangerous? Because if an attacker is connected to a blockchain system, the attacker likely know the range of the data being hashed (for example, hashed data could be trade ID, item name, bank name, address, cell phone number), so you can easily create dictionary attack to get the true data behind the hash.  

How about adding salt to each data to be hashed? Well, that’s one thing Hyperledger Fabric didn’t do.   To their defense, hyperledger didn’t implement salt because it is difficult to pass salts to counter parties. You can’t use DLT to pass salt value to counter parties because attackers would see it, so you have to create another p2p connection with counter party and send it over.

If you already have p2p connection with all the counter parties, what’s the point of using blockchain in the first place? just send your data over! It’s just scary that so many people are using this security hole and put their data in de facto clear text.  

Sure, if the hashed data is so big then it would harder to perform dictionary attack, but you better be very careful before using this feature because any mis-use will result in data leak, it is sad so many people actually believe this is a problem solver

Thanks for replying Yacov and Senthil. You're right that since the introduction of private data, Fabric recommends that private data be salted to avoid dictionary attacks. As this thread makes clear not everybody knows about the private data solution design considerations. I've opened Jira issue https://jira.hyperledger.org/browse/FAB-16885 to enhance the documentation with these considerations.


Dave Enyeart

"Senthil Nathan" ---10/21/2019 09:58:56 AM---Hi Ivan, Thank you for bringing this. We have discussed about including salt in

From: "Senthil Nathan" <cendhu@...>
To: Ivan Ch <acizlan@...>
Cc: fabric@...
Date: 10/21/2019 09:58 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Major security hole in Hyperledger Fabric - Private Data is not private #fabric #fabric-questions #fabric-dstorage #database #dstorage #dstorage-fabric #fabric-chaincode #ssl
Sent by: fabric@...

It amazes me that nobody had mentioned this before so I guess I better point this out now before more damages are being done.  

The logic behind Privated data is simple, it put data in a local embedded data store and put a hash of that data on blockchain.  

The issue is that cryptographic hash is not an encryption mechanism, same data hashed by anyone using the same hashing algorithm will always get you the same hash! This is exactly what hash functions are designed for, and that’s why we use hash in digital signature to allow anyone to validate signed data.   However, this also means that anyone can “decrypt” the data behind the hash by launching dictionary attack.  

Hashing is cheap, the cost of each hash on a normal laptop cpu core is about 3 microseconds, basically I can create 1 billion candidate result hashes within one hour on a single laptop cpu, and check if they match to the hashes on hyperledger fabric DLT.   And I am just talking about using a single cpu on my laptop, not even 50% of its processing power  

Why is it dangerous? Because if an attacker is connected to a blockchain system, the attacker likely know the range of the data being hashed (for example, hashed data could be trade ID, item name, bank name, address, cell phone number), so you can easily create dictionary attack to get the true data behind the hash.  

How about adding salt to each data to be hashed? Well, that’s one thing Hyperledger Fabric didn’t do.   To their defense, hyperledger didn’t implement salt because it is difficult to pass salts to counter parties. You can’t use DLT to pass salt value to counter parties because attackers would see it, so you have to create another p2p connection with counter party and send it over.

If you already have p2p connection with all the counter parties, what’s the point of using blockchain in the first place? just send your data over! It’s just scary that so many people are using this security hole and put their data in de facto clear text.  

Sure, if the hashed data is so big then it would harder to perform dictionary attack, but you better be very careful before using this feature because any mis-use will result in data leak, it is sad so many people actually believe this is a problem solver

Thanks again Ivan for pointing out the documentation hole - here's the doc update that describes how private data is secured:
https://hyperledger-fabric.readthedocs.io/en/latest/private-data-arch.html#protecting-private-data-content


Dave Enyeart

"David Enyeart" ---10/21/2019 11:03:49 AM---Thanks for replying Yacov and Senthil. You're right that since the introduction of private data, Fa

From: "David Enyeart" <enyeart@...>
To: "Senthil Nathan" <cendhu@...>
Cc: Ivan Ch <acizlan@...>, fabric@...
Date: 10/21/2019 11:03 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Major security hole in Hyperledger Fabric - Private Data is not private #fabric #fabric-questions #fabric-dstorage #database #dstorage #dstorage-fabric #fabric-chaincode #ssl
Sent by: fabric@...

It amazes me that nobody had mentioned this before so I guess I better point this out now before more damages are being done.  

The logic behind Privated data is simple, it put data in a local embedded data store and put a hash of that data on blockchain.  

The issue is that cryptographic hash is not an encryption mechanism, same data hashed by anyone using the same hashing algorithm will always get you the same hash! This is exactly what hash functions are designed for, and that’s why we use hash in digital signature to allow anyone to validate signed data.   However, this also means that anyone can “decrypt” the data behind the hash by launching dictionary attack.  

Hashing is cheap, the cost of each hash on a normal laptop cpu core is about 3 microseconds, basically I can create 1 billion candidate result hashes within one hour on a single laptop cpu, and check if they match to the hashes on hyperledger fabric DLT.   And I am just talking about using a single cpu on my laptop, not even 50% of its processing power  

Why is it dangerous? Because if an attacker is connected to a blockchain system, the attacker likely know the range of the data being hashed (for example, hashed data could be trade ID, item name, bank name, address, cell phone number), so you can easily create dictionary attack to get the true data behind the hash.  

How about adding salt to each data to be hashed? Well, that’s one thing Hyperledger Fabric didn’t do.   To their defense, hyperledger didn’t implement salt because it is difficult to pass salts to counter parties. You can’t use DLT to pass salt value to counter parties because attackers would see it, so you have to create another p2p connection with counter party and send it over.

If you already have p2p connection with all the counter parties, what’s the point of using blockchain in the first place? just send your data over! It’s just scary that so many people are using this security hole and put their data in de facto clear text.  

Sure, if the hashed data is so big then it would harder to perform dictionary attack, but you better be very careful before using this feature because any mis-use will result in data leak, it is sad so many people actually believe this is a problem solver

Thanks again Ivan for pointing out the documentation hole - here's the doc update that describes how private data is secured:
https://hyperledger-fabric.readthedocs.io/en/latest/private-data-arch.html#protecting-private-data-content


Dave Enyeart

"David Enyeart" ---10/21/2019 11:03:49 AM---Thanks for replying Yacov and Senthil. You're right that since the introduction of private data, Fa

From: "David Enyeart" <enyeart@...>
To: "Senthil Nathan" <cendhu@...>
Cc: Ivan Ch <acizlan@...>, fabric@...
Date: 10/21/2019 11:03 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Major security hole in Hyperledger Fabric - Private Data is not private #fabric #fabric-questions #fabric-dstorage #database #dstorage #dstorage-fabric #fabric-chaincode #ssl
Sent by: fabric@...

---

Thanks for replying Yacov and Senthil. You're right that since the introduction of private data, Fabric recommends that private data be salted to avoid dictionary attacks. As this thread makes clear not everybody knows about the private data solution design considerations. I've opened Jira issue https://jira.hyperledger.org/browse/FAB-16885 to enhance the documentation with these considerations.


Dave Enyeart

"Senthil Nathan" ---10/21/2019 09:58:56 AM---Hi Ivan, Thank you for bringing this. We have discussed about including salt in

From: "Senthil Nathan" <cendhu@...>
To: Ivan Ch <acizlan@...>
Cc: fabric@...
Date: 10/21/2019 09:58 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Major security hole in Hyperledger Fabric - Private Data is not private #fabric #fabric-questions #fabric-dstorage #database #dstorage #dstorage-fabric #fabric-chaincode #ssl
Sent by: fabric@...

---

Hi Ivan,

    Thank you for bringing this. We have discussed about including salt in the private data design document --
https://docs.google.com/document/d/1ShrgrYPWLznZSZrl5cnvmFq9LtLJ3tYUxjv9GN6rxuI/edit?usp=sharing
(please refer to section 2.6 Additional Consideration -- Salt Consideration).
We do have a JIRA for the same as well -- https://jira.hyperledger.org/browse/FAB-5101 but didn't implement
it as we have decided to leave it to the user for now (also for simplicity & flexibility).

    The salt to the data can always be added by the client which submits the transaction proposal. For example,
in the following JSON content, there can be an additional field called salt and the user can add any random data
to avoid a dictionary attack.
{"menu": {
"id": "file",
"value": "File",
"popup": {
  "menuitem": [
    {"value": "New", "onclick": "CreateNewDoc()"},
    {"value": "Open", "onclick": "OpenDoc()"},
    {"value": "Close", "onclick": "CloseDoc()"}
  ]
}
"salt": 88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589
}}

The same can be done for the keys too, not just values. As far as I know, many developers who use private data
follow this approach. I agree that a few might be unaware of this. As Yacov mentioned, we should add this approach
to our doc.

Regards,
Senthil

On Mon, Oct 21, 2019 at 6:51 PM Ivan Ch <acizlan@...> wrote:PrivateData is marketed as a data privacy solution in Hyperledger Fabric. Unfortunately, this is just another serious security hole somehow went under the radar, and all projects using this function are at risk.  

It amazes me that nobody had mentioned this before so I guess I better point this out now before more damages are being done.  

The logic behind Privated data is simple, it put data in a local embedded data store and put a hash of that data on blockchain.  

The issue is that cryptographic hash is not an encryption mechanism, same data hashed by anyone using the same hashing algorithm will always get you the same hash! This is exactly what hash functions are designed for, and that’s why we use hash in digital signature to allow anyone to validate signed data.   However, this also means that anyone can “decrypt” the data behind the hash by launching dictionary attack.  

Hashing is cheap, the cost of each hash on a normal laptop cpu core is about 3 microseconds, basically I can create 1 billion candidate result hashes within one hour on a single laptop cpu, and check if they match to the hashes on hyperledger fabric DLT.   And I am just talking about using a single cpu on my laptop, not even 50% of its processing power  

Why is it dangerous? Because if an attacker is connected to a blockchain system, the attacker likely know the range of the data being hashed (for example, hashed data could be trade ID, item name, bank name, address, cell phone number), so you can easily create dictionary attack to get the true data behind the hash.  

How about adding salt to each data to be hashed? Well, that’s one thing Hyperledger Fabric didn’t do.   To their defense, hyperledger didn’t implement salt because it is difficult to pass salts to counter parties. You can’t use DLT to pass salt value to counter parties because attackers would see it, so you have to create another p2p connection with counter party and send it over.

If you already have p2p connection with all the counter parties, what’s the point of using blockchain in the first place? just send your data over! It’s just scary that so many people are using this security hole and put their data in de facto clear text.  

Sure, if the hashed data is so big then it would harder to perform dictionary attack, but you better be very careful before using this feature because any mis-use will result in data leak, it is sad so many people actually believe this is a problem solver

-- 
Brian Behlendorf
Executive Director, Hyperledger
bbehlendorf@...
Twitter: @brianbehlendorf

"Private data is disseminated in a point to point manner among peers even now.The peers that posses the private data, send the peers that don't (but are eligible of receiving it) the hash pre-images, and the receiving peers validate the hash pre-images indeed correspond to the hashes on the public block.

I don't see any technical obstacle that prevents you to add a salt per collection name for a given transaction, that will be concatenated to the computation of the hash of the key and the value for the said collection.The salt can be part of the data element that is generated at the time of chaincode invocation, and will be passed along with the private data itself."

• You (as the user) have the ability to put on the blockchain hashes of salted data.
• The data is disseminated in a secure point to point connection between peers that are eligible of receiving the data.

Your second point is not specific to private data. Agreement on input data needs to be part of the application design, regardless of whether it is a private data scenario or not. For example the smart contract may require that each of the transactors submit their approval of a proposed data change on chain, before a final transaction verifies the approvals are in place and makes the change on chain.


Dave Enyeart

"Ivan Ch" ---10/23/2019 12:10:40 PM---Hi Alexandre, Yacov Thanks for your reply and I appreciate the discussion. my hands are tight now so

From: "Ivan Ch" <acizlan@...>
To: fabric@...
Date: 10/23/2019 12:10 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Major security hole in Hyperledger Fabric - Private Data is not private #fabric #fabric-questions #fabric-dstorage #database #dstorage #dstorage-fabric #fabric-chaincode #ssl
Sent by: fabric@...

Dave, Yacov, and Alex

Seems that the general response to this scenario is “this is an application design problem and should be solved by chaincode”

 
here is an example: my national ID is "1234567", but I am a bad guy and want others to believe that my national ID number is "7654321". so I put the false hash(salt, "7654321") on chain, and then send pre-images (salt, "7654321")  to whoever I want to convince. Since nobody can verify the hash(salt, "7654321")  when the hash was put on chain without prior knowledge of the data, an adversary can use the claims about private data functionality to trick people to believe forged data.

But my argument here is that chaincode design can’t solve this problem, and I can assure you that there is a large number of DLT deployments are at risk because of this.

As I stated earlier, hashes cannot be verified by third parties like digital signature or ZKP algorithm.  There is almost no way to guard against adversaries from putting fake data and then trick others to believe the fake data is real.

Since chaincode can’t decode hashes so the only thing a chaincode can perform is to limit on number of updates. In most financial use cases (e.g. trade transactions) this is irrelevant since pre-image data are not constants in the first place. Even for constant data such as “national ID” in the aforementioned scenario, chaincode most likely will still allow at least a few updates to cover typos.

Leaving it to applications is easier said than done since there are so few ways to get it right and this functionality simply opens door for attackers and yet offers almost nothing.

This bug is neither an application design issue nor fabric implementation issue, but a methodology problem that private data feature promotes. My humble recommendation is to depreciate this functionality or at least put warning signs to people still plan to use it

You are essentially suggesting to add a warning that private data content can't be known by non-members of the collection. That is the whole point of private data and anybody considering an implementation will already know this. The non-members only validate against a hash of the data. The members can later share the private data content with non-members if a need-to-know arises, and the non-member can then validate the pre-image content against the hash on chain, with an understanding that only the group of transactors may have come to agreement on the data. This is the fundamental design of private data. Like any feature, It will be fit for some use cases, and not fit for others. I believe these considerations were already obvious, but hopefully this thread has provided some clarification. I am glad the thread has at least helped to improve the documentation around the importance of including a salt in your private data if it is predictable, to keep it secure.


Dave Enyeart

"Ivan Ch" ---10/24/2019 06:02:26 AM---Dave, Yacov, and Alex Seems that the general response to this scenario is “this is an application de

From: "Ivan Ch" <acizlan@...>
To: fabric@...
Date: 10/24/2019 06:02 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Major security hole in Hyperledger Fabric - Private Data is not private #fabric #fabric-questions #fabric-dstorage #database #dstorage #dstorage-fabric #fabric-chaincode #ssl
Sent by: fabric@...

You are essentially suggesting to add a warning that private data content can't be known by non-members of the collection. That is the whole point of private data and anybody considering an implementation will already know this. The non-members only validate against a hash of the data. The members can later share the private data content with non-members if a need-to-know arises, and the non-member can then validate the pre-image content against the hash on chain, with an understanding that only the group of transactors may have come to agreement on the data. This is the fundamental design of private data.