Extremely strange behavior with Fabric - modifying ledger out of band
|
Siddharth Jain
Summary: we created a simple Fabric network that comes with the IBM Blockchain extension and uses CouchDB and one peer and one orderer. we made some chaincode invocation requests and created some data in the ledger. Then we made changes to CouchDB records from
the CouchDB web based UI (Fauxton) which can be accessed at
http://localhost:17055/_utils.
Observed: It resulted in new blocks being appended to the ledger! When we re-started the network we could see the block height increased and changes made by Fauxton showed up!
Expected: No new blocks should have been appended to the ledger. Fauxton doesn't even know who I am. It doesn't know my X.509 certificate. Who signed the new blocks? How did the changes get endorsed? How did making changes from Fauxton result in generation
of a Fabric transaction? This is truly bizarre.
Our understanding of Fabric was that the ledger data is stored in blocks under /var/hyperledger/production/ledgersData folder and the couchdb docker container would read these blocks and initialize the couchdb database. A user is not prevented from modifying
records using Fauxton but that won't change anything under the /var/hyperledger/production/ledgersData
folder which stores the actual blocks. So if the network is re-started, it should not show any out-of-band changes. But that simply did not turn out to be true.
Can anyone explain this?
e.g., we start with this:
we click on this record which gives
we change createdBy to david and click on save changes button
when we stop and re-start the network
the record has been permanently modified and its creator is now david! so anyone can make whatever changes they want to the ledger?!
|
|||
|
|
|||
|
David Enyeart
Something else must be going on, since what you've described is not possible. New blocks with validated transactions drive the CouchDB state database updates, never the other way around. Watch the peer log as you make updates and I expect you'll discovery something else going on. Summary: we created a simple Fabric network that comes with the IBM Blockchain extension and uses CouchDB and one peer and one orderer. we made some chaincode invocation requests and created some data in the ledger. Then we made changes to CouchDB records from the CouchDB web based UI (Fauxton) which can be accessed at http://localhost:17055/_utils. Observed: It resulted in new blocks being appended to the ledger! When we re-started the network we could see the block height increased and changes made by Fauxton showed up! Expected: No new blocks should have been appended to the ledger. Fauxton doesn't even know who I am. It doesn't know my X.509 certificate. Who signed the new blocks? How did the changes get endorsed? How did making changes from Fauxton result in generation of a Fabric transaction? This is truly bizarre. Our understanding of Fabric was that the ledger data is stored in blocks under /var/hyperledger/production/ledgersData folder and the couchdb docker container would read these blocks and initialize the couchdb database. A user is not prevented from modifying records using Fauxton but that won't change anything under the /var/hyperledger/production/ledgersData folder which stores the actual blocks. So if the network is re-started, it should not show any out-of-band changes. But that simply did not turn out to be true. Can anyone explain this? e.g., we start with this:  we click on this record which gives  we change createdBy to david and click on save changes button  when we stop and re-start the network the record has been permanently modified and its creator is now david! so anyone can make whatever changes they want to the ledger?! |
|||
|
|
|||
|
Siddharth Jain
see this video: https://youtu.be/KOS5CeP-mNY
the resolution is very poor but it contains the steps to repro the issue.
From: David Enyeart <enyeart@...>
Sent: Saturday, November 16, 2019 6:52 AM To: Siddharth Jain <siddjain@...> Cc: fabric@... <fabric@...> Subject: Re: [Hyperledger Fabric] Extremely strange behavior with Fabric - modifying ledger out of band Something else must be going on, since what you've described is not possible. New blocks with validated transactions drive the CouchDB state database updates, never the other way around. Watch the peer log as you make updates and I expect
you'll discovery something else going on. Summary: we created a simple Fabric network that comes with the IBM Blockchain extension and uses CouchDB and one peer and one orderer. we made some chaincode invocation requests and created some data in the ledger. Then we made changes to CouchDB records from the CouchDB web based UI (Fauxton) which can be accessed at http://localhost:17055/_utils. Observed: It resulted in new blocks being appended to the ledger! When we re-started the network we could see the block height increased and changes made by Fauxton showed up! Expected: No new blocks should have been appended to the ledger. Fauxton doesn't even know who I am. It doesn't know my X.509 certificate. Who signed the new blocks? How did the changes get endorsed? How did making changes from Fauxton result in generation of a Fabric transaction? This is truly bizarre. Our understanding of Fabric was that the ledger data is stored in blocks under /var/hyperledger/production/ledgersData folder and the couchdb docker container would read these blocks and initialize the couchdb database. A user is not prevented from modifying records using Fauxton but that won't change anything under the /var/hyperledger/production/ledgersData folder which stores the actual blocks. So if the network is re-started, it should not show any out-of-band changes. But that simply did not turn out to be true. Can anyone explain this? e.g., we start with this:  we click on this record which gives  we change createdBy to david and click on save changes button  when we stop and re-start the network the record has been permanently modified and its creator is now david! so anyone can make whatever changes they want to the ledger?! |
|||
|
|
|||
|
Joao Antunes
Hi, Are you storing the database locally (through a volume)? If yes, then the container is reloading CouchDB dataset And if you reset the peer (there is a new command in 1.4.3 for this), what is the value that it returns? |
|||
|
|
|||
|
David Enyeart
Remember that blockchains are decentralized ledgers. What you've done here is essentially tampered/corrupted your own local peer's state database so that it is not in sync with the blockchain blocks. This would be similar to tampering/corrupting a bitcoin node's chainstate database of unspent transactions. You won't be able to convince other nodes in the network that your data is correct. In fact they will detect the bad data upon the next endorsement request, and choose never to do business with you again, which was why you'd want to join a Fabric network consortium in the first place. see this video: https://youtu.be/KOS5CeP-mNY
the resolution is very poor but it contains the steps to repro the issue. From: David Enyeart <enyeart@...>
Sent: Saturday, November 16, 2019 6:52 AM To: Siddharth Jain <siddjain@...> Cc: fabric@... <fabric@...> Subject: Re: [Hyperledger Fabric] Extremely strange behavior with Fabric - modifying ledger out of band Something else must be going on, since what you've described is not possible. New blocks with validated transactions drive the CouchDB state database updates, never the other way around. Watch the peer log as you make updates and I expect you'll discovery something else going on. Use a logging string when starting peer such as: FABRIC_LOGGING_SPEC=info:kvledger,statecouchdb,couchdb=debug Dave Enyeart  "Siddharth Jain" ---11/15/2019 08:35:29 PM---Summary: we created a simple Fabric network that comes with the IBM Blockchain extension and uses CoFrom: "Siddharth Jain" <siddjain@...> To: "fabric@..." <fabric@...> Date: 11/15/2019 08:35 PM Subject: [EXTERNAL] [Hyperledger Fabric] Extremely strange behavior with Fabric - modifying ledger out of band Sent by: fabric@... Summary: we created a simple Fabric network that comes with the IBM Blockchain extension and uses CouchDB and one peer and one orderer. we made some chaincode invocation requests and created some data in the ledger. Then we made changes to CouchDB records from the CouchDB web based UI (Fauxton) which can be accessed at http://localhost:17055/_utils. Observed: It resulted in new blocks being appended to the ledger! When we re-started the network we could see the block height increased and changes made by Fauxton showed up! Expected: No new blocks should have been appended to the ledger. Fauxton doesn't even know who I am. It doesn't know my X.509 certificate. Who signed the new blocks? How did the changes get endorsed? How did making changes from Fauxton result in generation of a Fabric transaction? This is truly bizarre. Our understanding of Fabric was that the ledger data is stored in blocks under /var/hyperledger/production/ledgersData folder and the couchdb docker container would read these blocks and initialize the couchdb database. A user is not prevented from modifying records using Fauxton but that won't change anything under the /var/hyperledger/production/ledgersData folder which stores the actual blocks. So if the network is re-started, it should not show any out-of-band changes. But that simply did not turn out to be true. Can anyone explain this? e.g., we start with this:  we click on this record which gives  we change createdBy to david and click on save changes button  when we stop and re-start the network the record has been permanently modified and its creator is now david! so anyone can make whatever changes they want to the ledger?!
|
|||
|
|
|||
|
Siddharth Jain
Thanks for the reply. also adding another link to this thread about same question: https://stackoverflow.com/questions/49934312/how-your-data-is-safe-in-hyperledger-fabric-when-one-can-make-changes-to-couchdb
How do we rebuild the state database? we tried to drop the database (mychannel_test) (screenshot below) https://imagebin.ca/v/52UG2V44kOFz and restart the network but that does not rebuild the database. It is completely lost. |
|||
|
|
|||
|
David Enyeart
To rebuild peer's state database: Thanks for the reply. also adding another link to this thread about same question: https://stackoverflow.com/questions/49934312/how-your-data-is-safe-in-hyperledger-fabric-when-one-can-make-changes-to-couchdb How do we rebuild the state database? we tried to drop the database (mychannel_test) (screenshot below) https://imagebin.ca/v/52UG2V44kOFz and restart the network but that does not rebuild the database. It is completely lost. |
|||
|
|
|||
|
Siddharth Jain
see this video where it is shown that dropping the db and restarting the peer does not rebuild the db:
From: David Enyeart <enyeart@...>
Sent: Tuesday, November 19, 2019 10:32 PM To: Siddharth Jain <siddjain@...> Cc: fabric@... <fabric@...> Subject: Re: [Hyperledger Fabric] Extremely strange behavior with Fabric - modifying ledger out of band To rebuild peer's state database: Thanks for the reply. also adding another link to this thread about same question: https://stackoverflow.com/questions/49934312/how-your-data-is-safe-in-hyperledger-fabric-when-one-can-make-changes-to-couchdb How do we rebuild the state database? we tried to drop the database (mychannel_test) (screenshot below) https://imagebin.ca/v/52UG2V44kOFz and restart the network but that does not rebuild the database. It is completely lost. |
|||
|
|
|||
|
David Enyeart
You need to drop the entire CouchDB data volume (or /data directory within the volume) to trigger the state database rebuild, not individual databases within the CouchDB instance. see this video where it is shown that dropping the db and restarting the peer does not rebuild the db: https://youtu.be/h0NjZRH9RXE
From: David Enyeart <enyeart@...>
Sent: Tuesday, November 19, 2019 10:32 PM To: Siddharth Jain <siddjain@...> Cc: fabric@... <fabric@...> Subject: Re: [Hyperledger Fabric] Extremely strange behavior with Fabric - modifying ledger out of band To rebuild peer's state database: stop peer, drop state database, restart peer. State database will automatically be rebuilt from the blockchain. To rebuild peer's channel blockchains and state database: stop peer, drop state database, use "peer node reset" to reset channels to genesis blocks, restart peer. Peer will automatically re-pull and re-process blocks. Peer log will indicate rebuild progress. Dave Enyeart  "Siddharth Jain" ---11/19/2019 02:03:45 PM---Thanks for the reply. also adding another link to this thread about same question: https://urldefensFrom: "Siddharth Jain" <siddjain@...> To: fabric@... Date: 11/19/2019 02:03 PM Subject: [EXTERNAL] Re: [Hyperledger Fabric] Extremely strange behavior with Fabric - modifying ledger out of band Sent by: fabric@... Thanks for the reply. also adding another link to this thread about same question: https://stackoverflow.com/questions/49934312/how-your-data-is-safe-in-hyperledger-fabric-when-one-can-make-changes-to-couchdb How do we rebuild the state database? we tried to drop the database (mychannel_test) (screenshot below) https://imagebin.ca/v/52UG2V44kOFz and restart the network but that does not rebuild the database. It is completely lost. |
|||
|
|
"Siddharth Jain" ---11/15/2019 08:35:29 PM---Summary: we created a simple Fabric network that comes with the IBM Blockchain extension and uses Co
"Siddharth
Jain" ---11/15/2019 08:35:29 PM---Summary: we created a simple Fabric network that comes with the IBM Blockchain extension and uses Co
"Siddharth Jain" ---11/19/2019 02:03:45 PM---Thanks for the reply. also adding another link to this thread about same question: 
"Siddharth
Jain" ---11/19/2019 02:03:45 PM---Thanks for the reply. also adding another link to this thread about same question: