The sample demonstrates how to track block event progress so that your processing logic can resume where it left off if it is interrupted. Also the block events are replayable so you can re-process them from any block height if the downstream data is in doubt, assuming your downstream data store has an idempotent design. Of course, these are just tools, ultimately the data integrity is in your own hands once you pull the data out.


Dave Enyeart
IBM Blockchain
enyeart@...

"Nikos Karamolegkos" ---06/23/2021 08:05:33 AM---Thank you. I am not sure that I understand the part with the downstream data integrity. As I know, i

From: "Nikos Karamolegkos" <nkaram@...>
To: fabric@...
Date: 06/23/2021 08:05 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] off chain database that replicates the data from your peers
Sent by: fabric@...

Fabric Contributor Meeting

When:
06/23/2021
9:00am to 10:00am
(UTC-04:00) America/New York

Where:
https://zoom.us/my/hyperledger.community.3?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09

Organizer: Dave Enyeart enyeart@...

View Event

Description:
For meeting agendas, recordings, and more details, see https://wiki.hyperledger.org/display/fabric/Contributor+Meetings

Both LevelDB and CouchDB state databases are considered 'on-chain'.

You are correct - for advanced query, reporting, and analytics, the recommendation is to build an 'off-chain' data store based on the validated transactions.

https://hyperledger-fabric.readthedocs.io/en/latest/couchdb_as_state_database.html#good-practices-for-queries
"If you want to build a dashboard or collect aggregate data as part of your application, you can query an off-chain database that replicates the data from your blockchain network. This will allow you to query and analyze the blockchain data in a data store optimized for your needs, without degrading the performance of your network or disrupting transactions. To achieve this, applications may use block or chaincode events to write transaction data to an off-chain database or analytics engine. For each block received, the block listener application would iterate through the block transactions and build a data store using the key/value writes from each valid transaction’s rwset. The Peer channel-based event services provide replayable events to ensure the integrity of downstream data stores."

If you are concerned about downstream data store integrity, you can at any time re-build it or build another copy based on the prior validated transactions.

Sample that uses a separate CouchDB database for the off-chain database, but it could use any downstream data store:  https://github.com/hyperledger/fabric-samples/tree/main/off_chain_data


Dave Enyeart

"Nikos Karamolegkos" ---06/23/2021 06:25:17 AM---As I can understand from the documentation. I can use LevelDB or CouchDB  as state DB. Both of these

From: "Nikos Karamolegkos" <nkaram@...>
To: fabric <fabric@...>
Date: 06/23/2021 06:25 AM
Subject: [EXTERNAL] [Hyperledger Fabric] off chain database that replicates the data from your peers
Sent by: fabric@...

You are correct - CouchDB does full database scans with $or and $in clauses so that approach is expected to be awfully slow.

The solution is to make a GetState() call for each key you need. Although this will require a call to CouchDB for each key, each call will return very quickly since it will utilize a direct index hit. This will work well if you have a small number of keys to lookup, obviously it will not scale well if you have many keys to lookup.

To support many key lookups efficiently, it would be possible for Fabric to implement a new GetStateMultipleKeys() chaincode API which calls the CouchDB bulk retrieve API. I've created a story if anybody would like to contribute the feature - https://jira.hyperledger.org/browse/FAB-18507.


Dave Enyeart

"Samyak Jain | TraceX" ---06/22/2021 11:06:02 AM---Hi Community, We are trying to fetch multiple keys from Fabric ledger as part of traceability of a p

From: "Samyak Jain | TraceX" <samyakj@...>
To: fabric@...
Date: 06/22/2021 11:06 AM
Subject: [EXTERNAL] [Hyperledger Fabric] CouchDB performance awfully slow
Sent by: fabric@...

Hi Community,

We are trying to fetch multiple keys from Fabric ledger as part of traceability of a product. We are trying to do selector queries on a ledger containing already around ~40k transactions, and just fetching 2 ids (using primary key) with the $or clause is taking close to 8 seconds from the client application. The performance was much better when there were fewer records. I read in a lot of places that the index may not be supported for $or and $in clauses, in which case is there any other way to fetch multiple keys from the ledger in a single query? Note that keys are not part of a range. Please suggest optimization startegies.

I feel that this level of performance should not be the case, and if this is so, does this mean we have to store data off-chain? This would be a step backwards for showing traceability directly from the chain.

Example selector query: { $or: [ { asset_id: "xyz", }, { asset_id: "abc", }, ], }

Thanks, Samyak.

I'll plan to give a quick update on what has been going on in Fabric development recently. Any other agenda topics for the meeting this week?  

-------------------------------------------------------------------------

Hyperledger Fabric Contributor Meeting

When: Every other Wednesday 9am US Eastern, 13:00 UTC (during US daylight savings time)

Where: https ://zoom.us/j/51849 47650?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09

Agendas and Recordings: https://wiki.hyperledger.org/display/fabric/Contributor+Meetings

Hi Community,

We are trying to fetch multiple keys from Fabric ledger as part of traceability of a product. We are trying to do selector queries on a ledger containing already around ~40k transactions, and just fetching 2 ids (using primary key) with the $or clause is taking close to 8 seconds from the client application. The performance was much better when there were fewer records. I read in a lot of places that the index may not be supported for $or and $in clauses, in which case is there any other way to fetch multiple keys from the ledger in a single query? Note that keys are not part of a range. Please suggest optimization startegies.

I feel that this level of performance should not be the case, and if this is so, does this mean we have to store data off-chain? This would be a step backwards for showing traceability directly from the chain.

Example selector query: { $or: [ { asset_id: "xyz", }, { asset_id: "abc", }, ], }

Thanks, Samyak.

Private Chaincode Lab

When:
06/22/2021
8:00am to 9:00am
(UTC-07:00) America/Los Angeles

Where:
https://zoom.us/my/hyperledger.community.3?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09

Organizer: Marcus Brandenburger bur@...

View Event

Description:
Two of the Hyperleger Labs projects (private data objects and private chain code) are collaborating to develop a "private smart contracts" capability.

Join Zoom Meeting https://zoom.us/j/5184947650?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09 Meeting ID: 518 494 7650 Passcode: 475869

Hi,
Is it possible to cross-invoke a chaincode in other channel (channel-1) (using an identity that isn't part of the channel) from a chaincode in the channel-2 (identity is declared here)?

Example:
Org3 and Org4 are part of channel-2.
Org1, Org2 and Org3 are part of channel-1.
Org3 has chaincodeA (instantiated in channel-1) and chaincodeB (instantiated in channel-2) installed.

Org4.member invokes (query) chaincodeB (in Org3Peer) in channel-2 (ok, because Org4 is in channel-2), then chaincodeB internally invokes (query) chaincodeA (fails, because Org4Member MSP isn't defined in channel-1).

Is it possible to bypass this problem?
Any help appreciated.