Date   

Connection from Client Application to the (endorsing) Peers

Daniel
 

Hey guys,

as I understand, the client application proposes transactions to the peers. Is the client application sending it to only one peer or all the peers (as defined in endorsement policy)? Or is it possible, that the endorsing peer (that is receiving the proposal) is sending the proposal to the other endorsing peers? What if the peers are physical apart, how is client able to connect to nodes which are physical far away? 

Thanks in advance.
 


Re: off chain database that replicates the data from your peers

David Enyeart
 

The sample demonstrates how to track block event progress so that your processing logic can resume where it left off if it is interrupted. Also the block events are replayable so you can re-process them from any block height if the downstream data is in doubt, assuming your downstream data store has an idempotent design. Of course, these are just tools, ultimately the data integrity is in your own hands once you pull the data out.


Dave Enyeart
IBM Blockchain
enyeart@...


"Nikos Karamolegkos" ---06/23/2021 08:05:33 AM---Thank you. I am not sure that I understand the part with the downstream data integrity. As I know, i

From: "Nikos Karamolegkos" <nkaram@...>
To: fabric@...
Date: 06/23/2021 08:05 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] off chain database that replicates the data from your peers
Sent by: fabric@...





Thank you. I am not sure that I understand the part with the downstream data integrity. As I know, in a BC network with on-chain database if you want to retrieve some data you have to query the ledger of a peer (or more) and these data are build ZjQcmQRYFpfptBannerStart 
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Thank you. I am not sure that I understand the part with the downstream data integrity. As I know, in a BC network with on-chain database if you want to retrieve some data you have to query the ledger of a peer (or more) and these data are build from the transaction log so you are sure that these data are correct. In case there is an external (off-chain database) how is this feasible? I mean I can be sure for the data in on-chain DB but how can I be sure for the replicated data in off-chain DB.






Happening Now: Fabric Contributor Meeting - 06/23/2021 #cal-notice

fabric@lists.hyperledger.org Calendar <noreply@...>
 

Fabric Contributor Meeting

When:
06/23/2021
9:00am to 10:00am
(UTC-04:00) America/New York

Where:
https://zoom.us/my/hyperledger.community.3?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09

Organizer: Dave Enyeart enyeart@...

View Event

Description:
For meeting agendas, recordings, and more details, see https://wiki.hyperledger.org/display/fabric/Contributor+Meetings

Join Zoom Meeting
https://zoom.us/j/5184947650?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09
 
Meeting ID: 518 494 7650
Passcode: 475869


Re: off chain database that replicates the data from your peers

Nikos Karamolegkos <nkaram@...>
 

Thank you. I am not sure that I understand the part with the downstream data integrity. As I know, in a BC network with on-chain database if you want to retrieve some data you have to query the ledger of a peer (or more) and these data are build from the transaction log so you are sure that these data are correct. In case there is an external (off-chain database) how is this feasible? I mean I can be sure for the data in on-chain DB but how can I be sure for the replicated data in off-chain DB.


Re: off chain database that replicates the data from your peers

David Enyeart
 

Both LevelDB and CouchDB state databases are considered 'on-chain'.

You are correct - for advanced query, reporting, and analytics, the recommendation is to build an 'off-chain' data store based on the validated transactions.

https://hyperledger-fabric.readthedocs.io/en/latest/couchdb_as_state_database.html#good-practices-for-queries
"If you want to build a dashboard or collect aggregate data as part of your application, you can query an off-chain database that replicates the data from your blockchain network. This will allow you to query and analyze the blockchain data in a data store optimized for your needs, without degrading the performance of your network or disrupting transactions. To achieve this, applications may use block or chaincode events to write transaction data to an off-chain database or analytics engine. For each block received, the block listener application would iterate through the block transactions and build a data store using the key/value writes from each valid transaction’s rwset. The Peer channel-based event services provide replayable events to ensure the integrity of downstream data stores."

If you are concerned about downstream data store integrity, you can at any time re-build it or build another copy based on the prior validated transactions.

Sample that uses a separate CouchDB database for the off-chain database, but it could use any downstream data store:  https://github.com/hyperledger/fabric-samples/tree/main/off_chain_data


Dave Enyeart


"Nikos Karamolegkos" ---06/23/2021 06:25:17 AM---As I can understand from the documentation. I can use LevelDB or CouchDB  as state DB. Both of these

From: "Nikos Karamolegkos" <nkaram@...>
To: fabric <fabric@...>
Date: 06/23/2021 06:25 AM
Subject: [EXTERNAL] [Hyperledger Fabric] off chain database that replicates the data from your peers
Sent by: fabric@...





As I can understand from the documentation. I can use LevelDB or CouchDB
as state DB. Both of these two DB can be used as on-chain database.
Correct? (I am not sure if couchDB is used only as external (off-chain
database))

Also, in case I want to analyze the data from my network, the best
practice is to query an off chain database that replicates the data from
your peers. Is the CouchDB the only option or I can use hdfs or Spark?
How easy is that? I mean it is just a configuration issue?

Finally, I can use an event listener to write data to an external
database in case the transactions fulfill the smart-contracts but what
is happening during retrieve of the data from the off-chain DB, how am I
sure for the integrity of the retrieved data?

--
Nikos Karamolegkos
R & D engineer at ICS-FORTH
Telecommunications and Networks Lab (TNL)











off chain database that replicates the data from your peers

Nikos Karamolegkos <nkaram@...>
 

As I can understand from the documentation. I can use LevelDB or CouchDB as state DB. Both of these two DB can be used as on-chain database. Correct? (I am not sure if couchDB is used only as external (off-chain database))

Also, in case I want to analyze the data from my network, the best practice is to query an off chain database that replicates the data from your peers. Is the CouchDB the only option or I can use hdfs or Spark? How easy is that? I mean it is just a configuration issue?

Finally, I can use an event listener to write data to an external database in case the transactions fulfill the smart-contracts but what is happening during retrieve of the data from the off-chain DB, how am I sure for the integrity of the retrieved data?

--
Nikos Karamolegkos
R & D engineer at ICS-FORTH
Telecommunications and Networks Lab (TNL)


Re: Fabric Contributor Meeting - June 23, 2021 - any agenda topics?

Ry Jones
 


On Tue, Jun 22, 2021 at 3:32 PM Ry Jones <rjones@...> wrote:
I am interested in the appetite for simplification of who has commit bits - looking across all fabric repos, it seems fairly uneven.

The discussion doesn't need to be on this call.


On Tue, Jun 22, 2021 at 1:32 PM David Enyeart <enyeart@...> wrote:

    I'll plan to give a quick update on what has been going on in Fabric development recently. Any other agenda topics for the meeting this week?  

    -------------------------------------------------------------------------

    Hyperledger Fabric Contributor Meeting

    When: Every other Wednesday 9am US Eastern, 13:00 UTC (during US daylight savings time)

    Where: https ://zoom.us/j/51849 47650?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09

    Agendas and Recordings: https://wiki.hyperledger.org/display/fabric/Contributor+Meetings



Dave Enyeart
IBM Blockchain
enyeart@...




--
Ry Jones
Community Architect, Hyperledger


--
Ry Jones
Community Architect, Hyperledger


Re: CouchDB performance awfully slow

David Enyeart
 

You are correct - CouchDB does full database scans with $or and $in clauses so that approach is expected to be awfully slow.

The solution is to make a GetState() call for each key you need. Although this will require a call to CouchDB for each key, each call will return very quickly since it will utilize a direct index hit. This will work well if you have a small number of keys to lookup, obviously it will not scale well if you have many keys to lookup.

To support many key lookups efficiently, it would be possible for Fabric to implement a new GetStateMultipleKeys() chaincode API which calls the CouchDB bulk retrieve API. I've created a story if anybody would like to contribute the feature - https://jira.hyperledger.org/browse/FAB-18507.


Dave Enyeart


"Samyak Jain | TraceX" ---06/22/2021 11:06:02 AM---Hi Community, We are trying to fetch multiple keys from Fabric ledger as part of traceability of a p

From: "Samyak Jain | TraceX" <samyakj@...>
To: fabric@...
Date: 06/22/2021 11:06 AM
Subject: [EXTERNAL] [Hyperledger Fabric] CouchDB performance awfully slow
Sent by: fabric@...





Hi Community, We are trying to fetch multiple keys from Fabric ledger as part of traceability of a product. We are trying to do selector queries on a ledger containing already around ~40k transactions, and just fetching 2 ids (using primary ZjQcmQRYFpfptBannerStart 
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd

Hi Community,

We are trying to fetch multiple keys from Fabric ledger as part of traceability of a product. We are trying to do selector queries on a ledger containing already around ~40k transactions, and just fetching 2 ids (using primary key) with the $or clause is taking close to 8 seconds from the client application. The performance was much better when there were fewer records. I read in a lot of places that the index may not be supported for $or and $in clauses, in which case is there any other way to fetch multiple keys from the ledger in a single query? Note that keys are not part of a range. Please suggest optimization startegies.

I feel that this level of performance should not be the case, and if this is so, does this mean we have to store data off-chain? This would be a step backwards for showing traceability directly from the chain.

Example selector query: { $or: [ { asset_id: "xyz", }, { asset_id: "abc", }, ], }

Thanks, Samyak.






Re: Fabric Contributor Meeting - June 23, 2021 - any agenda topics?

Ry Jones
 

I am interested in the appetite for simplification of who has commit bits - looking across all fabric repos, it seems fairly uneven.

The discussion doesn't need to be on this call.


On Tue, Jun 22, 2021 at 1:32 PM David Enyeart <enyeart@...> wrote:

    I'll plan to give a quick update on what has been going on in Fabric development recently. Any other agenda topics for the meeting this week?  

    -------------------------------------------------------------------------

    Hyperledger Fabric Contributor Meeting

    When: Every other Wednesday 9am US Eastern, 13:00 UTC (during US daylight savings time)

    Where: https ://zoom.us/j/51849 47650?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09

    Agendas and Recordings: https://wiki.hyperledger.org/display/fabric/Contributor+Meetings



Dave Enyeart
IBM Blockchain
enyeart@...




--
Ry Jones
Community Architect, Hyperledger


Fabric Contributor Meeting - June 23, 2021 - any agenda topics?

David Enyeart
 

    I'll plan to give a quick update on what has been going on in Fabric development recently. Any other agenda topics for the meeting this week?  

    -------------------------------------------------------------------------

    Hyperledger Fabric Contributor Meeting

    When: Every other Wednesday 9am US Eastern, 13:00 UTC (during US daylight savings time)

    Where: https ://zoom.us/j/51849 47650?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09

    Agendas and Recordings: https://wiki.hyperledger.org/display/fabric/Contributor+Meetings



Dave Enyeart
IBM Blockchain
enyeart@...



CouchDB performance awfully slow

Samyak Jain | TraceX
 

Hi Community,

We are trying to fetch multiple keys from Fabric ledger as part of traceability of a product. We are trying to do selector queries on a ledger containing already around ~40k transactions, and just fetching 2 ids (using primary key) with the $or clause is taking close to 8 seconds from the client application. The performance was much better when there were fewer records. I read in a lot of places that the index may not be supported for $or and $in clauses, in which case is there any other way to fetch multiple keys from the ledger in a single query? Note that keys are not part of a range. Please suggest optimization startegies.

I feel that this level of performance should not be the case, and if this is so, does this mean we have to store data off-chain? This would be a step backwards for showing traceability directly from the chain.

Example selector query: { $or: [ { asset_id: "xyz", }, { asset_id: "abc", }, ], }

Thanks, Samyak.


Happening Now: Private Chaincode Lab - 06/22/2021 #cal-notice

fabric@lists.hyperledger.org Calendar <noreply@...>
 

Private Chaincode Lab

When:
06/22/2021
8:00am to 9:00am
(UTC-07:00) America/Los Angeles

Where:
https://zoom.us/my/hyperledger.community.3?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09

Organizer: Marcus Brandenburger bur@...

View Event

Description:
Two of the Hyperleger Labs projects (private data objects and private chain code) are collaborating to develop a "private smart contracts" capability.

Join Zoom Meeting https://zoom.us/j/5184947650?pwd=UE90WHhEaHRqOGEyMkV3cldKa2d2dz09 Meeting ID: 518 494 7650 Passcode: 475869


Re: connection is in TRANSIENT_FAILURE #fabric

Mark Lewis
 

You are absolutely correct, there is a significant overhead to tearing down and recreating connections. Definitely the right approach is to create one gateway client connection (per client identity) and use that gateway instance for multiple transaction invocations. A gateway instance can be used to invoke many transactions in parallel as well as sequentially. Glad you got it working OK.


Re: connection is in TRANSIENT_FAILURE #fabric

neeroz.kumar29@...
 

yea like you mentioned i am using InvokeChaincode on the chaincode stub to evaluate transactions on other chaincode. 

i might have figured out the issue in my use-case. i am creating 1000 connections for each transaction which are basically doing the same doing, just different args. i dug deeper in to this issue and found out that it has something to do with "connection caching" and every time you close the connection then releasing the resources and the cleaning up takes a while and due to caching the same connection is issued again which causes the TRANSIENT_FAILURE. 

the solution i applied to this was - use a single client connection to process all those transactions. so i for this i changed my application code to accept batches of transactions and `SubmitTransaction` using a single client connection (gateway client).

i apologise if i could not explain it well.


Re: connection is in TRANSIENT_FAILURE #fabric

Mark Lewis
 

Are you saying that, within your smart contract transaction function, you are using the client SDK to evaluate other transaction functions? If so, that really is not something you should be doing.

If you have some code you want to execute as part of two different transaction functions within the same chaincode, probably you should just structure the code there to allow it to be invoked as a local method/function from both transaction functions. If you want to invoke a transaction function from another chaincode from within a transaction function, probably you should be using InvokeChaincode on the chaincode stub.


Re: Distributed storage cluster using Hadoop or Spark

Nikos Karamolegkos <nkaram@...>
 

I would like to store data to the one channel BC network database. These data would be generated quite often so I need big amount of storage. Also, I would like to retrieve these data from the BC (query one or more peers) in large batches in order to use it machine learning algorithms. So I am thinking to use Hadoop or Spark with the hyperledger fabric. However, I am not sure how good idea is that and how difficult is to implement.


Re: Distributed storage cluster using Hadoop or Spark

Artem Barger
 

Do you have a specific use case in mind? Are you suggesting utilizing HDFS to manage world state? Or compute to improve chaincode computation?

Best regards,
                      Artem Barger.


On Tue, Jun 22, 2021 at 2:29 PM Nikos Karamolegkos <nkaram@...> wrote:
Hello, does anyone use distributed storage cluster like Hadoop or Spark
for scaling? Is this really possible with hyperledger fabric?

Thank you,







Distributed storage cluster using Hadoop or Spark

Nikos Karamolegkos <nkaram@...>
 

Hello, does anyone use distributed storage cluster like Hadoop or Spark for scaling? Is this really possible with hyperledger fabric?

Thank you,


Cross chaincode invoke, ACL policy problem (MSP not defined in called channel, but in caller) #fabric-chaincode

Sergio C
 

Hi,
Is it possible to cross-invoke a chaincode in other channel (channel-1) (using an identity that isn't part of the channel) from a chaincode in the channel-2 (identity is declared here)?

Example:
Org3 and Org4 are part of channel-2.
Org1, Org2 and Org3 are part of channel-1.
Org3 has chaincodeA (instantiated in channel-1) and chaincodeB (instantiated in channel-2) installed.

Org4.member invokes (query) chaincodeB (in Org3Peer) in channel-2 (ok, because Org4 is in channel-2), then chaincodeB internally invokes (query) chaincodeA (fails, because Org4Member MSP isn't defined in channel-1).

Is it possible to bypass this problem?
Any help appreciated.


connection is in TRANSIENT_FAILURE #fabric

neeroz.kumar29@...
 

I am maintaining 1000 concurrent connections (SubmitTransactions) which invokes the smart-contract.

The smart contract does the following operations:
1. "EvaluateTransaction" - on another chaincode
2. "EvaluateTransaction" - on the same chaincode as the "SubmitTransaction" is called
3. "PutState" - on the same chaincode as the "SubmitTransaction" is called

I am processing around 10,000 records while maintaining a concurrency of 1000 transactions (1000 SubmitTransactions which is 1000 connections). All these transactions are independent of each other, that is none of these transactions are reading or writing the same key.

It is working for few seconds, and after that one or more peer goes down! and sometimes all the peer goes down as well and the processing stops altogether

Error: dynamicdiscovery.(*ChannelService).evaluate -> WARN error from discovery request [grpcs://localhost:7051]: From target: grpcs://localhost:7051: could not connect to grpcs://localhost:7051: dialing connection on target [localhost:7051]: connection is in TRANSIENT_FAILURE

My fabric network v2.3 consists of:
- 2 orderer nodes
- 3 peer nodes (couchDB)

I reduced the concurrency up to 100, and i still get the same error.

1041 - 1060 of 11120