Date   

Re: Inconsistent Private Data Collection Write operation

Hritik Gupta <hritikgupta9@...>
 

I ran the network again, to double check. The inconsistency issue still exists. The issue occurs even when I pass OR("Org1MSP.Member") as the EP and try to update the key via Org1 client (which is an authorized peer).



On Fri, Jun 19, 2020 at 5:31 PM Cendhu U <cendhu@...> wrote:
but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

the above error seems to match your old definition -- "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

Anyway, I do not have expertise in this. If Yacov looks at this message, he might clarify.

Regards,
Senthil

On Fri, Jun 19, 2020 at 5:21 PM Hritik Gupta <hritikgupta9@...> wrote:
Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

"endorsementPolicy": {
      "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
    }

Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 



-- 


Re: Creation of couchdb index on an already running network #couchdb #fabric-chaincode

Senthil Nathan
 

Do you have the peer logs?

Regards,
Senthil


On Fri, Jun 19, 2020 at 5:22 PM Joao Antunes <joao.antunes@...> wrote:

Hi,

I currently trying to add a new index to my 1.4 network.

I have the folders correctly set too. When  I update the chaincode with the new index folder I can see that my peers have the package correctly but I can't see the index in the couchdb of my peer. Am I missing something?

Thank you.


Re: Inconsistent Private Data Collection Write operation

Senthil Nathan
 

but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

the above error seems to match your old definition -- "endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

Anyway, I do not have expertise in this. If Yacov looks at this message, he might clarify.

Regards,
Senthil

On Fri, Jun 19, 2020 at 5:21 PM Hritik Gupta <hritikgupta9@...> wrote:
Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

"endorsementPolicy": {
      "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
    }

Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 


Creation of couchdb index on an already running network #couchdb #fabric-chaincode

Joao Antunes
 

Hi,

I currently trying to add a new index to my 1.4 network.

I have the folders correctly set too. When  I update the chaincode with the new index folder I can see that my peers have the package correctly but I can't see the index in the couchdb of my peer. Am I missing something?

Thank you.


Re: Inconsistent Private Data Collection Write operation

Hritik Gupta <hritikgupta9@...>
 

Sorry for typing it wrong. The endorsement policy in my collections config is as follows:

"endorsementPolicy": {
      "signaturePolicy": "OR('Org1MSP.member', 'Org2MSP.member')"
    }

Setting memberOnlyWrite to false is deliberate because I want the peers of only Org2 and Org1 to be able to write to the private data collection; and not any other Org that might join the network in future. 


Re: Inconsistent Private Data Collection Write operation

Senthil Nathan
 

1) an authorized peer
2) an unauthorized peer after making the following change to the collection endorsement policy:
"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" }

  1. As the memberOnlyWrite is set to false by default (not configured in the collection config as well), all peers are authorized to write if it can satisfy the endorsement policy.
  2. Is the endorsement policy syntax correct for your scenario? Shouldn't it be "OR('Org1MSP.member', 'Org2MSP.member')" if you want to allow either of the organization to write to the collection?
Regards,
Senthil


On Fri, Jun 19, 2020 at 3:54 PM Hritik Gupta <hritikgupta9@...> wrote:
Hi all,

I have a fabric 2.0 network with 2 orgs having 1 peer each (similar to test-network). I implemented a private data collection that persists in the off-chain DB of the peer of one of the orgs (Org1, as shown below)
"name": "collectionMarblePrivateDetails", 
"policy": "OR('Org1MSP.member')", 
"requiredPeerCount": 0, 
"maxPeerCount": 3, 
"blockToLive": 0, 
"memberOnlyRead": true, 
"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member')" } 
}
I tried to change the value of the key in the private collection by invoking the relevant chaincode method from : (I'm using the submitTransaction method from the Contract class of fabric node-sdk) 
1) an authorized peer
2) an unauthorized peer after making the following change to the collection endorsement policy:
"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" } 

Although the above 2 operations work sometimes, but very inconsistently. It fails by giving the following endorsement policy error:

2020-06-19T10:01:19.660Z - warn: [TransactionEventHandler]: strategyFail: commit failure for transaction "8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d": Error: Commit of transaction 8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d failed on peer peer0.org1.example.com:7051 with status ENDORSEMENT_POLICY_FAILURE
Failed to submit transaction: Error: Commit of transaction 8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d failed on peer peer0.org1.example.com:7051 with status ENDORSEMENT_POLICY_FAILURE

The logs from the peer docker containers are as follows:
2020-06-19 10:01:14.584 UTC [committer.txvalidator] validateTx -> ERRO 0f2 Dispatch for transaction txId = 16c33a328f40e579577c1767b0b55054cf903c328cf1a236f1da0aafb1762099 returned error: validation of endorsement policy for chaincode fabcar in tx 15:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

Although, aforementioned methods are able to change the key-value in the private collection if executed successfully, it fails a lot many number of times too. I'm not sure why that happens.
Appreciate any help!

Thanks and regards.


--

Hritik Gupta

B.Tech, Computer Science and Engineering

Indian Institute of Technology

Mandi, Himachal Pradesh – 175005

+91-7838869482 | b16097@...



Re: How to verify RAFT health in HLF version 1.4.4 #raft

Senthil Nathan
 

On Fri, Jun 19, 2020 at 9:38 AM shrugupt via lists.hyperledger.org <shrugupt=microsoft.com@...> wrote:
Hi,

Would really appreciate if anybody has any thought/suggestion regarding my query.

Thanks,
Shruti



Inconsistent Private Data Collection Write operation

Hritik Gupta <hritikgupta9@...>
 

Hi all,

I have a fabric 2.0 network with 2 orgs having 1 peer each (similar to test-network). I implemented a private data collection that persists in the off-chain DB of the peer of one of the orgs (Org1, as shown below)
"name": "collectionMarblePrivateDetails", 
"policy": "OR('Org1MSP.member')", 
"requiredPeerCount": 0, 
"maxPeerCount": 3, 
"blockToLive": 0, 
"memberOnlyRead": true, 
"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member')" } 
}
I tried to change the value of the key in the private collection by invoking the relevant chaincode method from : (I'm using the submitTransaction method from the Contract class of fabric node-sdk) 
1) an authorized peer
2) an unauthorized peer after making the following change to the collection endorsement policy:
"endorsementPolicy": { "signaturePolicy": "OR('Org1MSP.member'), OR('Org2MSP.member')" } 

Although the above 2 operations work sometimes, but very inconsistently. It fails by giving the following endorsement policy error:

2020-06-19T10:01:19.660Z - warn: [TransactionEventHandler]: strategyFail: commit failure for transaction "8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d": Error: Commit of transaction 8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d failed on peer peer0.org1.example.com:7051 with status ENDORSEMENT_POLICY_FAILURE
Failed to submit transaction: Error: Commit of transaction 8a06bccc8afd555efb4c1a70ac0ce1905c17f135061ca1ec525d760c0101118d failed on peer peer0.org1.example.com:7051 with status ENDORSEMENT_POLICY_FAILURE

The logs from the peer docker containers are as follows:
2020-06-19 10:01:14.584 UTC [committer.txvalidator] validateTx -> ERRO 0f2 Dispatch for transaction txId = 16c33a328f40e579577c1767b0b55054cf903c328cf1a236f1da0aafb1762099 returned error: validation of endorsement policy for chaincode fabcar in tx 15:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Endorsement' sub-policies to be satisfied

Although, aforementioned methods are able to change the key-value in the private collection if executed successfully, it fails a lot many number of times too. I'm not sure why that happens.
Appreciate any help!

Thanks and regards.


--

Hritik Gupta

B.Tech, Computer Science and Engineering

Indian Institute of Technology

Mandi, Himachal Pradesh – 175005

+91-7838869482 | b16097@...



Hyperledger Fabric Documentation Workgroup call - Eastern hemisphere - Fri, 06/19/2020 #cal-notice

fabric@lists.hyperledger.org Calendar <noreply@...>
 

Hyperledger Fabric Documentation Workgroup call - Eastern hemisphere

When:
Friday, 19 June 2020
6:00am to 7:00am
(GMT+01:00) Europe/London

Where:
https://zoom.us/j/6223336701

Organizer:
a_o-dowd@... +441962816761

Description:
Documentation workgroup call.
Agenda, minutes and recordings: https://wiki.hyperledger.org/display/fabric/Documentation+Working+Group


Re: How to verify RAFT health in HLF version 1.4.4 #raft

shrugupt@...
 

Hi,

Would really appreciate if anybody has any thought/suggestion regarding my query.

Thanks,
Shruti



Documentation Workgroup: Agenda for Friday, 19 June

Anthony O'Dowd <a_o-dowd@...>
 

Hi All,

We will hold the documentation workgroup calls this Friday -- with both an Eastern hemisphere and Western hemisphere call. Please feel free to come along, you're always very welcome.

You can read about last week's calls at https://wiki.hyperledger.org/display/fabric/2020+06+12+DWG+Agenda You'll see significant minutes for both the Eastern and Western hemisphere calls, and recordings for both sessions. Our Eastern and Western hemisphere calls are very well attended at the moment -- thanks to all for your contributions and collaboration.

Our Eastern hemisphere had excellent contributions from the Japanese and Malayalam working group teams.  We reviewed the recent updated documentation and discussed other additions to help new language translators get started. Tomorrow we will review the new i18n repository structure and how to use it;  it will allow us to converge many of our international language repositories.

Our Western hemisphere call included the now regular V2.x status update from Pam and Joe on new content; keep up to date with these as we approach 2.2! There was an interesting discussion on the potential value of a forward looking Fabric architecture document, and a comprehensive update on Brazilian Portuguese language translation from Renato, who is ready to start to make content live to help boost wider collaboration.  David also shared some great interest in many other language translations that are starting: Arabic, Urdu, French, Spanish and more. Pam and Renato also discussed some recent pipfile updates; a small but important topic is you're building the documentation on Ubuntu.


You can catch up with the full recordings and other sessions: https://wiki.hyperledger.org/display/fabric/Recordings

See https://wiki.hyperledger.org/display/fabric/2020+06+19+DWG+Agenda for this week's agenda. Again, on the Eastern hemisphere call Aneena and her team will share progress including RTD builds, and PR mergin in the ML doc repo. On the Western hemisphere call, we also have a promising agenda.

Please feel free to contribute using the wiki, including helping to build next week's agenda: https://wiki.hyperledger.org/display/fabric/2020+06+26+DWG+Agenda

Thanks!

Pam, Anthony,  Joe, Nik

Meeting Details
-------------
Please use the following link to attend the meeting:  https://zoom.us/j/6223336701

The meeting times are as follows: https://wiki.hyperledger.org/display/fabric/Documentation+Working+Group

Meeting 133A: Friday 19 June
                   1130 India Standard Time
                   1400 China Standard Time
                   1500 Japan Standard Time
                   1700 Australia Eastern Time
                   1400 Singapore Time
                   1100 Gulf Standard Time
                   1000 Moscow Standard Time
                   0700 Greenwich Mean Time
                   0800 Central European Time    

Meeting 133B: Friday 19 June
              1100 Central Daylight Time
                   1200 Eastern Daylight Time
                   0900 Pacific Daylight Time
                   1400 Brasil Time (BRT)
                   1600 Greenwich Mean Time
                   1700 Central European Time
                   1800 Moscow Standard Time

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Re: Replacing CouchDB with a custom database

Senthil Nathan
 

  1. If you need to expose all features of a database, changes are needed at the database as well as peer code.
  2. if you need to plugin a database that is compatible with what currently Fabric offers, then you need to fork Fabric and implement the VersionedDB for the new database. 
I don't know why someone would want 2 when it does not provide any new feature. Whatever features that we provide with leveldb or couchDB is not going to be changed by plugging in an alternative DB (with 2). Only the query syntax would change (which does not justify the need for fork and other pains that would give). 

Regards,
Senthil


On Thu, Jun 18, 2020 at 1:17 PM <chintanr97@...> wrote:
Given that, I might be using cloud services to deploy custom database, let's say like SQL DB server, then I do not have the option to alter the databases, right? Plus, the fabric compatibility with the database should go with the master branch of the DB, else otherwise the changes in the DB versions will be difficult to back port to Fabric custom version of the DB. 

But let's say, if there are not much requirement changes on the database side, then possibly the migration would become much of modifying the Fabric code base alone, right? Like, only implementing the design in terms "Ledger changes", "Implementing the required interfaces", and "Few configuration updates with testing code". 

In the above JIRAs, I can conclude the requirements needed on the database side. Is there an understanding, which suggests the APIs that will be needed to be modified between the peer and DB layer, to accommodate a custom DB altogether?


Re: Replacing CouchDB with a custom database

chintanr97@...
 

Given that, I might be using cloud services to deploy custom database, let's say like SQL DB server, then I do not have the option to alter the databases, right? Plus, the fabric compatibility with the database should go with the master branch of the DB, else otherwise the changes in the DB versions will be difficult to back port to Fabric custom version of the DB. 

But let's say, if there are not much requirement changes on the database side, then possibly the migration would become much of modifying the Fabric code base alone, right? Like, only implementing the design in terms "Ledger changes", "Implementing the required interfaces", and "Few configuration updates with testing code". 

In the above JIRAs, I can conclude the requirements needed on the database side. Is there an understanding, which suggests the APIs that will be needed to be modified between the peer and DB layer, to accommodate a custom DB altogether?


Re: Replacing CouchDB with a custom database

Senthil Nathan
 

Yes, it is not in the roadmap (may change in the future if some drawbacks with go-plugins are addressed by the go team)

Yes. Let's say chaincode exposes Put() to update a kv pair. Based on this assumption, we do simulation, validation, and commit. If we plugin a new DB and try to pass an update query (SQL/MongoDB) via ExecuteQuery(), we need a lot of changes (both to Fabric and sometimes to the DB internals). An effort to plugin PostgreSQL was initiated 2 years ago. You can find the deck at https://jira.hyperledger.org/browse/FAB-8031 to understand what it takes to plugin a database and use all its futures. 

Regards,
Senthil


On Thu, Jun 18, 2020 at 12:22 PM <chintanr97@...> wrote:

As Gari suggests here, in the JIRA link you mentioned:

 

Go plugins have proven to be very problematic ... we are moving away from Go plugins as external plug points.
We could consider using a plugin architecture assuming we actually build the plugins with the Fabric tree as part of a release. But building and maintaining them outside the tree is no longer the direction we are going.


I was just trying to confirm, that building the plugins as native to Fabric tree in not currently under the timeline, right?

Let's say if I wish to deploy Mongo DB, then what does the following statement imply?

Note that it is not like if we plugin a MySQL/PostgreSQL/MongoDB, we can start using all its features. There are more to it.
Do you mean that we can only use the features that are parallely implemented by the chaincode shim API? 


Re: Replacing CouchDB with a custom database

chintanr97@...
 

As Gari suggests here, in the JIRA link you mentioned:

 

Go plugins have proven to be very problematic ... we are moving away from Go plugins as external plug points.
We could consider using a plugin architecture assuming we actually build the plugins with the Fabric tree as part of a release. But building and maintaining them outside the tree is no longer the direction we are going.


I was just trying to confirm, that building the plugins as native to Fabric tree in not currently under the timeline, right?

Let's say if I wish to deploy Mongo DB, then what does the following statement imply?

Note that it is not like if we plugin a MySQL/PostgreSQL/MongoDB, we can start using all its features. There are more to it.
Do you mean that we can only use the features that are parallely implemented by the chaincode shim API? 


Re: Replacing CouchDB with a custom database

Senthil Nathan
 

As discussed/decided in https://jira.hyperledger.org/browse/FAB-8665, pluggable stateDB is not going to be supported anytime soon. 

If you are okay to fork the Fabric, you can find a high level implementation details at https://docs.google.com/document/d/1ZdxPWdxUwEDwRAKY8tewgjqh2qTZCm6RZ6tU9ccm4dM/edit?usp=sharing but we wrote this document 2 years ago. The approach and technique would stay the same but the current VersionedDB structure has been extended with a bit more APIs https://github.com/hyperledger/fabric/blob/master/core/ledger/kvledger/txmgmt/statedb/statedb.go#L29

What database do you want to plugin? why? Note that it is not like if we plugin a MySQL/PostgreSQL/MongoDB, we can start using all its features. There are more to it.

Regards,
Senthil


On Thu, Jun 18, 2020 at 11:47 AM <chintanr97@...> wrote:
I learnt that initially there was a plan to implement the Go plugin architecture last year to provide pluggability of state databases in Fabric. However, looking at one of the JIRAs, I found that this implementation was dropped as well. Going ahead, in the same JIRA it was suggested that there are plans to provide true pluggability within the Fabric tree, like exposing interfaces which can be implemented by the users for their custom databases. 

Currently, if we look at the implementation of CouchDB as a state database then it implements the VersionedDB and VersionedDBProvider interfaces of the Fabric "kvledger". Is this the idea that reflects the above plan of exposing the interfaces? 

What I precisely mean to identify is that, what all interfaces will be needed to be altered for supporting our own custom DB for the world state?

Is the Fabric planning to make releases which will allow customers to implement the pluggable interfaces, so that individual networks will not be creating forks in the core Fabric branch?


Replacing CouchDB with a custom database

chintanr97@...
 

I learnt that initially there was a plan to implement the Go plugin architecture last year to provide pluggability of state databases in Fabric. However, looking at one of the JIRAs, I found that this implementation was dropped as well. Going ahead, in the same JIRA it was suggested that there are plans to provide true pluggability within the Fabric tree, like exposing interfaces which can be implemented by the users for their custom databases. 

Currently, if we look at the implementation of CouchDB as a state database then it implements the VersionedDB and VersionedDBProvider interfaces of the Fabric "kvledger". Is this the idea that reflects the above plan of exposing the interfaces? 

What I precisely mean to identify is that, what all interfaces will be needed to be altered for supporting our own custom DB for the world state?

Is the Fabric planning to make releases which will allow customers to implement the pluggable interfaces, so that individual networks will not be creating forks in the core Fabric branch?


Upcoming Event: Hyperledger Fabric Documentation Workgroup call - Eastern hemisphere - Fri, 06/19/2020 6:00am-7:00am #cal-reminder

fabric@lists.hyperledger.org Calendar <fabric@...>
 

Reminder: Hyperledger Fabric Documentation Workgroup call - Eastern hemisphere

When: Friday, 19 June 2020, 6:00am to 7:00am, (GMT+01:00) Europe/London

Where:https://zoom.us/j/6223336701

View Event

Organizer: Anthony O'Dowd a_o-dowd@... +441962816761

Description: Documentation workgroup call.
Agenda, minutes and recordings: https://wiki.hyperledger.org/display/fabric/Documentation+Working+Group


Re: Validating transactions including rich queries

Senthil Nathan
 

    However, we don't re-execute the couchdb's rich query on both in-memory and CouchDB to ensure serializable isolation. It is certainly possible to achieve serializability with rich queries. A very simple way to achieve that is -- after validating a transaction, instead of keeping the update batch in-memory, commit the updates to CouchDB and re-run CouchDB queries to validate. There are ways to optimize it as well.

A small clarification on this statement. We still validate all reads made by the CouchDB rich query to ensure repeatable read isolation. If any read version is not the same as the committed version even for the rich query, we abort the transaction. For performance reasons, we do not repeat the query execution to find phantom-reads.

Regards,
Senthil

On Thu, Jun 18, 2020 at 6:50 AM Senthil Nathan via lists.hyperledger.org <cendhu=gmail.com@...> wrote:

    As per the documentation, I understood that the support for SQL is very limited in the forked/proprietary oracle's Fabric (I believe it is not open-sourced). In other words, whatever we can do with CouchDB selector queries, only the same thing can be done using SQL. It is just the language preference (not exactly a feature improvement as far as I understood). I also want to point out that the Hyperledger Composer in fact supported SQL and converted the query to CouchDB rich query. It is just a matter of moving code from one repo to another to support SQL queries in the chaincode.

   On the serializable isolation with respect to richQueries, it is certainly possible to provide serializability with rich queries too in the officially supported Fabric version. The only reason that we didn't provide is the performance. Let me dive into a bit more implementation details to explain that.

   When the peer receives a block, it performs MVCC (part of optimistic concurrency control -- https://dl.acm.org/doi/10.1145/319566.319567)
  1. For each transaction, it executes the MVCC check serially. 
  2. Say that the transaction T in the block is validated. Instead of committing it immediately, we do keep the updates in an in-memory batch.
  3. When the transaction T+1 is being validated, it would consider states present in both the in-memory batch and disk. Even the re-executed range queries would consider that. Once it is found to be valid, the updates are added to the in-memory batch. 
    However, we don't re-execute the couchdb's rich query on both in-memory and CouchDB to ensure serializable isolation. It is certainly possible to achieve serializability with rich queries. A very simple way to achieve that is -- after validating a transaction, instead of keeping the update batch in-memory, commit the updates to CouchDB and re-run CouchDB queries to validate. There are ways to optimize it as well.

    Only for the performance reasons, we are not re-executing queries. If there are a few use-cases that require it, we can certainly make this configurable in Fabric and provide serializable isolation for all kinds of queries.

Regards,
Senthil 

On Thu, Jun 18, 2020 at 1:24 AM Yacov Manevich <YACOVM@...> wrote:
Hi Mark.

Do you mind explaining how you ensure serializability?

Do you record a digest of the results and then re-execute it at commit and compare or something else?

And how does that work with mixed networks where you can't do custom validation checks?

- Yacov



From:        "Mark Rakhmilevich" <mark.rakhmilevich@...>
To:        David Enyeart <enyeart@...>
Cc:        Senthil Nathan <cendhu@...>, fabric <fabric@...>, "Mr.Phuwanai Thummavet" <mr.thummavet@...>
Date:        06/17/2020 10:35 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by:        fabric@...




Oracle’s blockchain is fully compatible with other Fabric nodes and in fact being used in some mixed networks.  The state DB changes are under standard Fabric APIs.  If customers need to use rich data queries in chaincodes that run across Oracle and non-Oracle nodes, Couch DB syntax is supported.
 
Regards,
    Mark
  
 
From: David Enyeart <enyeart@...>
Sent:
Wednesday, June 17, 2020 12:30 PM
To:
Mark Rakhmilevich <mark.rakhmilevich@...>
Cc:
Senthil Nathan <cendhu@...>; fabric <fabric@...>; Mr.Phuwanai Thummavet <mr.thummavet@...>
Subject:
Re: [Hyperledger Fabric] Validating transactions including rich queries

 

If I understand this correctly, doesn't running a private fork of Fabric imply that Oracle nodes cannot be mixed with nodes from other clouds that support the open source Fabric?


Dave Enyeart


"Mark Rakhmilevich" ---06/17/2020 02:37:58 PM---As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Be

From:
"Mark Rakhmilevich" <mark.rakhmilevich@...>
To:
Senthil Nathan <cendhu@...>, David Enyeart <enyeart@...>
Cc:
"Mr.Phuwanai Thummavet" <mr.thummavet@...>, fabric <fabric@...>
Date:
06/17/2020 02:37 PM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by:
fabric@...






As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Berkeley DB as a replacement for Level DB and Couch DB, with full support for rich data queries and GetQueryResult validation at commit time to ensure the query result set is stable between endorsement and commit.  It’s available in the Cloud BaaS version as well as on-premises Enterprise  Edition.  


There’s also SQLite layer on top, enabling chaincodes to use SQL Selects for queries as well CouchDB query language for compatibility.  This solves a number of development and data consistency challenges in our enterprise deployments, and using row-level locking capabilities of Berkeley DB supports higher concurrency and throughput.  


Regards,
  Mark


From:
Senthil Nathan <
cendhu@...>
Sent:
Wednesday, June 17, 2020 5:58 AM
To:
David Enyeart <
enyeart@...>
Cc:
Mr.Phuwanai Thummavet <
mr.thummavet@...>; fabric <fabric@...>
Subject:
Re: [Hyperledger Fabric] Validating transactions including rich queries


I will add a few more things on top of what Dave said.

Yes, it is dependent on the application requirements.

1.        When a chaincode doesn't use rich queries, it archives serializable isolation.
2.        When a chaincode uses a rich query, it achieves only repeatable read.
Please read more about this here https://github.com/hyperledger/fabric-chaincode-go/blob/master/shim/interfaces.go#L187-L202(document well).

To understand various isolation levels, please refer to
Most databases support serializable and repeatable read isolations. In fact, the default setting is repeatable isolation in most databases. There are many use-cases that do use repeatable isolation (not carrying about the phantom reads).

Regards,
Senthil

On Wed, Jun 17, 2020 at 6:10 PM David Enyeart <
enyeart@...> wrote:

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.



Dave Enyeart


"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From:
"Mr.Phuwanai Thummavet" <mr.thummavet@...>
To:
fabric <fabric@...>
Date:
06/17/2020 01:59 AM
Subject:
[EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by:
fabric@...







Dear all,

I read the following article:
https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRangethe GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer






Re: Validating transactions including rich queries

Senthil Nathan
 


    As per the documentation, I understood that the support for SQL is very limited in the forked/proprietary oracle's Fabric (I believe it is not open-sourced). In other words, whatever we can do with CouchDB selector queries, only the same thing can be done using SQL. It is just the language preference (not exactly a feature improvement as far as I understood). I also want to point out that the Hyperledger Composer in fact supported SQL and converted the query to CouchDB rich query. It is just a matter of moving code from one repo to another to support SQL queries in the chaincode.

   On the serializable isolation with respect to richQueries, it is certainly possible to provide serializability with rich queries too in the officially supported Fabric version. The only reason that we didn't provide is the performance. Let me dive into a bit more implementation details to explain that.

   When the peer receives a block, it performs MVCC (part of optimistic concurrency control -- https://dl.acm.org/doi/10.1145/319566.319567)
  1. For each transaction, it executes the MVCC check serially. 
  2. Say that the transaction T in the block is validated. Instead of committing it immediately, we do keep the updates in an in-memory batch.
  3. When the transaction T+1 is being validated, it would consider states present in both the in-memory batch and disk. Even the re-executed range queries would consider that. Once it is found to be valid, the updates are added to the in-memory batch. 
    However, we don't re-execute the couchdb's rich query on both in-memory and CouchDB to ensure serializable isolation. It is certainly possible to achieve serializability with rich queries. A very simple way to achieve that is -- after validating a transaction, instead of keeping the update batch in-memory, commit the updates to CouchDB and re-run CouchDB queries to validate. There are ways to optimize it as well.

    Only for the performance reasons, we are not re-executing queries. If there are a few use-cases that require it, we can certainly make this configurable in Fabric and provide serializable isolation for all kinds of queries.

Regards,
Senthil 

On Thu, Jun 18, 2020 at 1:24 AM Yacov Manevich <YACOVM@...> wrote:
Hi Mark.

Do you mind explaining how you ensure serializability?

Do you record a digest of the results and then re-execute it at commit and compare or something else?

And how does that work with mixed networks where you can't do custom validation checks?

- Yacov



From:        "Mark Rakhmilevich" <mark.rakhmilevich@...>
To:        David Enyeart <enyeart@...>
Cc:        Senthil Nathan <cendhu@...>, fabric <fabric@...>, "Mr.Phuwanai Thummavet" <mr.thummavet@...>
Date:        06/17/2020 10:35 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by:        fabric@...




Oracle’s blockchain is fully compatible with other Fabric nodes and in fact being used in some mixed networks.  The state DB changes are under standard Fabric APIs.  If customers need to use rich data queries in chaincodes that run across Oracle and non-Oracle nodes, Couch DB syntax is supported.
 
Regards,
    Mark
  
 
From: David Enyeart <enyeart@...>
Sent:
Wednesday, June 17, 2020 12:30 PM
To:
Mark Rakhmilevich <mark.rakhmilevich@...>
Cc:
Senthil Nathan <cendhu@...>; fabric <fabric@...>; Mr.Phuwanai Thummavet <mr.thummavet@...>
Subject:
Re: [Hyperledger Fabric] Validating transactions including rich queries

 

If I understand this correctly, doesn't running a private fork of Fabric imply that Oracle nodes cannot be mixed with nodes from other clouds that support the open source Fabric?


Dave Enyeart


"Mark Rakhmilevich" ---06/17/2020 02:37:58 PM---As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Be

From:
"Mark Rakhmilevich" <mark.rakhmilevich@...>
To:
Senthil Nathan <cendhu@...>, David Enyeart <enyeart@...>
Cc:
"Mr.Phuwanai Thummavet" <mr.thummavet@...>, fabric <fabric@...>
Date:
06/17/2020 02:37 PM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by:
fabric@...






As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Berkeley DB as a replacement for Level DB and Couch DB, with full support for rich data queries and GetQueryResult validation at commit time to ensure the query result set is stable between endorsement and commit.  It’s available in the Cloud BaaS version as well as on-premises Enterprise  Edition.  


There’s also SQLite layer on top, enabling chaincodes to use SQL Selects for queries as well CouchDB query language for compatibility.  This solves a number of development and data consistency challenges in our enterprise deployments, and using row-level locking capabilities of Berkeley DB supports higher concurrency and throughput.  


Regards,
  Mark


From:
Senthil Nathan <
cendhu@...>
Sent:
Wednesday, June 17, 2020 5:58 AM
To:
David Enyeart <
enyeart@...>
Cc:
Mr.Phuwanai Thummavet <
mr.thummavet@...>; fabric <fabric@...>
Subject:
Re: [Hyperledger Fabric] Validating transactions including rich queries


I will add a few more things on top of what Dave said.

Yes, it is dependent on the application requirements.

1.        When a chaincode doesn't use rich queries, it archives serializable isolation.
2.        When a chaincode uses a rich query, it achieves only repeatable read.
Please read more about this here https://github.com/hyperledger/fabric-chaincode-go/blob/master/shim/interfaces.go#L187-L202(document well).

To understand various isolation levels, please refer to
Most databases support serializable and repeatable read isolations. In fact, the default setting is repeatable isolation in most databases. There are many use-cases that do use repeatable isolation (not carrying about the phantom reads).

Regards,
Senthil

On Wed, Jun 17, 2020 at 6:10 PM David Enyeart <
enyeart@...> wrote:

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.



Dave Enyeart


"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From:
"Mr.Phuwanai Thummavet" <mr.thummavet@...>
To:
fabric <fabric@...>
Date:
06/17/2020 01:59 AM
Subject:
[EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by:
fabric@...







Dear all,

I read the following article:
https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRangethe GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer





3021 - 3040 of 11527