1. #raft
  2. Topics

Validating transactions including rich queries

Senthil Nathan
 

    However, we don't re-execute the couchdb's rich query on both in-memory and CouchDB to ensure serializable isolation. It is certainly possible to achieve serializability with rich queries. A very simple way to achieve that is -- after validating a transaction, instead of keeping the update batch in-memory, commit the updates to CouchDB and re-run CouchDB queries to validate. There are ways to optimize it as well.

A small clarification on this statement. We still validate all reads made by the CouchDB rich query to ensure repeatable read isolation. If any read version is not the same as the committed version even for the rich query, we abort the transaction. For performance reasons, we do not repeat the query execution to find phantom-reads.

Regards,
Senthil

On Thu, Jun 18, 2020 at 6:50 AM Senthil Nathan via lists.hyperledger.org <cendhu=gmail.com@...> wrote:

    As per the documentation, I understood that the support for SQL is very limited in the forked/proprietary oracle's Fabric (I believe it is not open-sourced). In other words, whatever we can do with CouchDB selector queries, only the same thing can be done using SQL. It is just the language preference (not exactly a feature improvement as far as I understood). I also want to point out that the Hyperledger Composer in fact supported SQL and converted the query to CouchDB rich query. It is just a matter of moving code from one repo to another to support SQL queries in the chaincode.

   On the serializable isolation with respect to richQueries, it is certainly possible to provide serializability with rich queries too in the officially supported Fabric version. The only reason that we didn't provide is the performance. Let me dive into a bit more implementation details to explain that.

   When the peer receives a block, it performs MVCC (part of optimistic concurrency control -- https://dl.acm.org/doi/10.1145/319566.319567)
  1. For each transaction, it executes the MVCC check serially. 
  2. Say that the transaction T in the block is validated. Instead of committing it immediately, we do keep the updates in an in-memory batch.
  3. When the transaction T+1 is being validated, it would consider states present in both the in-memory batch and disk. Even the re-executed range queries would consider that. Once it is found to be valid, the updates are added to the in-memory batch. 
    However, we don't re-execute the couchdb's rich query on both in-memory and CouchDB to ensure serializable isolation. It is certainly possible to achieve serializability with rich queries. A very simple way to achieve that is -- after validating a transaction, instead of keeping the update batch in-memory, commit the updates to CouchDB and re-run CouchDB queries to validate. There are ways to optimize it as well.

    Only for the performance reasons, we are not re-executing queries. If there are a few use-cases that require it, we can certainly make this configurable in Fabric and provide serializable isolation for all kinds of queries.

Regards,
Senthil 

On Thu, Jun 18, 2020 at 1:24 AM Yacov Manevich <YACOVM@...> wrote:
Hi Mark.

Do you mind explaining how you ensure serializability?

Do you record a digest of the results and then re-execute it at commit and compare or something else?

And how does that work with mixed networks where you can't do custom validation checks?

- Yacov



From:        "Mark Rakhmilevich" <mark.rakhmilevich@...>
To:        David Enyeart <enyeart@...>
Cc:        Senthil Nathan <cendhu@...>, fabric <fabric@...>, "Mr.Phuwanai Thummavet" <mr.thummavet@...>
Date:        06/17/2020 10:35 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by:        fabric@...



Oracle’s blockchain is fully compatible with other Fabric nodes and in fact being used in some mixed networks.  The state DB changes are under standard Fabric APIs.  If customers need to use rich data queries in chaincodes that run across Oracle and non-Oracle nodes, Couch DB syntax is supported.
 
Regards,
    Mark
  
 
From: David Enyeart <enyeart@...>
Sent: Wednesday, June 17, 2020 12:30 PM
To: Mark Rakhmilevich <mark.rakhmilevich@...>
Cc: Senthil Nathan <cendhu@...>; fabric <fabric@...>; Mr.Phuwanai Thummavet <mr.thummavet@...>
Subject: Re: [Hyperledger Fabric] Validating transactions including rich queries
 

If I understand this correctly, doesn't running a private fork of Fabric imply that Oracle nodes cannot be mixed with nodes from other clouds that support the open source Fabric?


Dave Enyeart

"Mark Rakhmilevich" ---06/17/2020 02:37:58 PM---As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Be

From: "Mark Rakhmilevich" <mark.rakhmilevich@...>
To: Senthil Nathan <cendhu@...>, David Enyeart <enyeart@...>
Cc: "Mr.Phuwanai Thummavet" <mr.thummavet@...>, fabric <fabric@...>
Date: 06/17/2020 02:37 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...





As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Berkeley DB as a replacement for Level DB and Couch DB, with full support for rich data queries and GetQueryResult validation at commit time to ensure the query result set is stable between endorsement and commit.  It’s available in the Cloud BaaS version as well as on-premises Enterprise  Edition.  

There’s also SQLite layer on top, enabling chaincodes to use SQL Selects for queries as well CouchDB query language for compatibility.  This solves a number of development and data consistency challenges in our enterprise deployments, and using row-level locking capabilities of Berkeley DB supports higher concurrency and throughput.  

Regards,
  Mark

From: Senthil Nathan <cendhu@...>
Sent: Wednesday, June 17, 2020 5:58 AM
To: David Enyeart <enyeart@...>
Cc: Mr.Phuwanai Thummavet <mr.thummavet@...>; fabric <fabric@...>
Subject: Re: [Hyperledger Fabric] Validating transactions including rich queries

I will add a few more things on top of what Dave said.

Yes, it is dependent on the application requirements.
1.        When a chaincode doesn't use rich queries, it archives serializable isolation.
2.        When a chaincode uses a rich query, it achieves only repeatable read.
Please read more about this here https://github.com/hyperledger/fabric-chaincode-go/blob/master/shim/interfaces.go#L187-L202(document well).

To understand various isolation levels, please refer to Most databases support serializable and repeatable read isolations. In fact, the default setting is repeatable isolation in most databases. There are many use-cases that do use repeatable isolation (not carrying about the phantom reads).

Regards,
Senthil

On Wed, Jun 17, 2020 at 6:10 PM David Enyeart <enyeart@...> wrote:

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.


Dave Enyeart

"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From: "Mr.Phuwanai Thummavet" <mr.thummavet@...>
To: fabric <fabric@...>
Date: 06/17/2020 01:59 AM
Subject: [EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...






Dear all,

I read the following article: https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRangethe GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer





Senthil Nathan
 


    As per the documentation, I understood that the support for SQL is very limited in the forked/proprietary oracle's Fabric (I believe it is not open-sourced). In other words, whatever we can do with CouchDB selector queries, only the same thing can be done using SQL. It is just the language preference (not exactly a feature improvement as far as I understood). I also want to point out that the Hyperledger Composer in fact supported SQL and converted the query to CouchDB rich query. It is just a matter of moving code from one repo to another to support SQL queries in the chaincode.

   On the serializable isolation with respect to richQueries, it is certainly possible to provide serializability with rich queries too in the officially supported Fabric version. The only reason that we didn't provide is the performance. Let me dive into a bit more implementation details to explain that.

   When the peer receives a block, it performs MVCC (part of optimistic concurrency control -- https://dl.acm.org/doi/10.1145/319566.319567)
  1. For each transaction, it executes the MVCC check serially. 
  2. Say that the transaction T in the block is validated. Instead of committing it immediately, we do keep the updates in an in-memory batch.
  3. When the transaction T+1 is being validated, it would consider states present in both the in-memory batch and disk. Even the re-executed range queries would consider that. Once it is found to be valid, the updates are added to the in-memory batch. 
    However, we don't re-execute the couchdb's rich query on both in-memory and CouchDB to ensure serializable isolation. It is certainly possible to achieve serializability with rich queries. A very simple way to achieve that is -- after validating a transaction, instead of keeping the update batch in-memory, commit the updates to CouchDB and re-run CouchDB queries to validate. There are ways to optimize it as well.

    Only for the performance reasons, we are not re-executing queries. If there are a few use-cases that require it, we can certainly make this configurable in Fabric and provide serializable isolation for all kinds of queries.

Regards,
Senthil 

On Thu, Jun 18, 2020 at 1:24 AM Yacov Manevich <YACOVM@...> wrote:
Hi Mark.

Do you mind explaining how you ensure serializability?

Do you record a digest of the results and then re-execute it at commit and compare or something else?

And how does that work with mixed networks where you can't do custom validation checks?

- Yacov



From:        "Mark Rakhmilevich" <mark.rakhmilevich@...>
To:        David Enyeart <enyeart@...>
Cc:        Senthil Nathan <cendhu@...>, fabric <fabric@...>, "Mr.Phuwanai Thummavet" <mr.thummavet@...>
Date:        06/17/2020 10:35 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by:        fabric@...



Oracle’s blockchain is fully compatible with other Fabric nodes and in fact being used in some mixed networks.  The state DB changes are under standard Fabric APIs.  If customers need to use rich data queries in chaincodes that run across Oracle and non-Oracle nodes, Couch DB syntax is supported.
 
Regards,
    Mark
  
 
From: David Enyeart <enyeart@...>
Sent: Wednesday, June 17, 2020 12:30 PM
To: Mark Rakhmilevich <mark.rakhmilevich@...>
Cc: Senthil Nathan <cendhu@...>; fabric <fabric@...>; Mr.Phuwanai Thummavet <mr.thummavet@...>
Subject: Re: [Hyperledger Fabric] Validating transactions including rich queries
 

If I understand this correctly, doesn't running a private fork of Fabric imply that Oracle nodes cannot be mixed with nodes from other clouds that support the open source Fabric?


Dave Enyeart

"Mark Rakhmilevich" ---06/17/2020 02:37:58 PM---As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Be

From: "Mark Rakhmilevich" <mark.rakhmilevich@...>
To: Senthil Nathan <cendhu@...>, David Enyeart <enyeart@...>
Cc: "Mr.Phuwanai Thummavet" <mr.thummavet@...>, fabric <fabric@...>
Date: 06/17/2020 02:37 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...





As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Berkeley DB as a replacement for Level DB and Couch DB, with full support for rich data queries and GetQueryResult validation at commit time to ensure the query result set is stable between endorsement and commit.  It’s available in the Cloud BaaS version as well as on-premises Enterprise  Edition.  

There’s also SQLite layer on top, enabling chaincodes to use SQL Selects for queries as well CouchDB query language for compatibility.  This solves a number of development and data consistency challenges in our enterprise deployments, and using row-level locking capabilities of Berkeley DB supports higher concurrency and throughput.  

Regards,
  Mark

From: Senthil Nathan <cendhu@...>
Sent: Wednesday, June 17, 2020 5:58 AM
To: David Enyeart <enyeart@...>
Cc: Mr.Phuwanai Thummavet <mr.thummavet@...>; fabric <fabric@...>
Subject: Re: [Hyperledger Fabric] Validating transactions including rich queries

I will add a few more things on top of what Dave said.

Yes, it is dependent on the application requirements.
1.        When a chaincode doesn't use rich queries, it archives serializable isolation.
2.        When a chaincode uses a rich query, it achieves only repeatable read.
Please read more about this here https://github.com/hyperledger/fabric-chaincode-go/blob/master/shim/interfaces.go#L187-L202(document well).

To understand various isolation levels, please refer to Most databases support serializable and repeatable read isolations. In fact, the default setting is repeatable isolation in most databases. There are many use-cases that do use repeatable isolation (not carrying about the phantom reads).

Regards,
Senthil

On Wed, Jun 17, 2020 at 6:10 PM David Enyeart <enyeart@...> wrote:

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.


Dave Enyeart

"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From: "Mr.Phuwanai Thummavet" <mr.thummavet@...>
To: fabric <fabric@...>
Date: 06/17/2020 01:59 AM
Subject: [EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...






Dear all,

I read the following article: https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRangethe GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer





Yacov
 

Hi Mark.

Do you mind explaining how you ensure serializability?

Do you record a digest of the results and then re-execute it at commit and compare or something else?

And how does that work with mixed networks where you can't do custom validation checks?

- Yacov



From:        "Mark Rakhmilevich" <mark.rakhmilevich@...>
To:        David Enyeart <enyeart@...>
Cc:        Senthil Nathan <cendhu@...>, fabric <fabric@...>, "Mr.Phuwanai Thummavet" <mr.thummavet@...>
Date:        06/17/2020 10:35 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by:        fabric@...



Oracle’s blockchain is fully compatible with other Fabric nodes and in fact being used in some mixed networks.  The state DB changes are under standard Fabric APIs.  If customers need to use rich data queries in chaincodes that run across Oracle and non-Oracle nodes, Couch DB syntax is supported.
 
Regards,
    Mark
  
 

From: David Enyeart <enyeart@...>
Sent: Wednesday, June 17, 2020 12:30 PM
To: Mark Rakhmilevich <mark.rakhmilevich@...>
Cc: Senthil Nathan <cendhu@...>; fabric <fabric@...>; Mr.Phuwanai Thummavet <mr.thummavet@...>
Subject: Re: [Hyperledger Fabric] Validating transactions including rich queries
 

If I understand this correctly, doesn't running a private fork of Fabric imply that Oracle nodes cannot be mixed with nodes from other clouds that support the open source Fabric?


Dave Enyeart

"Mark Rakhmilevich" ---06/17/2020 02:37:58 PM---As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Be

From: "Mark Rakhmilevich" <mark.rakhmilevich@...>
To: Senthil Nathan <cendhu@...>, David Enyeart <enyeart@...>
Cc: "Mr.Phuwanai Thummavet" <mr.thummavet@...>, fabric <fabric@...>
Date: 06/17/2020 02:37 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...





As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Berkeley DB as a replacement for Level DB and Couch DB, with full support for rich data queries and GetQueryResult validation at commit time to ensure the query result set is stable between endorsement and commit.  It’s available in the Cloud BaaS version as well as on-premises Enterprise  Edition.  

There’s also SQLite layer on top, enabling chaincodes to use SQL Selects for queries as well CouchDB query language for compatibility.  This solves a number of development and data consistency challenges in our enterprise deployments, and using row-level locking capabilities of Berkeley DB supports higher concurrency and throughput.  

Regards,
  Mark

From: Senthil Nathan <cendhu@...>
Sent: Wednesday, June 17, 2020 5:58 AM
To: David Enyeart <enyeart@...>
Cc: Mr.Phuwanai Thummavet <mr.thummavet@...>; fabric <fabric@...>
Subject: Re: [Hyperledger Fabric] Validating transactions including rich queries

I will add a few more things on top of what Dave said.

Yes, it is dependent on the application requirements.
1.        When a chaincode doesn't use rich queries, it archives serializable isolation.
2.        When a chaincode uses a rich query, it achieves only repeatable read.
Please read more about this here https://github.com/hyperledger/fabric-chaincode-go/blob/master/shim/interfaces.go#L187-L202(document well).

To understand various isolation levels, please refer to Most databases support serializable and repeatable read isolations. In fact, the default setting is repeatable isolation in most databases. There are many use-cases that do use repeatable isolation (not carrying about the phantom reads).

Regards,
Senthil

On Wed, Jun 17, 2020 at 6:10 PM David Enyeart <enyeart@...> wrote:

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.


Dave Enyeart

"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From: "Mr.Phuwanai Thummavet" <mr.thummavet@...>
To: fabric <fabric@...>
Date: 06/17/2020 01:59 AM
Subject: [EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...






Dear all,

I read the following article: https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRangethe GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer





Mark Rakhmilevich
 

Oracle’s blockchain is fully compatible with other Fabric nodes and in fact being used in some mixed networks.  The state DB changes are under standard Fabric APIs.  If customers need to use rich data queries in chaincodes that run across Oracle and non-Oracle nodes, Couch DB syntax is supported.

 

Regards,

    Mark

  

 

From: David Enyeart <enyeart@...>
Sent: Wednesday, June 17, 2020 12:30 PM
To: Mark Rakhmilevich <mark.rakhmilevich@...>
Cc: Senthil Nathan <cendhu@...>; fabric <fabric@...>; Mr.Phuwanai Thummavet <mr.thummavet@...>
Subject: Re: [Hyperledger Fabric] Validating transactions including rich queries

 

If I understand this correctly, doesn't running a private fork of Fabric imply that Oracle nodes cannot be mixed with nodes from other clouds that support the open source Fabric?


Dave Enyeart

"Mark Rakhmilevich" ---06/17/2020 02:37:58 PM---As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Be

From: "Mark Rakhmilevich" <mark.rakhmilevich@...>
To: Senthil Nathan <cendhu@...>, David Enyeart <enyeart@...>
Cc: "Mr.Phuwanai Thummavet" <mr.thummavet@...>, fabric <fabric@...>
Date: 06/17/2020 02:37 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...




As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Berkeley DB as a replacement for Level DB and Couch DB, with full support for rich data queries and GetQueryResult validation at commit time to ensure the query result set is stable between endorsement and commit.  It’s available in the Cloud BaaS version as well as on-premises Enterprise  Edition. 

There’s also SQLite layer on top, enabling chaincodes to use SQL Selects for queries as well CouchDB query language for compatibility.  This solves a number of development and data consistency challenges in our enterprise deployments, and using row-level locking capabilities of Berkeley DB supports higher concurrency and throughput. 

Regards,
   Mark

From: Senthil Nathan <cendhu@...>
Sent: Wednesday, June 17, 2020 5:58 AM
To: David Enyeart <enyeart@...>
Cc: Mr.Phuwanai Thummavet <mr.thummavet@...>; fabric <fabric@...>
Subject: Re: [Hyperledger Fabric] Validating transactions including rich queries

I will add a few more things on top of what Dave said.

Yes, it is dependent on the application requirements.

    1. When a chaincode doesn't use rich queries, it archives serializable isolation.
    2. When a chaincode uses a rich query, it achieves only repeatable read.

Please read more about this here https://github.com/hyperledger/fabric-chaincode-go/blob/master/shim/interfaces.go#L187-L202 (document well).

To understand various isolation levels, please refer to

Most databases support serializable and repeatable read isolations. In fact, the default setting is repeatable isolation in most databases. There are many use-cases that do use repeatable isolation (not carrying about the phantom reads).

Regards,
Senthil

On Wed, Jun 17, 2020 at 6:10 PM David Enyeart <enyeart@...> wrote:

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.


Dave Enyeart

"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From: "Mr.Phuwanai Thummavet" <mr.thummavet@...>
To: fabric <fabric@...>
Date: 06/17/2020 01:59 AM
Subject: [EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...





Dear all,

I read the following article: https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRange the GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer



David Enyeart
 

If I understand this correctly, doesn't running a private fork of Fabric imply that Oracle nodes cannot be mixed with nodes from other clouds that support the open source Fabric?


Dave Enyeart

"Mark Rakhmilevich" ---06/17/2020 02:37:58 PM---As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Be

From: "Mark Rakhmilevich" <mark.rakhmilevich@...>
To: Senthil Nathan <cendhu@...>, David Enyeart <enyeart@...>
Cc: "Mr.Phuwanai Thummavet" <mr.thummavet@...>, fabric <fabric@...>
Date: 06/17/2020 02:37 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...




As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Berkeley DB as a replacement for Level DB and Couch DB, with full support for rich data queries and GetQueryResult validation at commit time to ensure the query result set is stable between endorsement and commit.  It’s available in the Cloud BaaS version as well as on-premises Enterprise  Edition. 

There’s also SQLite layer on top, enabling chaincodes to use SQL Selects for queries as well CouchDB query language for compatibility.  This solves a number of development and data consistency challenges in our enterprise deployments, and using row-level locking capabilities of Berkeley DB supports higher concurrency and throughput. 

Regards,
   Mark

From: Senthil Nathan <cendhu@...>
Sent: Wednesday, June 17, 2020 5:58 AM
To: David Enyeart <enyeart@...>
Cc: Mr.Phuwanai Thummavet <mr.thummavet@...>; fabric <fabric@...>
Subject: Re: [Hyperledger Fabric] Validating transactions including rich queries

I will add a few more things on top of what Dave said.

Yes, it is dependent on the application requirements.
    1. When a chaincode doesn't use rich queries, it archives serializable isolation.
    2. When a chaincode uses a rich query, it achieves only repeatable read.
Please read more about this here https://github.com/hyperledger/fabric-chaincode-go/blob/master/shim/interfaces.go#L187-L202 (document well).

To understand various isolation levels, please refer to Most databases support serializable and repeatable read isolations. In fact, the default setting is repeatable isolation in most databases. There are many use-cases that do use repeatable isolation (not carrying about the phantom reads).

Regards,
Senthil

On Wed, Jun 17, 2020 at 6:10 PM David Enyeart <enyeart@...> wrote:

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.


Dave Enyeart

"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From: "Mr.Phuwanai Thummavet" <mr.thummavet@...>
To: fabric <fabric@...>
Date: 06/17/2020 01:59 AM
Subject: [EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...





Dear all,

I read the following article: https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRange the GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer




Mark Rakhmilevich
 

As a point of interest, in Oracle Blockchain Platform implementation of Fabric we have been using Berkeley DB as a replacement for Level DB and Couch DB, with full support for rich data queries and GetQueryResult validation at commit time to ensure the query result set is stable between endorsement and commit.  It’s available in the Cloud BaaS version as well as on-premises Enterprise  Edition. 

 

There’s also SQLite layer on top, enabling chaincodes to use SQL Selects for queries as well CouchDB query language for compatibility.  This solves a number of development and data consistency challenges in our enterprise deployments, and using row-level locking capabilities of Berkeley DB supports higher concurrency and throughput. 

 

Regards,

   Mark

 

From: Senthil Nathan <cendhu@...>
Sent: Wednesday, June 17, 2020 5:58 AM
To: David Enyeart <enyeart@...>
Cc: Mr.Phuwanai Thummavet <mr.thummavet@...>; fabric <fabric@...>
Subject: Re: [Hyperledger Fabric] Validating transactions including rich queries

 

I will add a few more things on top of what Dave said.

 

Yes, it is dependent on the application requirements.

  1. When a chaincode doesn't use rich queries, it archives serializable isolation.
  2. When a chaincode uses a rich query, it achieves only repeatable read.

Please read more about this here https://github.com/hyperledger/fabric-chaincode-go/blob/master/shim/interfaces.go#L187-L202 (document well).

 

To understand various isolation levels, please refer to

Most databases support serializable and repeatable read isolations. In fact, the default setting is repeatable isolation in most databases. There are many use-cases that do use repeatable isolation (not carrying about the phantom reads).

 

Regards,

Senthil

 

On Wed, Jun 17, 2020 at 6:10 PM David Enyeart <enyeart@...> wrote:

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.


Dave Enyeart

"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From: "Mr.Phuwanai Thummavet" <mr.thummavet@...>
To: fabric <fabric@...>
Date: 06/17/2020 01:59 AM
Subject: [EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...




Dear all,

I read the following article: https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRange the GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer


Mr.Phuwanai Thummavet <mr.thummavet@...>
 

Thanks for the clarification.


On Wed, Jun 17, 2020 at 7:58 PM Cendhu U <cendhu@...> wrote:
I will add a few more things on top of what Dave said.

Yes, it is dependent on the application requirements.
  1. When a chaincode doesn't use rich queries, it archives serializable isolation.
  2. When a chaincode uses a rich query, it achieves only repeatable read.
Please read more about this here https://github.com/hyperledger/fabric-chaincode-go/blob/master/shim/interfaces.go#L187-L202 (document well).

To understand various isolation levels, please refer to
Most databases support serializable and repeatable read isolations. In fact, the default setting is repeatable isolation in most databases. There are many use-cases that do use repeatable isolation (not carrying about the phantom reads).

Regards,
Senthil

On Wed, Jun 17, 2020 at 6:10 PM David Enyeart <enyeart@...> wrote:

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.


Dave Enyeart

"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From: "Mr.Phuwanai Thummavet" <mr.thummavet@...>
To: fabric <fabric@...>
Date: 06/17/2020 01:59 AM
Subject: [EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...




Dear all,

I read the following article: https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRange the GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer





--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer

Senthil Nathan
 

I will add a few more things on top of what Dave said.

Yes, it is dependent on the application requirements.
  1. When a chaincode doesn't use rich queries, it archives serializable isolation.
  2. When a chaincode uses a rich query, it achieves only repeatable read.
Please read more about this here https://github.com/hyperledger/fabric-chaincode-go/blob/master/shim/interfaces.go#L187-L202 (document well).

To understand various isolation levels, please refer to
Most databases support serializable and repeatable read isolations. In fact, the default setting is repeatable isolation in most databases. There are many use-cases that do use repeatable isolation (not carrying about the phantom reads).

Regards,
Senthil

On Wed, Jun 17, 2020 at 6:10 PM David Enyeart <enyeart@...> wrote:

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.


Dave Enyeart

"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From: "Mr.Phuwanai Thummavet" <mr.thummavet@...>
To: fabric <fabric@...>
Date: 06/17/2020 01:59 AM
Subject: [EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...




Dear all,

I read the following article: https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRange the GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer



David Enyeart
 

That statement is still true. The risk is that other transactions may have inserted/updated/deleted data that would change the result set in the meantime, therefore the endorsed transaction can no longer be considered valid.
Some applications deal with this by first doing a read-only query to get a result set, and then doing a separate transaction to re-read the individual keys and update the keys. Still, other transactions may have updated data in the meantime - therefore, this may or may not be acceptable in your application depending on the scenario. Your application for example could do a final query after the transaction to determine if there are any new keys added that meet the query criteria.


Dave Enyeart

"Mr.Phuwanai Thummavet" ---06/17/2020 01:59:59 AM---Dear all, I read the following article:

From: "Mr.Phuwanai Thummavet" <mr.thummavet@...>
To: fabric <fabric@...>
Date: 06/17/2020 01:59 AM
Subject: [EXTERNAL] [Hyperledger Fabric] Validating transactions including rich queries
Sent by: fabric@...




Dear all,

I read the following article: https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRange the GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer



Mr.Phuwanai Thummavet <mr.thummavet@...>
 

Dear all,

I read the following article: https://medium.com/wearetheledger/hyperledger-fabric-couchdb-fantastic-queries-and-where-to-find-them-f8a3aecef767. The article stated, "unlike GetStateByRange the GetQueryResult query doesn’t get re-executed during the validation phase. This means that there is no guarantee the result set is stable between chaincode execution and commit time for rich queries, and therefore rich queries are not appropriate for use in update transactions".

It's a long time since the article was published. I would like to update my knowledge. Are there any protocol improvements on validating transactions including rich queries for the current Fabric implementations(v1.4.x and v2.x)? Is it safe to update values that are queried by GetQueryResult API for the current Fabric implementations?

Thank you.

--
Best Regards,
Phuwanai Thummavet
Blockchain Architect and Full-Stack Developer

1 - 10 of 10