Rich Query on levelDB (~7x faster queries) - iState package #fabric #fabric-chaincode


Prasanth Sundaravelu
 

Hi guys,

I just released v1.0 of iState package. Check it out: https://github.com/prasanths96/iState


iState is a state management package for Hyperledger fabric chaincode. It can be used to perform high performance rich queries on leveldb.

Features:

* Rich Query in levelDB.

* On an average case, query is **~7 times** faster than CouchDB's Rich Query with Index enabled.

* In-memory caching using ARC (Adaptive Replacement Cache) algorithm.

* Cache consistency is maintained. Data returned by query is always consistent.

 

* Easy to use.

My study on performance: https://docs.google.com/spreadsheets/d/1f3MenLWoq75ss5IvDJnGP3A30x1BqJLVVcQi7KEURt8/edit#gid=1695504678

I would encourage you to star the repo and use it in your application.

Suggestions, comments, insights and contributions are welcome. 

Note: This project is supposed to be an easy work around until leveldb rich queries are implemented at fabric core.


Yacov
 

Can you explain in a few sentences the mechanics behind it?

If I want to do a rich query that returns all keys that their value is "Yacov" - how do you do that without scanning the entire namespace ?





From:        "Prasanth Sundaravelu" <prasanths96@...>
To:        fabric@...
Date:        04/17/2020 09:09 PM
Subject:        [EXTERNAL] [Hyperledger Fabric] Rich Query on levelDB (~7x faster queries) - iState package #fabric #fabric-chaincode
Sent by:        fabric@...




Hi guys,

I just released v1.0 of iState package. Check it out: https://github.com/prasanths96/iState


iState is a state management package for Hyperledger fabric chaincode. It can be used to perform high performance rich queries on leveldb.

Features:

* Rich Query in levelDB.

* On an average case, query is **~7 times** faster than CouchDB's Rich Query with Index enabled.

* In-memory caching using ARC (Adaptive Replacement Cache) algorithm.

* Cache consistency is maintained. Data returned by query is always consistent.

 

* Easy to use.

My study on performance: https://docs.google.com/spreadsheets/d/1f3MenLWoq75ss5IvDJnGP3A30x1BqJLVVcQi7KEURt8/edit#gid=1695504678

I would encourage you to star the repo and use it in your application.

Suggestions, comments, insights and contributions are welcome.

Note: This project is supposed to be an easy work around until leveldb rich queries are implemented at fabric core.





Prasanth Sundaravelu
 

Hi Yacov,

I will try to explain in short.

Assume a structure: 

sturct {
    ID      string
   Name string
}

1. It will be converted to a series of index keys and one original key.
2. Index keys will hold information about the original key, original key will hold the actual marshalled struct value.

Eg: for ID="id1" and Name="Yacov", The following keys will be generated:

1. StructName_Name_Yavoc_id1: nil byte  (assuming only name field is tagged with istate tag)
2. id1: {ID:"id1", Name: "Yacov"}

During query, I will generate an index and perform range query.

Eg: query=[{"Name":"eq Yacov"}]  will generate index: "StructName_Name_Yacov_" and performs range query with:
1. startkey: "StructName_Name_Yacov_"
2. endkey: "StructName_Name_Yacov_~"  (~ being last usable ascii character)

This will return me the key: StructName_Name_Yavoc_id1, with which, I will extract the original key and fetch it using GetState()


Senthil Nathan
 

Hi Prasanth,


   One addition to above example is the usage of a cache at the chaincode. How is the consistency ensured? Each transaction could be simulated on a different set of peers (for load balancing or based on the endorsement policy). During the commit time, how would the chaincode know about the new updates to keep its cache consistent with the blockchain states?

Regards,
Senthil


On Sat, Apr 18, 2020 at 1:00 AM Prasanth Sundaravelu <prasanths96@...> wrote:

Hi Yacov,

I will try to explain in short.

Assume a structure: 

sturct {
    ID      string
   Name string
}

1. It will be converted to a series of index keys and one original key.
2. Index keys will hold information about the original key, original key will hold the actual marshalled struct value.

Eg: for ID="id1" and Name="Yacov", The following keys will be generated:

1. StructName_Name_Yavoc_id1: nil byte  (assuming only name field is tagged with istate tag)
2. id1: {ID:"id1", Name: "Yacov"}

During query, I will generate an index and perform range query.

Eg: query=[{"Name":"eq Yacov"}]  will generate index: "StructName_Name_Yacov_" and performs range query with:
1. startkey: "StructName_Name_Yacov_"
2. endkey: "StructName_Name_Yacov_~"  (~ being last usable ascii character)

This will return me the key: StructName_Name_Yavoc_id1, with which, I will extract the original key and fetch it using GetState()


Yacov
 

Could it be that the cache is only for reads and not for writes?

Otherwise, even if it's simulated on peers of equal height, as Senthil said - without putting these reads into the read set and short-circuiting via the cache essentially removes causal order in transactions and you basically get incorrect "blind writes" no?



From:        "Senthil Nathan" <cendhu@...>
To:        Prasanth Sundaravelu <prasanths96@...>
Cc:        fabric@...
Date:        04/17/2020 10:41 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Rich Query on levelDB (~7x faster queries) - iState package #fabric #fabric-chaincode
Sent by:        fabric@...




Hi Prasanth,

   Is it different from using a composite key https://github.com/hyperledger/fabric-samples/blob/master/chaincode/marbles02_private/go/marbles_chaincode_private.go#L276?

   One addition to above example is the usage of a cache at the chaincode. How is the consistency ensured? Each transaction could be simulated on a different set of peers (for load balancing or based on the endorsement policy). During the commit time, how would the chaincode know about the new updates to keep its cache consistent with the blockchain states?

Regards,
Senthil


On Sat, Apr 18, 2020 at 1:00 AM Prasanth Sundaravelu <prasanths96@...> wrote:
Hi Yacov,

I will try to explain in short.

Assume a structure: 

sturct {
    ID      string
   Name string
}

1. It will be converted to a series of index keys and one original key.
2. Index keys will hold information about the original key, original key will hold the actual marshalled struct value.

Eg: for ID="id1" and Name="Yacov", The following keys will be generated:

1. StructName_Name_Yavoc_id1: nil byte  (assuming only name field is tagged with istate tag)
2. id1: {ID:"id1", Name: "Yacov"}

During query, I will generate an index and perform range query.

Eg: query=[{"Name":"eq Yacov"}]  will generate index: "StructName_Name_Yacov_" and performs range query with:
1. startkey: "StructName_Name_Yacov_"
2. endkey: "StructName_Name_Yacov_~"  (~ being last usable ascii character)

This will return me the key: StructName_Name_Yavoc_id1, with which, I will extract the original key and fetch it using GetState()





Yacov
 

What if I have fields A, B, C, ... Z and I am asking about field F containing value "foo" ?

Can you describe what indices are built and what is the range query?



From:        "Prasanth Sundaravelu" <prasanths96@...>
To:        fabric@...
Date:        04/17/2020 10:30 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Rich Query on levelDB (~7x faster queries) - iState package #fabric #fabric-chaincode
Sent by:        fabric@...




Hi Yacov,

I will try to explain in short.

Assume a structure:

sturct {
   ID      string
  Name string
}

1. It will be converted to a series of index keys and one original key.
2. Index keys will hold information about the original key, original key will hold the actual marshalled struct value.

Eg: for ID="id1" and Name="Yacov", The following keys will be generated:

1. StructName_Name_Yavoc_id1: nil byte  (assuming only name field is tagged with istate tag)
2. id1: {ID:"id1", Name: "Yacov"}

During query, I will generate an index and perform range query.

Eg: query=[{"Name":"eq Yacov"}]  will generate index: "StructName_Name_Yacov_" and performs range query with:
1. startkey: "StructName_Name_Yacov_"
2. endkey: "StructName_Name_Yacov_~"  (~ being last usable ascii character)

This will return me the key: StructName_Name_Yavoc_id1, with which, I will extract the original key and fetch it using GetState()





Prasanth Sundaravelu
 

Hi Senthil, 

Question1: 
There is a problem with composite keys in performing full rich queries. 

Assume the same structure: 
sturct {
    ID      string
   Name string
}

With composite keys, I can put it as:  DocType~ID~Name~StructName~id1~Yacov

Here, if I want to use GetStateByPartialCompositeKey to query keys with Name as Yacov, I have the following info:

1. Composite Index: DocType~ID~Name~StructName
2. Values: <don't know>, Yacov

That will create a key:  DocType~ID~Name~StructName~<nil>~Yacov  - There is a hole in this information.
This key won't match the composite key that just got created. 

If the order of composite index is changed like: DocType~Name~ID~StructName~Yacov~id1, then, we have enough information without hole to query Yacov: DocType~Name~ID~StructName~Yacov

So, we cannot perform queries on all the fields individually. Also, comparative queries like "<", ">" , ">=", "<=" is not possible with composite keys. 

Question2:

When generating the index keys (while storing the state) as mentioned in the previous mail:


1. StructName_Name_Yavoc_id1: nil byte 
2. id1: {ID:"id1", Name: "Yacov"}

The index key will actually have CRC Hash of the actual data as a value instead of `nil byte`:  StructName_Name_Yavoc_id1: <hash>

I also will have stored the hash in cache as well. 

When performing query, I always fetch the index from db. After fetching it,

1. I will check if the hash present in index is same as the hash in cache. 

2. If matches, I fetch from cache. If not, I will drop it and get the data from db.


Prasanth Sundaravelu
 

Hi Yacov,

Multiple fields query:

Eg:

struct {
ID string
A string
..
F string
..
Z string
}

Generated Indeces: (Note: Field value is also present in the index)
1. StructName_A_foo_id1 
2. StructName_B_bar_id1
....
6. StructName_F_foo_id1
...
26. StructName_Z_bar_id1

+ Original key-val: id1:{...whole struct...}

Querying for F=foo: [{"F": "eq foo"}]:  
Index generated:  StructName_F_foo_<empty>  (F is the field name, foo is the value from query)

From this, I will perform range query with: (The start and end keys will be different for different operators, hence denoting it separately)

1. startkey: StructName_F_foo_
2. endkey: StructName_F_foo_~

This will return me key:

StructName_F_foo_id1

(Since this key contains "StructName_F_foo_"  and it comes under the range mentioned by startkey and endkey)

From this index key, I will get the original key: id1 and either read from cache or get from db.

 

About cache: 

You're actually right. If cache is used when writing, it may cause conflict in read/write sets during transaction and cache must only be used for reads. 

I've not considered this when implementing. I would appreciate, if you could create an issue in the repo. 

Can we detect an invoke vs query transaction from the tx info in stub? If so, I can make it so that, it uses cache only for queries.


Yacov
 

Well, if you want to do a write you would use a PutState.

In theory you can also persist to the Blockchain, transactions that have an empty write set, which are basically queries, but I doubt it's widely used.



From:        "Prasanth Sundaravelu" <prasanths96@...>
To:        fabric@...
Date:        04/17/2020 11:13 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Rich Query on levelDB (~7x faster queries) - iState package #fabric #fabric-chaincode
Sent by:        fabric@...




Hi Yacov,

Multiple fields query:

Eg:

struct {
ID string
A string
..
F string
..
Z string
}

Generated Indeces: (Note:
Field value is also present in the index)
1. StructName_A_foo_id1
2. StructName_B_bar_id1
....
6. StructName_F_foo_id1
...
26. StructName_Z_bar_id1

+ Original key-val: id1:{...whole struct...}

Querying for F=foo: [{"F": "eq foo"}]:  
Index generated:  StructName_F_foo_<empty>  (F is the field name, foo is the value from query)

From this, I will perform range query with: (The start and end keys will be different for different operators, hence denoting it separately)

1. startkey: StructName_F_foo_
2. endkey: StructName_F_foo_~

This will return me key:

StructName_F_foo_id1

(Since this key contains "StructName_F_foo_"  and it comes under the range mentioned by startkey and endkey)

From this index key, I will get the original key: id1 and either read from cache or get from db.

 

About cache:

You're actually right. If cache is used when writing, it may cause conflict in read/write sets during transaction and cache must only be used for reads.

I've not considered this when implementing. I would appreciate, if you could create an issue in the repo.

Can we detect an invoke vs query transaction from the tx info in stub? If so, I can make it so that, it uses cache only for queries.





Prasanth Sundaravelu
 

Oh yes,

Write will only happen when "Create", "Update" and "Delete". In that case. But still, User will probably perform Query before calling Create or Update, so Query still won't know if it is a write or read only transaction.

 

In theory you can also persist to the Blockchain, transactions that have an empty write set, which are basically queries, but I doubt it's widely used.

Conflict may still arise in a multi-peer network, if one peer has a copy of cache and another peer doesn't. In that case, the transaction will get rejected since peer1 and peer2 will have returned different read sets.


Yacov
 

The problem is not the query, but the cache.
You don't know whether to cache or not to cache until you have tried to do a PutState.

There is already a cache in the peer itself... maybe drop the cache?



From:        "Prasanth Sundaravelu" <prasanths96@...>
To:        fabric@...
Date:        04/17/2020 11:29 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Rich Query on levelDB (~7x faster queries) - iState package #fabric #fabric-chaincode
Sent by:        fabric@...




Oh yes,

Write will only happen when "Create", "Update" and "Delete". In that case. But still, User will probably perform Query before calling Create or Update, so Query still won't know if it is a write or read only transaction.

 

In theory you can also persist to the Blockchain, transactions that have an empty write set, which are basically queries, but I doubt it's widely used.

Conflict may still arise in a multi-peer network, if one peer has a copy of cache and another peer doesn't. In that case, the transaction will get rejected since peer1 and peer2 will have returned different read sets.





Prasanth Sundaravelu
 

Without cache, the performance is very bad when fetching a large number of matching records as discussed in here: https://lists.hyperledger.org/g/fabric/topic/72797648?p=Created,,,20,2,0,0

Short summary: Having the whole marshalled data directly inside index keys would be good performing. Another way maybe if GetStateMultipleKeys  from https://github.com/hyperledger/fabric/blob/master/core/ledger/ledger_interface.go#L172 is exposed. But still, it will be slightly slower than Cached data because of unmarshalling that takes place everytime when fetching data. In cache, I directly store unmarshalled data. 

 

Although, one simple way is to ask the user if this is an invoke transaction or not, as a parameter in the Query() function itself, and responsibility goes to the user to provide right information when coding.


Senthil Nathan
 

Hi Prasanth,

   Correct me if I am missing something.

With composite keys, I can put it as:  DocType~ID~Name~StructName~id1~Yacov

Here, if I want to use GetStateByPartialCompositeKey to query keys with Name as Yacov, I have the following info:

1. Composite Index: DocType~ID~Name~StructName
2. Values: <don't know>, Yacov

In our sample https://github.com/hyperledger/fabric-samples/blob/master/chaincode/marbles02_private/go/marbles_chaincode_private.go#L276, we create composite-key of format indexName~indexField~DocID which is similar to what you have used.

The index key will actually have CRC Hash of the actual data as a value instead of `nil byte`:  StructName_Name_Yavoc_id1: <hash>

I see. If there is a hash, we can detect stale values.

Regards,
Senthil

On Sat, Apr 18, 2020 at 2:12 AM Prasanth Sundaravelu <prasanths96@...> wrote:

Without cache, the performance is very bad when fetching a large number of matching records as discussed in here: https://lists.hyperledger.org/g/fabric/topic/72797648?p=Created,,,20,2,0,0

Short summary: Having the whole marshalled data directly inside index keys would be good performing. Another way maybe if GetStateMultipleKeys  from https://github.com/hyperledger/fabric/blob/master/core/ledger/ledger_interface.go#L172 is exposed. But still, it will be slightly slower than Cached data because of unmarshalling that takes place everytime when fetching data. In cache, I directly store unmarshalled data. 

 

Although, one simple way is to ask the user if this is an invoke transaction or not, as a parameter in the Query() function itself, and responsibility goes to the user to provide right information when coding.


Prasanth Sundaravelu
 

Hi Senthil,

You are right to some extent. My first inspiration is from the composite keys.

In fact, the first set of ideas involved just having composite key as index for each and every field in the struct. That would have been sufficient to query "equal to" and "not equal to". Other relational queries would not have been possible.

My next step was to figure out an encoding method to be able to store numbers in the correct numeric ascending order. This is necessary because, by default, when numbers are turned to strings and are ordered, the order of numbers from 1 to 10, will be like this: 1, 10, 2, 3, 4, 5, 6, 7, 8, 9. Getting these numbers ordered in numeric ascending order will now net me utilize range query itself to perform operations like >, <, >=, <=. Eg: For "> somevalue", I can have start and end keys as: "somevalue+1", "~". 

Composite key was not enough to let me do this, because this operation requires cutting of part of value while forming range query start,end keys and it cannot be done either by using GetStateByPartialCompositeKey() or by directly using custom generated composite key in range query (It complains that keys cannot start with null character. Apparently that is what a composite key start with (if I'm not mistaken)).

To explain this problem with an example, consider that you want perform ">" operation on a string value. Say, I have in db, this value set {apple, ant, ball, banana}. If I want all values that start with letter "b", I want to do a query "> a". That will need me to create a composite key DocType~Val~a and anyways, GetStateByPartialCompositeKey() can only match values and it will not return anything in this scenario.

Hence to be able to use range query over custom generated keys, I implemented a similar approach to that of a composite key.


Senthil Nathan
 

Hi Prasanth,

Hence to be able to use range query over custom generated keys, I implemented a similar approach to that of a composite key.

 I remember a similar request made in the past on enabling range queries on composite keys. Hence, we created this JIRA https://jira.hyperledger.org/browse/FAB-11281 but we didn't proceed with it.

My next step was to figure out an encoding method to be able to store numbers in the correct numeric ascending order. This is necessary because, by default, when numbers are turned to strings and are ordered, the order of numbers from 1 to 10, will be like this: 1, 10, 2, 3, 4, 5, 6, 7, 8, 9. Getting these numbers ordered in numeric ascending order will now net me utilize range query itself to perform operations like >, <, >=, <=. Eg: For "> somevalue", I can have start and end keys as: "somevalue+1", "~".

You can copy this code for order-preserving encoding/decoding:
  1. https://github.com/hyperledger/fabric/blob/master/common/ledger/util/util.go#L27
  2. https://github.com/hyperledger/fabric/blob/master/core/ledger/pvtdatastorage/kv_encoding.go#L227
Regards,
Senthil

On Sat, Apr 18, 2020 at 7:22 AM Prasanth Sundaravelu <prasanths96@...> wrote:

Hi Senthil,

You are right to some extent. My first inspiration is from the composite keys.

In fact, the first set of ideas involved just having composite key as index for each and every field in the struct. That would have been sufficient to query "equal to" and "not equal to". Other relational queries would not have been possible.

My next step was to figure out an encoding method to be able to store numbers in the correct numeric ascending order. This is necessary because, by default, when numbers are turned to strings and are ordered, the order of numbers from 1 to 10, will be like this: 1, 10, 2, 3, 4, 5, 6, 7, 8, 9. Getting these numbers ordered in numeric ascending order will now net me utilize range query itself to perform operations like >, <, >=, <=. Eg: For "> somevalue", I can have start and end keys as: "somevalue+1", "~". 

Composite key was not enough to let me do this, because this operation requires cutting of part of value while forming range query start,end keys and it cannot be done either by using GetStateByPartialCompositeKey() or by directly using custom generated composite key in range query (It complains that keys cannot start with null character. Apparently that is what a composite key start with (if I'm not mistaken)).

To explain this problem with an example, consider that you want perform ">" operation on a string value. Say, I have in db, this value set {apple, ant, ball, banana}. If I want all values that start with letter "b", I want to do a query "> a". That will need me to create a composite key DocType~Val~a and anyways, GetStateByPartialCompositeKey() can only match values and it will not return anything in this scenario.

Hence to be able to use range query over custom generated keys, I implemented a similar approach to that of a composite key.


Prasanth Sundaravelu
 

Sorry for not being clear. I have already formulated an encoding scheme that requires addition of 4 characters prefix and have implemented queries over that encoding format. 

But, If there is a good improvement in performance when using the code in the links you have mentioned, I would be happy to adopt it :), So thanks for sharing it.


Senthil Nathan
 

I see. Good to know. Anyway, keep up the good work.

I am sure that Manish would be more interested in this work. When we were designing ledger for v1.0, Manish, in fact, proposed to build a query layer on top of leveldb using these secondary indices (not at the chaincode-level but at the fabric-ledger level). I remember he wrote a brief design document too. You can ping him privately to get that doc. 

Traditionally, all database queries run using an either primary or secondary index. If there is no index, the database would do a full-range scan. If possible, you can also experiment with a complex indexing structure and share your experience with us. However, there might be challenges in implementing them on the chaincode side.
  1. https://www.postgresql.org/docs/12/indexes-types.html
  2. https://www.cs.ucr.edu/~vagelis/publications/LSM-secondary-indexing-sigmod2018.pdfA Comparative Study of Secondary Indexing Techniques in LSM-based NoSQL Databases, SIGMOD 2018. I recommend this paper to you as you have an interest in indexing techniques.

Regards,
Senthil

On Sat, 18 Apr 2020 at 8:07 AM, Prasanth Sundaravelu <prasanths96@...> wrote:

Sorry for not being clear. I have already formulated an encoding scheme that requires addition of 4 characters prefix and have implemented queries over that encoding format. 

But, If there is a good improvement in performance when using the code in the links you have mentioned, I would be happy to adopt it :), So thanks for sharing it.


Prasanth Sundaravelu
 

Thanks a lot Senthil :)

I have been meaning to contribute to Hyperledger project, since I started learning Fabric around 2 years back. I guess this is a right time for me to start contributing to the core code base too. I've never done any contribution to open source repos before. In fact, iState is my first meaningful open-source project. I would love to take support from you and Manish to get started in contributing. 

Thanks for those links. I will refer those.

I do already have done some experiments and have insights that might be helpful. 

1. For separators, I am using null character "\000" instead of "_". I chose "_" for explaining for simplicity. If not using null as a separator, there may be problem with performing relational queries over string data. 

Eg: When using "_" as separator, assume the following indeces: 

1. val_<nil>_id1 -> val__id1   
2. val_Hi_id5

Logically, we would expect nil string value to be ordered first followed by Hi, but in reality, it will be ordered like:
val_Hi_id1

val__id1

since, 5th character in the indeces are "H" and "_" and "H" is less than "_". Having "null" value as separator solves this issue, because then, 5th char will be "null" and "H" -> so null will be less than H and comes in front as expected.

 

2. Boolean type get encoded to characters "t" and "f". 

3. Numbers are encoded in the following format: 

<sign-digit><num-digits-tens-place><num-digits-ones-place><separator><actual-number>

Eg: For representing 100,  ->  103_100, where "1" (1st digit) represents positive sign, 03 represents the number has 3 digits. Only 2 digits is chosen to represent this, which is enough to accommodate up to 99 digits (which is already too big)

This way, the weight of the number can be recognized by the prefixed characters, rather than reading through all the digits.

Numbers 1 - 10:

101_1
101_2
101_3
101_4
..
101_9
102_10

Here, when 10 reaches, the 3rd prefix char raises to 2 and hence it is naturally numerically ordered, without going through rest of the characters.

In case of float, we can ignore coding for floating points and do the encoding just for decimal numbers before the "."

Because, the first character after the "." will always have the maximum weight-age and need not parse through the rest of the decimal points to figure out the weight (Unlike integers).

Eg:

100.10
100.95

Note: Having "1" as positive and "0" as negative does not make these encoded numbers fall in a perfect number scale.

Eg: In case of negative numbers, -1 is greater than -2. But encoded numbers will be ordered as:

001_1
001_2  

which denotes that -2 is greater.

4. Looks like JSON has a limitation in bitsize for numbers, which cannot hold 64bits worth of numbers in raw binary format. Hence, it encodes the number into scientific notation like "9e+10", which when unmarshalled back to go's type, will not be accurate, and sometimes gets rounded off. Eg: 99999999999999999 is rounded off to 100000000000000000. The maximum no. of integer digits upto which round off doesn't happen is 15.

 

5. When having bottle-neck due to GetState() API, we suspected the hard disk "not being able to" move to random memory locations quickly enough as in this discussion: https://lists.hyperledger.org/g/fabric/topic/72797648?p=Created,,,20,2,0,0. After that, I have tested moving all original keys to the last part of namespace by appending "~~" as prefix to the keys. 

This saved a 70-100ms time for a 6000 record match query that happened to take 850-900ms without this change (and without cache enabled). This does not seem to be a big difference, considering we blamed ramdom access from storage as the main bottle-neck.

I think there is some problem with GetState() API. Maybe is it because it creates db snapshot everytime it is called? whereas range query created snap for every 100 records? 

 

 

 


Jim
 

Hi

The  performance study on LevelDB secondary index options is well done.
That said, for Fabric, I think it's critical to keep CouchDB as an option.
Most of the solutions I look at need the functionality in CouchDB that LevelDB lacks.

If performance were the only issue for a key value store, there are other options to LevelDB other key store databases can be considered  - https://mozilla.github.io/firefox-browser-architecture/text/0017-lmdb-vs-leveldb.html

CouchDB has just released version 3.1 ( see notes below ).
CouchDB has decided to migrate to FoundationDB as the core for version 4x.( see https://www.foundationdb.org/  )
That work is in progress now.

Since version 3x is an interim to v4x, I'm not sure how Hyperledger Fabric will choose to update CouchDB versions on Fabric releases.

Thanks

Jim Mason




Notes on CouchDB 3.1


See the official release notes document for an exhaustive list of all changes:


Release Notes highlights from 3.0.1:

  - A memory leak when encoding large binary content was patched
 
  - Improvements in documentation and defaults
 
  - JavaScript will no longer corrupt UTF-8 strings in various JS functions
 
Release Notes highlights from 3.1.0:

Everything from 3.0.1, plus...

  - Support for Java Web Tokens

  - Support for SpiderMonkey 68, including binaries for Ubuntu 20.04 (Focal Fossa)

  - Up to a 40% performance improvement in the database compactor





On Friday, April 17, 2020, 10:56:31 PM EDT, Senthil Nathan <cendhu@...> wrote:


I see. Good to know. Anyway, keep up the good work.

I am sure that Manish would be more interested in this work. When we were designing ledger for v1.0, Manish, in fact, proposed to build a query layer on top of leveldb using these secondary indices (not at the chaincode-level but at the fabric-ledger level). I remember he wrote a brief design document too. You can ping him privately to get that doc. 

Traditionally, all database queries run using an either primary or secondary index. If there is no index, the database would do a full-range scan. If possible, you can also experiment with a complex indexing structure and share your experience with us. However, there might be challenges in implementing them on the chaincode side.
  1. https://www.postgresql.org/docs/12/indexes-types.html
  2. https://www.cs.ucr.edu/~vagelis/publications/LSM-secondary-indexing-sigmod2018.pdfA Comparative Study of Secondary Indexing Techniques in LSM-based NoSQL Databases, SIGMOD 2018. I recommend this paper to you as you have an interest in indexing techniques.

Regards,
Senthil

On Sat, 18 Apr 2020 at 8:07 AM, Prasanth Sundaravelu <prasanths96@...> wrote:

Sorry for not being clear. I have already formulated an encoding scheme that requires addition of 4 characters prefix and have implemented queries over that encoding format. 

But, If there is a good improvement in performance when using the code in the links you have mentioned, I would be happy to adopt it :), So thanks for sharing it.


Senthil Nathan
 

Hi Jim,

   I would like to understand the following:

   That said, for Fabric, I think it's critical to keep CouchDB as an option.

   Currently, we support only the query based on a secondary index and JSON from CouchDB (not supporting or planning to support map/reduce functions). However, CouchDB is designed to do much more and it is a distributed database that supports eventual consistency. In the longer-term, I would prefer forking LevelDB code to add both JSON support and secondary indexes for performance reasons alone and still provide backward compatibility with CouchDB in terms of the query (this idea originally came for Manish).

   Yes, CouchDB is planning to use FoundationDB as the storage engine going forward for the version maintenance. FoundationDB has some inherent limit on the transaction execution time (5s) till they replace SQLite engine with the RedWood storage engine. In fact, if you look at the FoundationDB transaction flow, it follows, tx simulation, rwset collection, validation, and commit. It has so many servers such as proxy servers, resolver servers, transaction log servers, and storage servers (with sharding of keys). Given that CouchDB is moving to use FoundationDB, IMO, Fabric needs to stick with CouchDB 2.1 or go with forked LevelDB as I mentioned earlier.

Regards,
Senthil

On Wed, May 6, 2020 at 6:48 PM Jim Mason <jmason900@...> wrote:
Hi

The  performance study on LevelDB secondary index options is well done.
That said, for Fabric, I think it's critical to keep CouchDB as an option.
Most of the solutions I look at need the functionality in CouchDB that LevelDB lacks.

If performance were the only issue for a key value store, there are other options to LevelDB other key store databases can be considered  - https://mozilla.github.io/firefox-browser-architecture/text/0017-lmdb-vs-leveldb.html

CouchDB has just released version 3.1 ( see notes below ).
CouchDB has decided to migrate to FoundationDB as the core for version 4x.( see https://www.foundationdb.org/  )
That work is in progress now.

Since version 3x is an interim to v4x, I'm not sure how Hyperledger Fabric will choose to update CouchDB versions on Fabric releases.

Thanks

Jim Mason




Notes on CouchDB 3.1


See the official release notes document for an exhaustive list of all changes:


Release Notes highlights from 3.0.1:

  - A memory leak when encoding large binary content was patched
 
  - Improvements in documentation and defaults
 
  - JavaScript will no longer corrupt UTF-8 strings in various JS functions
 
Release Notes highlights from 3.1.0:

Everything from 3.0.1, plus...

  - Support for Java Web Tokens

  - Support for SpiderMonkey 68, including binaries for Ubuntu 20.04 (Focal Fossa)

  - Up to a 40% performance improvement in the database compactor





On Friday, April 17, 2020, 10:56:31 PM EDT, Senthil Nathan <cendhu@...> wrote:


I see. Good to know. Anyway, keep up the good work.

I am sure that Manish would be more interested in this work. When we were designing ledger for v1.0, Manish, in fact, proposed to build a query layer on top of leveldb using these secondary indices (not at the chaincode-level but at the fabric-ledger level). I remember he wrote a brief design document too. You can ping him privately to get that doc. 

Traditionally, all database queries run using an either primary or secondary index. If there is no index, the database would do a full-range scan. If possible, you can also experiment with a complex indexing structure and share your experience with us. However, there might be challenges in implementing them on the chaincode side.
  1. https://www.postgresql.org/docs/12/indexes-types.html
  2. https://www.cs.ucr.edu/~vagelis/publications/LSM-secondary-indexing-sigmod2018.pdfA Comparative Study of Secondary Indexing Techniques in LSM-based NoSQL Databases, SIGMOD 2018. I recommend this paper to you as you have an interest in indexing techniques.

Regards,
Senthil

On Sat, 18 Apr 2020 at 8:07 AM, Prasanth Sundaravelu <prasanths96@...> wrote:

Sorry for not being clear. I have already formulated an encoding scheme that requires addition of 4 characters prefix and have implemented queries over that encoding format. 

But, If there is a good improvement in performance when using the code in the links you have mentioned, I would be happy to adopt it :), So thanks for sharing it.