Date   

peer container is down constantly

Park, Jungil
 

Hi Team,

 

I had issue that peer container is constantly down. It is really random like once every four months.

 

Here is error message. Could you please help with this why this happen?

 

Version : HF 1.4.0

 

2020-04-07 21:50:09.184 UTC [gossip.comm] readFromStream -> WARN 47bb149 Got error, aborting: Failed unmarshaling GossipMessage from envelope: unexpected EOF

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xc6ab8e]

goroutine 76685136 [running]:

github.com/hyperledger/fabric/gossip/comm.interceptAcks.func1(0x0)

        /opt/gopath/src/github.com/hyperledger/fabric/gossip/comm/ack.go:66 +0x2e

github.com/hyperledger/fabric/gossip/comm.(*connection).serviceConnection(0xc0018cbb80, 0x126d5a0, 0xc001b27440)

        /opt/gopath/src/github.com/hyperledger/fabric/gossip/comm/conn.go:299 +0x135

github.com/hyperledger/fabric/gossip/comm.(*commImpl).GossipStream(0xc001b27440, 0x13856c0, 0xc00fac6640, 0x0, 0x0)

        /opt/gopath/src/github.com/hyperledger/fabric/gossip/comm/comm_impl.go:599 +0x3c3

github.com/hyperledger/fabric/protos/gossip._Gossip_GossipStream_Handler(0x11ba000, 0xc001b27440, 0x1383aa0, 0xc002888c00, 0x40d1d8, 0x30)

        /opt/gopath/src/github.com/hyperledger/fabric/protos/gossip/message.pb.go:2758 +0xad

github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1(0x11ba000, 0xc001b27440, 0x1383aa0, 0xc002888c00, 0x1b, 0x13806fe)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:71 +0xf5

github.com/hyperledger/fabric/common/grpclogging.StreamServerInterceptor.func1(0x11ba000, 0xc001b27440, 0x1383b00, 0xc002888780, 0xc011ca82e0, 0xc00afe5d10, 0x5e8cd8b4, 0x29cab6cd)

        /opt/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server.go:128 +0x2b3

github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1(0x11ba000, 0xc001b27440, 0x1383b00, 0xc002888780, 0x136b2e0, 0x1f51e20)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:74 +0x9c

github.com/hyperledger/fabric/common/grpcmetrics.StreamServerInterceptor.func1(0x11ba000, 0xc001b27440, 0x1383f20, 0xc0018cb550, 0xc011ca82e0, 0xc00afe5d10, 0xc0061f8c48, 0x40d1d8)

        /opt/gopath/src/github.com/hyperledger/fabric/common/grpcmetrics/interceptor.go:63 +0x408

github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x11ba000, 0xc001b27440, 0x1383f20, 0xc0018cb550, 0xc011ca82e0, 0x126d968, 0x13809e0, 0xc0002d2640)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:79 +0x14d

github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc001b82180, 0x1387520, 0xc006495980, 0xc004174400, 0xc001924420, 0x1e358c0, 0xc002888630, 0x0, 0x0)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1131 +0x440

github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001b82180, 0x1387520, 0xc006495980, 0xc004174400, 0xc002888630)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1212 +0x12a1

github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000be83d0, 0xc001b82180, 0x1387520, 0xc006495980, 0xc004174400)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:686 +0x9f

created by github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:684 +0xa1

 

 

 

Thanks,

Jungil.

This message has been sent by ABN AMRO Bank N.V., which has its seat at Gustav Mahlerlaan 10 (1082 PP) Amsterdam, the Netherlands, and is registered in the Commercial Register of Amsterdam under number 34334259.


Re: [EXTERNAL] Re: [Hyperledger Fabric] Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

Park, Jungil
 

Hi Team,

 

I had issue that peer container is constantly down. It is really random like once every four months.

 

Here is error message. Could you please help with this why this happen?

 

Version : HF 1.4.0

 

2020-04-07 21:50:09.184 UTC [gossip.comm] readFromStream -> WARN 47bb149 Got error, aborting: Failed unmarshaling GossipMessage from envelope: unexpected EOF

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xc6ab8e]

goroutine 76685136 [running]:

github.com/hyperledger/fabric/gossip/comm.interceptAcks.func1(0x0)

        /opt/gopath/src/github.com/hyperledger/fabric/gossip/comm/ack.go:66 +0x2e

github.com/hyperledger/fabric/gossip/comm.(*connection).serviceConnection(0xc0018cbb80, 0x126d5a0, 0xc001b27440)

        /opt/gopath/src/github.com/hyperledger/fabric/gossip/comm/conn.go:299 +0x135

github.com/hyperledger/fabric/gossip/comm.(*commImpl).GossipStream(0xc001b27440, 0x13856c0, 0xc00fac6640, 0x0, 0x0)

        /opt/gopath/src/github.com/hyperledger/fabric/gossip/comm/comm_impl.go:599 +0x3c3

github.com/hyperledger/fabric/protos/gossip._Gossip_GossipStream_Handler(0x11ba000, 0xc001b27440, 0x1383aa0, 0xc002888c00, 0x40d1d8, 0x30)

        /opt/gopath/src/github.com/hyperledger/fabric/protos/gossip/message.pb.go:2758 +0xad

github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1(0x11ba000, 0xc001b27440, 0x1383aa0, 0xc002888c00, 0x1b, 0x13806fe)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:71 +0xf5

github.com/hyperledger/fabric/common/grpclogging.StreamServerInterceptor.func1(0x11ba000, 0xc001b27440, 0x1383b00, 0xc002888780, 0xc011ca82e0, 0xc00afe5d10, 0x5e8cd8b4, 0x29cab6cd)

        /opt/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server.go:128 +0x2b3

github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1(0x11ba000, 0xc001b27440, 0x1383b00, 0xc002888780, 0x136b2e0, 0x1f51e20)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:74 +0x9c

github.com/hyperledger/fabric/common/grpcmetrics.StreamServerInterceptor.func1(0x11ba000, 0xc001b27440, 0x1383f20, 0xc0018cb550, 0xc011ca82e0, 0xc00afe5d10, 0xc0061f8c48, 0x40d1d8)

        /opt/gopath/src/github.com/hyperledger/fabric/common/grpcmetrics/interceptor.go:63 +0x408

github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x11ba000, 0xc001b27440, 0x1383f20, 0xc0018cb550, 0xc011ca82e0, 0x126d968, 0x13809e0, 0xc0002d2640)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:79 +0x14d

github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc001b82180, 0x1387520, 0xc006495980, 0xc004174400, 0xc001924420, 0x1e358c0, 0xc002888630, 0x0, 0x0)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1131 +0x440

github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001b82180, 0x1387520, 0xc006495980, 0xc004174400, 0xc002888630)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1212 +0x12a1

github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000be83d0, 0xc001b82180, 0x1387520, 0xc006495980, 0xc004174400)

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:686 +0x9f

created by github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1

        /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:684 +0xa1

 

 

 

 

 

 

 

Met vriendelijke groet / Kind regards

Jungil Park

ABN AMRO | ABN AMRO CLEARING

Gustav Mahlerlaan 10 | 1082 PP Amsterdam | The Netherlands
Mob +31 (0)6 14667692
Email jungil.park@...
Internet
www.abnamro.com
Save a tree! Print this message only if it's absolutely necessary

 

From: fabric@... <fabric@...> On Behalf Of Manish
Sent: 08 April 2020 04:36
To: Prasanth Sundaravelu <prasanths96@...>
Cc: fabric@...
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

 

WARNING: External email – exercise caution. Please forward suspicious emails to your local security team.

Hi Prasanth,

 

Specifically, about your last point -

 

First, if you did not notice much difference when you use HDD v/s in-memory volume, then I would suspect for one of the following two things 

    (A) Perhaps you may be testing with small amount of data that perhaps is being served from the filesystem cache instead of hitting the disk 

    (B) the communication between peer and chaincode container is more subtle bottleneck here. For a single key lookup, you would have this overhead multiple times (akin to random access overhead) and in the case of a range query, the data is transferred between peer and chaincode in a batch of 100, so this overhead is amortized (akin to a sequential scan).

 

On the caching part, yes, there should not be a problem in caching the data on the chaincode side. However, you would need to consider if the values are not too big, there is no value in maintaining the hashes in the index and then matching over with the cache. You can maintain directly the values in your index (as was suggested by Dave in his reply). I am not certain that the container will be stopped but in any case, Fabric now supports custom runtime environment that you may have better control over.

 

Hope this helps

 

Thanks,

Manish

 

On Tue, Apr 7, 2020 at 4:58 AM Prasanth Sundaravelu <prasanths96@...> wrote:

@Gari,

There will be multiple index keys created for a single object. In fact, sometimes even multiple keys for single field which has multiple values. So, It will be highly redundant if I store data directly in those keys, hence using a pointing mechanism for the same.

 

@ Manish, 

Thanks for the response. 

 

1. That makes sense.

2. I agree, doing it in chaincode is not the best way, but it think it is the simpler way for still gaining some significant performance. Especially, for a frequently used casual query, which would return very less data. Earlier, I have been using composite keys with fields ordered in their significance of query necessity(Eg: Doctype at first always, Id will be second) and had a query processor at chaincode itself, that would take fetch all the records with the given docType and parse, in case the fields that you want to query is discontinued when forming as composite keys (Eg: doctype: doc1, Id:" ", count: 1000).

It also helps for relational queries. 

The main problem with this is that, when doing the above mentioned type query, there will be no best case, average case, All such queries run on worst case and directly proportional to the number of records in that docType. (eg: For 100k records, it takes 6 seconds to return response, no matter if it will return a single result or all 100k records as result).

Then, I also compared couch Db's performance with the same data sets, for un-indexed queries‌, it was slower than leveldb. Indexed queries were faster as long as it did not return like all of 100k records, and writes were far far slower than leveldb. Hence, this project was thought out, which could be a helpful and easy workaround until fabric gets this.

 

3. Yes, I am using a 7200RPM hdd. I agree it has got to be slower. I have also casually tested mounting a volume directly in memory and using it as the mount folder for /var/production. I did not see a significant improvement. I did not check it thoroughly though, it could also be bottleneck in my code. 

I am also storing meta data similar to cost model as you mentioned, which is used to choose the query field which would have least number of records and then doing further processing at code. Thanks for mentioning this, I think its a good idea to skip indexing altogether in some cases.

 

The next step for me is to build proper cache for the data in memory. I've implemented a simple one and tested it out, it has 40% improvement in the worst case scenario when compared to the old composite key implementation. 

To maintain reliability of the cached data: 

1. Will stored hash of actual data in index keys.

2. Will always fetch index keys from db, and verify if the hash in it and hash of data in cache are same. 

If same, then no change has occurred to data and will use the same.

Else, I will fetch the latest data from db. This ensures consistency.

 

I have questions in caching. 

1. I have read that in old version of fabric, chaincode container is stopped when idling. Is it still the case in 1.3+ versions? 

 

2. Is it ok to save decrypted data in memory?

 

- Prasanth

 

On Mon, 6 Apr 2020, 8:10 pm Manish Sethi, <manish.sethi@...> wrote:

Hello Prasanth,

Aw

My response to your questions would be three fold.

 

First, a chaincode is expected to be a sequential logic. Otherwise, providing a guarantee of deterministic behavior is very hard. If you go into theoretical aspects, the basic definition of a transaction is "a sequence of database operations….” - a logical sequence is key here. There are some mature systems that support nested transactions that could be executed in parallel but to keep the chaincode logic simple and deterministic, Fabric supports it in the form of a single sequential flow.

 

Second, for complex query support, you are experimenting in the right direction at a high level. However, implementing it on the chaincode side is not going to be efficient and ideally be implemented on the Fabric side. Since beginning, we have similar ideas to build a query engine on top, albeit on the Fabric side - however, this did not reach in the top of the priority list.

 

Third, on the performance behavior that you are observing - one of the main reason of this observation is that the random access is significantly costly in the terms of I/O. I am not sure what storage are you using but this would be magnified if you are using rotating hard disks as opposed to SSD. Similarly, this is going to be magnified if you measure this on the chaincode side as opposed to when implemented on the Fabric side. In general, this is a well understood behavior in traditional and mature databases that using an index is not always helpful and at times the using the index could in fact be more costly than scanning the whole data and filter what is not needed. For instance, if you employ a rotating disk and the data items that you need to retrieve is uniformly spread across in the entire range, and the data items that satisfy the query are more than 10% of the total data, you would perhaps be better off scanning full data than using the index. Most of the query engines employ a sophisticated cost model that estimates the cost of various potential query execution plans and select the one that appears to be the least expensive for the given query in the context of the data present. Most of the cost models maintains some metadata to derive some of these heuristics.

 

Thanks,

Manish

 

 

 

 

 

On Sun, Apr 5, 2020 at 4:02 PM Prasanth Sundaravelu <prasanths96@...> wrote:

Hi guys,

I have been recently developing custom rich query processor & indexing mechanism at chaincode level to exploit goleveldb's performance. I have designed this on the assumption that getting series of keys by GetStateByRange() and regular GetState() will be the same, but I could see now, that the range query is much faster. It makes sense as range query will read the storage in a sequential way and GetState() has to jump to different memory locations to fetch (Correct me if I am wrong). 

I have partially implemented it to be able to test the underlying design assumptions and I am in trouble now. 
The logic of the query operation is like this: 
1. Based on the query, make a range query on the relevant indexes.
2. After retrieving indexes, fetch the actual data pointed by the index using GetState().

This works charmingly well, if the number of matching records is less, but horrible if the record count is high:

- For a response with 1 record, it takes under ~30ms. 
- For 100 records: ~100ms
- For 1000 records: ~1sec! (See the trouble?)

I can see that the bottleneck is at the GetState(). I would like to have some pointers to resolve this issue. 
I have tried the following: 
- In range query, fetched a single iterator, synchronized it and fed it to multiple go routines. (Which doesn't improve the performance of range query at all, since it's still reading one at a time)
- Tried to get multiple iterators, but fabric threw error if I do that.
- Tried to run GetState() on multiple go routines. Still no improvement. 

I have the following questions:
1. Is the fabric running chaincode itself as a whole already as go routines? If so, does it help chaincode functions by default?
2. If having Go Routines in chaincode help at all? 
3. How can I make GetState() faster?

Pardon my long message. 

If you're interested in this project, you can help me implementing this by contributing: https://github.com/prasanths96/iState
This is my first open project, I would really appreciate comments and critiques that would positively impact the project. Suggestions are welcome.
More about the project: 
- Supports json like rich query in leveldb. (As the couch DB's performance is not "good enough")
- It aims to be a work around that can be easily used by any chaincode developer.
- In-memory cached data is going to be a feature that would increase the query performance to atleast 10 folds. 
- All this delivered as an interface to developers having simple CRUD functions.

- Prasanth

This message has been sent by ABN AMRO Bank N.V., which has its seat at Gustav Mahlerlaan 10 (1082 PP) Amsterdam, the Netherlands, and is registered in the Commercial Register of Amsterdam under number 34334259.


Re: Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

Manish
 

Hi Prasanth,

Specifically, about your last point -

First, if you did not notice much difference when you use HDD v/s in-memory volume, then I would suspect for one of the following two things 
    (A) Perhaps you may be testing with small amount of data that perhaps is being served from the filesystem cache instead of hitting the disk 
    (B) the communication between peer and chaincode container is more subtle bottleneck here. For a single key lookup, you would have this overhead multiple times (akin to random access overhead) and in the case of a range query, the data is transferred between peer and chaincode in a batch of 100, so this overhead is amortized (akin to a sequential scan).

On the caching part, yes, there should not be a problem in caching the data on the chaincode side. However, you would need to consider if the values are not too big, there is no value in maintaining the hashes in the index and then matching over with the cache. You can maintain directly the values in your index (as was suggested by Dave in his reply). I am not certain that the container will be stopped but in any case, Fabric now supports custom runtime environment that you may have better control over.

Hope this helps

Thanks,
Manish

On Tue, Apr 7, 2020 at 4:58 AM Prasanth Sundaravelu <prasanths96@...> wrote:
@Gari,
There will be multiple index keys created for a single object. In fact, sometimes even multiple keys for single field which has multiple values. So, It will be highly redundant if I store data directly in those keys, hence using a pointing mechanism for the same.

@ Manish, 
Thanks for the response. 

1. That makes sense.
2. I agree, doing it in chaincode is not the best way, but it think it is the simpler way for still gaining some significant performance. Especially, for a frequently used casual query, which would return very less data. Earlier, I have been using composite keys with fields ordered in their significance of query necessity(Eg: Doctype at first always, Id will be second) and had a query processor at chaincode itself, that would take fetch all the records with the given docType and parse, in case the fields that you want to query is discontinued when forming as composite keys (Eg: doctype: doc1, Id:" ", count: 1000).
It also helps for relational queries. 
The main problem with this is that, when doing the above mentioned type query, there will be no best case, average case, All such queries run on worst case and directly proportional to the number of records in that docType. (eg: For 100k records, it takes 6 seconds to return response, no matter if it will return a single result or all 100k records as result).
Then, I also compared couch Db's performance with the same data sets, for un-indexed queries‌, it was slower than leveldb. Indexed queries were faster as long as it did not return like all of 100k records, and writes were far far slower than leveldb. Hence, this project was thought out, which could be a helpful and easy workaround until fabric gets this.

3. Yes, I am using a 7200RPM hdd. I agree it has got to be slower. I have also casually tested mounting a volume directly in memory and using it as the mount folder for /var/production. I did not see a significant improvement. I did not check it thoroughly though, it could also be bottleneck in my code. 
I am also storing meta data similar to cost model as you mentioned, which is used to choose the query field which would have least number of records and then doing further processing at code. Thanks for mentioning this, I think its a good idea to skip indexing altogether in some cases.

The next step for me is to build proper cache for the data in memory. I've implemented a simple one and tested it out, it has 40% improvement in the worst case scenario when compared to the old composite key implementation. 
To maintain reliability of the cached data: 
1. Will stored hash of actual data in index keys.
2. Will always fetch index keys from db, and verify if the hash in it and hash of data in cache are same. 
If same, then no change has occurred to data and will use the same.
Else, I will fetch the latest data from db. This ensures consistency.

I have questions in caching. 
1. I have read that in old version of fabric, chaincode container is stopped when idling. Is it still the case in 1.3+ versions? 

2. Is it ok to save decrypted data in memory?

- Prasanth


On Mon, 6 Apr 2020, 8:10 pm Manish Sethi, <manish.sethi@...> wrote:
Hello Prasanth,
Aw
My response to your questions would be three fold.

First, a chaincode is expected to be a sequential logic. Otherwise, providing a guarantee of deterministic behavior is very hard. If you go into theoretical aspects, the basic definition of a transaction is "a sequence of database operations….” - a logical sequence is key here. There are some mature systems that support nested transactions that could be executed in parallel but to keep the chaincode logic simple and deterministic, Fabric supports it in the form of a single sequential flow.

Second, for complex query support, you are experimenting in the right direction at a high level. However, implementing it on the chaincode side is not going to be efficient and ideally be implemented on the Fabric side. Since beginning, we have similar ideas to build a query engine on top, albeit on the Fabric side - however, this did not reach in the top of the priority list.

Third, on the performance behavior that you are observing - one of the main reason of this observation is that the random access is significantly costly in the terms of I/O. I am not sure what storage are you using but this would be magnified if you are using rotating hard disks as opposed to SSD. Similarly, this is going to be magnified if you measure this on the chaincode side as opposed to when implemented on the Fabric side. In general, this is a well understood behavior in traditional and mature databases that using an index is not always helpful and at times the using the index could in fact be more costly than scanning the whole data and filter what is not needed. For instance, if you employ a rotating disk and the data items that you need to retrieve is uniformly spread across in the entire range, and the data items that satisfy the query are more than 10% of the total data, you would perhaps be better off scanning full data than using the index. Most of the query engines employ a sophisticated cost model that estimates the cost of various potential query execution plans and select the one that appears to be the least expensive for the given query in the context of the data present. Most of the cost models maintains some metadata to derive some of these heuristics.

Thanks,
Manish





On Sun, Apr 5, 2020 at 4:02 PM Prasanth Sundaravelu <prasanths96@...> wrote:

Hi guys,

I have been recently developing custom rich query processor & indexing mechanism at chaincode level to exploit goleveldb's performance. I have designed this on the assumption that getting series of keys by GetStateByRange() and regular GetState() will be the same, but I could see now, that the range query is much faster. It makes sense as range query will read the storage in a sequential way and GetState() has to jump to different memory locations to fetch (Correct me if I am wrong). 

I have partially implemented it to be able to test the underlying design assumptions and I am in trouble now. 
The logic of the query operation is like this: 
1. Based on the query, make a range query on the relevant indexes.
2. After retrieving indexes, fetch the actual data pointed by the index using GetState().

This works charmingly well, if the number of matching records is less, but horrible if the record count is high:

- For a response with 1 record, it takes under ~30ms. 
- For 100 records: ~100ms
- For 1000 records: ~1sec! (See the trouble?)

I can see that the bottleneck is at the GetState(). I would like to have some pointers to resolve this issue. 
I have tried the following: 
- In range query, fetched a single iterator, synchronized it and fed it to multiple go routines. (Which doesn't improve the performance of range query at all, since it's still reading one at a time)
- Tried to get multiple iterators, but fabric threw error if I do that.
- Tried to run GetState() on multiple go routines. Still no improvement. 

I have the following questions:
1. Is the fabric running chaincode itself as a whole already as go routines? If so, does it help chaincode functions by default?
2. If having Go Routines in chaincode help at all? 
3. How can I make GetState() faster?

Pardon my long message. 

If you're interested in this project, you can help me implementing this by contributing: https://github.com/prasanths96/iState
This is my first open project, I would really appreciate comments and critiques that would positively impact the project. Suggestions are welcome.
More about the project: 
- Supports json like rich query in leveldb. (As the couch DB's performance is not "good enough")
- It aims to be a work around that can be easily used by any chaincode developer.
- In-memory cached data is going to be a feature that would increase the query performance to atleast 10 folds. 
- All this delivered as an interface to developers having simple CRUD functions.

- Prasanth


Re: Install your own chaincode

Nikhil Gupta
 

Which version of Fabric are you using? If you are using 2.0+, use the test network instead: https://hyperledger-fabric.readthedocs.io/en/release-2.0/test_network.html

You can then use this tutorial to deploy your own CC the channel: https://hyperledger-fabric.readthedocs.io/en/release-2.0/deploy_chaincode.html



-----fabric@... wrote: -----
To: fabric@...
From: "music prime"
Sent by: fabric@...
Date: 04/07/2020 02:06PM
Subject: [EXTERNAL] [Hyperledger Fabric] Install your own chaincode

I have to install my own chaincode on byfn.
So where I need make the changes.
Can somebody help me. 


Re: Fabric 2.0 - Commit Chaincode

Nicholas Leonardi
 

That was it. Weird because I copied the configtx.yaml to fit my org but apparently 
it got erased at some point. 
Thanks a lot 

Em terça-feira, 7 de abril de 2020 15:04:02 BRT, Nikhil E Gupta <negupta@...> escreveu:


Hey,

Does your organization have the endorsement sub policy?

Endorsement:
Type: Signature
Rule: "OR('Org1MSP.peer')"


-----fabric@... wrote: -----
To: Fabric <fabric@...>
From: "Nicholas Leonardi via lists.hyperledger.org"
Sent by: fabric@...
Date: 04/07/2020 01:53PM
Subject: [EXTERNAL] [Hyperledger Fabric] Fabric 2.0 - Commit Chaincode

Hey guys,
I have a simple network with 1 org, 3 orderers and 1 peer.
I can't seem to commit the chaincode.
I'm able to package, install and approveForMyOrg. I checked
the commit readiness and it is showing as true for my org

"approvals": {
"Org1MSP": true
}

Only the commit isn't working. I checked everything, the MSP envs,
paths to the certs, --peerAddresses, fabric-cfg-path env

I keep getting ENDORSEMENT_POLICY_FAILURE.

The peer logs gives this error:

VSCC error: stateBasedValidator.Validate failed, err validation of endorsement policy for chaincode _lifecycle in tx 3:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Endorsement' sub-policies to be satisfied
 
The configtx ApplicationDefaults are correct. Also, the configtx.yaml was copied from the fabric samples so everything is there.

LifecycleEndorsement:
Type: ImplicitMeta
Rule: "MAJORITY Endorsement"
Endorsement:
Type: ImplicitMeta
Rule: "MAJORITY Endorsement"

Since there is only one org, it is the majority. I've tried everything, nothing seems to work. Can anyone
shed some light?

Thanks in advance. 

Below are the commands I'm using to install cc

peer lifecycle chaincode package chaincode.tar.gz --path /etc/hyperledger/chaincode --lang node --label ccv1

lifecycle chaincode install chaincode.tar.gz

peer lifecycle chaincode approveformyorg -o orderer1.org1.com:7050 --channelID channel --name chaincode --version 1.0 --init-required --sequence 1 --tls --cafile /etc/hyperledger/crypto-config/ordererOrganizations/orderers/orderer1.org1.com/tls/ca.crt --package-id myccv1:f997652c8722e223ccc7453128d5d8db797e9a1abf712863a255c8338530ed6d

peer lifecycle chaincode commit -o orderer1.org1.com:7050 --channelID channel --name chaincode --version 1.0 --sequence 1 --init-required --tls true --cafile /etc/hyperledger/crypto-config/ordererOrganizations/orderers/orderer1.org1.com/tls/ca.crt --peerAddresses peer.org1.com:7051 --tlsRootCertFiles /etc/hyperledger/crypto-config/peerOrganizations/peers/peer.org1.com/tls/ca.crt



Install your own chaincode

music prime
 

I have to install my own chaincode on byfn.
So where I need make the changes.
Can somebody help me. 


Re: Fabric 2.0 - Commit Chaincode

Nikhil Gupta
 

Hey,

Does your organization have the endorsement sub policy?

Endorsement:
Type: Signature
Rule: "OR('Org1MSP.peer')"


-----fabric@... wrote: -----
To: Fabric <fabric@...>
From: "Nicholas Leonardi via lists.hyperledger.org"
Sent by: fabric@...
Date: 04/07/2020 01:53PM
Subject: [EXTERNAL] [Hyperledger Fabric] Fabric 2.0 - Commit Chaincode

Hey guys,
I have a simple network with 1 org, 3 orderers and 1 peer.
I can't seem to commit the chaincode.
I'm able to package, install and approveForMyOrg. I checked
the commit readiness and it is showing as true for my org

"approvals": {
"Org1MSP": true
}

Only the commit isn't working. I checked everything, the MSP envs,
paths to the certs, --peerAddresses, fabric-cfg-path env

I keep getting ENDORSEMENT_POLICY_FAILURE.

The peer logs gives this error:

VSCC error: stateBasedValidator.Validate failed, err validation of endorsement policy for chaincode _lifecycle in tx 3:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Endorsement' sub-policies to be satisfied
 
The configtx ApplicationDefaults are correct. Also, the configtx.yaml was copied from the fabric samples so everything is there.

LifecycleEndorsement:
Type: ImplicitMeta
Rule: "MAJORITY Endorsement"
Endorsement:
Type: ImplicitMeta
Rule: "MAJORITY Endorsement"

Since there is only one org, it is the majority. I've tried everything, nothing seems to work. Can anyone
shed some light?

Thanks in advance. 

Below are the commands I'm using to install cc

peer lifecycle chaincode package chaincode.tar.gz --path /etc/hyperledger/chaincode --lang node --label ccv1

lifecycle chaincode install chaincode.tar.gz

peer lifecycle chaincode approveformyorg -o orderer1.org1.com:7050 --channelID channel --name chaincode --version 1.0 --init-required --sequence 1 --tls --cafile /etc/hyperledger/crypto-config/ordererOrganizations/orderers/orderer1.org1.com/tls/ca.crt --package-id myccv1:f997652c8722e223ccc7453128d5d8db797e9a1abf712863a255c8338530ed6d

peer lifecycle chaincode commit -o orderer1.org1.com:7050 --channelID channel --name chaincode --version 1.0 --sequence 1 --init-required --tls true --cafile /etc/hyperledger/crypto-config/ordererOrganizations/orderers/orderer1.org1.com/tls/ca.crt --peerAddresses peer.org1.com:7051 --tlsRootCertFiles /etc/hyperledger/crypto-config/peerOrganizations/peers/peer.org1.com/tls/ca.crt



Fabric 2.0 - Commit Chaincode

Nicholas Leonardi
 

Hey guys,
I have a simple network with 1 org, 3 orderers and 1 peer.
I can't seem to commit the chaincode.
I'm able to package, install and approveForMyOrg. I checked
the commit readiness and it is showing as true for my org

"approvals": {
"Org1MSP": true
}

Only the commit isn't working. I checked everything, the MSP envs,
paths to the certs, --peerAddresses, fabric-cfg-path env

I keep getting ENDORSEMENT_POLICY_FAILURE.

The peer logs gives this error:

VSCC error: stateBasedValidator.Validate failed, err validation of endorsement policy for chaincode _lifecycle in tx 3:0 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Endorsement' sub-policies to be satisfied
 
The configtx ApplicationDefaults are correct. Also, the configtx.yaml was copied from the fabric samples so everything is there.

LifecycleEndorsement:
Type: ImplicitMeta
Rule: "MAJORITY Endorsement"
Endorsement:
Type: ImplicitMeta
Rule: "MAJORITY Endorsement"

Since there is only one org, it is the majority. I've tried everything, nothing seems to work. Can anyone
shed some light?

Thanks in advance. 

Below are the commands I'm using to install cc

peer lifecycle chaincode package chaincode.tar.gz --path /etc/hyperledger/chaincode --lang node --label ccv1

lifecycle chaincode install chaincode.tar.gz

peer lifecycle chaincode approveformyorg -o orderer1.org1.com:7050 --channelID channel --name chaincode --version 1.0 --init-required --sequence 1 --tls --cafile /etc/hyperledger/crypto-config/ordererOrganizations/orderers/orderer1.org1.com/tls/ca.crt --package-id myccv1:f997652c8722e223ccc7453128d5d8db797e9a1abf712863a255c8338530ed6d

peer lifecycle chaincode commit -o orderer1.org1.com:7050 --channelID channel --name chaincode --version 1.0 --sequence 1 --init-required --tls true --cafile /etc/hyperledger/crypto-config/ordererOrganizations/orderers/orderer1.org1.com/tls/ca.crt --peerAddresses peer.org1.com:7051 --tlsRootCertFiles /etc/hyperledger/crypto-config/peerOrganizations/peers/peer.org1.com/tls/ca.crt


Re: Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

Prasanth Sundaravelu
 

@Senthilnathan N 
Thanks for info! I guess I have no trouble with range queries here.
1. In that case, first, I can just give an option whether or not to store data in indexes, then can code it more smartly to make things efficient and automated.
2. Absolutely, that would be the later step and will be out of scope for this project. If this package is improved to its utmost efficiency, any improvement outside it will have an even greater impact right?

- Prasanth

On Mon, Apr 6, 2020 at 8:46 PM David Enyeart <enyeart@...> wrote:

I'll just add two more points...

First, although it would increase the size of your chaincode created 'indexes', you could choose to include the base data that you need to retrieve in the index key's value, so that the range scan results can be used directly rather than having to do an additional GetState query. In the database world, this is called an INCLUDE in the index, as explained from PostreSQL as follows:
"The optional INCLUDE clause specifies a list of columns which will be included in the index as non-key columns. A non-key column cannot be used in an index scan search qualification, and it is disregarded for purposes of any uniqueness or exclusion constraint enforced by the index. However, an index-only scan can return the contents of non-key columns without having to visit the index's table, since they are available directly from the index entry. Thus, addition of non-key columns allows index-only scans to be used for queries that otherwise could not use them."
Of course, this may imply that you store the same base data in multiple 'indexes', so you'll have to balance the considerations.

Second, you can always decide to load balance queries across a number of peers, if you can find a way to split the query into multiple queries from the client perspective.


Dave Enyeart

"Manish" ---04/06/2020 10:56:26 AM---I would further like to elaborate on my first point as it may cause confusion in the context of Fabr

From: "Manish" <manish.sethi@...>
To: Prasanth Sundaravelu <prasanths96@...>
Cc: fabric@...
Date: 04/06/2020 10:56 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions
Sent by: fabric@...





I would further like to elaborate on my first point as it may cause confusion in the context of Fabric….
Like most of the other databases, Fabric 0.6 transaction semantics were to support read-your-own-writes (RYOW). With these semantics, it matters whether in the sequence of transaction, you performed a read or a write first. The query results may differ based on the sequence of operations.

However, with Fabric 1.0, we did not support RYOW and hence, there is no theoretical bottleneck to support parallel queries as such.

On Mon, Apr 6, 2020 at 10:40 AM Manish Sethi <manish.sethi@...> wrote:
    Hello Prasanth,

    My response to your questions would be three fold.

    First, a chaincode is expected to be a sequential logic. Otherwise, providing a guarantee of deterministic behavior is very hard. If you go into theoretical aspects, the basic definition of a transaction is "a sequence of database operations….” - a logical sequence is key here. There are some mature systems that support nested transactions that could be executed in parallel but to keep the chaincode logic simple and deterministic, Fabric supports it in the form of a single sequential flow.

    Second, for complex query support, you are experimenting in the right direction at a high level. However, implementing it on the chaincode side is not going to be efficient and ideally be implemented on the Fabric side. Since beginning, we have similar ideas to build a query engine on top, albeit on the Fabric side - however, this did not reach in the top of the priority list.

    Third, on the performance behavior that you are observing - one of the main reason of this observation is that the random access is significantly costly in the terms of I/O. I am not sure what storage are you using but this would be magnified if you are using rotating hard disks as opposed to SSD. Similarly, this is going to be magnified if you measure this on the chaincode side as opposed to when implemented on the Fabric side. In general, this is a well understood behavior in traditional and mature databases that using an index is not always helpful and at times the using the index could in fact be more costly than scanning the whole data and filter what is not needed. For instance, if you employ a rotating disk and the data items that you need to retrieve is uniformly spread across in the entire range, and the data items that satisfy the query are more than 10% of the total data, you would perhaps be better off scanning full data than using the index. Most of the query engines employ a sophisticated cost model that estimates the cost of various potential query execution plans and select the one that appears to be the least expensive for the given query in the context of the data present. Most of the cost models maintains some metadata to derive some of these heuristics.

    Thanks,
    Manish





    On Sun, Apr 5, 2020 at 4:02 PM Prasanth Sundaravelu <prasanths96@...> wrote:
    Hi guys,

    I have been recently developing custom rich query processor & indexing mechanism at chaincode level to exploit goleveldb's performance. I have designed this on the assumption that getting series of keys by GetStateByRange() and regular GetState() will be the same, but I could see now, that the range query is much faster. It makes sense as range query will read the storage in a sequential way and GetState() has to jump to different memory locations to fetch (Correct me if I am wrong). 

    I have partially implemented it to be able to test the underlying design assumptions and I am in trouble now. 
    The logic of the query operation is like this: 
    1. Based on the query, make a range query on the relevant indexes.
    2. After retrieving indexes, fetch the actual data pointed by the index using GetState().

    This works charmingly well, if the number of matching records is less, but horrible if the record count is high:

    - For a response with 1 record, it takes under ~30ms. 
    - For 100 records: ~100ms
    - For 1000 records: ~1sec! (See the trouble?)

    I can see that the bottleneck is at the GetState(). I would like to have some pointers to resolve this issue. 
    I have tried the following: 
    - In range query, fetched a single iterator, synchronized it and fed it to multiple go routines. (Which doesn't improve the performance of range query at all, since it's still reading one at a time)
    - Tried to get multiple iterators, but fabric threw error if I do that.
    - Tried to run GetState() on multiple go routines. Still no improvement. 

    I have the following questions:
    1. Is the fabric running chaincode itself as a whole already as go routines? If so, does it help chaincode functions by default?
    2. If having Go Routines in chaincode help at all? 
    3. How can I make GetState() faster?

    Pardon my long message. 

    If you're interested in this project, you can help me implementing this by contributing: https://github.com/prasanths96/iState
    This is my first open project, I would really appreciate comments and critiques that would positively impact the project. Suggestions are welcome.
    More about the project: 
    - Supports json like rich query in leveldb. (As the couch DB's performance is not "good enough")
    - It aims to be a work around that can be easily used by any chaincode developer.
    - In-memory cached data is going to be a feature that would increase the query performance to atleast 10 folds. 
    - All this delivered as an interface to developers having simple CRUD functions.

    - Prasanth






Re: Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

Prasanth Sundaravelu
 

@Gari,
There will be multiple index keys created for a single object. In fact, sometimes even multiple keys for single field which has multiple values. So, It will be highly redundant if I store data directly in those keys, hence using a pointing mechanism for the same.

@ Manish, 
Thanks for the response. 

1. That makes sense.
2. I agree, doing it in chaincode is not the best way, but it think it is the simpler way for still gaining some significant performance. Especially, for a frequently used casual query, which would return very less data. Earlier, I have been using composite keys with fields ordered in their significance of query necessity(Eg: Doctype at first always, Id will be second) and had a query processor at chaincode itself, that would take fetch all the records with the given docType and parse, in case the fields that you want to query is discontinued when forming as composite keys (Eg: doctype: doc1, Id:" ", count: 1000).
It also helps for relational queries. 
The main problem with this is that, when doing the above mentioned type query, there will be no best case, average case, All such queries run on worst case and directly proportional to the number of records in that docType. (eg: For 100k records, it takes 6 seconds to return response, no matter if it will return a single result or all 100k records as result).
Then, I also compared couch Db's performance with the same data sets, for un-indexed queries‌, it was slower than leveldb. Indexed queries were faster as long as it did not return like all of 100k records, and writes were far far slower than leveldb. Hence, this project was thought out, which could be a helpful and easy workaround until fabric gets this.

3. Yes, I am using a 7200RPM hdd. I agree it has got to be slower. I have also casually tested mounting a volume directly in memory and using it as the mount folder for /var/production. I did not see a significant improvement. I did not check it thoroughly though, it could also be bottleneck in my code. 
I am also storing meta data similar to cost model as you mentioned, which is used to choose the query field which would have least number of records and then doing further processing at code. Thanks for mentioning this, I think its a good idea to skip indexing altogether in some cases.

The next step for me is to build proper cache for the data in memory. I've implemented a simple one and tested it out, it has 40% improvement in the worst case scenario when compared to the old composite key implementation. 
To maintain reliability of the cached data: 
1. Will stored hash of actual data in index keys.
2. Will always fetch index keys from db, and verify if the hash in it and hash of data in cache are same. 
If same, then no change has occurred to data and will use the same.
Else, I will fetch the latest data from db. This ensures consistency.

I have questions in caching. 
1. I have read that in old version of fabric, chaincode container is stopped when idling. Is it still the case in 1.3+ versions? 

2. Is it ok to save decrypted data in memory?

- Prasanth


On Mon, 6 Apr 2020, 8:10 pm Manish Sethi, <manish.sethi@...> wrote:
Hello Prasanth,
Aw
My response to your questions would be three fold.

First, a chaincode is expected to be a sequential logic. Otherwise, providing a guarantee of deterministic behavior is very hard. If you go into theoretical aspects, the basic definition of a transaction is "a sequence of database operations….” - a logical sequence is key here. There are some mature systems that support nested transactions that could be executed in parallel but to keep the chaincode logic simple and deterministic, Fabric supports it in the form of a single sequential flow.

Second, for complex query support, you are experimenting in the right direction at a high level. However, implementing it on the chaincode side is not going to be efficient and ideally be implemented on the Fabric side. Since beginning, we have similar ideas to build a query engine on top, albeit on the Fabric side - however, this did not reach in the top of the priority list.

Third, on the performance behavior that you are observing - one of the main reason of this observation is that the random access is significantly costly in the terms of I/O. I am not sure what storage are you using but this would be magnified if you are using rotating hard disks as opposed to SSD. Similarly, this is going to be magnified if you measure this on the chaincode side as opposed to when implemented on the Fabric side. In general, this is a well understood behavior in traditional and mature databases that using an index is not always helpful and at times the using the index could in fact be more costly than scanning the whole data and filter what is not needed. For instance, if you employ a rotating disk and the data items that you need to retrieve is uniformly spread across in the entire range, and the data items that satisfy the query are more than 10% of the total data, you would perhaps be better off scanning full data than using the index. Most of the query engines employ a sophisticated cost model that estimates the cost of various potential query execution plans and select the one that appears to be the least expensive for the given query in the context of the data present. Most of the cost models maintains some metadata to derive some of these heuristics.

Thanks,
Manish





On Sun, Apr 5, 2020 at 4:02 PM Prasanth Sundaravelu <prasanths96@...> wrote:

Hi guys,

I have been recently developing custom rich query processor & indexing mechanism at chaincode level to exploit goleveldb's performance. I have designed this on the assumption that getting series of keys by GetStateByRange() and regular GetState() will be the same, but I could see now, that the range query is much faster. It makes sense as range query will read the storage in a sequential way and GetState() has to jump to different memory locations to fetch (Correct me if I am wrong). 

I have partially implemented it to be able to test the underlying design assumptions and I am in trouble now. 
The logic of the query operation is like this: 
1. Based on the query, make a range query on the relevant indexes.
2. After retrieving indexes, fetch the actual data pointed by the index using GetState().

This works charmingly well, if the number of matching records is less, but horrible if the record count is high:

- For a response with 1 record, it takes under ~30ms. 
- For 100 records: ~100ms
- For 1000 records: ~1sec! (See the trouble?)

I can see that the bottleneck is at the GetState(). I would like to have some pointers to resolve this issue. 
I have tried the following: 
- In range query, fetched a single iterator, synchronized it and fed it to multiple go routines. (Which doesn't improve the performance of range query at all, since it's still reading one at a time)
- Tried to get multiple iterators, but fabric threw error if I do that.
- Tried to run GetState() on multiple go routines. Still no improvement. 

I have the following questions:
1. Is the fabric running chaincode itself as a whole already as go routines? If so, does it help chaincode functions by default?
2. If having Go Routines in chaincode help at all? 
3. How can I make GetState() faster?

Pardon my long message. 

If you're interested in this project, you can help me implementing this by contributing: https://github.com/prasanths96/iState
This is my first open project, I would really appreciate comments and critiques that would positively impact the project. Suggestions are welcome.
More about the project: 
- Supports json like rich query in leveldb. (As the couch DB's performance is not "good enough")
- It aims to be a work around that can be easily used by any chaincode developer.
- In-memory cached data is going to be a feature that would increase the query performance to atleast 10 folds. 
- All this delivered as an interface to developers having simple CRUD functions.

- Prasanth


Fabric GO SDK Enroll & Register user

Antoni Massó Mola <antonimassomola@...>
 

Hello,

I'm trying to enroll & register new users with the Fabric GO SDK. 

This is my CA docker container configuration:

ca-root:
container_name: ca-root
image: hyperledger/fabric-ca:1.4.3
environment:
- FABRIC_CA_SERVER_HOME=/etc/hyperledger/fabric-ca-server
- ROOT_HOST=ca-root
- ROOT_PORT=7054
- ROOT_USERNAME=admin
- ROOT_PASSWORD=adminpw
volumes:
- ./state/ca-root:/etc/hyperledger/fabric-ca-server
- ./scripts:/scripts
command: sh /scripts/start-root.sh
ports:
- 7054:7054
networks:
- default.svc.cluster.local

The CA server is up & running.

This is my GO code:

mspClient, err := mspclient.New(sdk.Context(), mspclient.WithOrg(org), mspclient.WithCAInstance("ca-root:7054"))
if err != nil {
log.Println(err)
}

This line execution fails with the following error:

020/04/06 15:52:17 non-existent CA: 'ca-root:7054'
panic: runtime error: invalid memory address or nil pointer dereference

Any idea what I'm doing wrong?

Thank you


Re: Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

David Enyeart
 

I'll just add two more points...

First, although it would increase the size of your chaincode created 'indexes', you could choose to include the base data that you need to retrieve in the index key's value, so that the range scan results can be used directly rather than having to do an additional GetState query. In the database world, this is called an INCLUDE in the index, as explained from PostreSQL as follows:
"The optional INCLUDE clause specifies a list of columns which will be included in the index as non-key columns. A non-key column cannot be used in an index scan search qualification, and it is disregarded for purposes of any uniqueness or exclusion constraint enforced by the index. However, an index-only scan can return the contents of non-key columns without having to visit the index's table, since they are available directly from the index entry. Thus, addition of non-key columns allows index-only scans to be used for queries that otherwise could not use them."
Of course, this may imply that you store the same base data in multiple 'indexes', so you'll have to balance the considerations.

Second, you can always decide to load balance queries across a number of peers, if you can find a way to split the query into multiple queries from the client perspective.


Dave Enyeart

"Manish" ---04/06/2020 10:56:26 AM---I would further like to elaborate on my first point as it may cause confusion in the context of Fabr

From: "Manish" <manish.sethi@...>
To: Prasanth Sundaravelu <prasanths96@...>
Cc: fabric@...
Date: 04/06/2020 10:56 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions
Sent by: fabric@...





I would further like to elaborate on my first point as it may cause confusion in the context of Fabric….
Like most of the other databases, Fabric 0.6 transaction semantics were to support read-your-own-writes (RYOW). With these semantics, it matters whether in the sequence of transaction, you performed a read or a write first. The query results may differ based on the sequence of operations.

However, with Fabric 1.0, we did not support RYOW and hence, there is no theoretical bottleneck to support parallel queries as such.

On Mon, Apr 6, 2020 at 10:40 AM Manish Sethi <manish.sethi@...> wrote:
    Hello Prasanth,

    My response to your questions would be three fold.

    First, a chaincode is expected to be a sequential logic. Otherwise, providing a guarantee of deterministic behavior is very hard. If you go into theoretical aspects, the basic definition of a transaction is "a sequence of database operations….” - a logical sequence is key here. There are some mature systems that support nested transactions that could be executed in parallel but to keep the chaincode logic simple and deterministic, Fabric supports it in the form of a single sequential flow.

    Second, for complex query support, you are experimenting in the right direction at a high level. However, implementing it on the chaincode side is not going to be efficient and ideally be implemented on the Fabric side. Since beginning, we have similar ideas to build a query engine on top, albeit on the Fabric side - however, this did not reach in the top of the priority list.

    Third, on the performance behavior that you are observing - one of the main reason of this observation is that the random access is significantly costly in the terms of I/O. I am not sure what storage are you using but this would be magnified if you are using rotating hard disks as opposed to SSD. Similarly, this is going to be magnified if you measure this on the chaincode side as opposed to when implemented on the Fabric side. In general, this is a well understood behavior in traditional and mature databases that using an index is not always helpful and at times the using the index could in fact be more costly than scanning the whole data and filter what is not needed. For instance, if you employ a rotating disk and the data items that you need to retrieve is uniformly spread across in the entire range, and the data items that satisfy the query are more than 10% of the total data, you would perhaps be better off scanning full data than using the index. Most of the query engines employ a sophisticated cost model that estimates the cost of various potential query execution plans and select the one that appears to be the least expensive for the given query in the context of the data present. Most of the cost models maintains some metadata to derive some of these heuristics.

    Thanks,
    Manish





On Sun, Apr 5, 2020 at 4:02 PM Prasanth Sundaravelu <prasanths96@...> wrote:
Hi guys,

I have been recently developing custom rich query processor & indexing mechanism at chaincode level to exploit goleveldb's performance. I have designed this on the assumption that getting series of keys by GetStateByRange() and regular GetState() will be the same, but I could see now, that the range query is much faster. It makes sense as range query will read the storage in a sequential way and GetState() has to jump to different memory locations to fetch (Correct me if I am wrong). 

I have partially implemented it to be able to test the underlying design assumptions and I am in trouble now. 
The logic of the query operation is like this: 
1. Based on the query, make a range query on the relevant indexes.
2. After retrieving indexes, fetch the actual data pointed by the index using GetState().

This works charmingly well, if the number of matching records is less, but horrible if the record count is high:

- For a response with 1 record, it takes under ~30ms. 
- For 100 records: ~100ms
- For 1000 records: ~1sec! (See the trouble?)

I can see that the bottleneck is at the GetState(). I would like to have some pointers to resolve this issue. 
I have tried the following: 
- In range query, fetched a single iterator, synchronized it and fed it to multiple go routines. (Which doesn't improve the performance of range query at all, since it's still reading one at a time)
- Tried to get multiple iterators, but fabric threw error if I do that.
- Tried to run GetState() on multiple go routines. Still no improvement. 

I have the following questions:
1. Is the fabric running chaincode itself as a whole already as go routines? If so, does it help chaincode functions by default?
2. If having Go Routines in chaincode help at all? 
3. How can I make GetState() faster?

Pardon my long message. 

If you're interested in this project, you can help me implementing this by contributing: https://github.com/prasanths96/iState
This is my first open project, I would really appreciate comments and critiques that would positively impact the project. Suggestions are welcome.
More about the project: 
- Supports json like rich query in leveldb. (As the couch DB's performance is not "good enough")
- It aims to be a work around that can be easily used by any chaincode developer.
- In-memory cached data is going to be a feature that would increase the query performance to atleast 10 folds. 
- All this delivered as an interface to developers having simple CRUD functions.

- Prasanth






Re: Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

Senthil Nathan
 

Range query performance is dependent on the number of SSTables the goleveldb has at a given point of time and how the active states are distributed over these SSTables. There is an index per SSTable. If the workload is append-only, even then there would be some random IO (as there is an index per SSTable). If it is not an append-only but changes existing keys, the range query would result in a high random IO. The SSTable/LSM compaction merging these SSTable periodically (by removing non-active states) to avoid random IOs. Note that this compaction operation also creates a performance bottleneck. Still, the range query can be faster than GetState() depending on how IOs are scheduled/prioritized.

Regards,
Senthil

On Mon, Apr 6, 2020 at 8:26 PM Manish <manish.sethi@...> wrote:
I would further like to elaborate on my first point as it may cause confusion in the context of Fabric….
Like most of the other databases, Fabric 0.6 transaction semantics were to support read-your-own-writes (RYOW). With these semantics, it matters whether in the sequence of transaction, you performed a read or a write first. The query results may differ based on the sequence of operations.

However, with Fabric 1.0, we did not support RYOW and hence, there is no theoretical bottleneck to support parallel queries as such.

On Mon, Apr 6, 2020 at 10:40 AM Manish Sethi <manish.sethi@...> wrote:
Hello Prasanth,

My response to your questions would be three fold.

First, a chaincode is expected to be a sequential logic. Otherwise, providing a guarantee of deterministic behavior is very hard. If you go into theoretical aspects, the basic definition of a transaction is "a sequence of database operations….” - a logical sequence is key here. There are some mature systems that support nested transactions that could be executed in parallel but to keep the chaincode logic simple and deterministic, Fabric supports it in the form of a single sequential flow.

Second, for complex query support, you are experimenting in the right direction at a high level. However, implementing it on the chaincode side is not going to be efficient and ideally be implemented on the Fabric side. Since beginning, we have similar ideas to build a query engine on top, albeit on the Fabric side - however, this did not reach in the top of the priority list.

Third, on the performance behavior that you are observing - one of the main reason of this observation is that the random access is significantly costly in the terms of I/O. I am not sure what storage are you using but this would be magnified if you are using rotating hard disks as opposed to SSD. Similarly, this is going to be magnified if you measure this on the chaincode side as opposed to when implemented on the Fabric side. In general, this is a well understood behavior in traditional and mature databases that using an index is not always helpful and at times the using the index could in fact be more costly than scanning the whole data and filter what is not needed. For instance, if you employ a rotating disk and the data items that you need to retrieve is uniformly spread across in the entire range, and the data items that satisfy the query are more than 10% of the total data, you would perhaps be better off scanning full data than using the index. Most of the query engines employ a sophisticated cost model that estimates the cost of various potential query execution plans and select the one that appears to be the least expensive for the given query in the context of the data present. Most of the cost models maintains some metadata to derive some of these heuristics.

Thanks,
Manish





On Sun, Apr 5, 2020 at 4:02 PM Prasanth Sundaravelu <prasanths96@...> wrote:

Hi guys,

I have been recently developing custom rich query processor & indexing mechanism at chaincode level to exploit goleveldb's performance. I have designed this on the assumption that getting series of keys by GetStateByRange() and regular GetState() will be the same, but I could see now, that the range query is much faster. It makes sense as range query will read the storage in a sequential way and GetState() has to jump to different memory locations to fetch (Correct me if I am wrong). 

I have partially implemented it to be able to test the underlying design assumptions and I am in trouble now. 
The logic of the query operation is like this: 
1. Based on the query, make a range query on the relevant indexes.
2. After retrieving indexes, fetch the actual data pointed by the index using GetState().

This works charmingly well, if the number of matching records is less, but horrible if the record count is high:

- For a response with 1 record, it takes under ~30ms. 
- For 100 records: ~100ms
- For 1000 records: ~1sec! (See the trouble?)

I can see that the bottleneck is at the GetState(). I would like to have some pointers to resolve this issue. 
I have tried the following: 
- In range query, fetched a single iterator, synchronized it and fed it to multiple go routines. (Which doesn't improve the performance of range query at all, since it's still reading one at a time)
- Tried to get multiple iterators, but fabric threw error if I do that.
- Tried to run GetState() on multiple go routines. Still no improvement. 

I have the following questions:
1. Is the fabric running chaincode itself as a whole already as go routines? If so, does it help chaincode functions by default?
2. If having Go Routines in chaincode help at all? 
3. How can I make GetState() faster?

Pardon my long message. 

If you're interested in this project, you can help me implementing this by contributing: https://github.com/prasanths96/iState
This is my first open project, I would really appreciate comments and critiques that would positively impact the project. Suggestions are welcome.
More about the project: 
- Supports json like rich query in leveldb. (As the couch DB's performance is not "good enough")
- It aims to be a work around that can be easily used by any chaincode developer.
- In-memory cached data is going to be a feature that would increase the query performance to atleast 10 folds. 
- All this delivered as an interface to developers having simple CRUD functions.

- Prasanth


Re: Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

Manish
 

I would further like to elaborate on my first point as it may cause confusion in the context of Fabric….
Like most of the other databases, Fabric 0.6 transaction semantics were to support read-your-own-writes (RYOW). With these semantics, it matters whether in the sequence of transaction, you performed a read or a write first. The query results may differ based on the sequence of operations.

However, with Fabric 1.0, we did not support RYOW and hence, there is no theoretical bottleneck to support parallel queries as such.

On Mon, Apr 6, 2020 at 10:40 AM Manish Sethi <manish.sethi@...> wrote:
Hello Prasanth,

My response to your questions would be three fold.

First, a chaincode is expected to be a sequential logic. Otherwise, providing a guarantee of deterministic behavior is very hard. If you go into theoretical aspects, the basic definition of a transaction is "a sequence of database operations….” - a logical sequence is key here. There are some mature systems that support nested transactions that could be executed in parallel but to keep the chaincode logic simple and deterministic, Fabric supports it in the form of a single sequential flow.

Second, for complex query support, you are experimenting in the right direction at a high level. However, implementing it on the chaincode side is not going to be efficient and ideally be implemented on the Fabric side. Since beginning, we have similar ideas to build a query engine on top, albeit on the Fabric side - however, this did not reach in the top of the priority list.

Third, on the performance behavior that you are observing - one of the main reason of this observation is that the random access is significantly costly in the terms of I/O. I am not sure what storage are you using but this would be magnified if you are using rotating hard disks as opposed to SSD. Similarly, this is going to be magnified if you measure this on the chaincode side as opposed to when implemented on the Fabric side. In general, this is a well understood behavior in traditional and mature databases that using an index is not always helpful and at times the using the index could in fact be more costly than scanning the whole data and filter what is not needed. For instance, if you employ a rotating disk and the data items that you need to retrieve is uniformly spread across in the entire range, and the data items that satisfy the query are more than 10% of the total data, you would perhaps be better off scanning full data than using the index. Most of the query engines employ a sophisticated cost model that estimates the cost of various potential query execution plans and select the one that appears to be the least expensive for the given query in the context of the data present. Most of the cost models maintains some metadata to derive some of these heuristics.

Thanks,
Manish





On Sun, Apr 5, 2020 at 4:02 PM Prasanth Sundaravelu <prasanths96@...> wrote:

Hi guys,

I have been recently developing custom rich query processor & indexing mechanism at chaincode level to exploit goleveldb's performance. I have designed this on the assumption that getting series of keys by GetStateByRange() and regular GetState() will be the same, but I could see now, that the range query is much faster. It makes sense as range query will read the storage in a sequential way and GetState() has to jump to different memory locations to fetch (Correct me if I am wrong). 

I have partially implemented it to be able to test the underlying design assumptions and I am in trouble now. 
The logic of the query operation is like this: 
1. Based on the query, make a range query on the relevant indexes.
2. After retrieving indexes, fetch the actual data pointed by the index using GetState().

This works charmingly well, if the number of matching records is less, but horrible if the record count is high:

- For a response with 1 record, it takes under ~30ms. 
- For 100 records: ~100ms
- For 1000 records: ~1sec! (See the trouble?)

I can see that the bottleneck is at the GetState(). I would like to have some pointers to resolve this issue. 
I have tried the following: 
- In range query, fetched a single iterator, synchronized it and fed it to multiple go routines. (Which doesn't improve the performance of range query at all, since it's still reading one at a time)
- Tried to get multiple iterators, but fabric threw error if I do that.
- Tried to run GetState() on multiple go routines. Still no improvement. 

I have the following questions:
1. Is the fabric running chaincode itself as a whole already as go routines? If so, does it help chaincode functions by default?
2. If having Go Routines in chaincode help at all? 
3. How can I make GetState() faster?

Pardon my long message. 

If you're interested in this project, you can help me implementing this by contributing: https://github.com/prasanths96/iState
This is my first open project, I would really appreciate comments and critiques that would positively impact the project. Suggestions are welcome.
More about the project: 
- Supports json like rich query in leveldb. (As the couch DB's performance is not "good enough")
- It aims to be a work around that can be easily used by any chaincode developer.
- In-memory cached data is going to be a feature that would increase the query performance to atleast 10 folds. 
- All this delivered as an interface to developers having simple CRUD functions.

- Prasanth


Re: Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

Manish
 

Hello Prasanth,

My response to your questions would be three fold.

First, a chaincode is expected to be a sequential logic. Otherwise, providing a guarantee of deterministic behavior is very hard. If you go into theoretical aspects, the basic definition of a transaction is "a sequence of database operations….” - a logical sequence is key here. There are some mature systems that support nested transactions that could be executed in parallel but to keep the chaincode logic simple and deterministic, Fabric supports it in the form of a single sequential flow.

Second, for complex query support, you are experimenting in the right direction at a high level. However, implementing it on the chaincode side is not going to be efficient and ideally be implemented on the Fabric side. Since beginning, we have similar ideas to build a query engine on top, albeit on the Fabric side - however, this did not reach in the top of the priority list.

Third, on the performance behavior that you are observing - one of the main reason of this observation is that the random access is significantly costly in the terms of I/O. I am not sure what storage are you using but this would be magnified if you are using rotating hard disks as opposed to SSD. Similarly, this is going to be magnified if you measure this on the chaincode side as opposed to when implemented on the Fabric side. In general, this is a well understood behavior in traditional and mature databases that using an index is not always helpful and at times the using the index could in fact be more costly than scanning the whole data and filter what is not needed. For instance, if you employ a rotating disk and the data items that you need to retrieve is uniformly spread across in the entire range, and the data items that satisfy the query are more than 10% of the total data, you would perhaps be better off scanning full data than using the index. Most of the query engines employ a sophisticated cost model that estimates the cost of various potential query execution plans and select the one that appears to be the least expensive for the given query in the context of the data present. Most of the cost models maintains some metadata to derive some of these heuristics.

Thanks,
Manish





On Sun, Apr 5, 2020 at 4:02 PM Prasanth Sundaravelu <prasanths96@...> wrote:

Hi guys,

I have been recently developing custom rich query processor & indexing mechanism at chaincode level to exploit goleveldb's performance. I have designed this on the assumption that getting series of keys by GetStateByRange() and regular GetState() will be the same, but I could see now, that the range query is much faster. It makes sense as range query will read the storage in a sequential way and GetState() has to jump to different memory locations to fetch (Correct me if I am wrong). 

I have partially implemented it to be able to test the underlying design assumptions and I am in trouble now. 
The logic of the query operation is like this: 
1. Based on the query, make a range query on the relevant indexes.
2. After retrieving indexes, fetch the actual data pointed by the index using GetState().

This works charmingly well, if the number of matching records is less, but horrible if the record count is high:

- For a response with 1 record, it takes under ~30ms. 
- For 100 records: ~100ms
- For 1000 records: ~1sec! (See the trouble?)

I can see that the bottleneck is at the GetState(). I would like to have some pointers to resolve this issue. 
I have tried the following: 
- In range query, fetched a single iterator, synchronized it and fed it to multiple go routines. (Which doesn't improve the performance of range query at all, since it's still reading one at a time)
- Tried to get multiple iterators, but fabric threw error if I do that.
- Tried to run GetState() on multiple go routines. Still no improvement. 

I have the following questions:
1. Is the fabric running chaincode itself as a whole already as go routines? If so, does it help chaincode functions by default?
2. If having Go Routines in chaincode help at all? 
3. How can I make GetState() faster?

Pardon my long message. 

If you're interested in this project, you can help me implementing this by contributing: https://github.com/prasanths96/iState
This is my first open project, I would really appreciate comments and critiques that would positively impact the project. Suggestions are welcome.
More about the project: 
- Supports json like rich query in leveldb. (As the couch DB's performance is not "good enough")
- It aims to be a work around that can be easily used by any chaincode developer.
- In-memory cached data is going to be a feature that would increase the query performance to atleast 10 folds. 
- All this delivered as an interface to developers having simple CRUD functions.

- Prasanth


Re: Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

Gari Singh <garis@...>
 

Hi Prasanth -

Is there a reason why you are calling GetState() after you have called GetStateByRange()?
GetStateByRange() returns an iterator for KVs (https://github.com/hyperledger/fabric/blob/release-1.4/protos/ledger/queryresult/kv_query_result.pb.go#L23 ).
You should not need to call GetState() as you can simply iterate through the keys and values returned by GetStateByRange().

-- G

-----------------------------------------
Gari Singh
Distinguished Engineer, CTO - IBM Blockchain
IBM Middleware
550 King St
Littleton, MA 01460
Cell: 978-846-7499
garis@...
-----------------------------------------

-----fabric@... wrote: -----
To: fabric@...
From: "Prasanth Sundaravelu"
Sent by: fabric@...
Date: 04/05/2020 04:03PM
Subject: [EXTERNAL] [Hyperledger Fabric] Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

Hi guys,
I have been recently developing custom rich query processor & indexing mechanism at chaincode level to exploit goleveldb's performance. I have designed this on the assumption that getting series of keys by GetStateByRange() and regular GetState() will be the same, but I could see now, that the range query is much faster. It makes sense as range query will read the storage in a sequential way and GetState() has to jump to different memory locations to fetch (Correct me if I am wrong).

I have partially implemented it to be able to test the underlying design assumptions and I am in trouble now.
The logic of the query operation is like this:
1. Based on the query, make a range query on the relevant indexes.
2. After retrieving indexes, fetch the actual data pointed by the index using GetState().
This works charmingly well, if the number of matching records is less, but horrible if the record count is high:
- For a response with 1 record, it takes under ~30ms.
- For 100 records: ~100ms
- For 1000 records: ~1sec! (See the trouble?)

I can see that the bottleneck is at the GetState(). I would like to have some pointers to resolve this issue.
I have tried the following:
- In range query, fetched a single iterator, synchronized it and fed it to multiple go routines. (Which doesn't improve the performance of range query at all, since it's still reading one at a time)
- Tried to get multiple iterators, but fabric threw error if I do that.
- Tried to run GetState() on multiple go routines. Still no improvement.

I have the following questions:
1. Is the fabric running chaincode itself as a whole already as go routines? If so, does it help chaincode functions by default?
2. If having Go Routines in chaincode help at all?
3. How can I make GetState() faster?

Pardon my long message.


If you're interested in this project, you can help me implementing this by contributing: https://github.com/prasanths96/iState
This is my first open project, I would really appreciate comments and critiques that would positively impact the project. Suggestions are welcome.
More about the project:
- Supports json like rich query in leveldb. (As the couch DB's performance is not "good enough")
- It aims to be a work around that can be easily used by any chaincode developer.
- In-memory cached data is going to be a feature that would increase the query performance to atleast 10 folds.
- All this delivered as an interface to developers having simple CRUD functions.

- Prasanth


is it possible to create new channels with different raft orderers in fabric2.0 ?

kuku <loowenzi@...>
 





hi all :
I am using  fabric2.0 . I know that raft orderers can join different application channels  .Now I can add  a new raft orderer into the existed channel, 
But I want to konw  is it possible to create new channels with different raft orderers? 
For example:
      If the systemchannel genesis block has 5 raft orderers ( o1 ,o2,o3,o4,o5), and all the orderers works well,  now can I create a new channel  with 3 orderers ( o1 ,o2,o3 ) or create another channel with 4 orderers (o1,o2,o3,o5)? 
  if it is possible ,how can I do it ? 
  because, now I can only create new channels with the the same number orderers ( o1,o2,o3,o4,o5)  and then add/delete orderers through dynamic configuration
 
   


 



 


Re: introduce msp into chaincode for authentication

qs meng <qsmeng@...>
 

Hi Prasanth
    Yes, the issuer is not the creator of the transaction  and  I want to externally verify the issuer's idenitity.
   Currently, I store the issuer's CA certificate in the ledger state database and do verifying. However, as the state database gets bigger, I do not know which is faster between the current method and the msp method.  I think the CA certificate of msp will be faster.
 Thank you.
 regards,
meng






I didn't understand the use case correctly. Do you want to externally verify an Issuer's identity after he/she issues the digital asset? (I assume, when issuing the asset, he will be the creator of the transaction)

- You can store the issuer's x.509 certificate when he/she issues the asset and whoever needs to verify can acquire it from the ledger to do so.

- Prasanth

On Sun, Apr 5, 2020 at 8:14 AM qs meng <qsmeng@...> wrote:
Hi Prasanth,
   Thank you very much for your reply.  Indeed a transaction creator is already authenticated and attributes can be used to do acess control.
    Considier another use case. Using chaincode to issue digital assets for physical assets needs the asset issuer's identity being authenticated.  However, the issuer is not the transaction creator.  If the function to read msp CA can be moved intto chaincode part as an api it is very good.
   Is it possible and if it is, how should I do? Thank you.
  Best regards,
qs meng





At 2020-04-02 18:27:14, "Prasanth Sundaravelu" <prasanths96@...> wrote:

Hi Meng, 

If my understanding is correct,

You have considered one use-case, where you want to authenticate based on if the creator is from a certain org (To be more precise, a creator/user registered by a certain client/org). It is one of the application use cases. Similarly, there can be more such application use cases. We cannot move all such use cases to hyperledger's core.

Also, there is actually a way to implement your use case. Check out Attribute Based Access Control in hyprledger fabric. 

General idea:
For identifying if creator is from certain org:
- In an ideal case, each org will have different CA, so at chaincode, read the Certificate to find out Org Name.

For identifying if creator from a specific client in same org:
 - You can add a special attribute in the certificate, indicating the name/identifier of the client, when registering a user. 
- Then, at chaincode, you can read the certificate to find out client id.

In both cases, you have to write authentication logic at chaincode.

- Prasanth


On Wed, 1 Apr 2020, 12:34 pm qs meng, <qsmeng@...> wrote:
Hi Yacov,
    yes, a peer would authenticate the proposal creator, who is a member of  fabric network. But in chaincode container, there is no way to  authenticate an identity who belongs to one client application.  If a chaincode could get CA certificate, it is feasible for chaincode to authenticate identities who belongs to client application. 
   I do not know if I explain it clearly.
   Thank you.
  Regards,
qs meng





At 2020-04-01 14:45:32, "Yacov Manevich" <YACOVM@...> wrote:

The proposal is authenticated by the peer before it gets into the chaincode.



From:        "qs meng" <qsmeng@...>
To:        "fabric@..." <fabric@...>
Date:        04/01/2020 03:26 AM
Subject:        [EXTERNAL] [Hyperledger Fabric] introduce msp into chaincode for authentication
Sent by:        fabric@...




Hi,
   I suggest to add msp support into chaincode to authenticate identitis  in  client applications. The getCreator api only get the creator and take it as authenticated already.
  A way to do: for an endorsing peer, it has a function to get CA from configure block and autheniticate the transaction creator. Just copy the function to chaicode part. Is it feasible?
  Thank you.
 Regards,
qs meng

 






 



 



 


Go routines in chaincode helps? (especially with GetState() and GetStateByRange() APIs #fabric #hyperledger-fabric #fabric-questions

Prasanth Sundaravelu
 

Hi guys,

I have been recently developing custom rich query processor & indexing mechanism at chaincode level to exploit goleveldb's performance. I have designed this on the assumption that getting series of keys by GetStateByRange() and regular GetState() will be the same, but I could see now, that the range query is much faster. It makes sense as range query will read the storage in a sequential way and GetState() has to jump to different memory locations to fetch (Correct me if I am wrong). 

I have partially implemented it to be able to test the underlying design assumptions and I am in trouble now. 
The logic of the query operation is like this: 
1. Based on the query, make a range query on the relevant indexes.
2. After retrieving indexes, fetch the actual data pointed by the index using GetState().

This works charmingly well, if the number of matching records is less, but horrible if the record count is high:

- For a response with 1 record, it takes under ~30ms. 
- For 100 records: ~100ms
- For 1000 records: ~1sec! (See the trouble?)

I can see that the bottleneck is at the GetState(). I would like to have some pointers to resolve this issue. 
I have tried the following: 
- In range query, fetched a single iterator, synchronized it and fed it to multiple go routines. (Which doesn't improve the performance of range query at all, since it's still reading one at a time)
- Tried to get multiple iterators, but fabric threw error if I do that.
- Tried to run GetState() on multiple go routines. Still no improvement. 

I have the following questions:
1. Is the fabric running chaincode itself as a whole already as go routines? If so, does it help chaincode functions by default?
2. If having Go Routines in chaincode help at all? 
3. How can I make GetState() faster?

Pardon my long message. 

If you're interested in this project, you can help me implementing this by contributing: https://github.com/prasanths96/iState
This is my first open project, I would really appreciate comments and critiques that would positively impact the project. Suggestions are welcome.
More about the project: 
- Supports json like rich query in leveldb. (As the couch DB's performance is not "good enough")
- It aims to be a work around that can be easily used by any chaincode developer.
- In-memory cached data is going to be a feature that would increase the query performance to atleast 10 folds. 
- All this delivered as an interface to developers having simple CRUD functions.

- Prasanth


Re: How to write unit test with custom tx creator certificates?

Prasanth Sundaravelu
 

See if this package helps:

https://godoc.org/github.com/hyperledger/fabric-chaincode-go/pkg/cid

It can get the full certificate from chaincode stub (don't know if it works with mock stub)

- Prasanth