Topics

Performance improvement #fabric-questions #fabric-sdk-java


alexis.janin@...
 

Hi,
 
We are developing a solution on top of HLF using Java SDK but we are facing performance issues.
My goal is to understand if we are doing the things right or if we could optimize anything.
 
The configuration of our channel is :
 
"BatchSize": {
  "value": {
    "absolute_max_bytes": 103809024,
    "max_message_count": 300,
    "preferred_max_bytes": 1048576
},
},
"BatchTimeout": {
  "value": {
    "timeout": "2s"
}
 
The results of our performance tests shows that each request on our REST API takes an average time of 3000ms.
When we change the channel configuration to set the BatchTimeout to 30s, the average time explodes which seems to prove that the bottleneck is the wait of the event "transaction commited".
 
In our context we need to respond synchronously, which means we have to wait the transaction to be commited.
In our implementation we first send a transaction proposal then we wait for an event on BlockEvent.TransactionEvent
 
My questions are:
 - How could we improve the average time of reponses without changing the BatchTimeout ?
 - How do you guys implement this type of problems?
 - Do you send back a response after the proposal and expose an endpoint to check asynchronously the state of each transaction?
 
Thanks for the help.
 
Alexis


alexis.janin@...
 

Hello everyone,

First I wish you a happy new year with many successful Hyperledger projects ;-)

If anyone has an element to start the discussion on my questions you are welcome.

Thanks !

Alexis


Scott Zwierzynski <scottz@...>
 

Set your batchsize maxmessagecount to something small, like 10 (or even 1), since your application is synchronous. Set batchtimeout to a small value, too. Since your application implementation requires that each transaction depend on the previous one, raising the batchtimeout does nothing but introduce wait delays. You would need to redesign your application chaincode to allow parallel endorsements to speed up your throughput.
 
 
Scott Zwierzynski, Team Lead
Blockchain Test Engineering and Automation
 

----- Original message -----
From: alexis.janin@...
Sent by: fabric@...
To: fabric@...
Cc:
Subject: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Date: Wed, Dec 19, 2018 9:34 AM
 
Hi,
 
We are developing a solution on top of HLF using Java SDK but we are facing performance issues.
My goal is to understand if we are doing the things right or if we could optimize anything.
 
The configuration of our channel is :
 
"BatchSize": {
  "value": {
    "absolute_max_bytes": 103809024,
    "max_message_count": 300,
    "preferred_max_bytes": 1048576
},
},
"BatchTimeout": {
  "value": {
    "timeout": "2s"
}
 
The results of our performance tests shows that each request on our REST API takes an average time of 3000ms.
When we change the channel configuration to set the BatchTimeout to 30s, the average time explodes which seems to prove that the bottleneck is the wait of the event "transaction commited".
 
In our context we need to respond synchronously, which means we have to wait the transaction to be commited.
In our implementation we first send a transaction proposal then we wait for an event on BlockEvent.TransactionEvent
 
My questions are:
 - How could we improve the average time of reponses without changing the BatchTimeout ?
 - How do you guys implement this type of problems?
 - Do you send back a response after the proposal and expose an endpoint to check asynchronously the state of each transaction?
 
Thanks for the help.
 
Alexis
 


alexis.janin@...
 

Hello Scott,

Thank for the answer, it seems 'maxmessagecount' smaller is a bit better (which seems to be the default value ^^).
I don't know why but I forgot that the number of messages in a block is also a limiting parameter...

You said that we should redesign our application chaincode to allow parallel endorsements, but if we call the chaincode from multiple threads the execution will be multi-threaded no? Or do you mean that if we do not manage multi-threading in the nodejs chaincode application the execution will be sequential?

Which would mean that the Docker container build to execute the nodejs chaincode application should be configured any way to support multi-threading, is this your proposal?

Thank you for your help Scott.

Alexis


Marek Malik
 

Hi Alexis, 

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic. 

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable. 

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

  OrdererType: etcdraft

  BatchTimeout: 2s

  BatchSize:

    MaxMessageCount: 2

    AbsoluteMaxBytes: 99 MB

    PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.


Prasanth Sundaravelu
 

Hi Marek,

What's your hardware specs? 
And make sure you're performing query rather than invoke.


On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:

Hi Alexis, 

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic. 

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable. 

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

  OrdererType: etcdraft

  BatchTimeout: 2s

  BatchSize:

    MaxMessageCount: 2

    AbsoluteMaxBytes: 99 MB

    PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.


Marek Malik
 

Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer -  after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:

{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.

Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: niedziela, 3 maja 2020 08:42
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

 

What's your hardware specs? 

And make sure you're performing query rather than invoke.

 

 

On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:

Hi Alexis, 

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic. 

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable. 

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

  OrdererType: etcdraft

  BatchTimeout: 2s

  BatchSize:

    MaxMessageCount: 2

    AbsoluteMaxBytes: 99 MB

    PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.


Prasanth Sundaravelu
 

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook. 
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle. 
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke. 
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference. 

Regards,
Pras


On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:

Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer -  after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:

{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.

Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: niedziela, 3 maja 2020 08:42
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

 

What's your hardware specs? 

And make sure you're performing query rather than invoke.

 

 

On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:

Hi Alexis, 

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic. 

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable. 

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

  OrdererType: etcdraft

  BatchTimeout: 2s

  BatchSize:

    MaxMessageCount: 2

    AbsoluteMaxBytes: 99 MB

    PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.


Marek Malik
 

Hi Pras,

This network setup is also working on a 8GB MacBook Pro
😉

This are the docker stats, nothing but the peer two which I’m sending to the query is running -

The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).


I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

I have some small problems with configuration of the consortium when trying to setup a single org.


Pozdrawiam
Marek Malik

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: niedziela, 3 maja 2020 22:55
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook. 
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle. 
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke. 
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference. 

Regards,
Pras

 

On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:

Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer -  after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:

{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.

Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: niedziela, 3 maja 2020 08:42
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

 

What's your hardware specs? 

And make sure you're performing query rather than invoke.

 

 

On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:

Hi Alexis, 

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic. 

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable. 

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

  OrdererType: etcdraft

  BatchTimeout: 2s

  BatchSize:

    MaxMessageCount: 2

    AbsoluteMaxBytes: 99 MB

    PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.


Prasanth Sundaravelu
 

I'm not familiar with java chaincode. 
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric. 
Also you can try using some other language for chaincode and see if the problem persists.


On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:

Managed to run the network in single org setup – the time is exactly the same ;/

 


Pozdrawiam
Marek Malik

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: poniedziałek, 4 maja 2020 15:33
Do: Marek Malik <info@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power. 

- Stats from "top" command can be helpful to analyze further. 

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu. 

By any chance you are tweaking the "validatorPoolSize" to use fewer cores? 

 

On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:

Hi Pras,

This network setup is also working on a 8GB MacBook Pro
😉

This are the docker stats, nothing but the peer two which I’m sending to the query is running -

The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).


I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

I have some small problems with configuration of the consortium when trying to setup a single org.


Pozdrawiam
Marek Malik

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: niedziela, 3 maja 2020 22:55
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook. 
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle. 
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke. 
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference. 

Regards,
Pras

 

On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:

Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer -  after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:

{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.

Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: niedziela, 3 maja 2020 08:42
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

 

What's your hardware specs? 

And make sure you're performing query rather than invoke.

 

 

On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:

Hi Alexis, 

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic. 

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable. 

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

  OrdererType: etcdraft

  BatchTimeout: 2s

  BatchSize:

    MaxMessageCount: 2

    AbsoluteMaxBytes: 99 MB

    PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.


Marek Malik
 

Hi Guys,

Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

I’m trying to fetch 150 records from the db, the request lasts around 50 seconds (only the call to CouchDB).
I’m trying to fix this issue for a week now but with no success which makes it more frustrating.

This is the method in which I’m calling the couchDb using getQueryResult

public List<String> queryBy(String querySelector) {

hasText(querySelector, "QuerySelector is required!");

try (QueryResultsIterator<KeyValue> queryResult = ctx.getStub().getQueryResult(querySelector)) {

log.info("{}. {} query returned finished {} sec.", LocalDateTime.now(), domainName, querySelector);

return stream(queryResult.spliterator(), false)

.map(KeyValue::getStringValue)

.collect(toList());

} catch (Exception e)

{ log.error("error when querying the state db", e); throw new RuntimeException("Error when querying the state db", e); }

}

 

There is not much going on here, I’m executing the chaincode as query as suggested first.

 

Initially as Pras suggested I was worried that maybe ObjectMapper that I was using for deserializing the response into a POJO would be causing the lag, but after removing it the problem still persists.

When looking at logs I can see that couch is responding with the results quite quick, then peer logs with ‘[couchdb] QueryDocuments -> DEBU 25c1[0m [banyan-channel_poc-services] Adding json docment for id:`  and `
[lockbasedtxmgr] Next -> DEBU 25c4[0m queryResultsItr.Next() returned a record: `

I was trying to go by the classes used during the call:
InvocationStubImpl, QueryResultsIteratorImpl and somehow try to find the cause of this call being so long.
I’m running this on a 16GB, 6core i7 MacBookPro, A friend ran the same chaincode on a 32 GB dell machine – almost identical results.

I hope I’m making something wrong but if the Hyperledger has such technical  limitation to which transfer is so slow I would anticipate it knowing I cannot do anything about it. Then, I will try and search for something to cover such long query executions from outside the blockchain or change the architecture entirely. Only thing, I notices discussions on the mailing lists about working on datasets far bigger then my (2-3GB big or 80 million records) so I would imagine I still have some space to play with my 150 record.


Pozdrawiam
Marek Malik

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: poniedziałek, 4 maja 2020 20:01
Do: Marek Malik <info@...>, "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

I'm not familiar with java chaincode. 
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric. 
Also you can try using some other language for chaincode and see if the problem persists.

 

On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:

Managed to run the network in single org setup – the time is exactly the same ;/

 


Pozdrawiam
Marek Malik

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: poniedziałek, 4 maja 2020 15:33
Do: Marek Malik <info@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power. 

- Stats from "top" command can be helpful to analyze further. 

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu. 

By any chance you are tweaking the "validatorPoolSize" to use fewer cores? 

 

On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:

Hi Pras,

This network setup is also working on a 8GB MacBook Pro
😉

This are the docker stats, nothing but the peer two which I’m sending to the query is running -

The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).


I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

I have some small problems with configuration of the consortium when trying to setup a single org.


Pozdrawiam
Marek Malik

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: niedziela, 3 maja 2020 22:55
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook. 
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle. 
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke. 
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference. 

Regards,
Pras

 

On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:

Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer -  after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:

{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.

Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: niedziela, 3 maja 2020 08:42
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

 

What's your hardware specs? 

And make sure you're performing query rather than invoke.

 

 

On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:

Hi Alexis, 

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic. 

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable. 

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

  OrdererType: etcdraft

  BatchTimeout: 2s

  BatchSize:

    MaxMessageCount: 2

    AbsoluteMaxBytes: 99 MB

    PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.


David Enyeart
 

Hi Marek,

I don't know what is causing the slow response, but I do know that from analyzing your log in https://jira.hyperledger.org/browse/FAB-17842 that the chaincode container to peer to CouchDB roundtrip for your JSON query with 150 results only takes 158 milliseconds, with CouchDB itself taking 73 milliseconds. So focus should be on the chaincode or java chaincode shim side.


Dave Enyeart


"Marek Malik" ---05/06/2020 01:23:14 PM---Hi Guys, Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

From: "Marek Malik" <info@...>
To: Prasanth Sundaravelu <prasanths96@...>, "fabric@..." <fabric@...>
Date: 05/06/2020 01:23 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by: fabric@...





Hi Guys,

Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

I’m trying to fetch 150 records from the db, the request lasts around 50 seconds (only the call to CouchDB).
I’m trying to fix this issue for a week now but with no success which makes it more frustrating.

This is the method in which I’m calling the couchDb using getQueryResult

public List<String> queryBy(String querySelector) {

hasText(querySelector, "QuerySelector is required!");
try (QueryResultsIterator<KeyValue> queryResult = ctx.getStub().getQueryResult(querySelector)) {
log.info("{}. {} query returned finished {} sec.", LocalDateTime.now(), domainName, querySelector);
return stream(queryResult.spliterator(), false)
.map(KeyValue::getStringValue)
.collect(toList());
} catch (Exception e)
{ log.error("error when querying the state db", e); throw new RuntimeException("Error when querying the state db", e); }
}

There is not much going on here, I’m executing the chaincode as query as suggested first.

Initially as Pras suggested I was worried that maybe ObjectMapper that I was using for deserializing the response into a POJO would be causing the lag, but after removing it the problem still persists.

When looking at logs I can see that couch is responding with the results quite quick, then peer logs with ‘
[couchdb] QueryDocuments -> DEBU 25c1[0m [banyan-channel_poc-services] Adding json docment for id:` and ` [lockbasedtxmgr] Next -> DEBU 25c4[0m queryResultsItr.Next() returned a record: `

I was trying to go by the classes used during the call:
InvocationStubImpl, QueryResultsIteratorImpl and somehow try to find the cause of this call being so long.
I’m running this on a 16GB, 6core i7 MacBookPro, A friend ran the same chaincode on a 32 GB dell machine – almost identical results.

I hope I’m making something wrong but if the Hyperledger has such technical limitation to which transfer is so slow I would anticipate it knowing I cannot do anything about it. Then, I will try and search for something to cover such long query executions from outside the blockchain or change the architecture entirely. Only thing, I notices discussions on the mailing lists about working on datasets far bigger then my (2-3GB big or 80 million records) so I would imagine I still have some space to play with my 150 record.


Pozdrawiam
Marek Malik


Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 20:01
Do:
Marek Malik <info@...>, "fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

I'm not familiar with java chaincode.
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric.
Also you can try using some other language for chaincode and see if the problem persists.


On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:
Managed to run the network in single org setup – the time is exactly the same ;/


Pozdrawiam
Marek Malik


Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 15:33
Do:
Marek Malik <info@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power.

- Stats from "top" command can be helpful to analyze further.

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu.

By any chance you are tweaking the "validatorPoolSize" to use fewer cores?


On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:
    Hi Pras,

    This network setup is also working on a 8GB MacBook Pro
    ?
    This are the docker stats, nothing but the peer two which I’m sending to the query is running -

    The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).


    I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

    I have some small problems with configuration of the consortium when trying to setup a single org.


    Pozdrawiam
    Marek Malik


    Od: Prasanth Sundaravelu <prasanths96@...>
    Data:
    niedziela, 3 maja 2020 22:55
    Do:
    Marek Malik <info@...>
    DW:
    "fabric@..." <fabric@...>
    Temat:
    Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

    Hi Marek,

    1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
    2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook.
    - Try equivalent command for mac as "top" in linux and see how much load is taken when idle.
    - Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke.
    - Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference.

    Regards,
    Pras


    On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:
    Hi Prasanth,

    So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

    In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer - after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
    I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:

    {"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
    The index is created. When testing this query on Fauxton the query took 40 ms.

    Any ideas ?

    I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.


    Od: Prasanth Sundaravelu <prasanths96@...>
    Data:
    niedziela, 3 maja 2020 08:42
    Do:
    Marek Malik <info@...>
    DW:
    "fabric@..." <fabric@...>
    Temat:
    Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

    Hi Marek,

    What's your hardware specs?
    And make sure you're performing query rather than invoke.


    On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:
    Hi Alexis,

    This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic.

    Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable.

    The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

    The network setup is the following:
    ```

    Orderer: &OrdererDefaults

    OrdererType: etcdraft

    BatchTimeout: 2s

    BatchSize:

    MaxMessageCount: 2

    AbsoluteMaxBytes: 99 MB

    PreferredMaxBytes: 512 KB

    ```

    How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.






Marek Malik
 

Hi David,
Yes, this is also something that I noticed.
In the logs you can find the following process

peer0.org1.example.com|[36m2020-05-06 18:24:07.093 UTC [lockbasedtxmgr] Next -> DEBU 3078[0m queryResultsItr.Next() returned a record:{

It is spanned from the moment CouchDB returned the response and when my getQueryResult  method call returned with a response.

this log has a the gap that makes the process taking so long - > 44seconds.
I couldn’t find any reference about this lockbasedtxmgr  or a service that prints a similar logs in java lib, are you in any way aware about such process?

It looks like the first portion read 101 records, and the second read the remaining 44. It lasted exactly 27 ms and 10 ms the second one.

Only option that I found in the network configuration that was explisitly set was the max message count and that was in Orderer setup (I thought that with query calls orderers are not used).


--
Pozdrawiam | Best regards, 

Marek Malik, Solutions Engineer

 

Phone: +48 605 330 093

info@...

 

Od: David Enyeart <enyeart@...>
Data: środa, 6 maja 2020 20:32
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>, Prasanth Sundaravelu <prasanths96@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

I don't know what is causing the slow response, but I do know that from analyzing your log in https://jira.hyperledger.org/browse/FAB-17842 that the chaincode container to peer to CouchDB roundtrip for your JSON query with 150 results only takes 158 milliseconds, with CouchDB itself taking 73 milliseconds. So focus should be on the chaincode or java chaincode shim side.


Dave Enyeart


"Marek Malik" ---05/06/2020 01:23:14 PM---Hi Guys, Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

From: "Marek Malik" <info@...>
To: Prasanth Sundaravelu <prasanths96@...>, "fabric@..." <fabric@...>
Date: 05/06/2020 01:23 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by: fabric@...





Hi Guys,

Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?
I’m trying to fetch 150 records from the db, the request lasts around 50 seconds (only the call to CouchDB).
I’m trying to fix this issue for a week now but with no success which makes it more frustrating.

This is the method in which I’m calling the couchDb using getQueryResult

public List<String> queryBy(String querySelector) {
hasText(querySelector, "QuerySelector is required!");
try (QueryResultsIterator<KeyValue> queryResult = ctx.getStub().getQueryResult(querySelector)) {
log.info("{}. {} query returned finished {} sec.", LocalDateTime.now(), domainName, querySelector);
return stream(queryResult.spliterator(), false)
.map(KeyValue::getStringValue)
.collect(toList());
} catch (Exception e)
{ log.error("error when querying the state db", e); throw new RuntimeException("Error when querying the state db", e); }
}

There is not much going on here, I’m executing the chaincode as query as suggested first.

Initially as Pras suggested I was worried that maybe ObjectMapper that I was using for deserializing the response into a POJO would be causing the lag, but after removing it the problem still persists.

When looking at logs I can see that couch is responding with the results quite quick, then peer logs with ‘[couchdb] QueryDocuments -> DEBU 25c1[0m [banyan-channel_poc-services] Adding json docment for id:` and ` [lockbasedtxmgr] Next -> DEBU 25c4[0m queryResultsItr.Next() returned a record: `

I was trying to go by the classes used during the call: InvocationStubImpl, QueryResultsIteratorImpl and somehow try to find the cause of this call being so long.
I’m running this on a 16GB, 6core i7 MacBookPro, A friend ran the same chaincode on a 32 GB dell machine – almost identical results.

I hope I’m making something wrong but if the Hyperledger has such technical limitation to which transfer is so slow I would anticipate it knowing I cannot do anything about it. Then, I will try and search for something to cover such long query executions from outside the blockchain or change the architecture entirely. Only thing, I notices discussions on the mailing lists about working on datasets far bigger then my (2-3GB big or 80 million records) so I would imagine I still have some space to play with my 150 record.


Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 20:01
Do:
Marek Malik <info@...>, "fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

I'm not familiar with java chaincode.
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric.
Also you can try using some other language for chaincode and see if the problem persists.

On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:
Managed to run the network in single org setup – the time is exactly the same ;/


Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 15:33
Do:
Marek Malik <info@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power.

- Stats from "top" command can be helpful to analyze further.

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu.

By any chance you are tweaking the "validatorPoolSize" to use fewer cores?

On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:

Hi Pras,

This network setup is also working on a 8GB MacBook Pro ?
This are the docker stats, nothing but the peer two which I’m sending to the query is running -

The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).

I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

I have some small problems with configuration of the consortium when trying to setup a single org.

Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 22:55
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook.
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle.
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke.
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference.

Regards,
Pras

On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:
Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer - after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:
{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.
Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 08:42
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

What's your hardware specs?
And make sure you're performing query rather than invoke.


On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:
Hi Alexis,

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic.

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable.

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

OrdererType: etcdraft

BatchTimeout: 2s

BatchSize:

MaxMessageCount: 2

AbsoluteMaxBytes: 99 MB

PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.





Prasanth Sundaravelu
 

I think you need to focus only on chaincode. 

Why not try adding detailed logs in chaincode with time, from the first point of contact (from Invoke / Query) to the deepest function in the nest and see:
1. Which function takes lot of time
2. Then further add more logs to that function for every important / heavy line and see which line takes more time

Viewing chaincode container log will make it more clear (not the peer log)

I believe this might get us some good leads, since David also confirmed that there was no problem with the Chaincode container - peer - couchdb round-trip.


On Thu, May 7, 2020 at 12:56 AM Marek Malik <info@...> wrote:

Hi David,
Yes, this is also something that I noticed.
In the logs you can find the following process

peer0.org1.example.com|[36m2020-05-06 18:24:07.093 UTC [lockbasedtxmgr] Next -> DEBU 3078[0m queryResultsItr.Next() returned a record:{

It is spanned from the moment CouchDB returned the response and when my getQueryResult  method call returned with a response.

this log has a the gap that makes the process taking so long - > 44seconds.
I couldn’t find any reference about this lockbasedtxmgr  or a service that prints a similar logs in java lib, are you in any way aware about such process?

It looks like the first portion read 101 records, and the second read the remaining 44. It lasted exactly 27 ms and 10 ms the second one.

Only option that I found in the network configuration that was explisitly set was the max message count and that was in Orderer setup (I thought that with query calls orderers are not used).


--
Pozdrawiam | Best regards, 

Marek Malik, Solutions Engineer

 

Phone: +48 605 330 093

info@...

 

Od: David Enyeart <enyeart@...>
Data: środa, 6 maja 2020 20:32
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>, Prasanth Sundaravelu <prasanths96@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

I don't know what is causing the slow response, but I do know that from analyzing your log in https://jira.hyperledger.org/browse/FAB-17842 that the chaincode container to peer to CouchDB roundtrip for your JSON query with 150 results only takes 158 milliseconds, with CouchDB itself taking 73 milliseconds. So focus should be on the chaincode or java chaincode shim side.


Dave Enyeart


"Marek Malik" ---05/06/2020 01:23:14 PM---Hi Guys, Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

From: "Marek Malik" <info@...>
To: Prasanth Sundaravelu <prasanths96@...>, "fabric@..." <fabric@...>
Date: 05/06/2020 01:23 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by: fabric@...





Hi Guys,

Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?
I’m trying to fetch 150 records from the db, the request lasts around 50 seconds (only the call to CouchDB).
I’m trying to fix this issue for a week now but with no success which makes it more frustrating.

This is the method in which I’m calling the couchDb using getQueryResult

public List<String> queryBy(String querySelector) {
hasText(querySelector, "QuerySelector is required!");
try (QueryResultsIterator<KeyValue> queryResult = ctx.getStub().getQueryResult(querySelector)) {
log.info("{}. {} query returned finished {} sec.", LocalDateTime.now(), domainName, querySelector);
return stream(queryResult.spliterator(), false)
.map(KeyValue::getStringValue)
.collect(toList());
} catch (Exception e)
{ log.error("error when querying the state db", e); throw new RuntimeException("Error when querying the state db", e); }
}

There is not much going on here, I’m executing the chaincode as query as suggested first.

Initially as Pras suggested I was worried that maybe ObjectMapper that I was using for deserializing the response into a POJO would be causing the lag, but after removing it the problem still persists.

When looking at logs I can see that couch is responding with the results quite quick, then peer logs with ‘[couchdb] QueryDocuments -> DEBU 25c1[0m [banyan-channel_poc-services] Adding json docment for id:` and ` [lockbasedtxmgr] Next -> DEBU 25c4[0m queryResultsItr.Next() returned a record: `

I was trying to go by the classes used during the call: InvocationStubImpl, QueryResultsIteratorImpl and somehow try to find the cause of this call being so long.
I’m running this on a 16GB, 6core i7 MacBookPro, A friend ran the same chaincode on a 32 GB dell machine – almost identical results.

I hope I’m making something wrong but if the Hyperledger has such technical limitation to which transfer is so slow I would anticipate it knowing I cannot do anything about it. Then, I will try and search for something to cover such long query executions from outside the blockchain or change the architecture entirely. Only thing, I notices discussions on the mailing lists about working on datasets far bigger then my (2-3GB big or 80 million records) so I would imagine I still have some space to play with my 150 record.


Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 20:01
Do:
Marek Malik <info@...>, "fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

I'm not familiar with java chaincode.
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric.
Also you can try using some other language for chaincode and see if the problem persists.

On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:
Managed to run the network in single org setup – the time is exactly the same ;/


Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 15:33
Do:
Marek Malik <info@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power.

- Stats from "top" command can be helpful to analyze further.

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu.

By any chance you are tweaking the "validatorPoolSize" to use fewer cores?

On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:

Hi Pras,

This network setup is also working on a 8GB MacBook Pro ?
This are the docker stats, nothing but the peer two which I’m sending to the query is running -

The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).

I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

I have some small problems with configuration of the consortium when trying to setup a single org.

Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 22:55
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook.
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle.
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke.
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference.

Regards,
Pras

On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:
Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer - after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:
{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.
Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 08:42
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

What's your hardware specs?
And make sure you're performing query rather than invoke.


On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:
Hi Alexis,

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic.

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable.

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

OrdererType: etcdraft

BatchTimeout: 2s

BatchSize:

MaxMessageCount: 2

AbsoluteMaxBytes: 99 MB

PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.





Marek Malik
 

Hi Prasanth,

Please check the logs I have send in the last email.

You can search for log entries from Class: ByteStorageBaseService
This is the class where I make the
getQueryResult method call.
Between this call and the actual handling of the response you will find three logs (“query started”’ – log just before calling db, “query returned finished”- call returned, and “query copping finished”-  results copied to second variable). Between the second and third you should see around 9 sec. difference. The time between first and second is the one that hurts me.

For visualization, so you will know the this is the class method -> attachment.


Additionally, as pointed in the last email. I notices that the process named or tagged
lockbasedtxmgr which looks like something that is iterating thought the data from the CouchDB has stopped after 101 records, and restarted after about 45 seconds and iterated the remaining 44 records.
You can query this by using reqExp: 18:24:07 *.*queryResultsItr.Next() and 18:24:51 *.*queryResultsItr.Next()

 

These results are reoccurring, I tested it 4 times and the scenario is the same.
Again, does anyone have any idea what this process is
lockbasedtxmgr, and how it is controlled? Can I somehow influence the iterable counter there (if there is any) or at least lower the sleep time.


Best Regards,
Marek

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: środa, 6 maja 2020 22:46
Do: Marek Malik <info@...>
DW: David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

I think you need to focus only on chaincode. 

Why not try adding detailed logs in chaincode with time, from the first point of contact (from Invoke / Query) to the deepest function in the nest and see:
1. Which function takes lot of time
2. Then further add more logs to that function for every important / heavy line and see which line takes more time

Viewing chaincode container log will make it more clear (not the peer log)

I believe this might get us some good leads, since David also confirmed that there was no problem with the Chaincode container - peer - couchdb round-trip.

 

On Thu, May 7, 2020 at 12:56 AM Marek Malik <info@...> wrote:

Hi David,
Yes, this is also something that I noticed.
In the logs you can find the following process

peer0.org1.example.com|[36m2020-05-06 18:24:07.093 UTC [lockbasedtxmgr] Next -> DEBU 3078[0m queryResultsItr.Next() returned a record:{

It is spanned from the moment CouchDB returned the response and when my getQueryResult  method call returned with a response.

this log has a the gap that makes the process taking so long - > 44seconds.
I couldn’t find any reference about this lockbasedtxmgr  or a service that prints a similar logs in java lib, are you in any way aware about such process?

It looks like the first portion read 101 records, and the second read the remaining 44. It lasted exactly 27 ms and 10 ms the second one.

Only option that I found in the network configuration that was explisitly set was the max message count and that was in Orderer setup (I thought that with query calls orderers are not used).


--
Pozdrawiam | Best regards, 

Marek Malik, Solutions Engineer

 

Phone: +48 605 330 093

info@...

 

Od: David Enyeart <enyeart@...>
Data: środa, 6 maja 2020 20:32
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>, Prasanth Sundaravelu <prasanths96@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

I don't know what is causing the slow response, but I do know that from analyzing your log in https://jira.hyperledger.org/browse/FAB-17842 that the chaincode container to peer to CouchDB roundtrip for your JSON query with 150 results only takes 158 milliseconds, with CouchDB itself taking 73 milliseconds. So focus should be on the chaincode or java chaincode shim side.


Dave Enyeart


"Marek Malik" ---05/06/2020 01:23:14 PM---Hi Guys, Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

From: "Marek Malik" <info@...>
To: Prasanth Sundaravelu <prasanths96@...>, "fabric@..." <fabric@...>
Date: 05/06/2020 01:23 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by: fabric@...





Hi Guys,

Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?
I’m trying to fetch 150 records from the db, the request lasts around 50 seconds (only the call to CouchDB).
I’m trying to fix this issue for a week now but with no success which makes it more frustrating.

This is the method in which I’m calling the couchDb using getQueryResult

public List<String> queryBy(String querySelector) {
hasText(querySelector, "QuerySelector is required!");
try (QueryResultsIterator<KeyValue> queryResult = ctx.getStub().getQueryResult(querySelector)) {
log.info("{}. {} query returned finished {} sec.", LocalDateTime.now(), domainName, querySelector);
return stream(queryResult.spliterator(), false)
.map(KeyValue::getStringValue)
.collect(toList());
} catch (Exception e)
{ log.error("error when querying the state db", e); throw new RuntimeException("Error when querying the state db", e); }
}

There is not much going on here, I’m executing the chaincode as query as suggested first.

Initially as Pras suggested I was worried that maybe ObjectMapper that I was using for deserializing the response into a POJO would be causing the lag, but after removing it the problem still persists.

When looking at logs I can see that couch is responding with the results quite quick, then peer logs with ‘[couchdb] QueryDocuments -> DEBU 25c1[0m [banyan-channel_poc-services] Adding json docment for id:` and ` [lockbasedtxmgr] Next -> DEBU 25c4[0m queryResultsItr.Next() returned a record: `

I was trying to go by the classes used during the call: InvocationStubImpl, QueryResultsIteratorImpl and somehow try to find the cause of this call being so long.
I’m running this on a 16GB, 6core i7 MacBookPro, A friend ran the same chaincode on a 32 GB dell machine – almost identical results.

I hope I’m making something wrong but if the Hyperledger has such technical limitation to which transfer is so slow I would anticipate it knowing I cannot do anything about it. Then, I will try and search for something to cover such long query executions from outside the blockchain or change the architecture entirely. Only thing, I notices discussions on the mailing lists about working on datasets far bigger then my (2-3GB big or 80 million records) so I would imagine I still have some space to play with my 150 record.


Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 20:01
Do:
Marek Malik <info@...>, "fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

I'm not familiar with java chaincode.
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric.
Also you can try using some other language for chaincode and see if the problem persists.

On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:
Managed to run the network in single org setup – the time is exactly the same ;/


Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 15:33
Do:
Marek Malik <info@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power.

- Stats from "top" command can be helpful to analyze further.

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu.

By any chance you are tweaking the "validatorPoolSize" to use fewer cores?

On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:

Hi Pras,

This network setup is also working on a 8GB MacBook Pro ?
This are the docker stats, nothing but the peer two which I’m sending to the query is running -

The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).

I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

I have some small problems with configuration of the consortium when trying to setup a single org.

Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 22:55
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook.
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle.
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke.
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference.

Regards,
Pras

On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:
Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer - after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:
{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.
Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 08:42
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

What's your hardware specs?
And make sure you're performing query rather than invoke.


On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:
Hi Alexis,

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic.

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable.

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

OrdererType: etcdraft

BatchTimeout: 2s

BatchSize:

MaxMessageCount: 2

AbsoluteMaxBytes: 99 MB

PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.



Marek Malik
 

One more thing that I found. I’ve loaded 600+ records.
The query took: 259 seconds.


But the mysterious:
lockbasedtxmgr has been iterating over the data in 6 groups of 100 items and last that was 16 items only. Each time around 43 seconds gap between each iteration.

It looks like there is a process that accumulates the data send from the CouchDB, iterates through it and sends it to the peer. Can someone confirm this is correct, especially when only evaluating a chaincode?

I found a package in Hyperledger fabric, but I’m not 100% sure how to go throw with it.
 
https://github.com/hyperledger/fabric/tree/release-2.0/core/ledger/kvledger/txmgmt/txmgr/lockbasedtxmgr



Best Regards,  
Marek Malik

 

Od: <fabric@...> w imieniu użytkownika Marek Malik <info@...>
Data: środa, 6 maja 2020 23:26
Do: Prasanth Sundaravelu <prasanths96@...>
DW: David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Prasanth,

Please check the logs I have send in the last email.

You can search for log entries from Class: ByteStorageBaseService
This is the class where I make the
getQueryResult method call.
Between this call and the actual handling of the response you will find three logs (“query started”’ – log just before calling db, “query returned finished”- call returned, and “query copping finished”-  results copied to second variable). Between the second and third you should see around 9 sec. difference. The time between first and second is the one that hurts me.

For visualization, so you will know the this is the class method -> attachment.


Additionally, as pointed in the last email. I notices that the process named or tagged
lockbasedtxmgr which looks like something that is iterating thought the data from the CouchDB has stopped after 101 records, and restarted after about 45 seconds and iterated the remaining 44 records.
You can query this by using reqExp: 18:24:07 *.*queryResultsItr.Next() and 18:24:51 *.*queryResultsItr.Next()

 

These results are reoccurring, I tested it 4 times and the scenario is the same.
Again, does anyone have any idea what this process is
lockbasedtxmgr, and how it is controlled? Can I somehow influence the iterable counter there (if there is any) or at least lower the sleep time.


Best Regards,
Marek

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: środa, 6 maja 2020 22:46
Do: Marek Malik <info@...>
DW: David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

I think you need to focus only on chaincode. 

Why not try adding detailed logs in chaincode with time, from the first point of contact (from Invoke / Query) to the deepest function in the nest and see:
1. Which function takes lot of time
2. Then further add more logs to that function for every important / heavy line and see which line takes more time

Viewing chaincode container log will make it more clear (not the peer log)

I believe this might get us some good leads, since David also confirmed that there was no problem with the Chaincode container - peer - couchdb round-trip.

 

On Thu, May 7, 2020 at 12:56 AM Marek Malik <info@...> wrote:

Hi David,
Yes, this is also something that I noticed.
In the logs you can find the following process

peer0.org1.example.com|[36m2020-05-06 18:24:07.093 UTC [lockbasedtxmgr] Next -> DEBU 3078[0m queryResultsItr.Next() returned a record:{

It is spanned from the moment CouchDB returned the response and when my getQueryResult  method call returned with a response.

this log has a the gap that makes the process taking so long - > 44seconds.
I couldn’t find any reference about this lockbasedtxmgr  or a service that prints a similar logs in java lib, are you in any way aware about such process?

It looks like the first portion read 101 records, and the second read the remaining 44. It lasted exactly 27 ms and 10 ms the second one.

Only option that I found in the network configuration that was explisitly set was the max message count and that was in Orderer setup (I thought that with query calls orderers are not used).


--
Pozdrawiam | Best regards, 

Marek Malik, Solutions Engineer

 

Phone: +48 605 330 093

info@...

 

Od: David Enyeart <enyeart@...>
Data: środa, 6 maja 2020 20:32
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>, Prasanth Sundaravelu <prasanths96@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

I don't know what is causing the slow response, but I do know that from analyzing your log in https://jira.hyperledger.org/browse/FAB-17842 that the chaincode container to peer to CouchDB roundtrip for your JSON query with 150 results only takes 158 milliseconds, with CouchDB itself taking 73 milliseconds. So focus should be on the chaincode or java chaincode shim side.


Dave Enyeart


"Marek Malik" ---05/06/2020 01:23:14 PM---Hi Guys, Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

From: "Marek Malik" <info@...>
To: Prasanth Sundaravelu <prasanths96@...>, "fabric@..." <fabric@...>
Date: 05/06/2020 01:23 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by: fabric@...





Hi Guys,

Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?
I’m trying to fetch 150 records from the db, the request lasts around 50 seconds (only the call to CouchDB).
I’m trying to fix this issue for a week now but with no success which makes it more frustrating.

This is the method in which I’m calling the couchDb using getQueryResult

public List<String> queryBy(String querySelector) {
hasText(querySelector, "QuerySelector is required!");
try (QueryResultsIterator<KeyValue> queryResult = ctx.getStub().getQueryResult(querySelector)) {
log.info("{}. {} query returned finished {} sec.", LocalDateTime.now(), domainName, querySelector);
return stream(queryResult.spliterator(), false)
.map(KeyValue::getStringValue)
.collect(toList());
} catch (Exception e)
{ log.error("error when querying the state db", e); throw new RuntimeException("Error when querying the state db", e); }
}

There is not much going on here, I’m executing the chaincode as query as suggested first.

Initially as Pras suggested I was worried that maybe ObjectMapper that I was using for deserializing the response into a POJO would be causing the lag, but after removing it the problem still persists.

When looking at logs I can see that couch is responding with the results quite quick, then peer logs with ‘[couchdb] QueryDocuments -> DEBU 25c1[0m [banyan-channel_poc-services] Adding json docment for id:` and ` [lockbasedtxmgr] Next -> DEBU 25c4[0m queryResultsItr.Next() returned a record: `

I was trying to go by the classes used during the call: InvocationStubImpl, QueryResultsIteratorImpl and somehow try to find the cause of this call being so long.
I’m running this on a 16GB, 6core i7 MacBookPro, A friend ran the same chaincode on a 32 GB dell machine – almost identical results.

I hope I’m making something wrong but if the Hyperledger has such technical limitation to which transfer is so slow I would anticipate it knowing I cannot do anything about it. Then, I will try and search for something to cover such long query executions from outside the blockchain or change the architecture entirely. Only thing, I notices discussions on the mailing lists about working on datasets far bigger then my (2-3GB big or 80 million records) so I would imagine I still have some space to play with my 150 record.


Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 20:01
Do:
Marek Malik <info@...>, "fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

I'm not familiar with java chaincode.
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric.
Also you can try using some other language for chaincode and see if the problem persists.

On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:
Managed to run the network in single org setup – the time is exactly the same ;/


Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 15:33
Do:
Marek Malik <info@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power.

- Stats from "top" command can be helpful to analyze further.

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu.

By any chance you are tweaking the "validatorPoolSize" to use fewer cores?

On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:

Hi Pras,

This network setup is also working on a 8GB MacBook Pro ?
This are the docker stats, nothing but the peer two which I’m sending to the query is running -

The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).

I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

I have some small problems with configuration of the consortium when trying to setup a single org.

Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 22:55
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook.
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle.
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke.
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference.

Regards,
Pras

On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:
Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer - after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:
{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.
Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 08:42
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

What's your hardware specs?
And make sure you're performing query rather than invoke.


On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:
Hi Alexis,

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic.

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable.

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

OrdererType: etcdraft

BatchTimeout: 2s

BatchSize:

MaxMessageCount: 2

AbsoluteMaxBytes: 99 MB

PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.




Prasanth Sundaravelu
 

Query results iterator max count is hard coded to 100. It makes sense that it sends in batch of 100. 

What doesn't make sense is why the pause. A Query results iterator is also returned when performing range query as well. When using it, I never faced such issue. In-fact, it is much faster than fetching ids one by one using GetState API.


On Thu, 7 May 2020, 3:48 am Marek Malik, <info@...> wrote:

One more thing that I found. I’ve loaded 600+ records.
The query took: 259 seconds.


But the mysterious:
lockbasedtxmgr has been iterating over the data in 6 groups of 100 items and last that was 16 items only. Each time around 43 seconds gap between each iteration.

It looks like there is a process that accumulates the data send from the CouchDB, iterates through it and sends it to the peer. Can someone confirm this is correct, especially when only evaluating a chaincode?

I found a package in Hyperledger fabric, but I’m not 100% sure how to go throw with it.
 
https://github.com/hyperledger/fabric/tree/release-2.0/core/ledger/kvledger/txmgmt/txmgr/lockbasedtxmgr



Best Regards,  
Marek Malik

 

Od: <fabric@...> w imieniu użytkownika Marek Malik <info@...>
Data: środa, 6 maja 2020 23:26
Do: Prasanth Sundaravelu <prasanths96@...>
DW: David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Prasanth,

Please check the logs I have send in the last email.

You can search for log entries from Class: ByteStorageBaseService
This is the class where I make the
getQueryResult method call.
Between this call and the actual handling of the response you will find three logs (“query started”’ – log just before calling db, “query returned finished”- call returned, and “query copping finished”-  results copied to second variable). Between the second and third you should see around 9 sec. difference. The time between first and second is the one that hurts me.

For visualization, so you will know the this is the class method -> attachment.


Additionally, as pointed in the last email. I notices that the process named or tagged
lockbasedtxmgr which looks like something that is iterating thought the data from the CouchDB has stopped after 101 records, and restarted after about 45 seconds and iterated the remaining 44 records.
You can query this by using reqExp: 18:24:07 *.*queryResultsItr.Next() and 18:24:51 *.*queryResultsItr.Next()

 

These results are reoccurring, I tested it 4 times and the scenario is the same.
Again, does anyone have any idea what this process is
lockbasedtxmgr, and how it is controlled? Can I somehow influence the iterable counter there (if there is any) or at least lower the sleep time.


Best Regards,
Marek

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data: środa, 6 maja 2020 22:46
Do: Marek Malik <info@...>
DW: David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

I think you need to focus only on chaincode. 

Why not try adding detailed logs in chaincode with time, from the first point of contact (from Invoke / Query) to the deepest function in the nest and see:
1. Which function takes lot of time
2. Then further add more logs to that function for every important / heavy line and see which line takes more time

Viewing chaincode container log will make it more clear (not the peer log)

I believe this might get us some good leads, since David also confirmed that there was no problem with the Chaincode container - peer - couchdb round-trip.

 

On Thu, May 7, 2020 at 12:56 AM Marek Malik <info@...> wrote:

Hi David,
Yes, this is also something that I noticed.
In the logs you can find the following process

peer0.org1.example.com|[36m2020-05-06 18:24:07.093 UTC [lockbasedtxmgr] Next -> DEBU 3078[0m queryResultsItr.Next() returned a record:{

It is spanned from the moment CouchDB returned the response and when my getQueryResult  method call returned with a response.

this log has a the gap that makes the process taking so long - > 44seconds.
I couldn’t find any reference about this lockbasedtxmgr  or a service that prints a similar logs in java lib, are you in any way aware about such process?

It looks like the first portion read 101 records, and the second read the remaining 44. It lasted exactly 27 ms and 10 ms the second one.

Only option that I found in the network configuration that was explisitly set was the max message count and that was in Orderer setup (I thought that with query calls orderers are not used).


--
Pozdrawiam | Best regards, 

Marek Malik, Solutions Engineer

 

Phone: +48 605 330 093

info@...

 

Od: David Enyeart <enyeart@...>
Data: środa, 6 maja 2020 20:32
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>, Prasanth Sundaravelu <prasanths96@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

I don't know what is causing the slow response, but I do know that from analyzing your log in https://jira.hyperledger.org/browse/FAB-17842 that the chaincode container to peer to CouchDB roundtrip for your JSON query with 150 results only takes 158 milliseconds, with CouchDB itself taking 73 milliseconds. So focus should be on the chaincode or java chaincode shim side.


Dave Enyeart


"Marek Malik" ---05/06/2020 01:23:14 PM---Hi Guys, Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

From: "Marek Malik" <info@...>
To: Prasanth Sundaravelu <prasanths96@...>, "fabric@..." <fabric@...>
Date: 05/06/2020 01:23 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by: fabric@...





Hi Guys,

Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?
I’m trying to fetch 150 records from the db, the request lasts around 50 seconds (only the call to CouchDB).
I’m trying to fix this issue for a week now but with no success which makes it more frustrating.

This is the method in which I’m calling the couchDb using getQueryResult

public List<String> queryBy(String querySelector) {
hasText(querySelector, "QuerySelector is required!");
try (QueryResultsIterator<KeyValue> queryResult = ctx.getStub().getQueryResult(querySelector)) {
log.info("{}. {} query returned finished {} sec.", LocalDateTime.now(), domainName, querySelector);
return stream(queryResult.spliterator(), false)
.map(KeyValue::getStringValue)
.collect(toList());
} catch (Exception e)
{ log.error("error when querying the state db", e); throw new RuntimeException("Error when querying the state db", e); }
}

There is not much going on here, I’m executing the chaincode as query as suggested first.

Initially as Pras suggested I was worried that maybe ObjectMapper that I was using for deserializing the response into a POJO would be causing the lag, but after removing it the problem still persists.

When looking at logs I can see that couch is responding with the results quite quick, then peer logs with ‘[couchdb] QueryDocuments -> DEBU 25c1[0m [banyan-channel_poc-services] Adding json docment for id:` and ` [lockbasedtxmgr] Next -> DEBU 25c4[0m queryResultsItr.Next() returned a record: `

I was trying to go by the classes used during the call: InvocationStubImpl, QueryResultsIteratorImpl and somehow try to find the cause of this call being so long.
I’m running this on a 16GB, 6core i7 MacBookPro, A friend ran the same chaincode on a 32 GB dell machine – almost identical results.

I hope I’m making something wrong but if the Hyperledger has such technical limitation to which transfer is so slow I would anticipate it knowing I cannot do anything about it. Then, I will try and search for something to cover such long query executions from outside the blockchain or change the architecture entirely. Only thing, I notices discussions on the mailing lists about working on datasets far bigger then my (2-3GB big or 80 million records) so I would imagine I still have some space to play with my 150 record.


Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 20:01
Do:
Marek Malik <info@...>, "fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

I'm not familiar with java chaincode.
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric.
Also you can try using some other language for chaincode and see if the problem persists.

On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:
Managed to run the network in single org setup – the time is exactly the same ;/


Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 15:33
Do:
Marek Malik <info@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power.

- Stats from "top" command can be helpful to analyze further.

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu.

By any chance you are tweaking the "validatorPoolSize" to use fewer cores?

On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:

Hi Pras,

This network setup is also working on a 8GB MacBook Pro ?
This are the docker stats, nothing but the peer two which I’m sending to the query is running -

The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).

I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

I have some small problems with configuration of the consortium when trying to setup a single org.

Pozdrawiam
Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 22:55
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook.
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle.
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke.
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference.

Regards,
Pras

On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:
Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer - after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:
{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.
Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 08:42
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

What's your hardware specs?
And make sure you're performing query rather than invoke.


On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:
Hi Alexis,

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic.

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable.

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

OrdererType: etcdraft

BatchTimeout: 2s

BatchSize:

MaxMessageCount: 2

AbsoluteMaxBytes: 99 MB

PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.




David Enyeart
 

I've been able to reproduce the problem.
I updated FabCar to initialize 140 cars instead of 10. Query all cars takes the following times:
Go chaincode - 70ms
Javascript chaincode - 90ms
Java chaincode - 3500ms.

I added debug to the Java chaincode... the 3 second delay happens AFTER the chaincode container receives a batch of 100 records, but BEFORE control is passed to the user chaincode. Therefore the problem appears to be somewhere in the Java chaincode shim. I've updated the bug to Fabric Chaincode Java project. The experts in that project could take a look.
The new Jira number is: https://jira.hyperledger.org/browse/FABCJ-285


Dave Enyeart

"Prasanth Sundaravelu" ---05/06/2020 08:05:55 PM---Query results iterator max count is hard coded to 100. It makes sense that it sends in batch of 100.

From: "Prasanth Sundaravelu" <prasanths96@...>
To: Marek Malik <info@...>
Cc: David Enyeart <enyeart@...>, fabric@...
Date: 05/06/2020 08:05 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by: fabric@...





Query results iterator max count is hard coded to 100. It makes sense that it sends in batch of 100. 

What doesn't make sense is why the pause. A Query results iterator is also returned when performing range query as well. When using it, I never faced such issue. In-fact, it is much faster than fetching ids one by one using GetState API.

Can you try using classic way of iterating the query result similar to: https://github.com/hyperledger/fabric-samples/blob/master/chaincode/fabcar/java/src/main/java/org/hyperledger/fabric/samples/fabcar/FabCar.java#L152
instead of spliterator?

On Thu, 7 May 2020, 3:48 am Marek Malik, <info@...> wrote:
    One more thing that I found. I’ve loaded 600+ records.
    The query took: 259 seconds.


    But the mysterious: lockbasedtxmgr has been iterating over the data in 6 groups of 100 items and last that was 16 items only. Each time around 43 seconds gap between each iteration.

    It looks like there is a process that accumulates the data send from the CouchDB, iterates through it and sends it to the peer. Can someone confirm this is correct, especially when only evaluating a chaincode?

    I found a package in Hyperledger fabric, but I’m not 100% sure how to go throw with it.
     https://github.com/hyperledger/fabric/tree/release-2.0/core/ledger/kvledger/txmgmt/txmgr/lockbasedtxmgr



    Best Regards,  
    Marek Malik

     

    Od: <fabric@...> w imieniu użytkownika Marek Malik <info@...>
    Data:
    środa, 6 maja 2020 23:26
    Do:
    Prasanth Sundaravelu <prasanths96@...>
    DW:
    David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
    Temat:
    Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

     

    Hi Prasanth,

    Please check the logs I have send in the last email.

    You can search for log entries from Class: ByteStorageBaseService
    This is the class where I make the getQueryResult method call.
    Between this call and the actual handling of the response you will find three logs (“query started”’ – log just before calling db, “query returned finished”- call returned, and “query copping finished”-  results copied to second variable). Between the second and third you should see around 9 sec. difference. The time between first and second is the one that hurts me.

    For visualization, so you will know the this is the class method -> attachment.


    Additionally, as pointed in the last email. I notices that the process named or tagged lockbasedtxmgr which looks like something that is iterating thought the data from the CouchDB has stopped after 101 records, and restarted after about 45 seconds and iterated the remaining 44 records.
    You can query this by using reqExp: 18:24:07 *.*queryResultsItr.Next() and 18:24:51 *.*queryResultsItr.Next()

     

    These results are reoccurring, I tested it 4 times and the scenario is the same.
    Again, does anyone have any idea what this process is lockbasedtxmgr, and how it is controlled? Can I somehow influence the iterable counter there (if there is any) or at least lower the sleep time.


    Best Regards,
    Marek

     

    Od: Prasanth Sundaravelu <prasanths96@...>
    Data:
    środa, 6 maja 2020 22:46
    Do:
    Marek Malik <info@...>
    DW:
    David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
    Temat:
    Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

     

    I think you need to focus only on chaincode. 

    Why not try adding detailed logs in chaincode with time, from the first point of contact (from Invoke / Query) to the deepest function in the nest and see:
    1. Which function takes lot of time
    2. Then further add more logs to that function for every important / heavy line and see which line takes more time

    Viewing chaincode container log will make it more clear (not the peer log)

    I believe this might get us some good leads, since David also confirmed that there was no problem with the Chaincode container - peer - couchdb round-trip.

     

On Thu, May 7, 2020 at 12:56 AM Marek Malik <info@...> wrote:
Hi David,
Yes, this is also something that I noticed.
In the logs you can find the following process

peer0.org1.example.com|[36m2020-05-06 18:24:07.093 UTC [lockbasedtxmgr] Next -> DEBU 3078[0m queryResultsItr.Next() returned a record:{

It is spanned from the moment CouchDB returned the response and when my getQueryResult  method call returned with a response.

this log has a the gap that makes the process taking so long - > 44seconds.
I couldn’t find any reference about this lockbasedtxmgr  or a service that prints a similar logs in java lib, are you in any way aware about such process?

It looks like the first portion read 101 records, and the second read the remaining 44. It lasted exactly 27 ms and 10 ms the second one.

Only option that I found in the network configuration that was explisitly set was the max message count and that was in Orderer setup (I thought that with query calls orderers are not used).


--
Pozdrawiam | Best regards, 

Marek Malik, Solutions Engineer

 

Phone: +48 605 330 093

info@...

 

Od: David Enyeart <enyeart@...>
Data:
środa, 6 maja 2020 20:32
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>, Prasanth Sundaravelu <prasanths96@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

I don't know what is causing the slow response, but I do know that from analyzing your log in https://jira.hyperledger.org/browse/FAB-17842 that the chaincode container to peer to CouchDB roundtrip for your JSON query with 150 results only takes 158 milliseconds, with CouchDB itself taking 73 milliseconds. So focus should be on the chaincode or java chaincode shim side.


Dave Enyeart


"Marek Malik" ---05/06/2020 01:23:14 PM---Hi Guys, Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

From:
"Marek Malik" <info@...>
To:
Prasanth Sundaravelu <prasanths96@...>, "fabric@..." <fabric@...>
Date:
05/06/2020 01:23 PM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by:
fabric@...





Hi Guys,

Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?
I’m trying to fetch 150 records from the db, the request lasts around 50 seconds (only the call to CouchDB).
I’m trying to fix this issue for a week now but with no success which makes it more frustrating.

This is the method in which I’m calling the couchDb using getQueryResult

public List<String> queryBy(String querySelector) {
hasText(querySelector, "QuerySelector is required!");
try (QueryResultsIterator<KeyValue> queryResult = ctx.getStub().getQueryResult(querySelector)) {
log.info("{}. {} query returned finished {} sec.", LocalDateTime.now(), domainName, querySelector);
return stream(queryResult.spliterator(), false)
.map(KeyValue::getStringValue)
.collect(toList());
} catch (Exception e)
{ log.error("error when querying the state db", e); throw new RuntimeException("Error when querying the state db", e); }
}

There is not much going on here, I’m executing the chaincode as query as suggested first.

Initially as Pras suggested I was worried that maybe ObjectMapper that I was using for deserializing the response into a POJO would be causing the lag, but after removing it the problem still persists.

When looking at logs I can see that couch is responding with the results quite quick, then peer logs with ‘[couchdb] QueryDocuments -> DEBU 25c1[0m [banyan-channel_poc-services] Adding json docment for id:` and ` [lockbasedtxmgr] Next -> DEBU 25c4[0m queryResultsItr.Next() returned a record: `

I was trying to go by the classes used during the call: InvocationStubImpl, QueryResultsIteratorImpl and somehow try to find the cause of this call being so long.
I’m running this on a 16GB, 6core i7 MacBookPro, A friend ran the same chaincode on a 32 GB dell machine – almost identical results.

I hope I’m making something wrong but if the Hyperledger has such technical limitation to which transfer is so slow I would anticipate it knowing I cannot do anything about it. Then, I will try and search for something to cover such long query executions from outside the blockchain or change the architecture entirely. Only thing, I notices discussions on the mailing lists about working on datasets far bigger then my (2-3GB big or 80 million records) so I would imagine I still have some space to play with my 150 record.


Pozdrawiam
Marek Malik

Od:
Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 20:01
Do:
Marek Malik <info@...>, "fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

I'm not familiar with java chaincode.
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric.
Also you can try using some other language for chaincode and see if the problem persists.

On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:
Managed to run the network in single org setup – the time is exactly the same ;/


Pozdrawiam
Marek Malik

Od:
Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 15:33
Do:
Marek Malik <info@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power.

- Stats from "top" command can be helpful to analyze further.

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu.

By any chance you are tweaking the "validatorPoolSize" to use fewer cores?

On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:

      Hi Pras,

      This network setup is also working on a 8GB MacBook Pro ?
      This are the docker stats, nothing but the peer two which I’m sending to the query is running -

      The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).

      I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

      I have some small problems with configuration of the consortium when trying to setup a single org.

      Pozdrawiam
      Marek Malik

      Od:
      Prasanth Sundaravelu <prasanths96@...>
      Data:
      niedziela, 3 maja 2020 22:55
      Do:
      Marek Malik <info@...>
      DW:
      "fabric@..." <fabric@...>
      Temat:
      Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

      Hi Marek,

      1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
      2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook.
      - Try equivalent command for mac as "top" in linux and see how much load is taken when idle.
      - Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke.
      - Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference.

      Regards,
      Pras

      On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:
      Hi Prasanth,

      So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

      In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer - after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
      I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:
      {"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
      The index is created. When testing this query on Fauxton the query took 40 ms.
      Any ideas ?

      I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

      Od:
      Prasanth Sundaravelu <prasanths96@...>
      Data:
      niedziela, 3 maja 2020 08:42
      Do:
      Marek Malik <info@...>
      DW:
      "fabric@..." <fabric@...>
      Temat:
      Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

      Hi Marek,

      What's your hardware specs?
      And make sure you're performing query rather than invoke.


      On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:
      Hi Alexis,

      This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic.

      Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable.

      The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

      The network setup is the following:
      ```

      Orderer: &OrdererDefaults

      OrdererType: etcdraft

      BatchTimeout: 2s

      BatchSize:

      MaxMessageCount: 2

      AbsoluteMaxBytes: 99 MB

      PreferredMaxBytes: 512 KB

      ```

      How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.







Marek Malik
 

Hi David,

I have tried to replicate the same scenario you setup with FabCar, because your response was quite nice (3500ms for 140 records, this would entirely acceptable for my team at this stage).
I have setup a smaller data set – 128 cars and tried to play around with the chaincode. I even tried serialization/deserialization using Jackson lib, which gave a quicker response but very little (around 100-500ms).

I’m trying to identify main differences between the setup of FabCar with the test-network and the one that I build based on a combination of that and “basic-network” that I used.
I noticed among the chaincode and network setup the following differences:

  • The chaincode packaging doesn’t use shadowjar, but 'java-library-distribution'
  • The gateway is using the old 1.4.1 version, not sure if this has anything to do with the way the chaincode is executed or the time of the response.
  • fabric-ca is used instead of cryptogen
  • there is a general config folder with configtx.yaml, core.yaml and orderer.yaml files, which is but couldn’t find any reference too these files. I don’t have these files, I’m using the same configuration from the folder: test-network/configtx/configtx.yaml, should I instead point to that folder when running configtxgen?

 

 

The network seems to be working more responsively. Do you by any chance know what could be also be difference in the setup?

I’m testing FabCar with additional property types on the Car entity (like date parsed to LocalDateTime etc), also I don’t feel this could by any chance influence the results but in my chaincode I’m using multiple DataTypes that I;m storing in the ledger from one chaincode. This works well, I don’t imagine this could in anyway influence the results, right ?

 


Marek Malik

 

Od: David Enyeart <enyeart@...>
Data: czwartek, 7 maja 2020 05:57
Do: Prasanth Sundaravelu <prasanths96@...>
DW: "fabric@..." <fabric@...>, Marek Malik <info@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

I've been able to reproduce the problem.
I updated FabCar to initialize 140 cars instead of 10. Query all cars takes the following times:
Go chaincode - 70ms
Javascript chaincode - 90ms
Java chaincode - 3500ms.

I added debug to the Java chaincode... the 3 second delay happens AFTER the chaincode container receives a batch of 100 records, but BEFORE control is passed to the user chaincode. Therefore the problem appears to be somewhere in the Java chaincode shim. I've updated the bug to Fabric Chaincode Java project. The experts in that project could take a look.
The new Jira number is: https://jira.hyperledger.org/browse/FABCJ-285


Dave Enyeart


"Prasanth Sundaravelu" ---05/06/2020 08:05:55 PM---Query results iterator max count is hard coded to 100. It makes sense that it sends in batch of 100.

From: "Prasanth Sundaravelu" <prasanths96@...>
To: Marek Malik <info@...>
Cc: David Enyeart <enyeart@...>, fabric@...
Date: 05/06/2020 08:05 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by: fabric@...





Query results iterator max count is hard coded to 100. It makes sense that it sends in batch of 100. 

What doesn't make sense is why the pause. A Query results iterator is also returned when performing range query as well. When using it, I never faced such issue. In-fact, it is much faster than fetching ids one by one using GetState API.

Can you try using classic way of iterating the query result similar to: https://github.com/hyperledger/fabric-samples/blob/master/chaincode/fabcar/java/src/main/java/org/hyperledger/fabric/samples/fabcar/FabCar.java#L152
instead of spliterator?

On Thu, 7 May 2020, 3:48 am Marek Malik, <info@...> wrote:

One more thing that I found. I’ve loaded 600+ records.
The query took: 259 seconds.


But the mysterious: lockbasedtxmgr has been iterating over the data in 6 groups of 100 items and last that was 16 items only. Each time around 43 seconds gap between each iteration.

It looks like there is a process that accumulates the data send from the CouchDB, iterates through it and sends it to the peer. Can someone confirm this is correct, especially when only evaluating a chaincode?

I found a package in Hyperledger fabric, but I’m not 100% sure how to go throw with it.
 https://github.com/hyperledger/fabric/tree/release-2.0/core/ledger/kvledger/txmgmt/txmgr/lockbasedtxmgr



Best Regards,  
Marek Malik

 

Od: <fabric@...> w imieniu użytkownika Marek Malik <info@...>
Data:
środa, 6 maja 2020 23:26
Do:
Prasanth Sundaravelu <prasanths96@...>
DW:
David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Prasanth,

Please check the logs I have send in the last email.

You can search for log entries from Class: ByteStorageBaseService
This is the class where I make the getQueryResult method call.
Between this call and the actual handling of the response you will find three logs (“query started”’ – log just before calling db, “query returned finished”- call returned, and “query copping finished”-  results copied to second variable). Between the second and third you should see around 9 sec. difference. The time between first and second is the one that hurts me.

For visualization, so you will know the this is the class method -> attachment.


Additionally, as pointed in the last email. I notices that the process named or tagged lockbasedtxmgr which looks like something that is iterating thought the data from the CouchDB has stopped after 101 records, and restarted after about 45 seconds and iterated the remaining 44 records.
You can query this by using reqExp: 18:24:07 *.*queryResultsItr.Next() and 18:24:51 *.*queryResultsItr.Next()

 

These results are reoccurring, I tested it 4 times and the scenario is the same.
Again, does anyone have any idea what this process is lockbasedtxmgr, and how it is controlled? Can I somehow influence the iterable counter there (if there is any) or at least lower the sleep time.


Best Regards,
Marek

 

Od: Prasanth Sundaravelu <prasanths96@...>
Data:
środa, 6 maja 2020 22:46
Do:
Marek Malik <info@...>
DW:
David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

I think you need to focus only on chaincode. 

Why not try adding detailed logs in chaincode with time, from the first point of contact (from Invoke / Query) to the deepest function in the nest and see:
1. Which function takes lot of time
2. Then further add more logs to that function for every important / heavy line and see which line takes more time

Viewing chaincode container log will make it more clear (not the peer log)

I believe this might get us some good leads, since David also confirmed that there was no problem with the Chaincode container - peer - couchdb round-trip.

 

On Thu, May 7, 2020 at 12:56 AM Marek Malik <info@...> wrote:
Hi David,
Yes, this is also something that I noticed.
In the logs you can find the following process

peer0.org1.example.com|[36m2020-05-06 18:24:07.093 UTC [lockbasedtxmgr] Next -> DEBU 3078[0m queryResultsItr.Next() returned a record:{

It is spanned from the moment CouchDB returned the response and when my getQueryResult  method call returned with a response.

this log has a the gap that makes the process taking so long - > 44seconds.
I couldn’t find any reference about this lockbasedtxmgr  or a service that prints a similar logs in java lib, are you in any way aware about such process?

It looks like the first portion read 101 records, and the second read the remaining 44. It lasted exactly 27 ms and 10 ms the second one.

Only option that I found in the network configuration that was explisitly set was the max message count and that was in Orderer setup (I thought that with query calls orderers are not used).


--
Pozdrawiam | Best regards, 

Marek Malik, Solutions Engineer

 

Phone: +48 605 330 093

info@...

 

Od: David Enyeart <enyeart@...>
Data:
środa, 6 maja 2020 20:32
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>, Prasanth Sundaravelu <prasanths96@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

 

Hi Marek,

I don't know what is causing the slow response, but I do know that from analyzing your log in https://jira.hyperledger.org/browse/FAB-17842 that the chaincode container to peer to CouchDB roundtrip for your JSON query with 150 results only takes 158 milliseconds, with CouchDB itself taking 73 milliseconds. So focus should be on the chaincode or java chaincode shim side.


Dave Enyeart



"Marek Malik" ---05/06/2020 01:23:14 PM---Hi Guys, Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

From:
"Marek Malik" <info@...>
To:
Prasanth Sundaravelu <
prasanths96@...>, "fabric@..." <fabric@...>
Date:
05/06/2020 01:23 PM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by:
fabric@...





Hi Guys,

Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?
I’m trying to fetch 150 records from the db, the request lasts around 50 seconds (only the call to CouchDB).
I’m trying to fix this issue for a week now but with no success which makes it more frustrating.

This is the method in which I’m calling the couchDb using getQueryResult

public List<String> queryBy(String querySelector) {
hasText(querySelector, "QuerySelector is required!");
try (QueryResultsIterator<KeyValue> queryResult = ctx.getStub().getQueryResult(querySelector)) {
log.info("{}. {} query returned finished {} sec.", LocalDateTime.now(), domainName, querySelector);
return stream(queryResult.spliterator(), false)
.map(KeyValue::getStringValue)
.collect(toList());
} catch (Exception e)
{ log.error("error when querying the state db", e); throw new RuntimeException("Error when querying the state db", e); }
}

There is not much going on here, I’m executing the chaincode as query as suggested first.

Initially as Pras suggested I was worried that maybe ObjectMapper that I was using for deserializing the response into a POJO would be causing the lag, but after removing it the problem still persists.

When looking at logs I can see that couch is responding with the results quite quick, then peer logs with ‘[couchdb] QueryDocuments -> DEBU 25c1[0m [banyan-channel_poc-services] Adding json docment for id:` and ` [lockbasedtxmgr] Next -> DEBU 25c4[0m queryResultsItr.Next() returned a record: `

I was trying to go by the classes used during the call: InvocationStubImpl, QueryResultsIteratorImpl and somehow try to find the cause of this call being so long.
I’m running this on a 16GB, 6core i7 MacBookPro, A friend ran the same chaincode on a 32 GB dell machine – almost identical results.

I hope I’m making something wrong but if the Hyperledger has such technical limitation to which transfer is so slow I would anticipate it knowing I cannot do anything about it. Then, I will try and search for something to cover such long query executions from outside the blockchain or change the architecture entirely. Only thing, I notices discussions on the mailing lists about working on datasets far bigger then my (2-3GB big or 80 million records) so I would imagine I still have some space to play with my 150 record.


Pozdrawiam
Marek Malik

Od:
Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 20:01
Do:
Marek Malik <info@...>, "fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

I'm not familiar with java chaincode.
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric.
Also you can try using some other language for chaincode and see if the problem persists.

On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:
Managed to run the network in single org setup – the time is exactly the same ;/


Pozdrawiam
Marek Malik

Od:
Prasanth Sundaravelu <prasanths96@...>
Data:
poniedziałek, 4 maja 2020 15:33
Do:
Marek Malik <info@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power.

- Stats from "top" command can be helpful to analyze further.

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu.

By any chance you are tweaking the "validatorPoolSize" to use fewer cores?

On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:

Hi Pras,

This network setup is also working on a 8GB MacBook Pro ?
This are the docker stats, nothing but the peer two which I’m sending to the query is running -

The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).

I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

I have some small problems with configuration of the consortium when trying to setup a single org.

Pozdrawiam
Marek Malik

Od:
Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 22:55
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook.
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle.
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke.
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference.

Regards,
Pras

On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:
Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer - after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:
{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.
Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

Od:
Prasanth Sundaravelu <prasanths96@...>
Data:
niedziela, 3 maja 2020 08:42
Do:
Marek Malik <info@...>
DW:
"fabric@..." <fabric@...>
Temat:
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

What's your hardware specs?
And make sure you're performing query rather than invoke.


On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:
Hi Alexis,

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic.

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable.

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:
```

Orderer: &OrdererDefaults

OrdererType: etcdraft

BatchTimeout: 2s

BatchSize:

MaxMessageCount: 2

AbsoluteMaxBytes: 99 MB

PreferredMaxBytes: 512 KB

```

How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.