Re: Performance improvement #fabric-sdk-java #fabric-questions

Marek Malik <info@...>

One more thing that I found. I’ve loaded 600+ records.
The query took: 259 seconds.

But the mysterious:
lockbasedtxmgr has been iterating over the data in 6 groups of 100 items and last that was 16 items only. Each time around 43 seconds gap between each iteration.

It looks like there is a process that accumulates the data send from the CouchDB, iterates through it and sends it to the peer. Can someone confirm this is correct, especially when only evaluating a chaincode?

I found a package in Hyperledger fabric, but I’m not 100% sure how to go throw with it.

Best Regards,  
Marek Malik


Od: <fabric@...> w imieniu użytkownika Marek Malik <info@...>
Data: środa, 6 maja 2020 23:26
Do: Prasanth Sundaravelu <prasanths96@...>
DW: David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions


Hi Prasanth,

Please check the logs I have send in the last email.

You can search for log entries from Class: ByteStorageBaseService
This is the class where I make the
getQueryResult method call.
Between this call and the actual handling of the response you will find three logs (“query started”’ – log just before calling db, “query returned finished”- call returned, and “query copping finished”-  results copied to second variable). Between the second and third you should see around 9 sec. difference. The time between first and second is the one that hurts me.

For visualization, so you will know the this is the class method -> attachment.

Additionally, as pointed in the last email. I notices that the process named or tagged
lockbasedtxmgr which looks like something that is iterating thought the data from the CouchDB has stopped after 101 records, and restarted after about 45 seconds and iterated the remaining 44 records.
You can query this by using reqExp: 18:24:07 *.*queryResultsItr.Next() and 18:24:51 *.*queryResultsItr.Next()


These results are reoccurring, I tested it 4 times and the scenario is the same.
Again, does anyone have any idea what this process is
lockbasedtxmgr, and how it is controlled? Can I somehow influence the iterable counter there (if there is any) or at least lower the sleep time.

Best Regards,


Od: Prasanth Sundaravelu <prasanths96@...>
Data: środa, 6 maja 2020 22:46
Do: Marek Malik <info@...>
DW: David Enyeart <enyeart@...>, "fabric@..." <fabric@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions


I think you need to focus only on chaincode. 

Why not try adding detailed logs in chaincode with time, from the first point of contact (from Invoke / Query) to the deepest function in the nest and see:
1. Which function takes lot of time
2. Then further add more logs to that function for every important / heavy line and see which line takes more time

Viewing chaincode container log will make it more clear (not the peer log)

I believe this might get us some good leads, since David also confirmed that there was no problem with the Chaincode container - peer - couchdb round-trip.


On Thu, May 7, 2020 at 12:56 AM Marek Malik <info@...> wrote:

Hi David,
Yes, this is also something that I noticed.
In the logs you can find the following process|[36m2020-05-06 18:24:07.093 UTC [lockbasedtxmgr] Next -> DEBU 3078[0m queryResultsItr.Next() returned a record:{

It is spanned from the moment CouchDB returned the response and when my getQueryResult  method call returned with a response.

this log has a the gap that makes the process taking so long - > 44seconds.
I couldn’t find any reference about this lockbasedtxmgr  or a service that prints a similar logs in java lib, are you in any way aware about such process?

It looks like the first portion read 101 records, and the second read the remaining 44. It lasted exactly 27 ms and 10 ms the second one.

Only option that I found in the network configuration that was explisitly set was the max message count and that was in Orderer setup (I thought that with query calls orderers are not used).

Pozdrawiam | Best regards, 

Marek Malik, Solutions Engineer


Phone: +48 605 330 093



Od: David Enyeart <enyeart@...>
Data: środa, 6 maja 2020 20:32
Do: Marek Malik <info@...>
DW: "fabric@..." <fabric@...>, Prasanth Sundaravelu <prasanths96@...>
Temat: Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions


Hi Marek,

I don't know what is causing the slow response, but I do know that from analyzing your log in that the chaincode container to peer to CouchDB roundtrip for your JSON query with 150 results only takes 158 milliseconds, with CouchDB itself taking 73 milliseconds. So focus should be on the chaincode or java chaincode shim side.

Dave Enyeart

"Marek Malik" ---05/06/2020 01:23:14 PM---Hi Guys, Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?

From: "Marek Malik" <info@...>
To: Prasanth Sundaravelu <prasanths96@...>, "fabric@..." <fabric@...>
Date: 05/06/2020 01:23 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions
Sent by: fabric@...

Hi Guys,

Could I ask anyone for any advice or help regarding querying CouchDB from java chain code?
I’m trying to fetch 150 records from the db, the request lasts around 50 seconds (only the call to CouchDB).
I’m trying to fix this issue for a week now but with no success which makes it more frustrating.

This is the method in which I’m calling the couchDb using getQueryResult

public List<String> queryBy(String querySelector) {
hasText(querySelector, "QuerySelector is required!");
try (QueryResultsIterator<KeyValue> queryResult = ctx.getStub().getQueryResult(querySelector)) {"{}. {} query returned finished {} sec.",, domainName, querySelector);
return stream(queryResult.spliterator(), false)
} catch (Exception e)
{ log.error("error when querying the state db", e); throw new RuntimeException("Error when querying the state db", e); }

There is not much going on here, I’m executing the chaincode as query as suggested first.

Initially as Pras suggested I was worried that maybe ObjectMapper that I was using for deserializing the response into a POJO would be causing the lag, but after removing it the problem still persists.

When looking at logs I can see that couch is responding with the results quite quick, then peer logs with ‘[couchdb] QueryDocuments -> DEBU 25c1[0m [banyan-channel_poc-services] Adding json docment for id:` and ` [lockbasedtxmgr] Next -> DEBU 25c4[0m queryResultsItr.Next() returned a record: `

I was trying to go by the classes used during the call: InvocationStubImpl, QueryResultsIteratorImpl and somehow try to find the cause of this call being so long.
I’m running this on a 16GB, 6core i7 MacBookPro, A friend ran the same chaincode on a 32 GB dell machine – almost identical results.

I hope I’m making something wrong but if the Hyperledger has such technical limitation to which transfer is so slow I would anticipate it knowing I cannot do anything about it. Then, I will try and search for something to cover such long query executions from outside the blockchain or change the architecture entirely. Only thing, I notices discussions on the mailing lists about working on datasets far bigger then my (2-3GB big or 80 million records) so I would imagine I still have some space to play with my 150 record.

Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
poniedziałek, 4 maja 2020 20:01
Marek Malik <info@...>, "fabric@..." <fabric@...>
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

I'm not familiar with java chaincode.
Considering that the dev container took a lot of computing power, I will suggest you to time each major line on your query function in chaincode. It could be because of some package that you're using or it could also be the db fetch (GetState / Query).

You'll know what to fix then. If it's not any external package or your code, then it may also be because of the nature of the java chaincode in fabric.
Also you can try using some other language for chaincode and see if the problem persists.

On Mon, May 4, 2020 at 9:51 PM Marek Malik <info@...> wrote:
Managed to run the network in single org setup – the time is exactly the same ;/

Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
poniedziałek, 4 maja 2020 15:33
Marek Malik <info@...>
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

Hyperledger fabric peers are process-intensive. Having enough RAM would still affect the performance to a great extent if there is not enough processing power.

- Stats from "top" command can be helpful to analyze further.

- From docker stats that you have shared, I can see that the dev container is taking 100.45% - meaning, chaincode is being process intensive. It looks odd to see that it is taking this much process for a single query returning 150 records. Are you doing any intensive crypto operations in chaincode? Can I take a look at it?

- Assuming that the hardware is not the bottleneck (which I'm not yet satisfied that it is not) and assuming chaincode is normal, then I will then check if the setup is actually using all cores from the cpu.

By any chance you are tweaking the "validatorPoolSize" to use fewer cores?

On Mon, May 4, 2020 at 3:58 PM Marek Malik <info@...> wrote:

Hi Pras,

This network setup is also working on a 8GB MacBook Pro ?
This are the docker stats, nothing but the peer two which I’m sending to the query is running -

The setup uses around 8-9 GB of RAM but I’m still, 2-3 GB free so that is not any issue (and I’m running a lot of different tools that consumes ram so I have still some places to work).

I changed the way I persist data, wrapping the collection into a document that is stored, so I’m storing not 150 elements but one big that will have 150 elements inside, with this approach the speed didn’t increased, it took 51 sec to get the data. But with that approach I was able to construct a id key to get a single element, this also took 51 sec.

I have some small problems with configuration of the consortium when trying to setup a single org.

Marek Malik

Od: Prasanth Sundaravelu <prasanths96@...>
niedziela, 3 maja 2020 22:55
Marek Malik <info@...>
"fabric@..." <fabric@...>
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

1. CouchDB's fauxton interface returns paginated results. Check what's your per page results in that, only that many results are being returned and not all 150.
2. Two peers, two CouchDBs, 3 orderers - totally 7 containers. I'd say it is pretty heavy for a MacBook.
- Try equivalent command for mac as "top" in linux and see how much load is taken when idle.
- Also do "docker stats" to check the resource usage of each containers individually. Notice which containers take resources when a query call is made. Ideally, one peer, one couchdb only should be using resources. If orders use resources near the end, it means it is still using invoke.
- Also try querying with 1 peer, 1 couchdb and 1 orderer network, you should probably see a significant difference.


On Sun, May 3, 2020 at 8:13 PM Marek Malik <info@...> wrote:
Hi Prasanth,

So I tested the use-case with `evaluateTransaction` method from the Java SDK which should be doing the query as you suggested.

In the file you will see logs from the peer, the ChaincodeInvocationTask is logging that it received an answer - after ruffle 55 seconds (around 150 records of 200 in total), the log is printed twice, not sure why.
I think I have something else that is not working as excepted, I’m querying the CouchDB with a very simple query:
{"selector": {"projectId": "339d4be4-ad4e-4c8c-9f8f-0c96750600b3", ”data_type": "debt_service"}, "use_index": ["_design/pocServicesDoc","pocServicesIdx"]}
The index is created. When testing this query on Fauxton the query took 40 ms.
Any ideas ?

I’m running it on a 2018 MacBook Pro (16GB, 6-Core Intel Core i7), so I don’t think there is any limitation with the hardware.

Od: Prasanth Sundaravelu <prasanths96@...>
niedziela, 3 maja 2020 08:42
Marek Malik <info@...>
"fabric@..." <fabric@...>
Re: [Hyperledger Fabric] Performance improvement #fabric-sdk-java #fabric-questions

Hi Marek,

What's your hardware specs?
And make sure you're performing query rather than invoke.

On Sun, 3 May 2020, 1:00 am Marek Malik, <info@...> wrote:
Hi Alexis,

This is a fairly old thread but it is directly linked to my current problem. If you could give any feedback on your solution, that would be fantastic.

Basically, I'm building an API that will be responding with data from the blockchain, I require that the api will be responding rather quick, such that UX will be acceptable.

The current setup that I have locally is 2 orgs (one peer per organization) and 3 Rafi orderers. When fetching around 150 records from the CouchDB it takes around 15 seconds. I already identified that the db is not the cause of the problem, it fetches the data in around 20ms.

The network setup is the following:

Orderer: &OrdererDefaults

OrdererType: etcdraft

BatchTimeout: 2s


MaxMessageCount: 2

AbsoluteMaxBytes: 99 MB

PreferredMaxBytes: 512 KB


How did you guys resolve performance problems? It would seem that 150 small records returned as one collection from the blockchain should not be any problem.

Join to automatically receive all group messages.