MVCC conflict when I benchmark Java chaincode (update function) only 1tx per second. #hyperledger-fabric #fabric-chaincode #consensus #raft


Ali ِAlzubaidi
 

Hi everyone, I have this wiered MVCC-conflict issue. It appears occasionally as I benchmark my Java chaincode. I tried everything I possibly could to prevent its occurance.
Now, I am going to list the chaincode functionality and what I did to to try prevent it.

At the HLF network: The block batching is set to 1 second timeout, and one transaction per block! I even tried half second as well, and no avail.
At client side, using Hyperledger Caliper, I only use one worker (client) that only submits one transaction per second.

The network is composed of 2 org, one peer for each.
Consensus is Raft, three orderer.
HLF version is 1.4.6
storage DB is LevelDB


At chaincode side, it programmed using Java

The mothod under test, receives a key and updated value.
it first, lock up the current state using the method "ctx.getStub().getStringState(key)".
it takes, the value and do simple add operation to the value and then updates the ket with the new value as in this --- ctx.getStub().putStringState(key,value) ---
simple as that!

Now, given the extreme setting I explained before, such that client only submits one Tx per second, and the HLF block batching is set to take one T per block or half second timeout,
I WONDER, why the MVCC conflict happens??

Sometimes I get this error:
2020.05.08-06:19:40.258 error [caliper] [adapters/fabric] Transaction[4a28362802] commit errors:
- Commit error on peer0.org1.example.com with code MVCC_READ_CONFLICT
- Commit error on peer0.org2.example.com with code MVCC_READ_CONFLICT

Other times, I get this error.
2020-05-08T05:22:05.600Z - error: [Channel.js]: compareProposalResponseResults - read/writes result sets do not match index=1

Any explanation please?

Could it be a racing condition? such that a that when the transaction method tries to read a key and update the key, such that another version incremental happens between the read and update? 

I mean could that happen given my extreme settings?


Prasanth Sundaravelu
 

Hi Ali, 

Yes. MVCC - Multi Version Concurrency Control. It occurs exactly when like you mentioned.

You need to design the chaincode in such a way to avoid frequent same key update as much as possible. 

If for testing purposes alone, write a new key on every transaction.

On Fri, 8 May 2020, 11:49 am Ali Abdulaziz, <aakzubaidi@...> wrote:

Hi everyone, I have this wiered MVCC-conflict issue. It appears occasionally as I benchmark my Java chaincode. I tried everything I possibly could to prevent its occurance.
Now, I am going to list the chaincode functionality and what I did to to try prevent it.

At the HLF network: The block batching is set to 1 second timeout, and one transaction per block! I even tried half second as well, and no avail.
At client side, using Hyperledger Caliper, I only use one worker (client) that only submits one transaction per second.

The network is composed of 2 org, one peer for each.
Consensus is Raft, three orderer.
HLF version is 1.4.6
storage DB is LevelDB


At chaincode side, it programmed using Java

The mothod under test, receives a key and updated value.
it first, lock up the current state using the method "ctx.getStub().getStringState(key)".
it takes, the value and do simple add operation to the value and then updates the ket with the new value as in this --- ctx.getStub().putStringState(key,value) ---
simple as that!

Now, given the extreme setting I explained before, such that client only submits one Tx per second, and the HLF block batching is set to take one T per block or half second timeout,
I WONDER, why the MVCC conflict happens??

Sometimes I get this error:
2020.05.08-06:19:40.258 error [caliper] [adapters/fabric] Transaction[4a28362802] commit errors:
- Commit error on peer0.org1.example.com with code MVCC_READ_CONFLICT
- Commit error on peer0.org2.example.com with code MVCC_READ_CONFLICT

Other times, I get this error.
2020-05-08T05:22:05.600Z - error: [Channel.js]: compareProposalResponseResults - read/writes result sets do not match index=1

Any explanation please?

Could it be a racing condition? such that a that when the transaction method tries to read a key and update the key, such that another version incremental happens between the read and update? 

I mean could that happen given my extreme settings?


Ali ِAlzubaidi
 

ِThanks Prasnth for your reply. 

So, you think it is a racing condition. Just to confirm with you.

So, what the best way to update a value bases on its previous value? I need an example that does not impact throughput or latency that much. I do not want to go through key management and I do not want to trust client for making delta values for me or any interference from the client. I am seeking a solution at the chaincode level. So any suggestion or examples please? 


David Enyeart
 

If you send a steady flow of updates for the same key, you will always have some transactions in flight (endorsement or ordering phase) while earlier transactions are committing on peers. Since the later transaction's reads are now outdated, the chaincode execution is considered invalid, and those transactions will get invalidated as MVCC conflict.

Many performance tests will randomly select a key, e.g. key1 through key10000. The first time seen by chaincode it will read the key and since not found in current state, will create the key. The next time seen by chaincode it will read the key and since found, will update the key. Most will pass validation. A few will collide (same key chosen within 1-2 seconds) and get invalidated, and assuming client is listening to the block validation events, client can resubmit if you want a test with 100% pass rate.

There are a number of variables you can play with, for example a quicker block cut rate (BatchTimeout) or smaller blocks (MaxMessageCount/AbsoluteMaxBytes/PreferredMaxBytes) will result in fewer conflicts, but if too small may impact overall throughput, since block processing overhead wouldn't be amortized over as many transactions.


Dave Enyeart

"Ali Abdulaziz" ---05/08/2020 03:24:39 AM---ِThanks Prasnth for your reply. So, you think it is a racing condition. Just to confirm with you.

From: "Ali Abdulaziz" <aakzubaidi@...>
To: fabric@...
Date: 05/08/2020 03:24 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] MVCC conflict when I benchmark Java chaincode (update function) only 1tx per second. #hyperledger-fabric #fabric-chaincode #consensus #raft
Sent by: fabric@...





ِThanks Prasnth for your reply.

So, you think it is a racing condition. Just to confirm with you.

So, what the best way to update a value bases on its previous value? I need an example that does not impact throughput or latency that much. I do not want to go through key management and I do not want to trust client for making delta values for me or any interference from the client. I am seeking a solution at the chaincode level. So any suggestion or examples please?