Date   

Re: Performance Improvement: Max number of assets and max size of payload in single Transaction?

David Enyeart
 

On the other hand, large numbers of tiny transactions will not be as efficient due to the per-transaction overhead. It is quite typical to batch multiple writes into a single transaction (e.g. 100 at a time), especially upon initial ledger population. This provides a good balance between efficiency and keeping the transactions and blocks at reasonable sizes for distribution around the network. Some empirical trials will help you to find the sweet spot for your specific workload and data sizes.


Dave Enyeart

"Baohua Yang" ---02/14/2020 01:05:36 PM---Adhav The number of assets in a single transaction depends on several factors,

From: "Baohua Yang" <yangbaohua@...>
To: Adhav Pavan <adhavpavan@...>
Cc: Gari Singh <garis@...>, hyperledger-fabric <hyperledger-fabric@...>
Date: 02/14/2020 01:05 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Performance Improvement: Max number of assets and max size of payload in single Transaction?
Sent by: fabric@...





Adhav

The number of assets in a single transaction depends on several factors, e.g., the key-value size.

Putting many large data in one transaction may consume heavy CPU and memory at the peer, and hang other operations for a long time.

And large blocks are not efficient to distribute in the network.

This is a typical high-performance scenario, and assigning more hardware resource can accelerate the process.

On Fri, Feb 14, 2020 at 3:37 AM Gari Singh <garis@...> wrote:
    The maximum payload is ~100MB.  This is actually set at the transport protocol level and is not configurable.
    Given there is some additional overhead included in the Fabric protocol layer, you are looking at a max payload in terms of your keys/value of ~90MB (to be on the safe side).

    I'll assume that your chaincode is designed to insert multiple asset records for a single invoke.

    -----------------------------------------
    Gari Singh
    Distinguished Engineer, CTO - IBM Blockchain
    IBM Middleware
    550 King St
    Littleton, MA 01460
    Cell: 978-846-7499
    garis@...
    -----------------------------------------

    -----fabric@... wrote: -----
    To: hyperledger-fabric <hyperledger-fabric@...>
    From: "Adhav Pavan"
    Sent by: fabric@...
    Date: 02/14/2020 04:48AM
    Subject: [EXTERNAL] [Hyperledger Fabric] Performance Improvement: Max number of assets and max size of payload in single Transaction?

    Hello Experts,

    I have some concerns about number of assets and size of the payload in a single transaction.

    1)  How big payload in terms of size, we can add in a single transaction in Hyperledger Fabric. I have millions of assets, ingesting into Fabric network. Maximum assets(Key-Value) can be added into the single transaction are 1000(Correct if I am wrong). Can we customize this number, so that a high number of an asset can be added in a single transaction?

    2) What is an efficient way to trigger a huge number of assets into the network?

    Currently, I could add only 100 assets in a single transaction.

    I am trying to improve performance in terms of TPS and latency. I have already added necessary indexes and composite key (Couch DB)

    Thank you.
     Heartfelt Regards,

    Pavan Adhav Blockchain Developer, Infinichains phone:  8390114357 email:  pavan@... ------
    Please excuse my brevity.







--
Best wishes!

Baohua Yang





Re: Performance Improvement: Max number of assets and max size of payload in single Transaction?

Baohua Yang
 

Adhav

The number of assets in a single transaction depends on several factors, e.g., the key-value size.

Putting many large data in one transaction may consume heavy CPU and memory at the peer, and hang other operations for a long time.

And large blocks are not efficient to distribute in the network.

This is a typical high-performance scenario, and assigning more hardware resource can accelerate the process.

On Fri, Feb 14, 2020 at 3:37 AM Gari Singh <garis@...> wrote:
The maximum payload is ~100MB.  This is actually set at the transport protocol level and is not configurable.
Given there is some additional overhead included in the Fabric protocol layer, you are looking at a max payload in terms of your keys/value of ~90MB (to be on the safe side).

I'll assume that your chaincode is designed to insert multiple asset records for a single invoke.

-----------------------------------------
Gari Singh
Distinguished Engineer, CTO - IBM Blockchain
IBM Middleware
550 King St
Littleton, MA 01460
Cell: 978-846-7499
garis@...
-----------------------------------------

-----fabric@... wrote: -----
To: hyperledger-fabric <hyperledger-fabric@...>
From: "Adhav Pavan"
Sent by: fabric@...
Date: 02/14/2020 04:48AM
Subject: [EXTERNAL] [Hyperledger Fabric] Performance Improvement: Max number of assets and max size of payload in single Transaction?

Hello Experts,

I have some concerns about number of assets and size of the payload in a single transaction.

1)  How big payload in terms of size, we can add in a single transaction in Hyperledger Fabric. I have millions of assets, ingesting into Fabric network. Maximum assets(Key-Value) can be added into the single transaction are 1000(Correct if I am wrong). Can we customize this number, so that a high number of an asset can be added in a single transaction?

2) What is an efficient way to trigger a huge number of assets into the network?

Currently, I could add only 100 assets in a single transaction.

I am trying to improve performance in terms of TPS and latency. I have already added necessary indexes and composite key (Couch DB)

Thank you.
 Heartfelt Regards,

Pavan Adhav Blockchain Developer, Infinichains phone:  8390114357 email:  pavan@... ------
Please excuse my brevity.








--
Best wishes!

Baohua Yang


Hyperledger Fabric Documentation Workgroup call - Western hemisphere - Fri, 02/14/2020 #cal-notice

fabric@lists.hyperledger.org Calendar <noreply@...>
 

Hyperledger Fabric Documentation Workgroup call - Western hemisphere

When:
Friday, 14 February 2020
4:00pm to 5:00pm
(GMT+00:00) Europe/London

Where:
https://zoom.us/j/6223336701

Organizer:
a_o-dowd@... +441962816761

Description:
Documentation workgroup call.
Agenda, minutes and recordings :https://wiki.hyperledger.org/display/fabric/Documentation+Working+Group


Re: Error: failed to create deliver client

Marina Wanis <marinamaged1996@...>
 

Hi Faisal,

 

I copied the environment variable incomplete by mistake.

TLS_PARAMETERS=" --tls true --cafile $ORDERER_CA_ROOTFILE"

 

export ORDERER_CA_ROOTFILE=$PWD/crypto-config/ordererOrganizations/phone.com/orderers/orderer.phone.com/msp/tlscacerts/tlsca.phone.com-cert.pem

 

I’m still getting this error:

 

2020-02-14 19:10:28.886 +04 [main] InitCmd -> WARN 001 CORE_LOGGING_LEVEL is no longer supported, please use the FABRIC_LOGGING_SPEC environment variable

2020-02-14 19:10:28.888 +04 [main] SetOrdererEnv -> WARN 002 CORE_LOGGING_LEVEL is no longer supported, please use the FABRIC_LOGGING_SPEC environment variable

Error: failed to create deliver client: orderer client failed to connect to orderer.phone.com:7050: failed to create new connection: context deadline exceeded

 

 

Sent from Mail for Windows 10

 

From: Faisal
Sent: Friday, February 14, 2020 10:52 AM
To: fabric@...
Subject: Re: [Hyperledger Fabric] Error: failed to create deliver client

 

all the path to the file after the --cafile parameter your command should be include the tlsca.pem file 

export TLS_PARAMETERS = --tls true --cafile /PATH/tlsca.phone.com-cert.pem

 

 


Upcoming Event: Hyperledger Fabric Documentation Workgroup call - Western hemisphere - Fri, 02/14/2020 4:00pm-5:00pm #cal-reminder

fabric@lists.hyperledger.org Calendar <fabric@...>
 

Reminder: Hyperledger Fabric Documentation Workgroup call - Western hemisphere

When: Friday, 14 February 2020, 4:00pm to 5:00pm, (GMT+00:00) Europe/London

Where:https://zoom.us/j/6223336701

View Event

Organizer: Anthony O'Dowd a_o-dowd@... +441962816761

Description: Documentation workgroup call.
Agenda, minutes and recordings :https://wiki.hyperledger.org/display/fabric/Documentation+Working+Group


Re: Performance Improvement: Max number of assets and max size of payload in single Transaction?

Gari Singh <garis@...>
 

The maximum payload is ~100MB. This is actually set at the transport protocol level and is not configurable.
Given there is some additional overhead included in the Fabric protocol layer, you are looking at a max payload in terms of your keys/value of ~90MB (to be on the safe side).

I'll assume that your chaincode is designed to insert multiple asset records for a single invoke.

-----------------------------------------
Gari Singh
Distinguished Engineer, CTO - IBM Blockchain
IBM Middleware
550 King St
Littleton, MA 01460
Cell: 978-846-7499
garis@...
-----------------------------------------

-----fabric@... wrote: -----
To: hyperledger-fabric <hyperledger-fabric@...>
From: "Adhav Pavan"
Sent by: fabric@...
Date: 02/14/2020 04:48AM
Subject: [EXTERNAL] [Hyperledger Fabric] Performance Improvement: Max number of assets and max size of payload in single Transaction?

Hello Experts,

I have some concerns about number of assets and size of the payload in a single transaction.

1) How big payload in terms of size, we can add in a single transaction in Hyperledger Fabric. I have millions of assets, ingesting into Fabric network. Maximum assets(Key-Value) can be added into the single transaction are 1000(Correct if I am wrong). Can we customize this number, so that a high number of an asset can be added in a single transaction?

2) What is an efficient way to trigger a huge number of assets into the network?

Currently, I could add only 100 assets in a single transaction.

I am trying to improve performance in terms of TPS and latency. I have already added necessary indexes and composite key (Couch DB)

Thank you.
Heartfelt Regards,

Pavan Adhav Blockchain Developer, Infinichains phone: 8390114357 email: pavan@... ------
Please excuse my brevity.


Performance Improvement: Max number of assets and max size of payload in single Transaction?

Adhav Pavan
 

Hello Experts,

I have some concerns about number of assets and size of the payload in a single transaction.

1)  How big payload in terms of size, we can add in a single transaction in Hyperledger Fabric. I have millions of assets, ingesting into Fabric network. Maximum assets(Key-Value) can be added into the single transaction are 1000(Correct if I am wrong). Can we customize this number, so that a high number of an asset can be added in a single transaction?

2) What is an efficient way to trigger a huge number of assets into the network?

Currently, I could add only 100 assets in a single transaction.

I am trying to improve performance in terms of TPS and latency. I have already added necessary indexes and composite key (Couch DB)

Thank you.

Heartfelt Regards,

Pavan Adhav
Blockchain Developer, Infinichains
phone:  8390114357
email:  pavan@...
------
Please excuse my brevity.


Re: Error: failed to create deliver client

Faisal
 

all the path to the file after the --cafile parameter your command should be include the tlsca.pem file 

export TLS_PARAMETERS = --tls true --cafile /PATH/tlsca.phone.com-cert.pem

 


Error: failed to create deliver client

Marina Wanis <marinamaged1996@...>
 

Hi,

Does anyone know why am I getting this error when I try to create the channel with the command:
peer channel create -c phonechannel -f ./phonechannel.tx --outputBlock ./phonechannel.block -o $ORDERER_ADDRESS $TLS_PARAMETERS

The following is the environment variable $ORDERER_ADDRESS $TLS_PARAMETERS

/network$ echo $ORDERER_ADDRESS $TLS_PARAMETERS
orderer.phone.com:7050 --tls true –cafile

Error:
[cid:image002.png@...]

Is the problem in the TLS?

Thank you,
Marina

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10


Hyperledger Fabric Documentation Workgroup call - Eastern hemisphere - Fri, 02/14/2020 #cal-notice

fabric@lists.hyperledger.org Calendar <noreply@...>
 

Hyperledger Fabric Documentation Workgroup call - Eastern hemisphere

When:
Friday, 14 February 2020
6:00am to 7:00am
(GMT+00:00) Europe/London

Where:
https://zoom.us/j/6223336701

Organizer:
a_o-dowd@... +441962816761

Description:
Documentation workgroup call.
Agenda, minutes and recordings: https://wiki.hyperledger.org/display/fabric/Documentation+Working+Group


Documentation Workgroup: Agenda for Friday, 14 Feb

Anthony O'Dowd <a_o-dowd@...>
 

Hello!

We will hold the documentation workgroup call this Friday, both Western and Eastern hemispheres.   Thanks to everyone who attended last week's call.

The summary minutes for last week's meeting: https://wiki.hyperledger.org/display/fabric/Meetings

You can read all about the call at https://wiki.hyperledger.org/display/fabric/2020+02+07+DWG+Agenda It included a V2 status update from Pam and Joe, a review of our new Chinese language docs, an overview of the new style guide from Joe, and the new deployment guide.

Specifically, we'd would like to express our thanks to Rich Zhao and his team on the release of the Chinese language docs - a great piece of work! You can check them out: https://hyperledger-fabric.readthedocs.io/zh_CN/release-1.4/

All-in-all, a great session last week. You can catch up via the recording: https://wiki.hyperledger.org/display/fabric/Recordings

You'll see that there are lots of interesting items for this week: https://wiki.hyperledger.org/display/fabric/2020+02+14+DWG+Agenda
Please feel free to contribute using the wiki, including helping to build next week's agenda: https://wiki.hyperledger.org/display/fabric/2020+02+21+DWG+Agenda

Thanks!

Pam, Anthony,  Joe, Nik

Meeting Details
-------------
Please use the following link to attend the meeting:  https://zoom.us/j/6223336701

The meeting times are as follows: https://wiki.hyperledger.org/display/fabric/Documentation+Working+Group

Meeting 115A: Friday 14 Feb
                   1130 India Standard Time
                   1400 China Standard Time
                   1500 Japan Standard Time
                   1700 Australia Eastern Time
                   1400 Singapore Time
                   1000 Gulf Standard Time
                   0900 Moscow Standard Time
                   0600 Greenwich Mean Time
                   0700 Central European Time    

Meeting 115B: Friday 14 Feb
              1100 Central Daylight Time
                   1200 Eastern Daylight Time
                   0900 Pacific Daylight Time
                   1400 Brasil Time (BRT)
                   1700 Greenwich Mean Time
                   1800 Central European Time
                   1900 Moscow Standard Tim


Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


[common.deliver] deliverBlocks -> DEBU 371 Rejecting deliver for X.X.X.X:42926 because channel tracktrace not found

Siddharth Jain
 

we have a network that runs fine without Docker but when we try to containerize it we get this error in orderer logs:

2020-02-13 22:19:16.706 UTC [common.deliver] deliverBlocks -> DEBU 371 Rejecting deliver for X.X.X.X:42926 because channel tracktrace not found

we can actually see the channel if we look inside the orderer

root@a60672f8b829:/home# ls /var/hyperledger/production/chains
foo  tracktrace

we have mounted the orderer directory (when it was running without Docker) to /var/hyperledger/production

anyone know what's going on here and how to fix this? thanks


Upcoming Event: Hyperledger Fabric Documentation Workgroup call - Eastern hemisphere - Fri, 02/14/2020 6:00am-7:00am #cal-reminder

fabric@lists.hyperledger.org Calendar <fabric@...>
 

Reminder: Hyperledger Fabric Documentation Workgroup call - Eastern hemisphere

When: Friday, 14 February 2020, 6:00am to 7:00am, (GMT+00:00) Europe/London

Where:https://zoom.us/j/6223336701

View Event

Organizer: Anthony O'Dowd a_o-dowd@... +441962816761

Description: Documentation workgroup call.
Agenda, minutes and recordings: https://wiki.hyperledger.org/display/fabric/Documentation+Working+Group


Re: How to unpackage (or unpack, unzip, extract) a chaincode?

Brett T Logan <brett.t.logan@...>
 

Here is a very, very basic program to ingest the chaincode package and output a tarball of it: https://github.com/yeasy/blockchain_guide/blob/master/_code/unpack_chaincode.go
 
Credit to Baohua Yang for the answer:
 
 
Brett Logan
Software Engineer, IBM Blockchain
Phone: 1-984-242-6890
 
 
 

----- Original message -----
From: "Siddharth Jain" <siddjain@...>
Sent by: fabric@...
To: Matthew Sykes <matthew.sykes@...>
Cc: "fabric@..." <fabric@...>
Subject: [EXTERNAL] Re: [Hyperledger Fabric] How to unpackage (or unpack, unzip, extract) a chaincode?
Date: Wed, Feb 12, 2020 8:40 PM
 
How do i do that?
 

From: Matthew Sykes <matthew.sykes@...>
Sent: Wednesday, February 12, 2020 5:27:38 PM
To: Siddharth Jain <siddjain@...>
Cc: fabric@... <fabric@...>
Subject: Re: [Hyperledger Fabric] How to unpackage (or unpack, unzip, extract) a chaincode?
 
If you're using the new lifecycle commands, the chaincode package is a gzip compressed tar; if you're using the legacy (lscc) commands, the package is a protobuf encoded chaincode deployment spec. You'll need to unmarshal that message to get to the code package tar file.
 
On Wed, Feb 12, 2020 at 5:36 PM Siddharth Jain <siddjain@...> wrote:
Anyone knows how do we unpackage a chaincode? tar -xvf does not work.
 
$ tar -xvf mycc.1.0
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
 
in above mycc.1.0 is what got installed on the peer when we ran peer chaincode install.
 
$ ls -al mycc.1.0
-rw-r--r--  1 siddjain  staff  35163 Feb 12 14:19 mycc.1.0

 

 

 
 
--
Matthew Sykes
matthew.sykes@...
 


Re: How to unpackage (or unpack, unzip, extract) a chaincode?

Nye Liu <nye@...>
 

-xvzf

On 2/12/2020 5:39 PM, Siddharth Jain wrote:

How do i do that?


From: Matthew Sykes <matthew.sykes@...>
Sent: Wednesday, February 12, 2020 5:27:38 PM
To: Siddharth Jain <siddjain@...>
Cc: fabric@... <fabric@...>
Subject: Re: [Hyperledger Fabric] How to unpackage (or unpack, unzip, extract) a chaincode?
 
If you're using the new lifecycle commands, the chaincode package is a gzip compressed tar; if you're using the legacy (lscc) commands, the package is a protobuf encoded chaincode deployment spec. You'll need to unmarshal that message to get to the code package tar file.

On Wed, Feb 12, 2020 at 5:36 PM Siddharth Jain <siddjain@...> wrote:
Anyone knows how do we unpackage a chaincode? tar -xvf does not work.

$ tar -xvf mycc.1.0
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.

in above mycc.1.0 is what got installed on the peer when we ran peer chaincode install.

$ ls -al mycc.1.0
-rw-r--r--  1 siddjain  staff  35163 Feb 12 14:19 mycc.1.0


--
Matthew Sykes
matthew.sykes@...


Re: How to unpackage (or unpack, unzip, extract) a chaincode?

Siddharth Jain
 

How do i do that?


From: Matthew Sykes <matthew.sykes@...>
Sent: Wednesday, February 12, 2020 5:27:38 PM
To: Siddharth Jain <siddjain@...>
Cc: fabric@... <fabric@...>
Subject: Re: [Hyperledger Fabric] How to unpackage (or unpack, unzip, extract) a chaincode?
 
If you're using the new lifecycle commands, the chaincode package is a gzip compressed tar; if you're using the legacy (lscc) commands, the package is a protobuf encoded chaincode deployment spec. You'll need to unmarshal that message to get to the code package tar file.

On Wed, Feb 12, 2020 at 5:36 PM Siddharth Jain <siddjain@...> wrote:
Anyone knows how do we unpackage a chaincode? tar -xvf does not work.

$ tar -xvf mycc.1.0
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.

in above mycc.1.0 is what got installed on the peer when we ran peer chaincode install.

$ ls -al mycc.1.0
-rw-r--r--  1 siddjain  staff  35163 Feb 12 14:19 mycc.1.0



--
Matthew Sykes
matthew.sykes@...


Re: How to unpackage (or unpack, unzip, extract) a chaincode?

Matthew Sykes
 

If you're using the new lifecycle commands, the chaincode package is a gzip compressed tar; if you're using the legacy (lscc) commands, the package is a protobuf encoded chaincode deployment spec. You'll need to unmarshal that message to get to the code package tar file.


On Wed, Feb 12, 2020 at 5:36 PM Siddharth Jain <siddjain@...> wrote:
Anyone knows how do we unpackage a chaincode? tar -xvf does not work.

$ tar -xvf mycc.1.0
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.

in above mycc.1.0 is what got installed on the peer when we ran peer chaincode install.

$ ls -al mycc.1.0
-rw-r--r--  1 siddjain  staff  35163 Feb 12 14:19 mycc.1.0



--
Matthew Sykes
matthew.sykes@...


Re: Modify Chain code container

Matthew Sykes
 

The peer does not "fire the docker run cmd" but uses the docker API to create and start the container.


If you want to expose a port, you can either modify the fabric code or create your own base image for chaincode that exposes the port. If you create your own image, you only need to configure the peer(s) to use your image instead of the fabric image (peer.chaincode.${language}).


On Tue, Feb 11, 2020 at 7:38 AM <sanjaykumar3989@...> wrote:
I want to expose port from chain code container to host machine. From where peer node fire the docker run cmd .



--
Matthew Sykes
matthew.sykes@...


How to unpackage (or unpack, unzip, extract) a chaincode?

Siddharth Jain
 

Anyone knows how do we unpackage a chaincode? tar -xvf does not work.

$ tar -xvf mycc.1.0
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.

in above mycc.1.0 is what got installed on the peer when we ran peer chaincode install.

$ ls -al mycc.1.0
-rw-r--r--  1 siddjain  staff  35163 Feb 12 14:19 mycc.1.0


Re: Local Collections in Fabric 2.0 #fabric

Alessandro Sorniotti
 

Martin

A transaction that uses collections leaks information about the involved parties in 3 ways: 1) the creator; 2) the collection name and 3) endorsements. Point 1 can be addressed by using idemix. Local collections address point 2, and we have a (VERY ROUGH!) prototype here

https://github.com/IBM/fabric/blob/release-2.0/docs/source/local_collections.md

We're working on point 3, so take a look at our playground https://github.com/IBM/fabric/ for updates.

Cheers,
Ale

On Wed, 12 Feb 2020, at 10:50 AM, westerkamp@... wrote:
Hi everyone,

Some time ago there has been a discussion concerning local collections
with the goal to obscure members of a collection to other participants
of a channel in FAB-7593 <https://jira.hyperledger.org/browse/FAB-7593>.

I was wondering about the impact of novel Fabric 2.0 features on the
proposal or if Fabric 2.0 delivers functionality that enables such
collections already? In that context, I have checked out the private
data sharing patterns descibed here
<https://hyperledger-fabric.readthedocs.io/en/latest/private-data/private-data.html#private-data-sharing-patterns>. Especially the "*Keeping transactors private" *pattern seems very related.

I would like to introduce a short example to clearify the requirements:
Four organizations A, B, C and D participate in a shared channel. A and
B collaborate using chaincode containing a private data collection. C
and D should not be able to observe that A and B are collaborating.
Therefore, when A is submitting a transaction, C and D should not be
able to tell who is affected by that transaction. Currently, the data
collection definition reveals all members of the collections, so that C
and D can derive that A and B are collaborating (from my understanding).

So my questions are:
- Can Fabric 2.0 already implement such a use can, e.g. by applying the
"*Keeping transactors private" *pattern? If so, what would it look like
in detail?
- If it cannot be implemented by Fabric 2.0 yet, what do we need to do
so? Is FAB-7593 <https://jira.hyperledger.org/browse/FAB-7593> still
relevant in context of Fabric 2.0?

Thanks and best regards,
Martin

3821 - 3840 of 11527