Date
1 - 5 of 5
[Hyperledger Project TSC] [Hyperledger-fabric] Performance measurements
|
Brian,
toggle quoted message
Show quoted text
this is really exciting. What would be the way forward with this? I'm cc-ing this to the TSC list, too, because in light of the multiple, disconnected performance testing efforts emerging from many places you guys may want to consider setting up a WG, perhaps? (Sorry for the cross-post if the TSC list is not the right place for the suggestion.) As of now, performance testing/asssurance seems to be present in the project structure only through the busywork tool (and its project proposal), and that's somewhat orthogonal to this. At the same time, it's not necessarily the smartest for everybody to develop or duct tape together their own test harness (this seems to be happening), but that's another question. Anyhow, I assume that however big, this is a finite resource so access to it has to be controlled; either via some form of admission control for project members to send in jobs or simply periodically running sets of experiments agreed upon by the interested parties. One could even argue that "Continuous Performance Testing" wouldn't be a bad idea, either. Your thoughts on this? Best regards Imre --------------------------------------------------------------------------- Imre Kocsis assistant lecturer Fault Tolerant Systems Research Group Department of Measurement and Information Systems Budapest University of Technology and Economics mail: ikocsis@... phone: +36-1-463-2006 mobile: +36-20-514-6881 skype: kocsis_imre From: Brian Behlendorf <bbehlendorf@...> To: hyperledger-fabric@... Date: 06/17/2016 08:13 AM Subject: Re: [Hyperledger-fabric] Performance measurements Sent by: hyperledger-fabric-bounces@... I'm following this thread on the periphery, and wanted to highlight Imre's questions about the test environment and how it may compare to real-world deployments. I think it's crucial to start to articulate what that real-world looks like. For example it probably involves nodes that are not all on the same local network, because each node is "owned" by a different organization that doesn't trust the others, and thus connection latency across the chain could vary up to hundreds of milliseconds and face real-world packet loss from time to time. It also is extremely cheap these days to get high powered bare metal, so running everything in docker within a single VM may not reflect what's actually possible if we throw real-world hardware at the problem. The Linux Foundation has access to resources that could be used to help the community with performance testing, if this would be helpful, potentially including a 1000 CPU server farm that has been loaned to us for part-time experiments. Let me know if that's interesting (and no, you can't run a BTC or ETH miner on it) Brian On 06/16/2016 04:07 PM, ikocsis@... wrote: Yacov, measuring this is a good idea; however, I would have a few questions & comments: 1. First and foremost: could you maybe plot this? :) 2. Especially as you use (I have to assume) a single VM for the whole P2P network: have you controlled for the state of other resources than network? E.g. CPU saturation. 3. For the purpose of TPS computing: how do you define a "transaction" in this case? As I see it, client INVOKE queries simply ask the network something to be done on the world state; when they return the deed can be far from being done. (So there is no classic atomic transaction execution.) From the point of view of the clients, the invoke request is "done" when QUERY requests begin to show its effect. So the concept of TPS is not exactly trivial. Am I right on this? I have a more general question, too, open for discussion, if you like. (I'm honestly interested in this.) While the guys at Digital Asset seem to use (and I know that we do use :)) EC2 VMs for performance testing (not optimal, but good enough), there seem to be others simply using single-node (mostly meaning a single VM), purely Docker configuration. Is this really wise for performance testing? I mean containers can and do interfere performance-wise by simply competing for the access of time-shared resources, and while cgroups is there (assuming it's used), to the best of my knowledge it's not perfect for latency-sensitive scenarios. Also, slicing the HW resources of a single VM between containers can make each peer a bit anemic on its own. I understand that for off the cuff perf. measurements using e.g. the Vagrant devnet can be a rapid way to have some numbers, but is it representative enough for what can be expected "in production"? (The answer may very well be "yes and don't nitpick", but I'm having doubts at this point.) Best regards Imre From: "Yacov Manevich" <YACOVM@...> To: "Konstantinos Christidis" <kchristidis@...>, tamas@... Cc: Benjamin Mandler <MANDLER@...>, hyperledger-fabric@... Date: 06/16/2016 03:20 PM Subject: Re: [Hyperledger-fabric] Performance measurements Sent by: hyperledger-fabric-bounces@... Hi. On the same topic, I've also been doing some performance testings. Specifically I'm interested in correlating network latency, node number and transactions per second. I made a simple test that: 1. Spawns N validating peers with PBFT classic in docker containers 2. deploys a chaincode to them 3. Injects network latencies (a few milliseconds) to all containers but the first one using the tc command 4. runs a node.js script that does a few thousand invocations, (in parallel or not, when in parallel- each "thread" runs it against a different validating peer) 5. Counts the elapsed time and calculates the TPS (transactions per second) value. The results I got are attached: Regards, Yacov. From: "Konstantinos Christidis" <kchristidis@...> To: ikocsis@... Cc: hyperledger-fabric@... Date: 14/06/2016 19:16 Subject: Re: [Hyperledger-fabric] Performance measurements Sent by: hyperledger-fabric-bounces@... ----- Original message -----
From: ikocsis@... To: Konstantinos Christidis/Durham/IBM@IBMUS Cc: hyperledger-fabric@... Subject: Re: [Hyperledger-fabric] Performance measurements Date: Tue, Jun 14, 2016 7:24 AM On another note: we do use certainly the default "batch" pbft, but I saw the "sieve" and I think "crypto" options in the code, too. And I think I found the paper that explains these (haven't had the time to fully understand the specifics yet): http://arxiv.org/abs/1603.07351 Question is: do these already work? We certainly should stick to deterministic transactions at this point, I'm just curious. Imre, Sieve is a prototype, our development efforts are currently focused on PBFT. I would strongly suggest sticking to PBFT for your experiments. Best, Kostas _______________________________________________ Hyperledger-fabric mailing list Hyperledger-fabric@... https://lists.hyperledger.org/mailman/listinfo/hyperledger-fabric [attachment "results.pdf.zip" deleted by Imre Kocsis/ftsrg] _______________________________________________ Hyperledger-fabric mailing list Hyperledger-fabric@... https://lists.hyperledger.org/mailman/listinfo/hyperledger-fabric _______________________________________________ Hyperledger-fabric mailing list Hyperledger-fabric@... https://lists.hyperledger.org/mailman/listinfo/hyperledger-fabric -- Brian Behlendorf Executive Director at the Hyperledger Project bbehlendorf@... Twitter: @brianbehlendorf_______________________________________________ Hyperledger-fabric mailing list Hyperledger-fabric@... https://lists.hyperledger.org/mailman/listinfo/hyperledger-fabric |
||||||||||
|
|
||||||||||
|
Brian Behlendorf
Here is the form for requesting access
to the compute cluster.
https://github.com/cncf/cluster Note that it says it's focused on CNCF (the Cloud Native Computing Foundation, also an LF project like us), but I talked with the director of the project last night and we can use it too. It may be worth putting the workloads tested here also on EC2 or Azure for comparison, eventually. While I don't think we can use all 1000 nodes continuously forever, we don't currently have a lot of contention, and the more it becomes a demonstration vehicle for CNCF the more priority I think we'll get :) I would even feel comfortable with us suggesting to them a continuous performance testing process, though perhaps daily rather than upon every commit would be a good idea. But let's get together as a community and make a thoughtful, coordinated ask on that form for the cluster. I am looking for help on our side from the Linux world in performance testing and tuning, but I'd also encourage all our members to search their own organizations for people with these skills, as they are almost as rare as blockchain skills :) I would hesitate to break this activity out into a separate formal Working Group, only because then it may come across as Someone Else's Problem, rather than one of several parallel concerns devs should always have in mind while working on code (like security, too). I would suggest this list talk about what the ideal and likely production environments will look like, use that to drive a realistic set of targets and an ideal test harness, and then build collectively. Brian On 06/17/2016 02:52 AM, ikocsis@... wrote: Brian, -- Brian Behlendorf Executive Director at the Hyperledger Project bbehlendorf@... Twitter: @brianbehlendorf |
||||||||||
|
|
||||||||||
|
Jeremy Eder <jeder@...>
I'm a performance engineer for Red Hat -- we're also investigating
blockchain, and ramping up partnerships [1]. We have a scale-out project on the CNCF cluster ongoing as we speak wrt Kubernetes. If any Hyperledger participant has their implementation running on Kubernetes (one of the foundation projects for Red Hat OpenShift), we'd like to hear from you, specifically wrt performance testing. [1] https://www.redhat.com/en/about/press-releases/red-hat-introduces-new-partner-initiative-blockchain-software-vendors On Fri, Jun 17, 2016 at 11:27 AM, Brian Behlendorf via hyperledger-tsc <hyperledger-tsc@...> wrote: Here is the form for requesting access to the compute cluster. -- -- Jeremy Eder |
||||||||||
|
|
||||||||||
|
Zsolt Szilágyi <zsolt@...>
Hi all, I would like to share my latest performance measurements results.It has been done on current fabric and fabric-api master. For measurement, I used the following: - PerfTest test case with GrpcClient HLAPI implementation, 10000 transactions, rest of the things remained unchanged - 4 pbft nodes with pbft batch mode, default parameters Findings (tl;dr): - tx/sec dropped to 30/sec (earlier, we measured 300) - average transaction time (between sending it in, and getting it in event) is 15 sec - 2 transactions seem to be contained by two blocks (their listener called multiple times, from different block events) Non tl;dr Logs of the measurement can be found here, where output.txt is the fabric api log, and outputX.txt are vpX nodes of the pbft network. Here is the result from output.txt: ====== Test results ====== Total: 10000 transactions in 353.49 sec Average transaction/s: 28.29 Average transaction process time: 15027.30 ms Listener not called for 0 transactions Listener called multiple times for 2 transactions 0 transactions not found in the ledger 0 transactions rejected Distribution: 0 - 1000: tx/sec=34.83 avg_tx_time=1792.79 ms 1000 - 2000: tx/sec=30.65 avg_tx_time=7089.22 ms 2000 - 3000: tx/sec=26.52 avg_tx_time=13959.66 ms 3000 - 4000: tx/sec=26.18 avg_tx_time=19009.87 ms 4000 - 5000: tx/sec=26.60 avg_tx_time=18913.06 ms 5000 - 6000: tx/sec=26.61 avg_tx_time=19227.61 ms 6000 - 7000: tx/sec=27.60 avg_tx_time=18181.01 ms 7000 - 8000: tx/sec=26.87 avg_tx_time=18792.81 ms 8000 - 9000: tx/sec=26.39 avg_tx_time=19007.33 ms 9000 - 10000: tx/sec=38.16 avg_tx_time=14299.58 ms BR Zsolt On Sun, Jul 3, 2016 at 2:42 AM, Jeremy Eder <jeder@...> wrote: I'm a performance engineer for Red Hat -- we're also investigating --
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset. |
||||||||||
|
|
||||||||||
|
Zsolt Szilágyi <zsolt@...>
Documentation about how I ran these tests can be found here. On Fri, Jul 8, 2016 at 5:03 PM, Zsolt Szilágyi <zsolt@...> wrote:
--
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset. |
||||||||||
|
|
