David Huseby <dhuseby@...>
Does it really take big iron to run these tests? $10k/month/project
seems a bit much for this kind of thing. If the goal is to just kick
the tires or to have a soak test, why couldn't this be done on minimal
virtual machines or minimal hardware?
Security Maven, Hyperledger
The Linux Foundation
On Fri, Oct 26, 2018 at 12:30 PM Christopher Ferris
Interesting that Casey thinks of long-running tests, performance, chaos, etc. I always think of a testnet as a place where devs can kick the tires without having to do the heavy lifting of setting up a network themselves.
For the long-running tests, etc. we have nightly runs and we are gearing up a weekly as well that will be long running test that has kitchen sinks thrown at it and stuff. Not a large environment, but more than what we do for the merge and nightly CI.
IBM also runs a series of long running tests leading up to a release on our platforms. We do share the results indirectly and it would be nice if they were just public, frankly (this is me speaking not for IBM but as a member of the community). We are working on an ability to spin up networks arbitrary complexity and scale, so that tests can be run. It would be nice to have that running in the community, but again, not as a test net for users as much as for what Casey was advocating for.
On Fri, Oct 26, 2018 at 10:24 AM Middleton, Dan <dan.middleton@...> wrote:
Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?
I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.
I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.
Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.
LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.
This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.
And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.
Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for infrastructure would need to be on the order of $10k/month/project which is probably not tenable.
For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve in subsequent iterations.
From: <tsc@...> on behalf of Ry Jones <rjones@...>
Date: Friday, October 26, 2018 at 12:06 AM
To: Casey Kuhlman <casey@...>
Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...>
Subject: Re: [Hyperledger TSC] Test net thinking
For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.
Go here: https://github.com/cncf/cluster
Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via Lists.Hyperledger.Org <casey=monax.io@...> wrote:
From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
Community Architect, Hyperledger