Re: Test net thinking
Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?
I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.
I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.
Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.
LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.
This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.
And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.
Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for infrastructure would need to be on the order of $10k/month/project which is probably not tenable.
For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve in subsequent iterations.
From: <tsc@...> on behalf of Ry Jones <rjones@...>
Date: Friday, October 26, 2018 at 12:06 AM
To: Casey Kuhlman <casey@...>
Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...>
Subject: Re: [Hyperledger TSC] Test net thinking
For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.
Go here: https://github.com/cncf/cluster
Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via Lists.Hyperledger.Org <casey=monax.io@...> wrote:
From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).