Hi TSC,

I had a discussion with Ry this week while manning the booth at ATO in Raleigh and we both think the right way to go is a two pronged approach:

The first prong will be to engage with the free kubernetes hosting being paid for by (I think) the CNCF project. Ry knows more about it than I do. 

The second prong would be to set aside some budget to cover the cost of people running networks on Amazon EKS. We could set some rule like we cover the cost up to $200 for the first month with an opportunity to get additional months with approval. 

I agree with Chris and Ry and I’m concerned about the human overhead of trying to run test nets for people. I think the best compromise is to just cover the bills for people spinning up test networks on hosted kubernetes platforms. The oversight needs to be an easy application and approval process as well as an easy renewal process. I was thinking that $20k would be enough to cover. That’s at least 10 networks per month for a year. 

Thoughts?

Dave

--

---
David Huseby
Security Maven, Hyperledger
The Linux Foundation
+1-206-234-2392
dhuseby@...

I think that we need to consider what we think a "test net" is. Is it a whole network that a developer spins up themselves? Is it a pre-existing network that we have launched and that developers can use to deploy a smart contract and kick the tires? Is it permissioned? Permissionless? What are the implications? Will there be likelihood of noisy neighbor situations? Crypto Kitties? Who will monitor the experience? What if it gets really bogged down? Who is monitoring that it is running? Will users walk away from a bad experience with a bad taste in their mouths? How will the experience reflect on the project and the organization?

I don't disagree that we want to make it dead simple to engage with the various projects. However, I can say that for Fabric, the process could not be simpler.

download fabric-samples repo, images and binaries (one click)

cd first-network

byfn.sh up

three commands and you have it running on your laptop.

Could we create some helm charts that could be used to deploy our various projects to K8s? Let people deal with finding the compute however they like (maybe point at alternatives) but we don't host/pay?

I ask because I really don't understand what problem we think we are solving with a test net, nor do I believe that we have considered the broader implications of running a cloud service on a volunteer basis.

Chris

On Wed, Oct 24, 2018 at 5:56 PM David Huseby <dhuseby@...> wrote:

Hi TSC,

I had a discussion with Ry this week while manning the booth at ATO in Raleigh and we both think the right way to go is a two pronged approach:

The first prong will be to engage with the free kubernetes hosting being paid for by (I think) the CNCF project. Ry knows more about it than I do. 

The second prong would be to set aside some budget to cover the cost of people running networks on Amazon EKS. We could set some rule like we cover the cost up to $200 for the first month with an opportunity to get additional months with approval. 

I agree with Chris and Ry and I’m concerned about the human overhead of trying to run test nets for people. I think the best compromise is to just cover the bills for people spinning up test networks on hosted kubernetes platforms. The oversight needs to be an easy application and approval process as well as an easy renewal process. I was thinking that $20k would be enough to cover. That’s at least 10 networks per month for a year. 

Thoughts?

Dave

--

---
David Huseby
Security Maven, Hyperledger
The Linux Foundation
+1-206-234-2392
dhuseby@...

Chris,

I think you raise some very valid points regarding operating a test net. I'd like to inject an end user perspective if I may. 

I think Ethereum wins from an ease of use perspective because people can easily write a smart contract and deploy it to a test network. That's what the vast majority of developers want to get done when they first encounter blockchain and IMO that is where Fabric (and others) fall down. HLF v1.3 has made it (relatively) easy to *start* the network on your laptop, but once you actually want to deploy chaincode and interact with it, it is a whole different story: you start hacking on fabcar and trying to understand how ./startFabric.sh, enrollAdrmin.js, registerUser.js, hlf-key-store, docker etc all work. Contrast with using http://remix.ethereum.org running in the browser, or connected to https://infura.io or https://kaleido.io for writing your first Solidity code.

There's an opportunity to do something similar to remix using the Composer Playground connected to a test net which is worth exploring imo. I don't know if that should be the remit of HL/TSC but there's clearly a need for something like that to attract more developers.

Sincerely,

Dan

On Thu, Oct 25, 2018 at 2:54 PM Christopher Ferris <chris.ferris@...> wrote:

I think that we need to consider what we think a "test net" is. Is it a whole network that a developer spins up themselves? Is it a pre-existing network that we have launched and that developers can use to deploy a smart contract and kick the tires? Is it permissioned? Permissionless? What are the implications? Will there be likelihood of noisy neighbor situations? Crypto Kitties? Who will monitor the experience? What if it gets really bogged down? Who is monitoring that it is running? Will users walk away from a bad experience with a bad taste in their mouths? How will the experience reflect on the project and the organization?

I don't disagree that we want to make it dead simple to engage with the various projects. However, I can say that for Fabric, the process could not be simpler.

download fabric-samples repo, images and binaries (one click)

cd first-network

byfn.sh up

three commands and you have it running on your laptop.

Could we create some helm charts that could be used to deploy our various projects to K8s? Let people deal with finding the compute however they like (maybe point at alternatives) but we don't host/pay?

I ask because I really don't understand what problem we think we are solving with a test net, nor do I believe that we have considered the broader implications of running a cloud service on a volunteer basis.

Chris

On Wed, Oct 24, 2018 at 5:56 PM David Huseby <dhuseby@...> wrote:

Hi TSC,

I had a discussion with Ry this week while manning the booth at ATO in Raleigh and we both think the right way to go is a two pronged approach:

The first prong will be to engage with the free kubernetes hosting being paid for by (I think) the CNCF project. Ry knows more about it than I do. 

The second prong would be to set aside some budget to cover the cost of people running networks on Amazon EKS. We could set some rule like we cover the cost up to $200 for the first month with an opportunity to get additional months with approval. 

I agree with Chris and Ry and I’m concerned about the human overhead of trying to run test nets for people. I think the best compromise is to just cover the bills for people spinning up test networks on hosted kubernetes platforms. The oversight needs to be an easy application and approval process as well as an easy renewal process. I was thinking that $20k would be enough to cover. That’s at least 10 networks per month for a year. 

Thoughts?

Dave

--

---
David Huseby
Security Maven, Hyperledger
The Linux Foundation
+1-206-234-2392
dhuseby@...

--

Dan Selman

CTO, Clause Inc.

@danielselman

Dan, yes, but my response to that would be that Fabric should improve its UX around deploying and engaging chaincode/smart contract.

Having a testnet does nothing to improve that experience IMNSHO.

Chris

On Thu, Oct 25, 2018 at 10:18 AM Dan Selman <dan@...> wrote:

Chris,

I think you raise some very valid points regarding operating a test net. I'd like to inject an end user perspective if I may. 

I think Ethereum wins from an ease of use perspective because people can easily write a smart contract and deploy it to a test network. That's what the vast majority of developers want to get done when they first encounter blockchain and IMO that is where Fabric (and others) fall down. HLF v1.3 has made it (relatively) easy to *start* the network on your laptop, but once you actually want to deploy chaincode and interact with it, it is a whole different story: you start hacking on fabcar and trying to understand how ./startFabric.sh, enrollAdrmin.js, registerUser.js, hlf-key-store, docker etc all work. Contrast with using http://remix.ethereum.org running in the browser, or connected to https://infura.io or https://kaleido.io for writing your first Solidity code.

There's an opportunity to do something similar to remix using the Composer Playground connected to a test net which is worth exploring imo. I don't know if that should be the remit of HL/TSC but there's clearly a need for something like that to attract more developers.

Sincerely,

Dan

On Thu, Oct 25, 2018 at 2:54 PM Christopher Ferris <chris.ferris@...> wrote:

I think that we need to consider what we think a "test net" is. Is it a whole network that a developer spins up themselves? Is it a pre-existing network that we have launched and that developers can use to deploy a smart contract and kick the tires? Is it permissioned? Permissionless? What are the implications? Will there be likelihood of noisy neighbor situations? Crypto Kitties? Who will monitor the experience? What if it gets really bogged down? Who is monitoring that it is running? Will users walk away from a bad experience with a bad taste in their mouths? How will the experience reflect on the project and the organization?

I don't disagree that we want to make it dead simple to engage with the various projects. However, I can say that for Fabric, the process could not be simpler.

download fabric-samples repo, images and binaries (one click)

cd first-network

byfn.sh up

three commands and you have it running on your laptop.

Could we create some helm charts that could be used to deploy our various projects to K8s? Let people deal with finding the compute however they like (maybe point at alternatives) but we don't host/pay?

I ask because I really don't understand what problem we think we are solving with a test net, nor do I believe that we have considered the broader implications of running a cloud service on a volunteer basis.

Chris

On Wed, Oct 24, 2018 at 5:56 PM David Huseby <dhuseby@...> wrote:

Hi TSC,

I had a discussion with Ry this week while manning the booth at ATO in Raleigh and we both think the right way to go is a two pronged approach:

The first prong will be to engage with the free kubernetes hosting being paid for by (I think) the CNCF project. Ry knows more about it than I do. 

The second prong would be to set aside some budget to cover the cost of people running networks on Amazon EKS. We could set some rule like we cover the cost up to $200 for the first month with an opportunity to get additional months with approval. 

I agree with Chris and Ry and I’m concerned about the human overhead of trying to run test nets for people. I think the best compromise is to just cover the bills for people spinning up test networks on hosted kubernetes platforms. The oversight needs to be an easy application and approval process as well as an easy renewal process. I was thinking that $20k would be enough to cover. That’s at least 10 networks per month for a year. 

Thoughts?

Dave

--

---
David Huseby
Security Maven, Hyperledger
The Linux Foundation
+1-206-234-2392
dhuseby@...

--

Dan Selman

CTO, Clause Inc.

@danielselman

From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle. 

While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.

My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).

Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?

 

I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.

 

I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.

 

Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.

 

LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.

 

This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.

And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.

 

Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for infrastructure would need to be on the order of $10k/month/project which is probably not tenable.

 

For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve in subsequent iterations.

 

Regards,
Dan

 

 

From: <tsc@...> on behalf of Ry Jones <rjones@...>
Date: Friday, October 26, 2018 at 12:06 AM
To: Casey Kuhlman <casey@...>
Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...>
Subject: Re: [Hyperledger TSC] Test net thinking

 

For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.

 

Go here: https://github.com/cncf/cluster

 

Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.

 

Ry

 

On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via Lists.Hyperledger.Org <casey=monax.io@...> wrote:

From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle. 

 

While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.

 

My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).

--

Ry Jones

Community Architect, Hyperledger

Chat: @ry

Does it really take big iron to run these tests?  $10k/month/project
seems a bit much for this kind of thing.  If the goal is to just kick
the tires or to have a soak test, why couldn't this be done on minimal
virtual machines or minimal hardware?

Dave
---
David Huseby
Security Maven, Hyperledger
The Linux Foundation
+1-206-234-2392
dhuseby@...

On Fri, Oct 26, 2018 at 12:30 PM Christopher Ferris
<chris.ferris@...> wrote:
>
> Interesting that Casey thinks of long-running tests, performance, chaos, etc. I always think of a testnet as a place where devs can kick the tires without having to do the heavy lifting of setting up a network themselves.
>
> For the long-running tests, etc. we have nightly runs and we are gearing up a weekly as well that will be long running test that has kitchen sinks thrown at it and stuff. Not a large environment, but more than what we do for the merge and nightly CI.
>
> IBM also runs a series of long running tests leading up to a release on our platforms. We do share the results indirectly and it would be nice if they were just public, frankly (this is me speaking not for IBM but as a member of the community). We are working on an ability to spin up networks arbitrary complexity and scale, so that tests can be run. It would be nice to have that running in the community, but again, not as a test net for users as much as for what Casey was advocating for.
>
> Cheers
>
> Chris
>
> On Fri, Oct 26, 2018 at 10:24 AM Middleton, Dan <dan.middleton@...> wrote:
>>
>> Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?
>>
>>
>>
>> I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.
>>
>>
>>
>> I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.
>>
>>
>>
>> Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.
>>
>>
>>
>> LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.
>>
>>
>>
>> This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.
>>
>> And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.
>>
>>
>>
>> Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for infrastructure would need to be on the order of $10k/month/project which is probably not tenable.
>>
>>
>>
>> For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve in subsequent iterations.
>>
>>
>>
>> Regards,
>> Dan
>>
>>
>>
>>
>>
>> From: <tsc@...> on behalf of Ry Jones <rjones@...>
>> Date: Friday, October 26, 2018 at 12:06 AM
>> To: Casey Kuhlman <casey@...>
>> Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...>
>> Subject: Re: [Hyperledger TSC] Test net thinking
>>
>>
>>
>> For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.
>>
>>
>>
>> Go here: https://github.com/cncf/cluster
>>
>>
>>
>> Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
>>
>>
>>
>> Ry
>>
>>
>>
>> On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via Lists.Hyperledger.Org <casey=monax.io@...> wrote:
>>
>> From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
>>
>>
>>
>> While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
>>
>>
>>
>> My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
>>
>> --
>>
>> Ry Jones
>>
>> Community Architect, Hyperledger
>>
>> Chat: @ry
>

You cannot do performance and scale testing on nominal hardware. Long running and chaos tests, sure.

Chris

On Mon, Oct 29, 2018 at 9:15 PM David Huseby <dhuseby@...> wrote:

Does it really take big iron to run these tests?  $10k/month/project
seems a bit much for this kind of thing.  If the goal is to just kick
the tires or to have a soak test, why couldn't this be done on minimal
virtual machines or minimal hardware?

Dave
---
David Huseby
Security Maven, Hyperledger
The Linux Foundation
+1-206-234-2392
dhuseby@...

On Fri, Oct 26, 2018 at 12:30 PM Christopher Ferris
<chris.ferris@...> wrote:
>
> Interesting that Casey thinks of long-running tests, performance, chaos, etc. I always think of a testnet as a place where devs can kick the tires without having to do the heavy lifting of setting up a network themselves.
>
> For the long-running tests, etc. we have nightly runs and we are gearing up a weekly as well that will be long running test that has kitchen sinks thrown at it and stuff. Not a large environment, but more than what we do for the merge and nightly CI.
>
> IBM also runs a series of long running tests leading up to a release on our platforms. We do share the results indirectly and it would be nice if they were just public, frankly (this is me speaking not for IBM but as a member of the community). We are working on an ability to spin up networks arbitrary complexity and scale, so that tests can be run. It would be nice to have that running in the community, but again, not as a test net for users as much as for what Casey was advocating for.
>
> Cheers
>
> Chris
>
> On Fri, Oct 26, 2018 at 10:24 AM Middleton, Dan <dan.middleton@...> wrote:
>>
>> Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?
>>
>>
>>
>> I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.
>>
>>
>>
>> I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.
>>
>>
>>
>> Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.
>>
>>
>>
>> LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.
>>
>>
>>
>> This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.
>>
>> And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.
>>
>>
>>
>> Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for infrastructure would need to be on the order of $10k/month/project which is probably not tenable.
>>
>>
>>
>> For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve in subsequent iterations.
>>
>>
>>
>> Regards,
>> Dan
>>
>>
>>
>>
>>
>> From: <tsc@...> on behalf of Ry Jones <rjones@...>
>> Date: Friday, October 26, 2018 at 12:06 AM
>> To: Casey Kuhlman <casey@...>
>> Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...>
>> Subject: Re: [Hyperledger TSC] Test net thinking
>>
>>
>>
>> For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.
>>
>>
>>
>> Go here: https://github.com/cncf/cluster
>>
>>
>>
>> Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
>>
>>
>>
>> Ry
>>
>>
>>
>> On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via Lists.Hyperledger.Org <casey=monax.io@...> wrote:
>>
>> From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
>>
>>
>>
>> While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
>>
>>
>>
>> My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
>>
>> --
>>
>> Ry Jones
>>
>> Community Architect, Hyperledger
>>
>> Chat: @ry
>