Re: Test net thinking
Duc Trinh <duc.m.trinh@...>
Both would useful. However, if there is a need to prioritize, I would like to throw a +1 for standing up a test net - especially for Fabric. I think the motivations for having a test are similar to what has been articulated for Stellar (@ https://www.stellar.org/developers/guides/concepts/test-net.html)
To keep hosting costs in check, HL can reset the network periodically (again, similar to what Stellar does).
Cheers,
Duc
On 10/25/18, 11:13 AM, "tsc@... on behalf of David Huseby" <tsc@... on behalf of dhuseby@...> wrote:
I think all of you are making very good points. There seem to be two
camps of "itches":
1) chaincode and developer UX focused "let me quickly test this".
2) developer/integration testing of an instance of a blockchain for
purposes of running it through the gamut of tests and letting it soak
to test stability over time.
I draw the lines there because addressing #1 is not really a concern
of test nets. I would think that something like a universal HL
Composer would be the way to go. Or better yet, a plugin for VSCode
or Vim/Emacs that allows for quick deployment of chaincode to a
running blockchain and then has useful hooks for testing. Again, this
is more developer interfacing than test nets and outside of the scope
of the proposal.
#2 is absolutely test nets and I think the correct way to address this
is for HL to pay for managed Kubernetes infrastructure that teams can
use Kubernetes configs and/or Helm charts to deploy a blockchain
network on for whatever they need to do, be it chaos testing, load
testing, or just a soak test. We have already begun the effort to
standardize Kubernetes configs/Helm charts in several projects. The
HL Cello meeting is where those people have been congregating and the
next step there is to have the different blockchain projects start
publishing official, composable configs/charts as part of their build
process and published outputs. This would make it trivial for a
developer/tester to "kubectl apply fabric-1.3.yaml" on Amazon EKS or
GCE or whatever Kubernetes infrastructure we set up.
I prefer Amazon because it has a very good cost predictor calculator
that can be used to see how much the tests would cost to run and it
would help with our accounting.
As for HL managment, my plan is that we only have very minimal
management that is focused around giving out credentials to use the
Kubernetes infrastructure and handling re-embursements for the bills.
I think it should be entirely upon the user/tester to handle spinning
up/down their blockchain. IMO HL shouldn't be in the business of
being IT for everybody who wants to spin up a blockchain for testing.
The cost of that would be huge and the time spent on that would
greatly impact our ability to grow the community in all other ways.
Dave
---
David Huseby
Security Maven, Hyperledger
The Linux Foundation
+1-206-234-2392
dhuseby@...
toggle quoted message
Show quoted text
To keep hosting costs in check, HL can reset the network periodically (again, similar to what Stellar does).
Cheers,
Duc
On 10/25/18, 11:13 AM, "tsc@... on behalf of David Huseby" <tsc@... on behalf of dhuseby@...> wrote:
I think all of you are making very good points. There seem to be two
camps of "itches":
1) chaincode and developer UX focused "let me quickly test this".
2) developer/integration testing of an instance of a blockchain for
purposes of running it through the gamut of tests and letting it soak
to test stability over time.
I draw the lines there because addressing #1 is not really a concern
of test nets. I would think that something like a universal HL
Composer would be the way to go. Or better yet, a plugin for VSCode
or Vim/Emacs that allows for quick deployment of chaincode to a
running blockchain and then has useful hooks for testing. Again, this
is more developer interfacing than test nets and outside of the scope
of the proposal.
#2 is absolutely test nets and I think the correct way to address this
is for HL to pay for managed Kubernetes infrastructure that teams can
use Kubernetes configs and/or Helm charts to deploy a blockchain
network on for whatever they need to do, be it chaos testing, load
testing, or just a soak test. We have already begun the effort to
standardize Kubernetes configs/Helm charts in several projects. The
HL Cello meeting is where those people have been congregating and the
next step there is to have the different blockchain projects start
publishing official, composable configs/charts as part of their build
process and published outputs. This would make it trivial for a
developer/tester to "kubectl apply fabric-1.3.yaml" on Amazon EKS or
GCE or whatever Kubernetes infrastructure we set up.
I prefer Amazon because it has a very good cost predictor calculator
that can be used to see how much the tests would cost to run and it
would help with our accounting.
As for HL managment, my plan is that we only have very minimal
management that is focused around giving out credentials to use the
Kubernetes infrastructure and handling re-embursements for the bills.
I think it should be entirely upon the user/tester to handle spinning
up/down their blockchain. IMO HL shouldn't be in the business of
being IT for everybody who wants to spin up a blockchain for testing.
The cost of that would be huge and the time spent on that would
greatly impact our ability to grow the community in all other ways.
Dave
---
David Huseby
Security Maven, Hyperledger
The Linux Foundation
+1-206-234-2392
dhuseby@...
On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman <casey@...> wrote:
>
> From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
>
> While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
>
> My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
> ____________________________________
>
> Casey Kuhlman, Monax
> CEO
> Site: https://monax.io
> Twitter: @monaxhq, @compleatang
> Email: casey@...
> Phone (US): +1-423-523-9531; (UK): +44-75-073-96359
> ____________________________________
>
>
> On Thu, Oct 25, 2018 at 4:03 PM Christopher Ferris <chris.ferris@...> wrote:
>>
>> Dan, yes, but my response to that would be that Fabric should improve its UX around deploying and engaging chaincode/smart contract.
>> Having a testnet does nothing to improve that experience IMNSHO.
>>
>> Chris
>>
>> On Thu, Oct 25, 2018 at 10:18 AM Dan Selman <dan@...> wrote:
>>>
>>> Chris,
>>>
>>> I think you raise some very valid points regarding operating a test net. I'd like to inject an end user perspective if I may.
>>>
>>> I think Ethereum wins from an ease of use perspective because people can easily write a smart contract and deploy it to a test network. That's what the vast majority of developers want to get done when they first encounter blockchain and IMO that is where Fabric (and others) fall down. HLF v1.3 has made it (relatively) easy to *start* the network on your laptop, but once you actually want to deploy chaincode and interact with it, it is a whole different story: you start hacking on fabcar and trying to understand how ./startFabric.sh, enrollAdrmin.js, registerUser.js, hlf-key-store, docker etc all work. Contrast with using http://remix.ethereum.org running in the browser, or connected to https://infura.io or https://kaleido.io for writing your first Solidity code.
>>>
>>> There's an opportunity to do something similar to remix using the Composer Playground connected to a test net which is worth exploring imo. I don't know if that should be the remit of HL/TSC but there's clearly a need for something like that to attract more developers.
>>>
>>> Sincerely,
>>> Dan
>>>
>>>
>>> On Thu, Oct 25, 2018 at 2:54 PM Christopher Ferris <chris.ferris@...> wrote:
>>>>
>>>> I think that we need to consider what we think a "test net" is. Is it a whole network that a developer spins up themselves? Is it a pre-existing network that we have launched and that developers can use to deploy a smart contract and kick the tires? Is it permissioned? Permissionless? What are the implications? Will there be likelihood of noisy neighbor situations? Crypto Kitties? Who will monitor the experience? What if it gets really bogged down? Who is monitoring that it is running? Will users walk away from a bad experience with a bad taste in their mouths? How will the experience reflect on the project and the organization?
>>>>
>>>> I don't disagree that we want to make it dead simple to engage with the various projects. However, I can say that for Fabric, the process could not be simpler.
>>>>
>>>> download fabric-samples repo, images and binaries (one click)
>>>> cd first-network
>>>> byfn.sh up
>>>>
>>>> three commands and you have it running on your laptop.
>>>>
>>>> Could we create some helm charts that could be used to deploy our various projects to K8s? Let people deal with finding the compute however they like (maybe point at alternatives) but we don't host/pay?
>>>>
>>>> I ask because I really don't understand what problem we think we are solving with a test net, nor do I believe that we have considered the broader implications of running a cloud service on a volunteer basis.
>>>>
>>>> Chris
>>>>
>>>> On Wed, Oct 24, 2018 at 5:56 PM David Huseby <dhuseby@...> wrote:
>>>>>
>>>>> Hi TSC,
>>>>>
>>>>> I had a discussion with Ry this week while manning the booth at ATO in Raleigh and we both think the right way to go is a two pronged approach:
>>>>>
>>>>> The first prong will be to engage with the free kubernetes hosting being paid for by (I think) the CNCF project. Ry knows more about it than I do.
>>>>>
>>>>> The second prong would be to set aside some budget to cover the cost of people running networks on Amazon EKS. We could set some rule like we cover the cost up to $200 for the first month with an opportunity to get additional months with approval.
>>>>>
>>>>> I agree with Chris and Ry and I’m concerned about the human overhead of trying to run test nets for people. I think the best compromise is to just cover the bills for people spinning up test networks on hosted kubernetes platforms. The oversight needs to be an easy application and approval process as well as an easy renewal process. I was thinking that $20k would be enough to cover. That’s at least 10 networks per month for a year.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> Dave
>>>>> --
>>>>> ---
>>>>> David Huseby
>>>>> Security Maven, Hyperledger
>>>>> The Linux Foundation
>>>>> +1-206-234-2392
>>>>> dhuseby@...
>>>
>>>
>>>
>>> --
>>> Dan Selman
>>> CTO, Clause Inc.
>>> @danielselman
>>>
>>
>
> From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
>
> While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
>
> My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
> ____________________________________
>
> Casey Kuhlman, Monax
> CEO
> Site: https://monax.io
> Twitter: @monaxhq, @compleatang
> Email: casey@...
> Phone (US): +1-423-523-9531; (UK): +44-75-073-96359
> ____________________________________
>
>
> On Thu, Oct 25, 2018 at 4:03 PM Christopher Ferris <chris.ferris@...> wrote:
>>
>> Dan, yes, but my response to that would be that Fabric should improve its UX around deploying and engaging chaincode/smart contract.
>> Having a testnet does nothing to improve that experience IMNSHO.
>>
>> Chris
>>
>> On Thu, Oct 25, 2018 at 10:18 AM Dan Selman <dan@...> wrote:
>>>
>>> Chris,
>>>
>>> I think you raise some very valid points regarding operating a test net. I'd like to inject an end user perspective if I may.
>>>
>>> I think Ethereum wins from an ease of use perspective because people can easily write a smart contract and deploy it to a test network. That's what the vast majority of developers want to get done when they first encounter blockchain and IMO that is where Fabric (and others) fall down. HLF v1.3 has made it (relatively) easy to *start* the network on your laptop, but once you actually want to deploy chaincode and interact with it, it is a whole different story: you start hacking on fabcar and trying to understand how ./startFabric.sh, enrollAdrmin.js, registerUser.js, hlf-key-store, docker etc all work. Contrast with using http://remix.ethereum.org running in the browser, or connected to https://infura.io or https://kaleido.io for writing your first Solidity code.
>>>
>>> There's an opportunity to do something similar to remix using the Composer Playground connected to a test net which is worth exploring imo. I don't know if that should be the remit of HL/TSC but there's clearly a need for something like that to attract more developers.
>>>
>>> Sincerely,
>>> Dan
>>>
>>>
>>> On Thu, Oct 25, 2018 at 2:54 PM Christopher Ferris <chris.ferris@...> wrote:
>>>>
>>>> I think that we need to consider what we think a "test net" is. Is it a whole network that a developer spins up themselves? Is it a pre-existing network that we have launched and that developers can use to deploy a smart contract and kick the tires? Is it permissioned? Permissionless? What are the implications? Will there be likelihood of noisy neighbor situations? Crypto Kitties? Who will monitor the experience? What if it gets really bogged down? Who is monitoring that it is running? Will users walk away from a bad experience with a bad taste in their mouths? How will the experience reflect on the project and the organization?
>>>>
>>>> I don't disagree that we want to make it dead simple to engage with the various projects. However, I can say that for Fabric, the process could not be simpler.
>>>>
>>>> download fabric-samples repo, images and binaries (one click)
>>>> cd first-network
>>>> byfn.sh up
>>>>
>>>> three commands and you have it running on your laptop.
>>>>
>>>> Could we create some helm charts that could be used to deploy our various projects to K8s? Let people deal with finding the compute however they like (maybe point at alternatives) but we don't host/pay?
>>>>
>>>> I ask because I really don't understand what problem we think we are solving with a test net, nor do I believe that we have considered the broader implications of running a cloud service on a volunteer basis.
>>>>
>>>> Chris
>>>>
>>>> On Wed, Oct 24, 2018 at 5:56 PM David Huseby <dhuseby@...> wrote:
>>>>>
>>>>> Hi TSC,
>>>>>
>>>>> I had a discussion with Ry this week while manning the booth at ATO in Raleigh and we both think the right way to go is a two pronged approach:
>>>>>
>>>>> The first prong will be to engage with the free kubernetes hosting being paid for by (I think) the CNCF project. Ry knows more about it than I do.
>>>>>
>>>>> The second prong would be to set aside some budget to cover the cost of people running networks on Amazon EKS. We could set some rule like we cover the cost up to $200 for the first month with an opportunity to get additional months with approval.
>>>>>
>>>>> I agree with Chris and Ry and I’m concerned about the human overhead of trying to run test nets for people. I think the best compromise is to just cover the bills for people spinning up test networks on hosted kubernetes platforms. The oversight needs to be an easy application and approval process as well as an easy renewal process. I was thinking that $20k would be enough to cover. That’s at least 10 networks per month for a year.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> Dave
>>>>> --
>>>>> ---
>>>>> David Huseby
>>>>> Security Maven, Hyperledger
>>>>> The Linux Foundation
>>>>> +1-206-234-2392
>>>>> dhuseby@...
>>>
>>>
>>>
>>> --
>>> Dan Selman
>>> CTO, Clause Inc.
>>> @danielselman
>>>
>>