Who will be at DevCon in Prague in two weeks? I'll be there but only Thursday the 1st through Sunday morning. Happy to meet up with anyone there. We are also doing a Meetup on Friday night: https://www.meetup.com/Hyperledger-Prague/events/255766163/ Marta will be there I believe all week. Let me know if you'll be there.
Or in Hong Kong next week for HK Fintech week? Or Singapore for SG Fintech Week the 11th-14th?
Brian
-- Brian Behlendorf Executive Director, Hyperledger bbehlendorf@... Twitter: @brianbehlendorf
Thanks for the feedback! Either here or on the Google doc is fine. If you have direct suggestions, feel free to comment them in to the Google doc so we can merge them correctly.
I’ll try to address your questions and comments here:
1.Tiering is always going to be somewhat of a judgement call, particularly between tiers 2 and 3. Post-quantum algorithms are a good example of this, and I think (in my opinion) some might go either
way. For instance, I would put Frodo (an LWE-based key exchange that is extremely conservative in its parameter choices and has had an open implementation for quite some time) clearly in tier 2. On the other hand, I would put something like WalnutDSA (a
braid group-based signature algorithm that has had many parameter changes over the past year) in tier 3. It wouldn’t be unreasonable for maintainers to have disagreements on tiers for certain things, so we would have to vote on tier issues. However, I think
that most of the stuff that we will begin with should be clear.
2.Yes, this is a good point. I’ll make this change.
3.Ah, the joys of UC-security. The “theoretical maintainers” should block any pull requests that combine things in such an insecure manner, since any such mechanisms use crypto in a non-black box manner.
However, you do bring up a good point: standardized APIs are really nice (that’s actually the goal of the base crypto-lib signature library!). We hope to coalesce into simple APIs that are universally adopted throughout the project, but we think it will
hinder development and adoption if we force people to use certain APIs at the start of the project. Backwards compatibility with existing projects is another reason why we can’t standardize APIs completely—we would like to have an API that all of the new
stuff uses, but backwards compatibility breaks the old stuff. I guess, to summarize: we hope that some kind of de facto API Council is formed, but we don’t want to mandate it right now because we think it might hinder adoption and development.
4.Yes, that’s a good point. While I’m perfectly comfortable saying that rolling your own hash functions is “incredibly foolish” and having that enshrined and public, I have replaced that phrase with
“poor.” In addition, sadly the FAQ about the dunking booth will have to go before the document is finalized…
5.The base lib project is planning to incorporate shims to many different languages. I think that there are many people interested in a Golang interface, so I imagine this will get built. As the project
matures, I imagine we will have more discussions about safe languages. I think this is also why many people involved in the project are really excited about Rust.
6.Tier 0, clearly! ;)
I hope this cleared things up. Let me know if you have further questions or comments.
From: Geater, Jon [mailto:Jon.Geater@...]
Sent: Tuesday, October 30, 2018 2:07 AM To: Montgomery, Hart <hmontgomery@...>; Hyperledger List <tsc@...> Subject: Re: [Hyperledger TSC] crypto-lib Proposal Update
Hi Hart, all,
Hope this is the right forum for feedback: if I should be on the wiki or inside the Google Doc just let me know and I’ll shift over.
By and large, looks great, and I appreciate the clear thought that has gone into making this a practical and actionable structured project.
I have a handful of comments/questions:
1. The tiering between 2 and 3 is not completely clear to me in one regard: specifically, we’re talking firmly in tier 2 about things in that are used (widely?) in practise, but then tier 3 goes right to science experiments. What do we
do with things like leading post-quantum algorithms that are relatively well known and trod but are not yet standards approved and are still being refined? Would such things default to tier 3 and then be promoted when we’re happy with the implementation?
Or default to tier 2 and then be taken out if they’re judged dangerous?
2. I appreciate (and agree with) the intent of the ‘overwhelming consensus’ requirement for doing important things, but as time goes on and people come and go I’m concerned that such loose language could cause unnecessary controversy at
some point. Why not define this more concretely as 2/3 super majority or similar as it is later on in the doc? Note that this also implies a requirement to manage the voting eligibility of members, but ‘twas ever thus.
3. Since sub-projects are defined essentially as creating code modules for people to incorporate into their super projects, we should assume that dependent super projects will incorporate a number of these modules in order to satisfy all
their Crypto-related needs. It follows therefore that there’s a clear need to ensure that (most of) those modules follow a consistent architectural pattern, implement the same API Design philosophy, and very strictly agree on the semantics and data typing
of parameters that may be shared, or at least commonly expressed, between them. Apart from the obvious programming sanity requirement here, we’ve all seen plenty of cases where plugging together 2 secure things makes one big insecure thing. Who/what role
will do this? It’s good that the sub projects are independent and self-governing most of the time but Inthink something like an API Council is required here.
4. Assuming this proposal will at some point become enshrined and semi-public, it would probably be polite to replace the “incredibly foolish” language. You’re right, but still. Optics.
5. I want a Golang version. Would that be a subproject shim, or just a separate output of the one base lib project?
5a. I’d like a discussion about safe languages in general, but that can happen in slower time.
6. What tier of Project is the dunking booth? It doesn’t seem to quite fit any consistent set of the criteria for any band.
Jon
JonGeater CTO
Tel: +44 1223 703479
Mob: +44 7966 995312
@thalesesecurity
Thales eSecurity One Station Square
Cambridge CB1 2GA United Kingdom
First of all, thank you all for the feedback during and after the TSC meeting last week. It was good to hear suggestions from people, particularly those that are open source veterans.
We have incorporated most of these suggestions into a new draft of our crypto-lib proposal. Please take a look—we’d love more feedback. In particular, we thought Chris Ferris’s suggestion to eliminate the steward position and instead
rely on maintainer lists was a good one, and it is reflected in the current proposal.
On Tue, Oct 30, 2018, 6:19 PM Alejandro Sasha Vicente-Grabovetsky <alex.vice.grab@...> wrote:
Hi all,
We (AID:Tech) are here also, if someone wants to meet up and discuss Hyperledger Fabric/Composer DevOps on Kubernetes, we are free to meet up.
Best, Sasha
On Mon, Oct 29, 2018 at 6:15 AM Vipin Bharathan <vipinsun@...> wrote:
Hello Brian,
I will be at the Malta Blockchain Conference on November 1 and 2.
I will promote Hyperledger as a collaborative project and also get a read on the Blockchain activity in Malta.
Best,
Vipin
On Tue, Oct 23, 2018 at 9:35 PM Brian Behlendorf <bbehlendorf@...> wrote:
Who will be at DevCon in Prague in two weeks? I'll be there but only
Thursday the 1st through Sunday morning. Happy to meet up with anyone
there. We are also doing a Meetup on Friday night: https://www.meetup.com/Hyperledger-Prague/events/255766163/ Marta will
be there I believe all week. Let me know if you'll be there.
Or in Hong Kong next week for HK Fintech week? Or Singapore for SG
Fintech Week the 11th-14th?
Brian
--
Brian Behlendorf
Executive Director, Hyperledger bbehlendorf@...
Twitter: @brianbehlendorf
On Mon, Oct 29, 2018 at 6:15 AM Vipin Bharathan <vipinsun@...> wrote:
Hello Brian,
I will be at the Malta Blockchain Conference on November 1 and 2.
I will promote Hyperledger as a collaborative project and also get a read on the Blockchain activity in Malta.
Best,
Vipin
On Tue, Oct 23, 2018 at 9:35 PM Brian Behlendorf <bbehlendorf@...> wrote:
Who will be at DevCon in Prague in two weeks? I'll be there but only
Thursday the 1st through Sunday morning. Happy to meet up with anyone
there. We are also doing a Meetup on Friday night: https://www.meetup.com/Hyperledger-Prague/events/255766163/ Marta will
be there I believe all week. Let me know if you'll be there.
Or in Hong Kong next week for HK Fintech week? Or Singapore for SG
Fintech Week the 11th-14th?
Brian
--
Brian Behlendorf
Executive Director, Hyperledger bbehlendorf@...
Twitter: @brianbehlendorf
Hope this is the right forum for feedback: if I should be on the wiki or inside the Google Doc just let me know and I’ll shift over.
By and large, looks great, and I appreciate the clear thought that has gone into making this a practical and actionable structured project.
I have a handful of comments/questions:
1. The tiering between 2 and 3 is not completely clear to me in one regard: specifically, we’re talking firmly in tier 2 about things in that are used (widely?) in practise, but then tier 3 goes right to science experiments. What
do we do with things like leading post-quantum algorithms that are relatively well known and trod but are not yet standards approved and are still being refined? Would such things default to tier 3 and then be promoted when we’re happy with the implementation?
Or default to tier 2 and then be taken out if they’re judged dangerous?
2. I appreciate (and agree with) the intent of the ‘overwhelming consensus’ requirement for doing important things, but as time goes on and people come and go I’m concerned that such loose language could cause unnecessary controversy
at some point. Why not define this more concretely as 2/3 super majority or similar as it is later on in the doc? Note that this also implies a requirement to manage the voting eligibility of members, but ‘twas ever thus.
3. Since sub-projects are defined essentially as creating code modules for people to incorporate into their super projects, we should assume that dependent super projects will incorporate a number of these modules in order to satisfy
all their Crypto-related needs. It follows therefore that there’s a clear need to ensure that (most of) those modules follow a consistent architectural pattern, implement the same API Design philosophy, and very strictly agree on the semantics and data typing
of parameters that may be shared, or at least commonly expressed, between them. Apart from the obvious programming sanity requirement here, we’ve all seen plenty of cases where plugging together 2 secure things makes one big insecure thing. Who/what role
will do this? It’s good that the sub projects are independent and self-governing most of the time but Inthink something like an API Council is required here.
4. Assuming this proposal will at some point become enshrined and semi-public, it would probably be polite to replace the “incredibly foolish” language. You’re right, but still. Optics.
5. I want a Golang version. Would that be a subproject shim, or just a separate output of the one base lib project?
5a. I’d like a discussion about safe languages in general, but that can happen in slower time.
6. What tier of Project is the dunking booth? It doesn’t seem to quite fit any consistent set of the criteria for any band.
Jon
Jon Geater
CTO
Tel: +44 1223 703479
Mob: +44 7966 995312
@thalesesecurity
Thales eSecurity
One Station Square
Cambridge
CB1 2GA
United Kingdom
On Mon, Oct 29, 2018 at 11:55 PM +0100, "hmontgomery@..."
<hmontgomery@...> wrote:
Hi Everyone,
First of all, thank you all for the feedback during and after the TSC meeting last week. It was good to hear suggestions from people, particularly those that are open source veterans.
We have incorporated most of these suggestions into a new draft of our crypto-lib proposal. Please take a look—we’d love more feedback. In particular, we thought Chris Ferris’s suggestion to eliminate the steward position and instead
rely on maintainer lists was a good one, and it is reflected in the current proposal.
It strikes me, though, that not everyone is aware of what Hyperledger already has available to the various top-level projects.
Fabric uses Jenkins to drive testing of CRs and merge commits, and we also do nightly and weekly tests leveraging the LF cloud provider's VMs. They aren't fast, the network is slow, but they get the job done. We spin up networks on-demand, run the tests and tear it all down. Lather, rinse and repeat. These aren't large networks (yet), just simple docker-compose configurations designed to run in a single VM.
Do we want to have the ability to test across multiple VMs leveraging a k8s environment? You bet we do. I think that for the most part, that we'd continue for CRs and merges we would continue down the path we currently have chosen.
For integration, performance, scale, long-running and chaos testing having a persistent k8s platform in which to launch pre-configured networks leveraging the latest images and binaries for testing purposes would be "a good thing". Managing that platform is probably something that the LF is not set up to do. Leveraging a provider such as IBM, or AWS or Azure kubernetes service would make sense. Then there is the problem of managing the use of that (it is still a scarce resource that needs managing) across the various projects.
Again, this all presumes we are talking effectively about testing FOR the projects, not OF the projects by end users.
On Tue, Oct 30, 2018 at 9:59 AM Christopher Ferris <chris.ferris@...> wrote:
You cannot do performance and scale testing on nominal hardware. Long running and chaos tests, sure.
Chris
On Mon, Oct 29, 2018 at 9:15 PM David Huseby <dhuseby@...> wrote:
Does it really take big iron to run these tests? $10k/month/project
seems a bit much for this kind of thing. If the goal is to just kick
the tires or to have a soak test, why couldn't this be done on minimal
virtual machines or minimal hardware?
Dave
---
David Huseby
Security Maven, Hyperledger
The Linux Foundation
+1-206-234-2392 dhuseby@...
On Fri, Oct 26, 2018 at 12:30 PM Christopher Ferris
<chris.ferris@...> wrote:
>
> Interesting that Casey thinks of long-running tests, performance, chaos, etc. I always think of a testnet as a place where devs can kick the tires without having to do the heavy lifting of setting up a network themselves.
>
> For the long-running tests, etc. we have nightly runs and we are gearing up a weekly as well that will be long running test that has kitchen sinks thrown at it and stuff. Not a large environment, but more than what we do for the merge and nightly CI.
>
> IBM also runs a series of long running tests leading up to a release on our platforms. We do share the results indirectly and it would be nice if they were just public, frankly (this is me speaking not for IBM but as a member of the community). We are working on an ability to spin up networks arbitrary complexity and scale, so that tests can be run. It would be nice to have that running in the community, but again, not as a test net for users as much as for what Casey was advocating for.
>
> Cheers
>
> Chris
>
> On Fri, Oct 26, 2018 at 10:24 AM Middleton, Dan <dan.middleton@...> wrote:
>>
>> Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?
>>
>>
>>
>> I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.
>>
>>
>>
>> I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.
>>
>>
>>
>> Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.
>>
>>
>>
>> LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.
>>
>>
>>
>> This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.
>>
>> And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.
>>
>>
>>
>> Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for infrastructure would need to be on the order of $10k/month/project which is probably not tenable.
>>
>>
>>
>> For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve in subsequent iterations.
>>
>>
>>
>> Regards,
>> Dan
>>
>>
>>
>>
>>
>> From: <tsc@...> on behalf of Ry Jones <rjones@...>
>> Date: Friday, October 26, 2018 at 12:06 AM
>> To: Casey Kuhlman <casey@...>
>> Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...>
>> Subject: Re: [Hyperledger TSC] Test net thinking
>>
>>
>>
>> For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.
>>
>>
>>
>> Go here: https://github.com/cncf/cluster
>>
>>
>>
>> Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
>>
>>
>>
>> Ry
>>
>>
>>
>> On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via Lists.Hyperledger.Org <casey=monax.io@...> wrote:
>>
>> From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
>>
>>
>>
>> While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
>>
>>
>>
>> My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
>>
>> --
>>
>> Ry Jones
>>
>> Community Architect, Hyperledger
>>
>> Chat: @ry
>
On Mon, Oct 29, 2018 at 9:15 PM David Huseby <dhuseby@...> wrote:
Does it really take big iron to run these tests? $10k/month/project
seems a bit much for this kind of thing. If the goal is to just kick
the tires or to have a soak test, why couldn't this be done on minimal
virtual machines or minimal hardware?
Dave
---
David Huseby
Security Maven, Hyperledger
The Linux Foundation
+1-206-234-2392 dhuseby@...
On Fri, Oct 26, 2018 at 12:30 PM Christopher Ferris
<chris.ferris@...> wrote:
>
> Interesting that Casey thinks of long-running tests, performance, chaos, etc. I always think of a testnet as a place where devs can kick the tires without having to do the heavy lifting of setting up a network themselves.
>
> For the long-running tests, etc. we have nightly runs and we are gearing up a weekly as well that will be long running test that has kitchen sinks thrown at it and stuff. Not a large environment, but more than what we do for the merge and nightly CI.
>
> IBM also runs a series of long running tests leading up to a release on our platforms. We do share the results indirectly and it would be nice if they were just public, frankly (this is me speaking not for IBM but as a member of the community). We are working on an ability to spin up networks arbitrary complexity and scale, so that tests can be run. It would be nice to have that running in the community, but again, not as a test net for users as much as for what Casey was advocating for.
>
> Cheers
>
> Chris
>
> On Fri, Oct 26, 2018 at 10:24 AM Middleton, Dan <dan.middleton@...> wrote:
>>
>> Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?
>>
>>
>>
>> I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.
>>
>>
>>
>> I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.
>>
>>
>>
>> Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.
>>
>>
>>
>> LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.
>>
>>
>>
>> This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.
>>
>> And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.
>>
>>
>>
>> Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for infrastructure would need to be on the order of $10k/month/project which is probably not tenable.
>>
>>
>>
>> For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve in subsequent iterations.
>>
>>
>>
>> Regards,
>> Dan
>>
>>
>>
>>
>>
>> From: <tsc@...> on behalf of Ry Jones <rjones@...>
>> Date: Friday, October 26, 2018 at 12:06 AM
>> To: Casey Kuhlman <casey@...>
>> Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...>
>> Subject: Re: [Hyperledger TSC] Test net thinking
>>
>>
>>
>> For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.
>>
>>
>>
>> Go here: https://github.com/cncf/cluster
>>
>>
>>
>> Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
>>
>>
>>
>> Ry
>>
>>
>>
>> On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via Lists.Hyperledger.Org <casey=monax.io@...> wrote:
>>
>> From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
>>
>>
>>
>> While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
>>
>>
>>
>> My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
>>
>> --
>>
>> Ry Jones
>>
>> Community Architect, Hyperledger
>>
>> Chat: @ry
>
If we contribute a basic K8s cluster (1 master/3 nodes) running 3 Sawtooth validators with m4.large machines that’s approx 40c/hour on demand.
Obviously this can be reduced further using 1 year reserved instances (m5.large with no upfront payment is ~$45/month) so on this basis in round figures if we had 10 participants each contributing a 3-validator cluster we could create a 30-validitor Sawtooth testnet running across multiple regions for ~$2k/month - possibly more depending on network/storage requirements.
On 30 Oct 2018, at 01:15, David Huseby <dhuseby@...> wrote:
Does it really take big iron to run these tests? $10k/month/project seems a bit much for this kind of thing. If the goal is to just kick the tires or to have a soak test, why couldn't this be done on minimal virtual machines or minimal hardware?
Dave --- David Huseby Security Maven, Hyperledger The Linux Foundation +1-206-234-2392 dhuseby@...
On Fri, Oct 26, 2018 at 12:30 PM Christopher Ferris <chris.ferris@...> wrote:
Interesting that Casey thinks of long-running tests, performance, chaos, etc. I always think of a testnet as a place where devs can kick the tires without having to do the heavy lifting of setting up a network themselves.
For the long-running tests, etc. we have nightly runs and we are gearing up a weekly as well that will be long running test that has kitchen sinks thrown at it and stuff. Not a large environment, but more than what we do for the merge and nightly CI.
IBM also runs a series of long running tests leading up to a release on our platforms. We do share the results indirectly and it would be nice if they were just public, frankly (this is me speaking not for IBM but as a member of the community). We are working on an ability to spin up networks arbitrary complexity and scale, so that tests can be run. It would be nice to have that running in the community, but again, not as a test net for users as much as for what Casey was advocating for.
Cheers
Chris
On Fri, Oct 26, 2018 at 10:24 AM Middleton, Dan <dan.middleton@...> wrote:
Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?
I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.
I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.
Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.
LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.
This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.
And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.
Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for infrastructure would need to be on the order of $10k/month/project which is probably not tenable.
For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve in subsequent iterations.
Regards, Dan
From: <tsc@...> on behalf of Ry Jones <rjones@...> Date: Friday, October 26, 2018 at 12:06 AM To: Casey Kuhlman <casey@...> Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...> Subject: Re: [Hyperledger TSC] Test net thinking
For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.
Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
Ry
On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via Lists.Hyperledger.Org <casey=monax.io@...> wrote:
From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
Description:
The Hyperledger Composer project update to the TSC was due October 29, 2018, and it will be presented to the TSC on November 1, 2018. Please review prior to the meeting and bring your questions.
Does it really take big iron to run these tests? $10k/month/project seems a bit much for this kind of thing. If the goal is to just kick the tires or to have a soak test, why couldn't this be done on minimal virtual machines or minimal hardware?
Dave --- David Huseby Security Maven, Hyperledger The Linux Foundation +1-206-234-2392 dhuseby@...
On Fri, Oct 26, 2018 at 12:30 PM Christopher Ferris <chris.ferris@...> wrote:
Interesting that Casey thinks of long-running tests, performance, chaos, etc. I always think of a testnet as a place where devs can kick the tires without having to do the heavy lifting of setting up a network themselves.
For the long-running tests, etc. we have nightly runs and we are gearing up a weekly as well that will be long running test that has kitchen sinks thrown at it and stuff. Not a large environment, but more than what we do for the merge and nightly CI.
IBM also runs a series of long running tests leading up to a release on our platforms. We do share the results indirectly and it would be nice if they were just public, frankly (this is me speaking not for IBM but as a member of the community). We are working on an ability to spin up networks arbitrary complexity and scale, so that tests can be run. It would be nice to have that running in the community, but again, not as a test net for users as much as for what Casey was advocating for.
Cheers
Chris
On Fri, Oct 26, 2018 at 10:24 AM Middleton, Dan <dan.middleton@...> wrote:
Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?
I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.
I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.
Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.
LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.
This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.
And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.
Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for infrastructure would need to be on the order of $10k/month/project which is probably not tenable.
For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve in subsequent iterations.
Regards, Dan
From: <tsc@...> on behalf of Ry Jones <rjones@...> Date: Friday, October 26, 2018 at 12:06 AM To: Casey Kuhlman <casey@...> Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...> Subject: Re: [Hyperledger TSC] Test net thinking
For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.
Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
Ry
On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via Lists.Hyperledger.Org <casey=monax.io@...> wrote:
From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
First of all, thank you all for the feedback during and after the TSC meeting last week. It was good to hear suggestions from people, particularly those that are open source veterans.
We have incorporated most of these suggestions into a new draft of our crypto-lib proposal. Please take a look—we’d love more feedback. In particular, we thought Chris Ferris’s suggestion to eliminate the steward position and instead
rely on maintainer lists was a good one, and it is reflected in the current proposal.
On Tue, Oct 23, 2018 at 9:35 PM Brian Behlendorf <bbehlendorf@...> wrote:
Who will be at DevCon in Prague in two weeks? I'll be there but only
Thursday the 1st through Sunday morning. Happy to meet up with anyone
there. We are also doing a Meetup on Friday night: https://www.meetup.com/Hyperledger-Prague/events/255766163/ Marta will
be there I believe all week. Let me know if you'll be there.
Or in Hong Kong next week for HK Fintech week? Or Singapore for SG
Fintech Week the 11th-14th?
Brian
--
Brian Behlendorf
Executive Director, Hyperledger bbehlendorf@...
Twitter: @brianbehlendorf
Who will be at DevCon in Prague in two weeks? I'll be there but only Thursday the 1st through Sunday morning. Happy to meet up with anyone there. We are also doing a Meetup on Friday night: https://www.meetup.com/Hyperledger-Prague/events/255766163/ Marta will be there I believe all week. Let me know if you'll be there.
Or in Hong Kong next week for HK Fintech week? Or Singapore for SG Fintech Week the 11th-14th?
Brian
-- Brian Behlendorf Executive Director, Hyperledger bbehlendorf@... Twitter: @brianbehlendorf
On Oct 24, 2018, at 5:06 AM, Brian Behlendorf <bbehlendorf@...> wrote:
That's great. And yes my ask was of anyone following the list, not just TSC members. :) See you in SG.
Brian
On 10/23/18 6:47 PM, Duncan Johnston-Watt wrote:
Brian
I appreciate that we are not TSC members but FWIW myself and Kai Davenport will be at SG Fintech Festival where we will be delivering a two day workshop as part of their industry events program Nov 15&16 - https://fintechfestival.sg/festival-line-up/industry/
- which we will be promoting shortly.
If you or Julian know of folk locally in SG who would benefit from this but workshop who don't have the budget please connect us and we will give them a discount code.
Best
Duncan
On Wed, Oct 24, 2018 at 2:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
Who will be at DevCon in Prague in two weeks? I'll be there but only
Thursday the 1st through Sunday morning. Happy to meet up with anyone
there. We are also doing a Meetup on Friday night: https://www.meetup.com/Hyperledger-Prague/events/255766163/ Marta will
be there I believe all week. Let me know if you'll be there.
Or in Hong Kong next week for HK Fintech week? Or Singapore for SG
Fintech Week the 11th-14th?
Brian
--
Brian Behlendorf
Executive Director, Hyperledger bbehlendorf@...
Twitter: @brianbehlendorf
Or iPhone one-tap : US: +16699006833,,6223336701# or +16465588656,,6223336701# Or Telephone: Dial(for higher quality, dial a number based on your current location): US: +1 669 900 6833 or +1 646 558 8656 or +1 877 369 0926 (Toll Free) or +1 855 880 1246 (Toll Free) Meeting ID: 622-333-6701 International numbers available: https://zoom.us/zoomconference?m=BYDz1WGXJTTJ_s4_zumD9hqKjJv-Whgs
You are receiving this courtesy email at the account tsc@... because you are an attendee of this event.
To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar.
Forwarding this invitation could allow any recipient to modify your RSVP response. Learn More.
Or iPhone one-tap : US: +16699006833,,6223336701# or +16465588656,,6223336701# Or Telephone: Dial(for higher quality, dial a number based on your current location): US: +1 669 900 6833 or +1 646 558 8656 or +1 877 369 0926 (Toll Free) or +1 855 880 1246 (Toll Free) Meeting ID: 622-333-6701 International numbers available: https://zoom.us/zoomconference?m=BYDz1WGXJTTJ_s4_zumD9hqKjJv-Whgs
You are receiving this courtesy email at the account tsc@... because you are an attendee of this event.
To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar.
Forwarding this invitation could allow any recipient to modify your RSVP response. Learn More.
Interesting that Casey thinks of long-running tests, performance, chaos, etc. I always think of a testnet as a place where devs can kick the tires without having to do the heavy lifting of setting up a network themselves.
For the long-running tests, etc. we have nightly runs and we are gearing up a weekly as well that will be long running test that has kitchen sinks thrown at it and stuff. Not a large environment, but more than what we do for the merge and nightly CI.
IBM also runs a series of long running tests leading up to a release on our platforms. We do share the results indirectly and it would be nice if they were just public, frankly (this is me speaking not for IBM but as a member of the community). We are working on an ability to spin up networks arbitrary complexity and scale, so that tests can be run. It would be nice to have that running in the community, but again, not as a test net for users as much as for what Casey was advocating for.
On Fri, Oct 26, 2018 at 10:24 AM Middleton, Dan <dan.middleton@...> wrote:
Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?
I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain
smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.
I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.
Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.
LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.
This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.
And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.
Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for
infrastructure would need to be on the order of $10k/month/project which is probably not tenable.
For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that
multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve
in subsequent iterations.
Regards,
Dan
From: <tsc@...> on behalf of Ry Jones <rjones@...> Date: Friday, October 26, 2018 at 12:06 AM To: Casey Kuhlman <casey@...> Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...> Subject: Re: [Hyperledger TSC] Test net thinking
For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.
Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure
which can help us.... test.
My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy
to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
Tldr: We only scale this if we have multiple companies hosting. What kind of HL support is required to facilitate that?
I agree with Chris that at least for native contracts like transaction families and chaincode our projects have scripts so developers can launch mini environments on their laptops (in sawtooth it’s basically `docker-compose up`). For on-chain
smart contracts like burrow, sabre, and so forth developers might appreciate a running testnet to deploy to.
I kind of expected Casey to come in along those lines, so it’s really helpful to read his priority for stress testing infrastructure.
Last year, running up to our 1.0 launch, the sawtooth team found they needed to automate the long running tests and associated cloud deployments. We’ve taken to calling these our LR networks.
LRs are typically 10 node networks + load generation and stats collections for 1 to 7 days. These LR networks have been invaluable to testing features before issuing PRs.
This stress testing is not cheap though. It’s one thing to pay for an idle instance and it’s another to pay for 10 fully utilized instances.
And as we are normally testing nightly builds along with experimental branches etc., it’s really more like a multiple of 10.
Based on the CNCF readme I’d bet the sawtooth project alone could easily consume all their credits. CNCF, then is probably a good start but not a complete solution to support all the HL projects. Similarly any direct budget estimate for
infrastructure would need to be on the order of $10k/month/project which is probably not tenable.
For sawtooth, we are looking at a crawl, walk, run with the community to expand from LR nets to larger longer testnets. Start with some machines provided by companies x, y, z and then add more users/instances. Actually demonstrating that
multiple companies can attach to a running network is a highly valuable exercise. We’re shooting to have the crawl version of the testnet up imminently. In getting that underway we may discover some of the administrative or other challenges that HL can solve
in subsequent iterations.
Regards,
Dan
From: <tsc@...> on behalf of Ry Jones <rjones@...> Date: Friday, October 26, 2018 at 12:06 AM To: Casey Kuhlman <casey@...> Cc: Christopher Ferris <chris.ferris@...>, "tsc@..." <tsc@...> Subject: Re: [Hyperledger TSC] Test net thinking
For people who are interested in CNCF services, the gatekeeper is CNCF, not Hyperledger.
Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure
which can help us.... test.
My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy
to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
Following the template, file an issue, make your case. Note it is a limited resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple of times and he wants to use the resource as much as possible.
From the burrow perspective, we'd be very interested in test networks to actually run testing in terms of performance testing, chaos testing, soak testing, long-lived network testing which a simple CI run post merge cannot really handle.
While I think both Dan and Chris make valid points in terms of positioning the test networks as things to help users get started faster, our interest from the burrow team is less in that area and more in having an HL managed infrastructure which can help us.... test.
My point, and the reason that Silas brought this up a few months ago to the TSC is that from a testing and performance perspective I think we could do much better as a community. CCNF and the Kubernetes project have a super clean and easy to approach CI system that made it dead simple for me as a contributor when I started making contributions to helm. From my perspective we'd be very interested to see if HL could coordinate with CCNF (as Ry mentioned on RocketChat).
