Hi,

Some of the ideas in today's call were

a. Look at Emerald and suggest ways in which this particular front-end can be hooked into Caliper, we have to look at duplication in both these libraries as well as the interface to decide whether this is worthwhile.

b. What does it take (smart contract wise)  to have Emerald tests run.

c. TPC/STAC Mark says that they run at a glacial pace plus they do not admit non-members (STAC does not) and they dont have Blockchain efforts

d. We spoke about Mark's presentation on May 7th and that he needed volunteers to run the call.

e. We spoke about Identity capabilities in Openshift as well as Oracle Blockchain in a side conversation

f. Spoke about having a quick look at Emerald.

A preliminary look at the github/gitlab repos say.

That you need the following capability (smart contracts etc.) in the platform you are going to test:

"In order for the test harness to measure the performance of a blockchain payents platform. The platform must support the following use cases:

• Create several accounts that can store tokens
• Issue tokens to specific accounts
• Transfer tokens from account to account between nodes.

The third use case is what is measured by this benchmark."

So you have to have a solution that contains these actions

Documentation is here https://gitlab.com/emerald-platform/emerald/wikis/Emerald-Benchmark

The hyperledger lab is empty except for the Readme file (https://github.com/hyperledger-labs/payments-performance-test-harness)

 

Conclusions:

1. It would require considerable effort to architect a solution that interacts with caliper; there are also some duplicate facilities around load generation.
2. The multi location load generation is neat
3. There is some neat stuff there for the payments use case, since it follows ISO standards for payments payload.
4. In order to run it against HL Fabric or HL Sawtooth or anything else the DLTs need a prebuilt payment solution (accounts storing tokens, issue tokens, transfer tokens between accounts)
5. HL Fabric has the new FabToken project maybe this can help
6. Without Mark Simpson to guide us it would take resources that we do not currently have. Caliper guys have to opine too.
7. In short not now, not with what we have in PSWG is my conclusion- obviously I am willing to learn if someone says why this isn't so.

Best,

Vipin

Vipin Bharathan

Enterprise Blockchain Consultant

vip@...

---

From: tsc@... <tsc@...> on behalf of mark wagner via Lists.Hyperledger.Org <mwagner=redhat.com@...>
Sent: Monday, April 22, 2019 9:38 AM
To: Dan Hyperledger
Cc: tsc@...
Subject: Re: [Hyperledger Performance and Scale WG] [Hyperledger TSC] some thoughts on the future of the PSWG

 

Hi Dan

Welcome back!

While I agree that integrity and availability are important and need to be quantified, I think that we still need baseline metrics to do so. "This is what the system does with faults, this is what happens when faults are introduced. "

One of the topics we had previously discussed was the Emerald benchmark that Mark Simpson had been involved in. It is now a Hyperledger lab. It is designed to measure payments. Should we consider using it with Caliper as our next deliverable ?

One of reasons for suggesting this is that it is already defined and has working code, abeit for Corda. However, by taking something that we know works, it should save us some time. The fact that it is a Hyperledger lab means that the code should be clean to start. Attila / Imre, did you ever get time to evaluate how difficult it would be to use emerald benchmark with Caliper ?  Mark Simpson, feel encouraged to jump in here as well!,

With respect to having TPC, SPEC etc to own a benchmark, I have mixed feelings. Red Hat is involved in both organizations and neither appears to be working with blockchain at this point in time. Based on my personal experience with the development of SPECVirt and SPECCloud, it will be at least a year before there is anything usable. Salman do you concur ?

Also, both orgs have fairly stringent rules and use policies, so a benchmark is typically not freely available for anyone to run. So there is no open source component.

Anyway, enough rambling. What do others think ?

-mark

On Tue, Apr 16, 2019 at 5:10 PM Dan Hyperledger <dan.hyperledger@...> wrote:

With respect to PSWG specifically, I think the most significant thing that working group can do is mature our ability to quantify integrity and availability.

The thing that makes blockchain different from other enterprise transaction systems is the use of decentralization in protecting the integrity and availability of the system.

The first output of the PSWG was a metrics document that made strides in measuring performance in that context. What does it mean if a network can execute 1000 TPS but can be taken down by a single node failure? The best we could do was be clear that performance should always be published in the context of the size of the network (as a proxy for a measure of availability and integrity).

We had a lot of discussion about measurements in the presence of faults. We even had some material started by a few of our contributors and while it wasn’t a fit for that draft of the metrics document, it seemed like a good seed for the next round of work.

A different direction of exploration is in workload design. In this case contributors from companies adopting blockchain are invaluable to help inform actual workload characteristics. One reason I don’t list benchmark design with higher importance is I believe it’s better developed by an organization like TPC.

Regards,

Dan Middleton

On Tue, Mar 19, 2019 at 7:44 AM Vipin Bharathan <vipinsun@...> wrote:

Hello Mark and others,

We have been having similar issues with the Architecture and Identity Working Groups.

You do not join a WG to mature into a developer for a particular solution. The work in a WG is complementary to the work in a project. You could do both. The WGs are not just about "documentation".

I have been thinking about the function of Working Groups. In my view it is the following.

The Working Groups were meant to be technical and focused on cross-cutting or primary concerns; like Architecture, Identity, Performance & Scale and now Smart Contracts.

Building stuff does require thinking about what to build. The WGs were meant to be a place where technical concerns from the various projects are aired out and where the collective wisdom of the WG participants could inform us all and prevent the isolation of frameworks from each other. In other words what makes Hyperledger, Hyperledger (beyond a single ledger); to connect projects together. This is evident in the papers published by the Architecture and PSWG groups.  

The Identity WG for example had people from Legal, Technical, Business as well as practitioners from  Identity specific areas all coming together to discuss topics like GDPR, Aadhar etc. and its relevance to the DLTs we are building. We brought Indy into the mix and are now working on L2 protocols in the Identity space. There is no comparable venue in HL for this. Somehow this message has been diluted and projects do not take this seriously enough nor is it effectively communicated.

The WGs were never highlighted in the wiki, and their benefits and contributions were seen as less than "landing patches".  

I see similarities in the way SIGs are forming now. In contrast, to the WGs the SIGs focus on verticals. WGs focus on horizontals. Both are needed. There should be collaboration between the SIGs and the WGs. 

Technical Working Group China and now one for India were formed because of differences in time zones and languages; and they are like all the WGs and SIGs rolled into one for the region. 

So now for some suggestions and ideas to make Working Groups more effective and encourage participation. This is based on conversations I have had with many people:

- It is important to do cross SIG/cross Regional/cross WG presentations

-Promotion in wiki, during bootcamps, messaging and Community Architect Support

-Experimentation with meeting times to encourage worldwide participation

-For PSWG, engagement with TPC, STAC, actual functional use case(Provenance) to look for numbers to measure, Look at ExactPro methodology etc.

-Engagement with Architecture for improvements in Performance like the waterloo project and implications on DLT architecture. What are the implications for other DLTs (other than Fabric)

Hope this long email may be of some use in rebooting the WGs and also rethinking of a more collaborative future. 

Best,

Vipin

On Tue, Mar 19, 2019 at 1:22 AM Kulkarni, Amol <amol.kulkarni@...> wrote:

One of the issues with the WGs as they’re currently set up is that interested folks from Asia cannot call into them.

 

The PSWG, for example, currently meets from 6:30 – 7:30 pm India time. While I’m working on performance and benchmarking specifically, my colleagues and I cannot call into any of these meetings. Perhaps rescheduling meetings to encourage participants from different geos might be one way to increase participation in WGs.

 

Amol

 

From: tsc@... [mailto:tsc@...] On Behalf Of Silona Bonewald
Sent: Tuesday, March 19, 2019 7:49 AM
To: Christopher Ferris <chris.ferris@...>
Cc: Todd Little <todd.little@...>; Montgomery, Hart <hmontgomery@...>; mark wagner <mwagner@...>; perf-and-scale-wg@...; Hyperledger List <tsc@...>
Subject: Re: [Hyperledger Performance and Scale WG] [Hyperledger TSC] some thoughts on the future of the PSWG

 

The Technical Working Group China participated in the HKBootcamp and was highly visible.  I think it is a good idea for other working groups to think about how to recruit and engage more with the community thru events like Bootcamps etc. 

 

For example, my team has been discussing with the Learning Materials Working Group about how they will recruit.  First from the projects themselves, and then how do we help them market themselves to get more volunteers?  Of course, having a good landing page on the wiki helps!

 

Also, Hyperledger has been starting up a bunch of SIGs.  I believe some people are moving to those SIGs.  And Ecosystem team is actively recruiting new members to participate there first.  Should we look at how we cross "advertise" internally?  Or look at how we "advertise" to new members and externally.

 

What other paths to recruitment should we explore? 

 

Silona

 

On Mon, Mar 18, 2019 at 4:15 PM Christopher Ferris <chris.ferris@...> wrote:

Todd, the Fabric maintainers would welcome additional help in documenting Fabric or providing and curating samples. They don’t write themselves.

 

I tend to agree with Hart’s analysis, and while I have a regular call that conflicts, I join occasionally and am definitely interested in the subject matter.

 

I’d agree with Todd that working with a traditional benchmarking org would be good, as would input into Caliper and other performance testing efforts.

 

We aren’t yet at a point where we can effectively provide guidance on comparative performance characteristics, which is ultimately what people will be seeking.

Chris

On Mar 18, 2019, at 6:49 PM, Todd Little <todd.little@...> wrote:

Hi Hart,

I tend to agree, but I also think another factor is that as we've moved from initial versions of the various platforms to production ready platforms, people are probably getting tied up working on those production deployments.  I know for me it's mostly a matter of I participate when I don't have conflicts or not traveling.

Personally I'd like to see us move in the direction of helping some standards organizations define blockchain benchmarks that can be used to evaluate blockchain platforms, as I'm not sure we're in the business of setting those kinds of standards.  Someone please correct me if I'm wrong.  I may have asked this before, but has anyone reached out to TPC to see if they're interested in creating some blockchain benchmarks?

With respect to the documentation, agreed it's fairly weak, and in particular design documentation is sorely lacking, at least for Fabric.  I can't speak to the other platforms.  Samples/examples is another area that is pretty weak.  Our platforms are so feature rich that it's hard to pick them up without a set of rich samples/examples.

-tl

PS FWIW I'll be on the call tomorrow.

On 3/18/2019 5:28 PM, Montgomery, Hart wrote:

Hi Mark,

 

I think this is probably true for most working groups.  At least from what I’ve seen and heard (and to be fair, this is not a huge sample size), fewer people are participating in the working groups, and those who are generally are participating less.

 

I’m not sure this is a bad thing, in many cases.  I think some people have replaced spending time on working groups with time spent on regular projects.  Personally, I’ve spent less time on working group stuff since Ursa has begun (and, obviously, more time on Ursa), which is probably a kind of progression we want to encourage in Hyperledger.  Having people that come into the working groups looking to learn become regular contributors surely is a positive thing. 

 

On the other hand, as the working groups seem to focus more and more on work products—which is typically documentation--fewer people seem to be interested since there is often substantial outside work involved, and not necessarily of the fun kind.  In particular, I think every SWOT analysis we have done on Hyperledger involves documentation being listed as a weakness.   While documentation has gotten better, it’s still something that I think most people would agree needs a lot of work on most aspects of Hyperledger.

 

So, for a TL;DR:  from my viewpoint, people are leaving working groups to contribute (great!)  or avoid documentation work (not so great). 

 

Do these experiences jibe with yours (or others’)?

 

Sorry for another long email.

 

Thanks,

Hart

 

 

 

From: tsc@... [mailto:tsc@...] On Behalf Of mark wagner
Sent: Monday, March 18, 2019 12:33 PM
To: perf-and-scale-wg@...
Cc: Hyperledger List <tsc@...>
Subject: [Hyperledger TSC] some thoughts on the future of the PSWG

 

Hi

 

I would like to carry this discussion mostly in the mailing lists in order to be as inclusive as possible.

 

Over the last few months attendance and participation on the Performance and Scale WG (PSWG) calls has been declining. I am wondering if we need to shift focus a bit to get more people involved and make progress.

 

There are many possible directions we can head in and I have included some thoughts here, but if others have additional ideas please feel encouraged to add them.

 

First is stay the course. We are currently struggling to get our thoughts on describing metrics for provenance captured to paper.

 

Do we want to focus more on actual performance work vs just defining metrics / use case papers. This does not need to be "just running tests".

 

We can do things like examine trade offs between different design decisions, examine different crypto solutions / technologies from a perf and scale perspective, help understand cost performance tradeoffs.

 

Perhaps we help spin up testnets and drive testing of different DLT frameworks ?

 

Do we provide a performance analysis service to the different projects that is based on architecture decisions. Perhaps something similar to what several research students have done (Harish and his team, FastFabric, etc).

 

I am trying to cut a wide swath here to avoid limiting peoples thoughts, so any and all comments are encouraged. If you have not been involved in the PSWG and would be interested in participating if we did "X" then speak up!

 

thanks

 

-mark

--

Mark Wagner

Chair, Performance and Scale Working Group

 

--

Silona Bonewald

VP of Community Architecture, Hyperledger

Mobile/Text: 512.750.9220
https://calendly.com/silona

The Linux Foundation
http://hyperledger.org

--

Mark Wagner

Senior Principal Software Engineer

Performance and Scalability

Red Hat, Inc

With respect to PSWG specifically, I think the most significant thing that working group can do is mature our ability to quantify integrity and availability.

The thing that makes blockchain different from other enterprise transaction systems is the use of decentralization in protecting the integrity and availability of the system.

The first output of the PSWG was a metrics document that made strides in measuring performance in that context. What does it mean if a network can execute 1000 TPS but can be taken down by a single node failure? The best we could do was be clear that performance should always be published in the context of the size of the network (as a proxy for a measure of availability and integrity).

We had a lot of discussion about measurements in the presence of faults. We even had some material started by a few of our contributors and while it wasn’t a fit for that draft of the metrics document, it seemed like a good seed for the next round of work.

A different direction of exploration is in workload design. In this case contributors from companies adopting blockchain are invaluable to help inform actual workload characteristics. One reason I don’t list benchmark design with higher importance is I believe it’s better developed by an organization like TPC.

Regards,

Dan Middleton

On Tue, Mar 19, 2019 at 7:44 AM Vipin Bharathan <vipinsun@...> wrote:

Hello Mark and others,

We have been having similar issues with the Architecture and Identity Working Groups.

You do not join a WG to mature into a developer for a particular solution. The work in a WG is complementary to the work in a project. You could do both. The WGs are not just about "documentation".

I have been thinking about the function of Working Groups. In my view it is the following.

The Working Groups were meant to be technical and focused on cross-cutting or primary concerns; like Architecture, Identity, Performance & Scale and now Smart Contracts.

Building stuff does require thinking about what to build. The WGs were meant to be a place where technical concerns from the various projects are aired out and where the collective wisdom of the WG participants could inform us all and prevent the isolation of frameworks from each other. In other words what makes Hyperledger, Hyperledger (beyond a single ledger); to connect projects together. This is evident in the papers published by the Architecture and PSWG groups.  

The Identity WG for example had people from Legal, Technical, Business as well as practitioners from  Identity specific areas all coming together to discuss topics like GDPR, Aadhar etc. and its relevance to the DLTs we are building. We brought Indy into the mix and are now working on L2 protocols in the Identity space. There is no comparable venue in HL for this. Somehow this message has been diluted and projects do not take this seriously enough nor is it effectively communicated.

The WGs were never highlighted in the wiki, and their benefits and contributions were seen as less than "landing patches".  

I see similarities in the way SIGs are forming now. In contrast, to the WGs the SIGs focus on verticals. WGs focus on horizontals. Both are needed. There should be collaboration between the SIGs and the WGs. 

Technical Working Group China and now one for India were formed because of differences in time zones and languages; and they are like all the WGs and SIGs rolled into one for the region. 

So now for some suggestions and ideas to make Working Groups more effective and encourage participation. This is based on conversations I have had with many people:

- It is important to do cross SIG/cross Regional/cross WG presentations

-Promotion in wiki, during bootcamps, messaging and Community Architect Support

-Experimentation with meeting times to encourage worldwide participation

-For PSWG, engagement with TPC, STAC, actual functional use case(Provenance) to look for numbers to measure, Look at ExactPro methodology etc.

-Engagement with Architecture for improvements in Performance like the waterloo project and implications on DLT architecture. What are the implications for other DLTs (other than Fabric)

Hope this long email may be of some use in rebooting the WGs and also rethinking of a more collaborative future. 

Best,

Vipin

On Tue, Mar 19, 2019 at 1:22 AM Kulkarni, Amol <amol.kulkarni@...> wrote:

One of the issues with the WGs as they’re currently set up is that interested folks from Asia cannot call into them.

 

The PSWG, for example, currently meets from 6:30 – 7:30 pm India time. While I’m working on performance and benchmarking specifically, my colleagues and I cannot call into any of these meetings. Perhaps rescheduling meetings to encourage participants from different geos might be one way to increase participation in WGs.

 

Amol

 

From: tsc@... [mailto:tsc@...] On Behalf Of Silona Bonewald
Sent: Tuesday, March 19, 2019 7:49 AM
To: Christopher Ferris <chris.ferris@...>
Cc: Todd Little <todd.little@...>; Montgomery, Hart <hmontgomery@...>; mark wagner <mwagner@...>; perf-and-scale-wg@...; Hyperledger List <tsc@...>
Subject: Re: [Hyperledger Performance and Scale WG] [Hyperledger TSC] some thoughts on the future of the PSWG

 

The Technical Working Group China participated in the HKBootcamp and was highly visible.  I think it is a good idea for other working groups to think about how to recruit and engage more with the community thru events like Bootcamps etc. 

 

For example, my team has been discussing with the Learning Materials Working Group about how they will recruit.  First from the projects themselves, and then how do we help them market themselves to get more volunteers?  Of course, having a good landing page on the wiki helps!

 

Also, Hyperledger has been starting up a bunch of SIGs.  I believe some people are moving to those SIGs.  And Ecosystem team is actively recruiting new members to participate there first.  Should we look at how we cross "advertise" internally?  Or look at how we "advertise" to new members and externally.

 

What other paths to recruitment should we explore? 

 

Silona

 

On Mon, Mar 18, 2019 at 4:15 PM Christopher Ferris <chris.ferris@...> wrote:

Todd, the Fabric maintainers would welcome additional help in documenting Fabric or providing and curating samples. They don’t write themselves.

 

I tend to agree with Hart’s analysis, and while I have a regular call that conflicts, I join occasionally and am definitely interested in the subject matter.

 

I’d agree with Todd that working with a traditional benchmarking org would be good, as would input into Caliper and other performance testing efforts.

 

We aren’t yet at a point where we can effectively provide guidance on comparative performance characteristics, which is ultimately what people will be seeking.

Chris

On Mar 18, 2019, at 6:49 PM, Todd Little <todd.little@...> wrote:

Hi Hart,

I tend to agree, but I also think another factor is that as we've moved from initial versions of the various platforms to production ready platforms, people are probably getting tied up working on those production deployments.  I know for me it's mostly a matter of I participate when I don't have conflicts or not traveling.

Personally I'd like to see us move in the direction of helping some standards organizations define blockchain benchmarks that can be used to evaluate blockchain platforms, as I'm not sure we're in the business of setting those kinds of standards.  Someone please correct me if I'm wrong.  I may have asked this before, but has anyone reached out to TPC to see if they're interested in creating some blockchain benchmarks?

With respect to the documentation, agreed it's fairly weak, and in particular design documentation is sorely lacking, at least for Fabric.  I can't speak to the other platforms.  Samples/examples is another area that is pretty weak.  Our platforms are so feature rich that it's hard to pick them up without a set of rich samples/examples.

-tl

PS FWIW I'll be on the call tomorrow.

On 3/18/2019 5:28 PM, Montgomery, Hart wrote:

Hi Mark,

 

I think this is probably true for most working groups.  At least from what I’ve seen and heard (and to be fair, this is not a huge sample size), fewer people are participating in the working groups, and those who are generally are participating less.

 

I’m not sure this is a bad thing, in many cases.  I think some people have replaced spending time on working groups with time spent on regular projects.  Personally, I’ve spent less time on working group stuff since Ursa has begun (and, obviously, more time on Ursa), which is probably a kind of progression we want to encourage in Hyperledger.  Having people that come into the working groups looking to learn become regular contributors surely is a positive thing. 

 

On the other hand, as the working groups seem to focus more and more on work products—which is typically documentation--fewer people seem to be interested since there is often substantial outside work involved, and not necessarily of the fun kind.  In particular, I think every SWOT analysis we have done on Hyperledger involves documentation being listed as a weakness.   While documentation has gotten better, it’s still something that I think most people would agree needs a lot of work on most aspects of Hyperledger.

 

So, for a TL;DR:  from my viewpoint, people are leaving working groups to contribute (great!)  or avoid documentation work (not so great). 

 

Do these experiences jibe with yours (or others’)?

 

Sorry for another long email.

 

Thanks,

Hart

 

 

 

From: tsc@... [mailto:tsc@...] On Behalf Of mark wagner
Sent: Monday, March 18, 2019 12:33 PM
To: perf-and-scale-wg@...
Cc: Hyperledger List <tsc@...>
Subject: [Hyperledger TSC] some thoughts on the future of the PSWG

 

Hi

 

I would like to carry this discussion mostly in the mailing lists in order to be as inclusive as possible.

 

Over the last few months attendance and participation on the Performance and Scale WG (PSWG) calls has been declining. I am wondering if we need to shift focus a bit to get more people involved and make progress.

 

There are many possible directions we can head in and I have included some thoughts here, but if others have additional ideas please feel encouraged to add them.

 

First is stay the course. We are currently struggling to get our thoughts on describing metrics for provenance captured to paper.

 

Do we want to focus more on actual performance work vs just defining metrics / use case papers. This does not need to be "just running tests".

 

We can do things like examine trade offs between different design decisions, examine different crypto solutions / technologies from a perf and scale perspective, help understand cost performance tradeoffs.

 

Perhaps we help spin up testnets and drive testing of different DLT frameworks ?

 

Do we provide a performance analysis service to the different projects that is based on architecture decisions. Perhaps something similar to what several research students have done (Harish and his team, FastFabric, etc).

 

I am trying to cut a wide swath here to avoid limiting peoples thoughts, so any and all comments are encouraged. If you have not been involved in the PSWG and would be interested in participating if we did "X" then speak up!

 

thanks

 

-mark

--

Mark Wagner

Chair, Performance and Scale Working Group

 

--

Silona Bonewald

VP of Community Architecture, Hyperledger

Mobile/Text: 512.750.9220
https://calendly.com/silona

The Linux Foundation
http://hyperledger.org

Hello Mark and others,

We have been having similar issues with the Architecture and Identity Working Groups.

You do not join a WG to mature into a developer for a particular solution. The work in a WG is complementary to the work in a project. You could do both. The WGs are not just about "documentation".

I have been thinking about the function of Working Groups. In my view it is the following.

The Working Groups were meant to be technical and focused on cross-cutting or primary concerns; like Architecture, Identity, Performance & Scale and now Smart Contracts.

Building stuff does require thinking about what to build. The WGs were meant to be a place where technical concerns from the various projects are aired out and where the collective wisdom of the WG participants could inform us all and prevent the isolation of frameworks from each other. In other words what makes Hyperledger, Hyperledger (beyond a single ledger); to connect projects together. This is evident in the papers published by the Architecture and PSWG groups.  

The Identity WG for example had people from Legal, Technical, Business as well as practitioners from  Identity specific areas all coming together to discuss topics like GDPR, Aadhar etc. and its relevance to the DLTs we are building. We brought Indy into the mix and are now working on L2 protocols in the Identity space. There is no comparable venue in HL for this. Somehow this message has been diluted and projects do not take this seriously enough nor is it effectively communicated.

The WGs were never highlighted in the wiki, and their benefits and contributions were seen as less than "landing patches".  

I see similarities in the way SIGs are forming now. In contrast, to the WGs the SIGs focus on verticals. WGs focus on horizontals. Both are needed. There should be collaboration between the SIGs and the WGs. 

Technical Working Group China and now one for India were formed because of differences in time zones and languages; and they are like all the WGs and SIGs rolled into one for the region. 

So now for some suggestions and ideas to make Working Groups more effective and encourage participation. This is based on conversations I have had with many people:

- It is important to do cross SIG/cross Regional/cross WG presentations

-Promotion in wiki, during bootcamps, messaging and Community Architect Support

-Experimentation with meeting times to encourage worldwide participation

-For PSWG, engagement with TPC, STAC, actual functional use case(Provenance) to look for numbers to measure, Look at ExactPro methodology etc.

-Engagement with Architecture for improvements in Performance like the waterloo project and implications on DLT architecture. What are the implications for other DLTs (other than Fabric)

Hope this long email may be of some use in rebooting the WGs and also rethinking of a more collaborative future. 

Best,

Vipin

On Tue, Mar 19, 2019 at 1:22 AM Kulkarni, Amol <amol.kulkarni@...> wrote:

One of the issues with the WGs as they’re currently set up is that interested folks from Asia cannot call into them.

 

The PSWG, for example, currently meets from 6:30 – 7:30 pm India time. While I’m working on performance and benchmarking specifically, my colleagues and I cannot call into any of these meetings. Perhaps rescheduling meetings to encourage participants from different geos might be one way to increase participation in WGs.

 

Amol

 

From: tsc@... [mailto:tsc@...] On Behalf Of Silona Bonewald
Sent: Tuesday, March 19, 2019 7:49 AM
To: Christopher Ferris <chris.ferris@...>
Cc: Todd Little <todd.little@...>; Montgomery, Hart <hmontgomery@...>; mark wagner <mwagner@...>; perf-and-scale-wg@...; Hyperledger List <tsc@...>
Subject: Re: [Hyperledger Performance and Scale WG] [Hyperledger TSC] some thoughts on the future of the PSWG

 

The Technical Working Group China participated in the HKBootcamp and was highly visible.  I think it is a good idea for other working groups to think about how to recruit and engage more with the community thru events like Bootcamps etc. 

 

For example, my team has been discussing with the Learning Materials Working Group about how they will recruit.  First from the projects themselves, and then how do we help them market themselves to get more volunteers?  Of course, having a good landing page on the wiki helps!

 

Also, Hyperledger has been starting up a bunch of SIGs.  I believe some people are moving to those SIGs.  And Ecosystem team is actively recruiting new members to participate there first.  Should we look at how we cross "advertise" internally?  Or look at how we "advertise" to new members and externally.

 

What other paths to recruitment should we explore? 

 

Silona

 

On Mon, Mar 18, 2019 at 4:15 PM Christopher Ferris <chris.ferris@...> wrote:

Todd, the Fabric maintainers would welcome additional help in documenting Fabric or providing and curating samples. They don’t write themselves.

 

I tend to agree with Hart’s analysis, and while I have a regular call that conflicts, I join occasionally and am definitely interested in the subject matter.

 

I’d agree with Todd that working with a traditional benchmarking org would be good, as would input into Caliper and other performance testing efforts.

 

We aren’t yet at a point where we can effectively provide guidance on comparative performance characteristics, which is ultimately what people will be seeking.

Chris

On Mar 18, 2019, at 6:49 PM, Todd Little <todd.little@...> wrote:

Hi Hart,

I tend to agree, but I also think another factor is that as we've moved from initial versions of the various platforms to production ready platforms, people are probably getting tied up working on those production deployments.  I know for me it's mostly a matter of I participate when I don't have conflicts or not traveling.

Personally I'd like to see us move in the direction of helping some standards organizations define blockchain benchmarks that can be used to evaluate blockchain platforms, as I'm not sure we're in the business of setting those kinds of standards.  Someone please correct me if I'm wrong.  I may have asked this before, but has anyone reached out to TPC to see if they're interested in creating some blockchain benchmarks?

With respect to the documentation, agreed it's fairly weak, and in particular design documentation is sorely lacking, at least for Fabric.  I can't speak to the other platforms.  Samples/examples is another area that is pretty weak.  Our platforms are so feature rich that it's hard to pick them up without a set of rich samples/examples.

-tl

PS FWIW I'll be on the call tomorrow.

On 3/18/2019 5:28 PM, Montgomery, Hart wrote:

Hi Mark,

 

I think this is probably true for most working groups.  At least from what I’ve seen and heard (and to be fair, this is not a huge sample size), fewer people are participating in the working groups, and those who are generally are participating less.

 

I’m not sure this is a bad thing, in many cases.  I think some people have replaced spending time on working groups with time spent on regular projects.  Personally, I’ve spent less time on working group stuff since Ursa has begun (and, obviously, more time on Ursa), which is probably a kind of progression we want to encourage in Hyperledger.  Having people that come into the working groups looking to learn become regular contributors surely is a positive thing. 

 

On the other hand, as the working groups seem to focus more and more on work products—which is typically documentation--fewer people seem to be interested since there is often substantial outside work involved, and not necessarily of the fun kind.  In particular, I think every SWOT analysis we have done on Hyperledger involves documentation being listed as a weakness.   While documentation has gotten better, it’s still something that I think most people would agree needs a lot of work on most aspects of Hyperledger.

 

So, for a TL;DR:  from my viewpoint, people are leaving working groups to contribute (great!)  or avoid documentation work (not so great). 

 

Do these experiences jibe with yours (or others’)?

 

Sorry for another long email.

 

Thanks,

Hart

 

 

 

From: tsc@... [mailto:tsc@...] On Behalf Of mark wagner
Sent: Monday, March 18, 2019 12:33 PM
To: perf-and-scale-wg@...
Cc: Hyperledger List <tsc@...>
Subject: [Hyperledger TSC] some thoughts on the future of the PSWG

 

Hi

 

I would like to carry this discussion mostly in the mailing lists in order to be as inclusive as possible.

 

Over the last few months attendance and participation on the Performance and Scale WG (PSWG) calls has been declining. I am wondering if we need to shift focus a bit to get more people involved and make progress.

 

There are many possible directions we can head in and I have included some thoughts here, but if others have additional ideas please feel encouraged to add them.

 

First is stay the course. We are currently struggling to get our thoughts on describing metrics for provenance captured to paper.

 

Do we want to focus more on actual performance work vs just defining metrics / use case papers. This does not need to be "just running tests".

 

We can do things like examine trade offs between different design decisions, examine different crypto solutions / technologies from a perf and scale perspective, help understand cost performance tradeoffs.

 

Perhaps we help spin up testnets and drive testing of different DLT frameworks ?

 

Do we provide a performance analysis service to the different projects that is based on architecture decisions. Perhaps something similar to what several research students have done (Harish and his team, FastFabric, etc).

 

I am trying to cut a wide swath here to avoid limiting peoples thoughts, so any and all comments are encouraged. If you have not been involved in the PSWG and would be interested in participating if we did "X" then speak up!

 

thanks

 

-mark

--

Mark Wagner

Chair, Performance and Scale Working Group

 

--

Silona Bonewald

VP of Community Architecture, Hyperledger

Mobile/Text: 512.750.9220
https://calendly.com/silona

The Linux Foundation
http://hyperledger.org

One of the issues with the WGs as they’re currently set up is that interested folks from Asia cannot call into them.

 

The PSWG, for example, currently meets from 6:30 – 7:30 pm India time. While I’m working on performance and benchmarking specifically, my colleagues and I cannot call into any of these meetings. Perhaps rescheduling meetings to encourage participants from different geos might be one way to increase participation in WGs.

 

Amol

 

From: tsc@... [mailto:tsc@...] On Behalf Of Silona Bonewald
Sent: Tuesday, March 19, 2019 7:49 AM
To: Christopher Ferris <chris.ferris@...>
Cc: Todd Little <todd.little@...>; Montgomery, Hart <hmontgomery@...>; mark wagner <mwagner@...>; perf-and-scale-wg@...; Hyperledger List <tsc@...>
Subject: Re: [Hyperledger Performance and Scale WG] [Hyperledger TSC] some thoughts on the future of the PSWG

 

The Technical Working Group China participated in the HKBootcamp and was highly visible.  I think it is a good idea for other working groups to think about how to recruit and engage more with the community thru events like Bootcamps etc. 

 

For example, my team has been discussing with the Learning Materials Working Group about how they will recruit.  First from the projects themselves, and then how do we help them market themselves to get more volunteers?  Of course, having a good landing page on the wiki helps!

 

Also, Hyperledger has been starting up a bunch of SIGs.  I believe some people are moving to those SIGs.  And Ecosystem team is actively recruiting new members to participate there first.  Should we look at how we cross "advertise" internally?  Or look at how we "advertise" to new members and externally.

 

What other paths to recruitment should we explore? 

 

Silona

 

On Mon, Mar 18, 2019 at 4:15 PM Christopher Ferris <chris.ferris@...> wrote:

Todd, the Fabric maintainers would welcome additional help in documenting Fabric or providing and curating samples. They don’t write themselves.

 

I tend to agree with Hart’s analysis, and while I have a regular call that conflicts, I join occasionally and am definitely interested in the subject matter.

 

I’d agree with Todd that working with a traditional benchmarking org would be good, as would input into Caliper and other performance testing efforts.

 

We aren’t yet at a point where we can effectively provide guidance on comparative performance characteristics, which is ultimately what people will be seeking.

Chris

On Mar 18, 2019, at 6:49 PM, Todd Little <todd.little@...> wrote:

Hi Hart,

I tend to agree, but I also think another factor is that as we've moved from initial versions of the various platforms to production ready platforms, people are probably getting tied up working on those production deployments.  I know for me it's mostly a matter of I participate when I don't have conflicts or not traveling.

Personally I'd like to see us move in the direction of helping some standards organizations define blockchain benchmarks that can be used to evaluate blockchain platforms, as I'm not sure we're in the business of setting those kinds of standards.  Someone please correct me if I'm wrong.  I may have asked this before, but has anyone reached out to TPC to see if they're interested in creating some blockchain benchmarks?

With respect to the documentation, agreed it's fairly weak, and in particular design documentation is sorely lacking, at least for Fabric.  I can't speak to the other platforms.  Samples/examples is another area that is pretty weak.  Our platforms are so feature rich that it's hard to pick them up without a set of rich samples/examples.

-tl

PS FWIW I'll be on the call tomorrow.

On 3/18/2019 5:28 PM, Montgomery, Hart wrote:

Hi Mark,

 

I think this is probably true for most working groups.  At least from what I’ve seen and heard (and to be fair, this is not a huge sample size), fewer people are participating in the working groups, and those who are generally are participating less.

 

I’m not sure this is a bad thing, in many cases.  I think some people have replaced spending time on working groups with time spent on regular projects.  Personally, I’ve spent less time on working group stuff since Ursa has begun (and, obviously, more time on Ursa), which is probably a kind of progression we want to encourage in Hyperledger.  Having people that come into the working groups looking to learn become regular contributors surely is a positive thing. 

 

On the other hand, as the working groups seem to focus more and more on work products—which is typically documentation--fewer people seem to be interested since there is often substantial outside work involved, and not necessarily of the fun kind.  In particular, I think every SWOT analysis we have done on Hyperledger involves documentation being listed as a weakness.   While documentation has gotten better, it’s still something that I think most people would agree needs a lot of work on most aspects of Hyperledger.

 

So, for a TL;DR:  from my viewpoint, people are leaving working groups to contribute (great!)  or avoid documentation work (not so great). 

 

Do these experiences jibe with yours (or others’)?

 

Sorry for another long email.

 

Thanks,

Hart

 

 

 

From: tsc@... [mailto:tsc@...] On Behalf Of mark wagner
Sent: Monday, March 18, 2019 12:33 PM
To: perf-and-scale-wg@...
Cc: Hyperledger List <tsc@...>
Subject: [Hyperledger TSC] some thoughts on the future of the PSWG

 

Hi

 

I would like to carry this discussion mostly in the mailing lists in order to be as inclusive as possible.

 

Over the last few months attendance and participation on the Performance and Scale WG (PSWG) calls has been declining. I am wondering if we need to shift focus a bit to get more people involved and make progress.

 

There are many possible directions we can head in and I have included some thoughts here, but if others have additional ideas please feel encouraged to add them.

 

First is stay the course. We are currently struggling to get our thoughts on describing metrics for provenance captured to paper.

 

Do we want to focus more on actual performance work vs just defining metrics / use case papers. This does not need to be "just running tests".

 

We can do things like examine trade offs between different design decisions, examine different crypto solutions / technologies from a perf and scale perspective, help understand cost performance tradeoffs.

 

Perhaps we help spin up testnets and drive testing of different DLT frameworks ?

 

Do we provide a performance analysis service to the different projects that is based on architecture decisions. Perhaps something similar to what several research students have done (Harish and his team, FastFabric, etc).

 

I am trying to cut a wide swath here to avoid limiting peoples thoughts, so any and all comments are encouraged. If you have not been involved in the PSWG and would be interested in participating if we did "X" then speak up!

 

thanks

 

-mark

--

Mark Wagner

Chair, Performance and Scale Working Group

 

--

Silona Bonewald

VP of Community Architecture, Hyperledger

Mobile/Text: 512.750.9220
https://calendly.com/silona

The Linux Foundation
http://hyperledger.org

Todd, the Fabric maintainers would welcome additional help in documenting Fabric or providing and curating samples. They don’t write themselves.

I tend to agree with Hart’s analysis, and while I have a regular call that conflicts, I join occasionally and am definitely interested in the subject matter.

I’d agree with Todd that working with a traditional benchmarking org would be good, as would input into Caliper and other performance testing efforts.

We aren’t yet at a point where we can effectively provide guidance on comparative performance characteristics, which is ultimately what people will be seeking.

Chris

On Mar 18, 2019, at 6:49 PM, Todd Little <todd.little@...> wrote:

Hi Hart,

I tend to agree, but I also think another factor is that as we've moved from initial versions of the various platforms to production ready platforms, people are probably getting tied up working on those production deployments.  I know for me it's mostly a matter of I participate when I don't have conflicts or not traveling.

Personally I'd like to see us move in the direction of helping some standards organizations define blockchain benchmarks that can be used to evaluate blockchain platforms, as I'm not sure we're in the business of setting those kinds of standards.  Someone please correct me if I'm wrong.  I may have asked this before, but has anyone reached out to TPC to see if they're interested in creating some blockchain benchmarks?

With respect to the documentation, agreed it's fairly weak, and in particular design documentation is sorely lacking, at least for Fabric.  I can't speak to the other platforms.  Samples/examples is another area that is pretty weak.  Our platforms are so feature rich that it's hard to pick them up without a set of rich samples/examples.

-tl

PS FWIW I'll be on the call tomorrow.

On 3/18/2019 5:28 PM, Montgomery, Hart wrote:

Hi Mark,

 

I think this is probably true for most working groups.  At least from what I’ve seen and heard (and to be fair, this is not a huge sample size), fewer people are participating in the working groups, and those who are generally are participating less.

 

I’m not sure this is a bad thing, in many cases.  I think some people have replaced spending time on working groups with time spent on regular projects.  Personally, I’ve spent less time on working group stuff since Ursa has begun (and, obviously, more time on Ursa), which is probably a kind of progression we want to encourage in Hyperledger.  Having people that come into the working groups looking to learn become regular contributors surely is a positive thing. 

 

On the other hand, as the working groups seem to focus more and more on work products—which is typically documentation--fewer people seem to be interested since there is often substantial outside work involved, and not necessarily of the fun kind.  In particular, I think every SWOT analysis we have done on Hyperledger involves documentation being listed as a weakness.   While documentation has gotten better, it’s still something that I think most people would agree needs a lot of work on most aspects of Hyperledger.

 

So, for a TL;DR:  from my viewpoint, people are leaving working groups to contribute (great!)  or avoid documentation work (not so great). 

 

Do these experiences jibe with yours (or others’)?

 

Sorry for another long email.

 

Thanks,

Hart

 

 

 

From: tsc@... [mailto:tsc@...] On Behalf Of mark wagner
Sent: Monday, March 18, 2019 12:33 PM
To: perf-and-scale-wg@...
Cc: Hyperledger List <tsc@...>
Subject: [Hyperledger TSC] some thoughts on the future of the PSWG

 

Hi

 

I would like to carry this discussion mostly in the mailing lists in order to be as inclusive as possible.

 

Over the last few months attendance and participation on the Performance and Scale WG (PSWG) calls has been declining. I am wondering if we need to shift focus a bit to get more people involved and make progress.

 

There are many possible directions we can head in and I have included some thoughts here, but if others have additional ideas please feel encouraged to add them.

 

First is stay the course. We are currently struggling to get our thoughts on describing metrics for provenance captured to paper.

 

Do we want to focus more on actual performance work vs just defining metrics / use case papers. This does not need to be "just running tests".

 

We can do things like examine trade offs between different design decisions, examine different crypto solutions / technologies from a perf and scale perspective, help understand cost performance tradeoffs.

 

Perhaps we help spin up testnets and drive testing of different DLT frameworks ?

 

Do we provide a performance analysis service to the different projects that is based on architecture decisions. Perhaps something similar to what several research students have done (Harish and his team, FastFabric, etc).

 

I am trying to cut a wide swath here to avoid limiting peoples thoughts, so any and all comments are encouraged. If you have not been involved in the PSWG and would be interested in participating if we did "X" then speak up!

 

thanks

 

-mark

--

Mark Wagner

Chair, Performance and Scale Working Group