Hi all,

It is clear that there are many sub-projects already in an informal sense. I agree with Shawn that these have to be managed and governed by the maintainers of the enveloping projects. Maybe they can be mentioned in the project report if they become important enough.

Another one is sub-projects masquerading as projects. They have to be identified and moved back as sub-projects (or not as is suggested by Mic's comment ).  This depends on the marginal cost of projects (see below). 

Is there any quantification of the resources needed to add a project? All I hear about is budget and resource limitations, no hard or even ballpark numbers. 

Silas listed some; let us look at them.

- TSC time - can be mitigated by the WGs, pre-reading and socializing on the mailing list

- Developer time - Presumably the proposers will address this or even attract new ones 

- Community support - WGs, plus SIGs etc. Also events.

- Political will - Is there a cost associated with this?

- Brand value/Attention - A few marketing materials, stickers, press releases

- Consumables (hosting (in the wiki?, github? others?), CI/CD, stickers, etc) - What is the marginal cost?

 

There may be others expenses we are not even considering. Let us put something together.

If possible there should be more top level projects, especially if the marginal cost of adding them is low. Of course they have to make sense, maybe incubate in labs to float trial balloons . We should not be refusing the emergence of an important technology since we cannot predict winners, especially when we evaluate them in the young stage (otherwise we would all have been very rich, mining Bitcoin in 2009 (and Hodling) or buying Amazon in 1998).

 

Arnaud and I discussed the fact that we should have a budget projection section in the project proposal. Of course this needs input from LF staff.

In terms of cross project conversations and bridges, the natural place for them are the technical working groups, at least to spark the interaction. Have been trying to do this in all the WGs that I am involved in. However, we have been noticing lesser participation which has to be addressed from a community and other levels. Even today, the landing page does not include references to the Technical WGs. 

 

Best,

Vipin

On Fri, May 24, 2019 at 1:38 PM Mic Bowman <cmickeyb@...> wrote:

> Frankly, I think that before we add in any additional top level framework 

> projects, we should have a good long think about the implications.  

I completely agree though I would drop "framework" and simply say that we need to be more aware of the implications.

(And the rest of my comments assume that there are already informal "subprojects" like the ones Chris and Shawn talk about.)

What constrains the number of top level projects that we approve? Or to put it more bluntly... why would I NOT vote for a project that might "feel" like a sub-project (assuming that it was well written & within scope)? What are the constrained resources? 

I think the list Silas put together is very good. Given that list... how would we recognize project "saturation" (in terms of "we will run out of some form of resource if we accept this project")? And if we cannot recognize that, then what drawback is there to continuing to add more and more projects? 

I'm not opposed to some formal notion of "sub-project", but with or without sub-projects there needs to be clearer definition/understanding of the back-pressure on the process of accepting projects whether they are top level or not.

--mic

On Thu, May 23, 2019 at 2:58 AM Christopher Ferris <chrisfer@...> wrote:

This is a good discussion, and I am sorry I will miss the call.

 

I think that sub-projects is an interesting thought, but we kind of have that now, do we not?

I agree that there isn't necessarily consistency, but that is due to the fact that we allowed projects

to expand their repo footprint to support not only new features but refactoring, etc.

 

I do agree that it would be useful to have a more formal relationship between (sub)projects such as SDKs and the related top-level project(s) such that there can be a more formal path to integration into the formal release process. Of course, as each top-level project has its own governance, it is less clear how this would play out.

 

Frankly, I think that before we add in any additional top level framework projects, we should have a good long think about the implications.

 

As to Hart's last point, I think that the TSC and community architects should invest more time and energy in working on how we can encourage more organic cross-project interaction/integration/etc and that discussion should include the maintainers/leads of the various top level projects as they are most certainly invested in any outcome and have important insights that we should take under advisement.

 

Cheers,

Christopher Ferris
IBM Fellow, CTO Open Technology
IBM Cognitive Applications
email: chrisfer@...
twitter: @christo4ferris

blog: https://developer.ibm.com/code/author/chrisfer/

IBM Open Source white paper: https://developer.ibm.com/articles/cl-open-architecture-update/
phone: +1 508 667 0402

 

 

----- Original message -----
From: "Silas Davis via Lists.Hyperledger.Org" <silas=monax.io@...>
Sent by: tsc@...
To: "hmontgomery@..." <hmontgomery@...>
Cc: tsc@...
Subject: [EXTERNAL] Re: [Hyperledger TSC] Projects and Subprojects
Date: Thu, May 23, 2019 4:42 AM
 

Thanks for putting this together Hart, the history is useful. Sorry for taking nearly 3 weeks to respond to it.

 

It seems like there are a few different resources that people are concerned with when thinking about projects, some of which are necessarily scarce and some of which are artificially scarce:

 

- TSC time

- Developer time

- Community support

- Political will

- Brand value/Attention

- Consumables (hosting, CI/CD, stickers, etc)
 

To scale the number of projects we need away to allocate these resources fairly and to maximise the resources a project brings with it (in terms of developer and community effort).
 

Looking at Brian's and other's previous statements it seems the desire for a flat project structure is motivated by giving all projects formal structures in terms of wiki, roadmap, maintainers and also opening the possibility of cross-project work. In practice it seems for the latter often amounts to wishful thinking in the absence of clear plans and motivation to do so.

 

It seems to me (though I haven't been around here long) that often new projects tend to have their origins/support from people in an existing top-level project (I think this has to do with the political will resource...) in which case it might be natural for a new project to 'report to' or be associated with an established project for the purposes of quarterly reviews and elements of governance and perhaps sharing resources (consumables and developer time. This would have the effect of scaling the amount of projects we could support and would give some kind of responsibility to the 'mentoring' projects.

 

The project/sub-project relationship seems to equate technical subsumption/sub-ordinance with an organisational subdivision. Could these concerns be meaningfully separated? Though it occurs to me as I write this that I might just be describing incubation with a twist... Also if a project is completely technically independent it might not make sense to shoe-horn TSC updates and testing under the same umbrella. Where there is significant overlap however, as with Transact and Sawtooth (at the time of writing), from a practical point of view it would probably work well, it''s just that that project has scope that is different to, and goes beyond Sawtooth in general.
 

I think the other issue particularly for incumbent projects is what proliferation of top level projects does for the Hyperledger brand and the distribution of attention across projects. If there are projects that are less mature or lower quality at the top level what effect does that have on the perceived value of existing projects? What effect does that have on levels of contribution or use of projects in the wild?

 

Another issue I think I raised on a TSC call is we don't really have a way to prune top level projects, which would be another way to address proliferation. Currently it would seem rather mean and drastic to strip a project of top-level status, perhaps we could address that and encourage more top-level project 'startups' - maybe they return to incubation or get decommissioned after a certain period of time. Maybe that kind of churn would degrade trust in Hyperledger.

 

I have managed to write quite a few paragraphs there without providing any answers. So:

 

- I broadly think sub-projects are a good idea (whether by that name or not)

- If organising cacucusses of projects in a hierarchy for the purposes of reporting and governance (i.e. projects give TSC update together, projects share CI resources) makes sense in practice I would support it

 

Perhaps we don't have a problem acute enough to be worth solving today. But I'd be interested in people's thoughts on the negative effects of project proliferation in terms of the types of resources listed above.

 

Silas

 

On Thu, 2 May 2019 at 17:27, hmontgomery@... <hmontgomery@...> wrote:

Hi Everyone,

 

In light of today’s discussion at the TSC meeting, I wanted to write up some of my thoughts on the project approval process.  I apologize in advance for the wall of text, but I wanted to summarize a lot of things.  I’ll start with a bit of history (we have been around long enough for that to matter!) and then explain what I think we should do.

 

At some point two or (more like) three years ago, we had a number of discussions about new projects in Hyperledger.  We were beginning to get quite a lot of interest from people who wanted to have their own project.  At the time, the popular opinion was that we should be very inclusive as to what qualified for a project approval, and even things like SDKs for DLT platforms (i.e. a single language SDK for a single DLT platform) deserved their own projects.  For instance, here (https://lists.hyperledger.org/g/tsc/message/668?p=,,,20,0,0,0::relevance,,Fabric-Go-SDK,20,2,0,17551967) and here (https://lists.hyperledger.org/g/tsc/message/668?p=,,,20,0,0,0::relevance,,Fabric-Go-SDK,20,2,0,17551967 ) are archived emails from Brian explaining his thoughts on this.  I believe this was the most popular (or at least the most commonly expressed) opinion at the time.

 

However, there was some worry about too much project proliferation (which seems to be one of the main concerns today).  I recall proposing that we have a tiered project structure, with projects and sub-projects.  The goal of this was to allow for subproject independence (and to give subprojects all the independent tools they needed) without putting a huge burden on the TSC through too many projects.  I don’t know that I was the originator of this idea, but here is an email from me to TSC list at the time:  https://lists.hyperledger.org/g/tsc/message/681?p=,,,20,0,0,0::relevance,,RFC+1149,20,2,0,17551967.  This idea got shot down, though—people thought it was best to have a flat project hierarchy, in an approach more like Apache.

 

Of course, we all know what happened.  We approved a bunch of projects that really didn’t become independent at all.  Chris brought up Composer today in the TSC meeting as an example, but perhaps an even more illustrative example is the Fabric Java SDK.  From a purely rules-based perspective, this is still officially a top-level Hyperledger project!  You can see this email where we approved it:  https://lists.hyperledger.org/g/tsc/message/321?p=,,,20,0,0,0::relevance,,Fabric-Java-SDK,20,2,0,17551816.  It doesn’t function as a top-level project at all:  we have neglected to require updates from it at TSC meetings (or any of the other “new” requirements we have made of projects), and the governance, as far as I can tell, is done completely within Fabric.  But we also haven’t done anything official to take away its status as a fully independent, top-level Hyperledger project, so it is sitting in some kind of mixed state where it is officially an independent project but not one in practice at all.

 

And now we have come full circle.  Yet again, we are in a period where people have proposed a lot of new projects (which is definitely a good thing!) and others are worried about project proliferation.  Yet again, I’d like to bring up the topic of project tiers and sub-projects.  I am curious if opinions have changed over the past two or three years as Hyperledger has matured.

 

I would like to get people’s feedback on the following idea:  in the future, in addition to project proposals, we allow for sub-project proposals (and thus, sub-projects).  If you don’t like my naming conventions (looking at you Silona) please feel free to suggest something else.  We define a sub-project to be a project that is exclusively dependent on or forked from a single (higher-level) project. Sub-projects have all of the tools available to them that regular projects do, but could potentially have slightly different governance:  they could do some reporting to the higher-level project, rather than directly to the TSC (although the TSC can step in to manage disputes as necessary), which would decrease the burden on the TSC.  If a sub-project matures to the point where it works with/supports/is used for multiple higher tier projects, then it should be promoted to full project without extensive discussion.  We can also allow the TSC to (with fair warning) demote projects that only support one higher-tier project into sub-projects.

 

The upside of this is that Hyperledger could support more projects this way (even the Fabric-Java-SDK, for instance, could be a sub-project), giving people perhaps more incentive to take ownership and try building new things.  Outsiders could very clearly see the project structure and how things work as well, and we could market sub-projects effectively too if this was desired or needed.  The downside of the hierarchical project structure is that it may decrease cross-project interaction as people might see sub-projects as part of a different project and not something to interoperate with (this was my interpretation of the core of Brian’s argument two and a half years ago).  Over time, I think we have learned that cross-project interaction doesn’t really seem to happen organically or spontaneously—it takes a lot of effort and, at least for Hyperledger, seems to take some central planning by project maintainers.  So I’m not sure this argument turned out to be correct.  

 

What do people think about this?  Is this reasonable?  Should we just continue with the flat project hierarchy?  Is there some other way to gracefully handle project proliferation?  I am curious to hear what people think about this.  I’m substantially less experienced in open source projects than many of the other people here, some of whom have been working on open source for longer than I have been alive, so it’s entirely possible that I am just completely missing the boat, and I welcome critical feedback.

 

If you’ve made it this far, thank you for reading, and sorry again for the wall of text. 

 

Thanks,

Hart