toggle quoted messageShow quoted text
Chris is right that the more mature large projects already kind of have sub-projects. In Sawtooth, we are usually careful to call them components to avoid confusion. But basically, each repo within Sawtooth (and there are a lot) have an independent maintainers list and most of those repos are distinct components (some are example apps). While there is a lot of overlap in maintainers, they do differ between component. There are specific people within the project coordinating for release purposes. An RFC process is used to propose/approve/discuss new components on various metrics: is it aligned with the project's long-term vision, will it be maintainable as part of the project long-term, is it interesting enough, would it be better if it were in labs or a separate project, etc. All the components are expected to behave in a uniform way (such as the PR process, build processes, etc.) so it's easy (for maintainers) to go between working on component A and component B. They are also expected to make up the overall vision of the project. All of these expectations mean they aren't really a separate project. And, they are normally proposed by existing maintainers (though this need not be the case, it can help w/long-term support).
Deciding on whether to adopt a component can be very difficult as it has the potential to shift the core values of the project. For example, in Sawtooth we adopted both Raft and PBFT as components. Adopting Raft presented somewhat of a crisis in identity of the project, because part of Sawtooth's identity is a ledger that is truly distributed including solving BFT. Obviously, Raft fails that test. PBFT on the other hand, was easy, because it fit in with the vision. There are a lot of dimensions though - Raft we had a nice pre-existing Rust library, and PBFT we had to put in more coding effort. In the end, we accepted both Raft and PBFT primarily because it was consistent with another aspect of Sawtooth's identity which is modularity. I couldn't imagine delegating these decisions to the TSC because for any given project, a good portion of the TSC members are not deeply involved with that project day-to-day -- much better to let the project maintainers do it. (Would we have put a ton of effort into Sawtooth's iOS SDK if there was the possibility that the TSC would decide that Sawtooth shouldn't have an iOS component? Should Fabric folks be forced to accept new UI components that don't fit with the vision of the next major release? etc. etc.)
So, I guess I'm against sub-projects in any sort of formalization sense. Seems completely unnecessary and potentially actively harmful.
I do believe that more projects is better. But specifically, more projects geared toward reusable components like libraries. Because the those are the projects that will encourage cross-project code re-use and help unify Hyperledger as we move forward. I think the effort should be put into the taxonomy and classification of projects and to what extent HL focuses and markets them. Things like Ursa and Transact obviously need far less marketing than the "framework" projects.
Another thought would be to strengthen labs. Remove any reason not to use labs. And maybe even require new projects exist in labs for a short time before becoming a project.
On Thu, May 23, 2019 at 4:58 AM Christopher Ferris <chrisfer@...
This is a good discussion, and I am sorry I will miss the call.
I think that sub-projects is an interesting thought, but we kind of have that now, do we not?
I agree that there isn't necessarily consistency, but that is due to the fact that we allowed projects
to expand their repo footprint to support not only new features but refactoring, etc.
I do agree that it would be useful to have a more formal relationship between (sub)projects such as SDKs and the related top-level project(s) such that there can be a more formal path to integration into the formal release process. Of course, as each top-level project has its own governance, it is less clear how this would play out.
Frankly, I think that before we add in any additional top level framework projects, we should have a good long think about the implications.
As to Hart's last point, I think that the TSC and community architects should invest more time and energy in working on how we can encourage more organic cross-project interaction/integration/etc and that discussion should include the maintainers/leads of the various top level projects as they are most certainly invested in any outcome and have important insights that we should take under advisement.
IBM Fellow, CTO Open Technology
IBM Cognitive Applications
----- Original message -----
From: "Silas Davis via Lists.Hyperledger.Org" <silas=monax.io@...>
Sent by: tsc@...
To: "hmontgomery@..." <hmontgomery@...>
Subject: [EXTERNAL] Re: [Hyperledger TSC] Projects and Subprojects
Date: Thu, May 23, 2019 4:42 AM
Thanks for putting this together Hart, the history is useful. Sorry for taking nearly 3 weeks to respond to it.
It seems like there are a few different resources that people are concerned with when thinking about projects, some of which are necessarily scarce and some of which are artificially scarce:
- TSC time
- Developer time
- Community support
- Political will
- Brand value/Attention
- Consumables (hosting, CI/CD, stickers, etc)
To scale the number of projects we need away to allocate these resources fairly and to maximise the resources a project brings with it (in terms of developer and community effort).
Looking at Brian's and other's previous statements it seems the desire for a flat project structure is motivated by giving all projects formal structures in terms of wiki, roadmap, maintainers and also opening the possibility of cross-project work. In practice it seems for the latter often amounts to wishful thinking in the absence of clear plans and motivation to do so.
It seems to me (though I haven't been around here long) that often new projects tend to have their origins/support from people in an existing top-level project (I think this has to do with the political will resource...) in which case it might be natural for a new project to 'report to' or be associated with an established project for the purposes of quarterly reviews and elements of governance and perhaps sharing resources (consumables and developer time. This would have the effect of scaling the amount of projects we could support and would give some kind of responsibility to the 'mentoring' projects.
The project/sub-project relationship seems to equate technical subsumption/sub-ordinance with an organisational subdivision. Could these concerns be meaningfully separated? Though it occurs to me as I write this that I might just be describing incubation with a twist... Also if a project is completely technically independent it might not make sense to shoe-horn TSC updates and testing under the same umbrella. Where there is significant overlap however, as with Transact and Sawtooth (at the time of writing), from a practical point of view it would probably work well, it''s just that that project has scope that is different to, and goes beyond Sawtooth in general.
I think the other issue particularly for incumbent projects is what proliferation of top level projects does for the Hyperledger brand and the distribution of attention across projects. If there are projects that are less mature or lower quality at the top level what effect does that have on the perceived value of existing projects? What effect does that have on levels of contribution or use of projects in the wild?
Another issue I think I raised on a TSC call is we don't really have a way to prune top level projects, which would be another way to address proliferation. Currently it would seem rather mean and drastic to strip a project of top-level status, perhaps we could address that and encourage more top-level project 'startups' - maybe they return to incubation or get decommissioned after a certain period of time. Maybe that kind of churn would degrade trust in Hyperledger.
I have managed to write quite a few paragraphs there without providing any answers. So:
- I broadly think sub-projects are a good idea (whether by that name or not)
- If organising cacucusses of projects in a hierarchy for the purposes of reporting and governance (i.e. projects give TSC update together, projects share CI resources) makes sense in practice I would support it
Perhaps we don't have a problem acute enough to be worth solving today. But I'd be interested in people's thoughts on the negative effects of project proliferation in terms of the types of resources listed above.
In light of today’s discussion at the TSC meeting, I wanted to write up some of my thoughts on the project approval process. I apologize in advance for the wall of text, but I wanted to summarize a lot of things. I’ll start with a bit of history (we have been around long enough for that to matter!) and then explain what I think we should do.
At some point two or (more like) three years ago, we had a number of discussions about new projects in Hyperledger. We were beginning to get quite a lot of interest from people who wanted to have their own project. At the time, the popular opinion was that we should be very inclusive as to what qualified for a project approval, and even things like SDKs for DLT platforms (i.e. a single language SDK for a single DLT platform) deserved their own projects. For instance, here (https://lists.hyperledger.org/g/tsc/message/668?p=,,,20,0,0,0::relevance,,Fabric-Go-SDK,20,2,0,17551967) and here (https://lists.hyperledger.org/g/tsc/message/668?p=,,,20,0,0,0::relevance,,Fabric-Go-SDK,20,2,0,17551967 ) are archived emails from Brian explaining his thoughts on this. I believe this was the most popular (or at least the most commonly expressed) opinion at the time.
However, there was some worry about too much project proliferation (which seems to be one of the main concerns today). I recall proposing that we have a tiered project structure, with projects and sub-projects. The goal of this was to allow for subproject independence (and to give subprojects all the independent tools they needed) without putting a huge burden on the TSC through too many projects. I don’t know that I was the originator of this idea, but here is an email from me to TSC list at the time: https://lists.hyperledger.org/g/tsc/message/681?p=,,,20,0,0,0::relevance,,RFC+1149,20,2,0,17551967. This idea got shot down, though—people thought it was best to have a flat project hierarchy, in an approach more like Apache.
Of course, we all know what happened. We approved a bunch of projects that really didn’t become independent at all. Chris brought up Composer today in the TSC meeting as an example, but perhaps an even more illustrative example is the Fabric Java SDK. From a purely rules-based perspective, this is still officially a top-level Hyperledger project! You can see this email where we approved it: https://lists.hyperledger.org/g/tsc/message/321?p=,,,20,0,0,0::relevance,,Fabric-Java-SDK,20,2,0,17551816. It doesn’t function as a top-level project at all: we have neglected to require updates from it at TSC meetings (or any of the other “new” requirements we have made of projects), and the governance, as far as I can tell, is done completely within Fabric. But we also haven’t done anything official to take away its status as a fully independent, top-level Hyperledger project, so it is sitting in some kind of mixed state where it is officially an independent project but not one in practice at all.
And now we have come full circle. Yet again, we are in a period where people have proposed a lot of new projects (which is definitely a good thing!) and others are worried about project proliferation. Yet again, I’d like to bring up the topic of project tiers and sub-projects. I am curious if opinions have changed over the past two or three years as Hyperledger has matured.
I would like to get people’s feedback on the following idea: in the future, in addition to project proposals, we allow for sub-project proposals (and thus, sub-projects). If you don’t like my naming conventions (looking at you Silona) please feel free to suggest something else. We define a sub-project to be a project that is exclusively dependent on or forked from a single (higher-level) project. Sub-projects have all of the tools available to them that regular projects do, but could potentially have slightly different governance: they could do some reporting to the higher-level project, rather than directly to the TSC (although the TSC can step in to manage disputes as necessary), which would decrease the burden on the TSC. If a sub-project matures to the point where it works with/supports/is used for multiple higher tier projects, then it should be promoted to full project without extensive discussion. We can also allow the TSC to (with fair warning) demote projects that only support one higher-tier project into sub-projects.
The upside of this is that Hyperledger could support more projects this way (even the Fabric-Java-SDK, for instance, could be a sub-project), giving people perhaps more incentive to take ownership and try building new things. Outsiders could very clearly see the project structure and how things work as well, and we could market sub-projects effectively too if this was desired or needed. The downside of the hierarchical project structure is that it may decrease cross-project interaction as people might see sub-projects as part of a different project and not something to interoperate with (this was my interpretation of the core of Brian’s argument two and a half years ago). Over time, I think we have learned that cross-project interaction doesn’t really seem to happen organically or spontaneously—it takes a lot of effort and, at least for Hyperledger, seems to take some central planning by project maintainers. So I’m not sure this argument turned out to be correct.
What do people think about this? Is this reasonable? Should we just continue with the flat project hierarchy? Is there some other way to gracefully handle project proliferation? I am curious to hear what people think about this. I’m substantially less experienced in open source projects than many of the other people here, some of whom have been working on open source for longer than I have been alive, so it’s entirely possible that I am just completely missing the boat, and I welcome critical feedback.
If you’ve made it this far, thank you for reading, and sorry again for the wall of text.