Sorry for the incoming long wall of text. I wanted to comment on the incubation/active status debates so that hopefully we could clarify things, frustrate people less, and spend less time on what is essentially a rubber stamp in the future.
With that being said, here goes:
In light of all of the discussion we’ve had on the Besu proposal for active status, I wanted to dig down into the details of why people care so much about active status versus incubation. We discussed this a little bit in the project lifecycle
task force (https://wiki.hyperledger.org/display/HYP/Project+Incubation+Exit+Criteria), but in light of all of the discussion at the last TSC meeting, I thought it might
be a good idea to bring it up again.
My fundamental question is the following: why do we care so much (or at all) about active/incubation status, and do we even need a distinction?
The two documents that are relevant to the discussion are:
Let’s ignore for a moment that the exit criteria in these two documents are not exactly consistent (for instance, the project lifecycle document requires a “history of releases” like that of an active project, while the actual exit criteria
document only would require a “developer preview”). This is something that definitely needs to be cleaned up, but I’ll mostly go by the exit criteria document, since it is more recent.
The core overriding requirement for active status, as put by Chris first (I think), is that “the community and infrastructure necessary for a project to succeed are in place.” This means that all people and things that a project will need
to generate good quality, useful code are there. With this in mind, let’s take a look at the core requirements from the exit criteria document:
Legal (Apache 2.0, etc.)
Community—the seemingly most discussed metric
Sufficient test coverage
Sufficient user documentation
Alignment (i.e., the project still fits within Hyperledger)
Infrastructure is set up
We also require TSC approval for a project’s 1.0 release. This hasn’t been migrated over to the official TSC documentation list (so, there’s even more inconsistency here in terms of what is written in the official TSC documentation).
But here (https://wiki.hyperledger.org/pages/viewpage.action?pageId=16321784 ) is the document and here is a summary:
Project has reached active status
Release features meet the project’s charter
Satisfies “internal project requirements”—mostly things like appropriate test coverage, bug handling, and documentation
Satisfies “technical criteria”—CII badge, security audit, crypto code auditing, license scan, etc.
Obviously there is a lot of overlap between these two.
In particular, of the active status criteria, #3 (test coverage), #4 (sufficient user documentation), and #7 (CII badge) are explicitly covered in the 1.0 release criteria. In addition, #6 (infrastructure) and #1 (legal) are mentioned
in both, although the requirements for 1.0 release seem a little bit more stringent (i.e. more infrastructure is required).
Let’s go back to the beginning of the project lifecycle and look at what is required for incubation status:
Have a clear description
Have a well-defined scope
Identify committed development resources
Identify initial maintainers
Be vendor neutral
We have been implicitly requiring incoming code to be correctly licensed by incubation (rather than active status), so one easy move I think we could do is to formalize this requirement. It’s not mentioned in either the project lifecycle
document or the project proposal document.
“Well-defined scope” corresponds roughly to “Alignment” in the incubation exit criteria, and we ask the “context” question in the project lifecycle document, so I’d assume that any project that fails the #2 requirement “Alignment” for active
status will probably not have made it to incubation in the first place, barring some sort of pivot (but this would, of course, fail the #2 requirement for 1.0 release (release meet’s the project’s charter).
So if we assume metrics #1 (legal) and #5 (alignment) are already a given due to the project getting into incubation in the first place, then the only thing active/incubation is really giving us that is unique from both the project proposal
requirements and the 1.0 release criteria is the community metric.
Perhaps not coincidentally, we have talked about the community metric the most for active status discussion, and it seems to be the bottleneck in most past, current, and planned proposals (i.e., people are waiting on proposing due to the
community metric). This metric is also perhaps the most subjective. On the most recent TSC call, it seemed like everyone had a different interpretation of the community metric.
The exact community metric requirements are:
The project must have an active and diverse set of contributing members representing various constituencies
The project is not highly dependent on any single contributor (there are at least 3 legally independent committers and there is no single company or entity that is vital to the success of the project)
Release plans are developed and executed in public by the community.
Out of these, the first two are very open to interpretation. If one is so inclined, it’s easy to make the case that many (or, if you want to be argumentative, all) of the Hyperledger projects are highly dependent on a “single entity.”
Regardless of whether you agree with what I’m about to propose, I think it’s a relatively universal good idea that if we do keep the incubation exit criteria in its current form, we should provide more quantification here so that projects that want to apply
for active status have a better idea of what they need to attain.
I’ve mostly stuck with established facts so far. Now I’m going to veer off into what might be a controversial opinion. I’m of the opinion that the only real criterion (and purpose) of the active status/incubation exit criteria checkpoint
is the community status evaluation—everything else either follows from the project being approved for incubation in the first place or is present in the 1.0 release criteria.
What does the general community get from incubation/active status? A binary value that says whether the TSC thought the community support of a certain project was “good” or “not good” on some particular day. In the grand scheme of things,
this is not particularly useful information, since even the TSC can’t exactly agree on what “good” community support is.
On the other hand, there are several notable tools that would provide community metrics that could be very useful for outsiders. Bitergia (https://blog.bitergia.com/) is an example that comes to
mind. If we had strong public metrics on code contributions and community support (distributions on contribution by company, contributor, geography, etc.) this active/incubation bit wouldn’t really be needed: rather than have the TSC assess the status of
a community, why not give people all of the appropriate information and let them judge for themselves?
We can argue that something like security or cryptography requires expert review: it’s very hard for someone who isn’t an expert in these areas to decide whether or not something is secure, even given time to examine it. But community
support is not exactly something that requires an expert review—any average person with some knowledge of software development can make a pretty reasonable determination of community support if they are given enough facts. Since this is something that is
very subjective but also doesn’t require expert knowledge to analyze, it might be best to leave the TSC out of this.
So, while I don’t think this is feasible now since we don’t have the software in place to property handle contributor analytics, I think it might be worth considering in the long run to do the following:
Move all of the requirements of the incubation exit criteria to either the 1.0 release or the project proposal.
Get rid of the active/incubation status altogether.
Collect and publish a large and useful amount of contributor statistics so outsiders can determine the level of community support for a project themselves.
What do people think about this? Does this make sense?
I’d also like to point out that we currently require projects to achieve active status before releasing a 1.0 version. If we keep active status as a milestone, I’d like to push back on this requirement. At least in my opinion, we approved
this assuming the bar for active status was much lower than it currently is today. The bar for active status seems to be very high. Something like Explorer may never get enough contributors to have active status under the current metrics—and this is OK,
because it may not need a lot of contributors to build successful software--but that doesn’t mean it shouldn’t be able to release a 1.0 if the software is excellent.
Finally, regardless of what you think about this idea, we do need to clean up our documentation on this. It’s not clear at all and is in some cases contradictory. Putting all of the useful documentation in one place and adding some examples
of when (and when not) projects are ready for milestones would be quite useful for projects and hopefully end a lot of the frustration involved in the project lifecycle process for both the projects and the TSC.
Please let me know what you all think about this (positive and negative). If this gets a lot of responses, I’ll move it to a wiki so it can be further taken apart.
If you’ve made it this far, bless your heart and thanks for reading.