Hi,

The agenda for this week's call is online. There are several quarterly reports that are due and haven't been posted yet though.

https://wiki.hyperledger.org/display/TSC/2019+12+12+TSC+Agenda

Thanks.
--
Arnaud  Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM

Reminder: Hyperledger Quilt Quarterly Update Due #tsc-project-update

When: Thursday, 12 December 2019

View Event

Organizer: community-architects@...

Description: Please review the update at TSC Project Updates prior to the meeting and add your questions to the update.

Hi Everyone,

 

Sorry for the incoming long wall of text.  I wanted to comment on the incubation/active status debates so that hopefully we could clarify things, frustrate people less, and spend less time on what is essentially a rubber stamp in the future.  With that being said, here goes:

 

In light of all of the discussion we’ve had on the Besu proposal for active status, I wanted to dig down into the details of why people care so much about active status versus incubation.  We discussed this a little bit in the project lifecycle task force (https://wiki.hyperledger.org/display/HYP/Project+Incubation+Exit+Criteria), but in light of all of the discussion at the last TSC meeting, I thought it might be a good idea to bring it up again.

 

My fundamental question is the following:  why do we care so much (or at all) about active/incubation status, and do we even need a distinction? 

 

The two documents that are relevant to the discussion are:

https://wiki.hyperledger.org/display/HYP/Project+Lifecycle

https://wiki.hyperledger.org/display/HYP/Project+Incubation+Exit+Criteria

 

Let’s ignore for a moment that the exit criteria in these two documents are not exactly consistent (for instance, the project lifecycle document requires a “history of releases” like that of an active project, while the actual exit criteria document only would require a “developer preview”).  This is something that definitely needs to be cleaned up, but I’ll mostly go by the exit criteria document, since it is more recent.

 

The core overriding requirement for active status, as put by Chris first (I think), is that “the community and infrastructure necessary for a project to succeed are in place.”  This means that all people and things that a project will need to generate good quality, useful code are there.  With this in mind, let’s take a look at the core requirements from the exit criteria document:

 

1.       Legal (Apache 2.0, etc.)

2.       Community—the seemingly most discussed metric

3.       Sufficient test coverage

4.       Sufficient user documentation

5.       Alignment (i.e., the project still fits within Hyperledger)

6.       Infrastructure is set up

7.       CII Badge

 

We also require TSC approval for a project’s 1.0 release.  This hasn’t been migrated over to the official TSC documentation list (so, there’s even more inconsistency here in terms of what is written in the official TSC documentation).  But here (https://wiki.hyperledger.org/pages/viewpage.action?pageId=16321784 ) is the document and here is a summary:

 

1.       Project has reached active status

2.       Release features meet the project’s charter

3.       Satisfies “internal project requirements”—mostly things like appropriate test coverage, bug handling, and documentation

4.       Satisfies “technical criteria”—CII badge, security audit, crypto code auditing, license scan, etc.

 

Obviously there is a lot of overlap between these two. 

 

In particular, of the active status criteria, #3 (test coverage), #4 (sufficient user documentation), and #7 (CII badge) are explicitly covered in the 1.0 release criteria.  In addition, #6 (infrastructure) and #1 (legal) are mentioned in both, although the requirements for 1.0 release seem a little bit more stringent (i.e. more infrastructure is required).

 

Let’s go back to the beginning of the project lifecycle and look at what is required for incubation status:

 

1.       Have a clear description

2.       Have a well-defined scope

3.       Identify committed development resources

4.       Identify initial maintainers

5.       Be vendor neutral

 

We have been implicitly requiring incoming code to be correctly licensed by incubation (rather than active status), so one easy move I think we could do is to formalize this requirement.  It’s not mentioned in either the project lifecycle document or the project proposal document.

 

“Well-defined scope” corresponds roughly to “Alignment” in the incubation exit criteria, and we ask the “context” question in the project lifecycle document, so I’d assume that any project that fails the #2 requirement “Alignment” for active status will probably not have made it to incubation in the first place, barring some sort of pivot (but this would, of course, fail the #2 requirement for 1.0 release (release meet’s the project’s charter).

 

So if we assume metrics #1 (legal) and #5 (alignment) are already a given due to the project getting into incubation in the first place, then the only thing active/incubation is really giving us that is unique from both the project proposal requirements and the 1.0 release criteria is the community metric. 

 

Perhaps not coincidentally, we have talked about the community metric the most for active status discussion, and it seems to be the bottleneck in most past, current, and planned proposals (i.e., people are waiting on proposing due to the community metric).  This metric is also perhaps the most subjective.  On the most recent TSC call, it seemed like everyone had a different interpretation of the community metric. 

 

The exact community metric requirements are:

·         The project must have an active and diverse set of contributing members representing various constituencies

·         The project is not highly dependent on any single contributor (there are at least 3 legally independent committers and there is no single company or entity that is vital to the success of the project)

·         Release plans are developed and executed in public by the community.

 

Out of these, the first two are very open to interpretation.  If one is so inclined, it’s easy to make the case that many (or, if you want to be argumentative, all) of the Hyperledger projects are highly dependent on a “single entity.”  Regardless of whether you agree with what I’m about to propose, I think it’s a relatively universal good idea that if we do keep the incubation exit criteria in its current form, we should provide more quantification here so that projects that want to apply for active status have a better idea of what they need to attain.

 

I’ve mostly stuck with established facts so far.  Now I’m going to veer off into what might be a controversial opinion.  I’m of the opinion that the only real criterion (and purpose) of the active status/incubation exit criteria checkpoint is the community status evaluation—everything else either follows from the project being approved for incubation in the first place or is present in the 1.0 release criteria. 

 

What does the general community get from incubation/active status?  A binary value that says whether the TSC thought the community support of a certain project was “good” or “not good” on some particular day.  In the grand scheme of things, this is not particularly useful information, since even the TSC can’t exactly agree on what “good” community support is.

 

On the other hand, there are several notable tools that would provide community metrics that could be very useful for outsiders.  Bitergia (https://blog.bitergia.com/) is an example that comes to mind.  If we had strong public metrics on code contributions and community support (distributions on contribution by company, contributor, geography, etc.) this active/incubation bit wouldn’t really be needed:  rather than have the TSC assess the status of a community, why not give people all of the appropriate information and let them judge for themselves?

 

We can argue that something like security or cryptography requires expert review:  it’s very hard for someone who isn’t an expert in these areas to decide whether or not something is secure, even given time to examine it.  But community support is not exactly something that requires an expert review—any average person with some knowledge of software development can make a pretty reasonable determination of community support if they are given enough facts.  Since this is something that is very subjective but also doesn’t require expert knowledge to analyze, it might be best to leave the TSC out of this.

 

So, while I don’t think this is feasible now since we don’t have the software in place to property handle contributor analytics, I think it might be worth considering in the long run to do the following:

 

1.       Move all of the requirements of the incubation exit criteria to either the 1.0 release or the project proposal.

2.       Get rid of the active/incubation status altogether.

3.       Collect and publish a large and useful amount of contributor statistics so outsiders can determine the level of community support for a project themselves.

 

What do people think about this?  Does this make sense?

 

I’d also like to point out that we currently require projects to achieve active status before releasing a 1.0 version.  If we keep active status as a milestone, I’d like to push back on this requirement.  At least in my opinion, we approved this assuming the bar for active status was much lower than it currently is today.  The bar for active status seems to be very high.  Something like Explorer may never get enough contributors to have active status under the current metrics—and this is OK, because it may not need a lot of contributors to build successful software--but that doesn’t mean it shouldn’t be able to release a 1.0 if the software is excellent.

 

Finally, regardless of what you think about this idea, we do need to clean up our documentation on this.  It’s not clear at all and is in some cases contradictory.  Putting all of the useful documentation in one place and adding some examples of when (and when not) projects are ready for milestones would be quite useful for projects and hopefully end a lot of the frustration involved in the project lifecycle process for both the projects and the TSC.

 

Please let me know what you all think about this (positive and negative).  If this gets a lot of responses, I’ll move it to a wiki so it can be further taken apart.

 

If you’ve made it this far, bless your heart and thanks for reading.

 

Thanks,

Hart

 

I think the incubation / active status serves an important function. It demands of a project that it spends effort to get a diverse contributor/maintainer base and learns how to operate in a community.

That kind of diversity is important for a healthy and reliable project and inhibits single vendor control.

 

The requirement that a project fulfill this milestone before it’s first major release is directly in line with this. We do not want to advertise a project is production ready if it relies on an immature community under single vendor control.

 

My $0.02,

Dan

 

From: <tsc@...> on behalf of "hmontgomery@..." <hmontgomery@...>
Date: Wednesday, December 4, 2019 at 11:11 PM
To: "tsc@..." <tsc@...>
Subject: [Hyperledger TSC] Incubation/Active Status

 

Hi Everyone,

 

Sorry for the incoming long wall of text.  I wanted to comment on the incubation/active status debates so that hopefully we could clarify things, frustrate people less, and spend less time on what is essentially a rubber stamp in the future.  With that being said, here goes:

 

In light of all of the discussion we’ve had on the Besu proposal for active status, I wanted to dig down into the details of why people care so much about active status versus incubation.  We discussed this a little bit in the project lifecycle task force (https://wiki.hyperledger.org/display/HYP/Project+Incubation+Exit+Criteria), but in light of all of the discussion at the last TSC meeting, I thought it might be a good idea to bring it up again.

 

My fundamental question is the following:  why do we care so much (or at all) about active/incubation status, and do we even need a distinction? 

 

The two documents that are relevant to the discussion are:

https://wiki.hyperledger.org/display/HYP/Project+Lifecycle

https://wiki.hyperledger.org/display/HYP/Project+Incubation+Exit+Criteria

 

Let’s ignore for a moment that the exit criteria in these two documents are not exactly consistent (for instance, the project lifecycle document requires a “history of releases” like that of an active project, while the actual exit criteria document only would require a “developer preview”).  This is something that definitely needs to be cleaned up, but I’ll mostly go by the exit criteria document, since it is more recent.

 

The core overriding requirement for active status, as put by Chris first (I think), is that “the community and infrastructure necessary for a project to succeed are in place.”  This means that all people and things that a project will need to generate good quality, useful code are there.  With this in mind, let’s take a look at the core requirements from the exit criteria document:

 

1. Legal (Apache 2.0, etc.)
2. Community—the seemingly most discussed metric
3. Sufficient test coverage
4. Sufficient user documentation
5. Alignment (i.e., the project still fits within Hyperledger)
6. Infrastructure is set up
7. CII Badge

 

We also require TSC approval for a project’s 1.0 release.  This hasn’t been migrated over to the official TSC documentation list (so, there’s even more inconsistency here in terms of what is written in the official TSC documentation).  But here (https://wiki.hyperledger.org/pages/viewpage.action?pageId=16321784 ) is the document and here is a summary:

 

1. Project has reached active status
2. Release features meet the project’s charter
3. Satisfies “internal project requirements”—mostly things like appropriate test coverage, bug handling, and documentation
4. Satisfies “technical criteria”—CII badge, security audit, crypto code auditing, license scan, etc.

 

Obviously there is a lot of overlap between these two. 

 

In particular, of the active status criteria, #3 (test coverage), #4 (sufficient user documentation), and #7 (CII badge) are explicitly covered in the 1.0 release criteria.  In addition, #6 (infrastructure) and #1 (legal) are mentioned in both, although the requirements for 1.0 release seem a little bit more stringent (i.e. more infrastructure is required).

 

Let’s go back to the beginning of the project lifecycle and look at what is required for incubation status:

 

1. Have a clear description
2. Have a well-defined scope
3. Identify committed development resources
4. Identify initial maintainers
5. Be vendor neutral

 

We have been implicitly requiring incoming code to be correctly licensed by incubation (rather than active status), so one easy move I think we could do is to formalize this requirement.  It’s not mentioned in either the project lifecycle document or the project proposal document.

 

“Well-defined scope” corresponds roughly to “Alignment” in the incubation exit criteria, and we ask the “context” question in the project lifecycle document, so I’d assume that any project that fails the #2 requirement “Alignment” for active status will probably not have made it to incubation in the first place, barring some sort of pivot (but this would, of course, fail the #2 requirement for 1.0 release (release meet’s the project’s charter).

 

So if we assume metrics #1 (legal) and #5 (alignment) are already a given due to the project getting into incubation in the first place, then the only thing active/incubation is really giving us that is unique from both the project proposal requirements and the 1.0 release criteria is the community metric. 

 

Perhaps not coincidentally, we have talked about the community metric the most for active status discussion, and it seems to be the bottleneck in most past, current, and planned proposals (i.e., people are waiting on proposing due to the community metric).  This metric is also perhaps the most subjective.  On the most recent TSC call, it seemed like everyone had a different interpretation of the community metric. 

 

The exact community metric requirements are:

• The project must have an active and diverse set of contributing members representing various constituencies
• The project is not highly dependent on any single contributor (there are at least 3 legally independent committers and there is no single company or entity that is vital to the success of the project)
• Release plans are developed and executed in public by the community.

 

Out of these, the first two are very open to interpretation.  If one is so inclined, it’s easy to make the case that many (or, if you want to be argumentative, all) of the Hyperledger projects are highly dependent on a “single entity.”  Regardless of whether you agree with what I’m about to propose, I think it’s a relatively universal good idea that if we do keep the incubation exit criteria in its current form, we should provide more quantification here so that projects that want to apply for active status have a better idea of what they need to attain.

 

I’ve mostly stuck with established facts so far.  Now I’m going to veer off into what might be a controversial opinion.  I’m of the opinion that the only real criterion (and purpose) of the active status/incubation exit criteria checkpoint is the community status evaluation—everything else either follows from the project being approved for incubation in the first place or is present in the 1.0 release criteria. 

 

What does the general community get from incubation/active status?  A binary value that says whether the TSC thought the community support of a certain project was “good” or “not good” on some particular day.  In the grand scheme of things, this is not particularly useful information, since even the TSC can’t exactly agree on what “good” community support is.

 

On the other hand, there are several notable tools that would provide community metrics that could be very useful for outsiders.  Bitergia (https://blog.bitergia.com/) is an example that comes to mind.  If we had strong public metrics on code contributions and community support (distributions on contribution by company, contributor, geography, etc.) this active/incubation bit wouldn’t really be needed:  rather than have the TSC assess the status of a community, why not give people all of the appropriate information and let them judge for themselves?

 

We can argue that something like security or cryptography requires expert review:  it’s very hard for someone who isn’t an expert in these areas to decide whether or not something is secure, even given time to examine it.  But community support is not exactly something that requires an expert review—any average person with some knowledge of software development can make a pretty reasonable determination of community support if they are given enough facts.  Since this is something that is very subjective but also doesn’t require expert knowledge to analyze, it might be best to leave the TSC out of this.

 

So, while I don’t think this is feasible now since we don’t have the software in place to property handle contributor analytics, I think it might be worth considering in the long run to do the following:

 

1. Move all of the requirements of the incubation exit criteria to either the 1.0 release or the project proposal.
2. Get rid of the active/incubation status altogether.
3. Collect and publish a large and useful amount of contributor statistics so outsiders can determine the level of community support for a project themselves.

 

What do people think about this?  Does this make sense?

 

I’d also like to point out that we currently require projects to achieve active status before releasing a 1.0 version.  If we keep active status as a milestone, I’d like to push back on this requirement.  At least in my opinion, we approved this assuming the bar for active status was much lower than it currently is today.  The bar for active status seems to be very high.  Something like Explorer may never get enough contributors to have active status under the current metrics—and this is OK, because it may not need a lot of contributors to build successful software--but that doesn’t mean it shouldn’t be able to release a 1.0 if the software is excellent.

 

Finally, regardless of what you think about this idea, we do need to clean up our documentation on this.  It’s not clear at all and is in some cases contradictory.  Putting all of the useful documentation in one place and adding some examples of when (and when not) projects are ready for milestones would be quite useful for projects and hopefully end a lot of the frustration involved in the project lifecycle process for both the projects and the TSC.

 

Please let me know what you all think about this (positive and negative).  If this gets a lot of responses, I’ll move it to a wiki so it can be further taken apart.

 

If you’ve made it this far, bless your heart and thanks for reading.

 

Thanks,

Hart