Thanks for your response. I tend to agree completely that code review is not adequate alone. However, it is none-the-less important and for a critical project, ideally there are multiple sets of eyes on reviews and ideally, one cannot review and merge their own code (non-author code review or NACR). My preference is that projects should have a 2+2 review policy with the author precluded from approving their own contributions.

I also think that we should look at code and endpoint scanning utilities that we might leverage for formally released "GA" level code releases.

I also agree that it really doesn't belong as part of the "incubation" exit criteria as that is intended to be more about the processes of the project team than the code.

Cheers,

Chris

On Fri, Nov 11, 2016 at 7:47 PM, Hart Montgomery via hyperledger-tsc <hyperledger-tsc@...> wrote:

Hi Everyone,

 

I just wanted to clarify some of the points I brought up in the discussion during the TSC meeting yesterday about the incubation exit criteria and see what other people have to say about them.

 

The main point I wanted to make:  I am not advocating that a project have done an extensive security review prior to exit from incubation.  I think it’s important that some sort of plan is in place, although whether we want to formally include this in the exit criteria document is another matter entirely (since there are certainly some projects, i.e. Hyperledger Explorer, for which security will not be a key component). 

 

As Brian often likes to say, the exit criteria should be about having a healthy community working on a project.  I think that having some people who have at least thought about security is an important part of a healthy community for certainly any of the core blockchain projects.  Whether we want to address this in the exit criteria document or elsewhere is something that is open to debate, but I think it still should be a general consideration.

 

However, “code coverage” is not an effective way of measuring security.  Almost by definition, attacks on software end up hitting code in unintended ways that are not meant to be part of the functionality of the software, so it is really impossible to argue for security by saying that “code coverage” exists.  It’s impossible to run most complicated programs on all of their theoretically possible inputs—there are usually exponentially many of them—and attacks typically focus on inputs/program uses that are far from intended functionality, so most security analyses use some sort of formal verification, expert penetration testing, automated security testing tools, or some combination of these (someone whose day job is entirely security analysis can probably offer a better analysis than I can here). 

 

Thus, if we do include the additional requirements section of the exit criteria document (and I don’t think we necessarily need to include it), I think that we should revise the security requirements to not involve code coverage.  Some of the additional requirements are not really relevant to a lot of projects (again, the Hyperledger Explorer probably shouldn’t have to dictate a maximum block size requirement), so we do probably need to either drop or rewrite this section.  If I recall, Arnaud wrote this document (very nicely, I might add) before we really had adopted the concept of the umbrella organization, and this section doesn’t reflect the fact that projects might not be full blockchains.

 

TL;DR:

I agree with what I perceived to be the general consensus among the group that, while security is an extremely important facet of blockchain development, perhaps the exit criteria document is not the place to put in strict requirements.

 

Sorry for the long email, and thanks for your time.  If anyone has any thoughts or comments on this, I’d love to hear them.

 

Thanks,

Hart

 

 

_______________________________________________
hyperledger-tsc mailing list
hyperledger-tsc@...
https://lists.hyperledger.org/mailman/listinfo/hyperledger-tsc