Re: [Hyperledger Project TSC] Project Proposal: Consensus Platform


Hart Montgomery
 

Hi Brian,

 

Thanks for your reply.  I’ve got some questions and comments to your responses below.

 

That doesn't seem to map to what I've seen happen in Open Source projects that implement open protocols or crypto routines/suites.  You always have implementation bugs, and partial implementations of protocols, and unrealized optimizations, all of which drive ongoing development work.  And protocols that get actual widespread usage almost always see updates - HTTP/1.0 to HTTP/1.1 to 2.0, for instance.  TLS, too, for instance.  And despite the huge bugs found constantly in protocols, the standards get patched, then the code gets patched and iterated and updates made, sometimes all within hours if the vulnerability is critical enough.  In any event, even a project that is born, supported, but then discovered to be hopelessly broken and then is closed down, serves a purpose: either it can be a warning sign (with enough code to substantiate the warning) or it can be mulch for a non-broken idea that salvages the code and the community around it”

 

I’m curious.  Do you know of any widespread, successful open source crypto protocols or standards that aren’t based upon extremely heavy peer review?  TLS and all of the others I can think of that are used by a large number of people have been heavily reviewed.  They are generally administered by large, long-term standards bodies that have general goals—secure crypto for a set of problems XYZ—rather than specific algorithms in mind.  None of these standards bodies are wedded to any particular implementation—they are around for the long-term, and if someone breaks the crypto, then they update and change it, sometimes in big ways (the history of hash functions from the RCs to the SHAs is a good example of this, and updating some of these standards to quantum-resistant algorithms is an example of something happening now).  In particular, TLS has a huge number of different protocols inside of it and is really a “general purpose crypto for the internet” project rather than an implementation of any particular protocol.

 

Isn't Fabric already a "general pluggable consensus project"?  If not, could you really have one that wasn't better done as enhancements to turn Fabric into that (or Sawtooth/Iroha et al)?  Or is there something generic to DLTs, in the same way the GSL proposal was generic and potentially common to all DLTs, that could be implemented?”


I view a “general pluggable consensus project” as a project that implements a number of different consensus algorithms that are compatible with a number of different DLTs.  So no, at this point I wouldn’t consider Fabric a “general pluggable consensus project”. 

 

Consensus algorithms are notoriously tricky to implement, and I believe that much of the consensus code would be independent of the DLT, so I do think there is indeed something that is generic to all DLTs (I think the same way about crypto algorithms).  I think it would, in most cases, be far easier to write one consensus or crypto algorithm and port it to N different DLTs than to write N different crypto or consensus algorithms.  I don’t have as much experience writing consensus code as some other people on this list, but I can say for sure that I would rather implement, say, a signature scheme and port it to Sawtooth, Fabric, and Iroha rather than code three different signature protocols, one for each platform, securely.

 

Thanks for reading, and sorry for the pessimism.  I think it’s acquired somewhere in standard crypto education. Also, please let me know what you think!

 

Thanks,

Hart

 

From: Brian Behlendorf [mailto:bbehlendorf@...]
Sent: Wednesday, June 28, 2017 9:43 AM
To: Hart Montgomery <hmontgomery@...>
Cc: hyperledger-tsc <hyperledger-tsc@...>
Subject: Re: [Hyperledger Project TSC] Project Proposal: Consensus Platform

 

On 06/28/2017 09:31 AM, Hart Montgomery wrote:

The issue is the following:  projects that aim to implement a specific protocol are likely to be relatively front-loaded.  All of the effort happens in the beginning to get the protocol up and running, and then essentially it is just maintenance from there on out.  If something goes wrong and a large bug is found in the algorithm (or the algorithm is found to be fundamentally incorrect, or just useless in that it doesn’t have any advantages over other existing algorithms), the project is likely to be abandoned since it really wouldn’t be fixable, which may cause very bad developments for users or other projects that depend on it.


That doesn't seem to map to what I've seen happen in Open Source projects that implement open protocols or crypto routines/suites.  You always have implementation bugs, and partial implementations of protocols, and unrealized optimizations, all of which drive ongoing development work.  And protocols that get actual widespread usage almost always see updates - HTTP/1.0 to HTTP/1.1 to 2.0, for instance.  TLS, too, for instance.  And despite the huge bugs found constantly in protocols, the standards get patched, then the code gets patched and iterated and updates made, sometimes all within hours if the vulnerability is critical enough.  In any event, even a project that is born, supported, but then discovered to be hopelessly broken and then is closed down, serves a purpose: either it can be a warning sign (with enough code to substantiate the warning) or it can be mulch for a non-broken idea that salvages the code and the community around it. 
 

On the other hand, projects with a general approach are likely to be more permanent and less likely to be completely derailed by a huge bug in an algorithm.  These projects will either have multiple algorithms that can be plugged in at will, which will mitigate the impact of bugs, or be more willing to completely switch algorithms.  In general, I think that projects with a more general approach will be more stable long term as well, since the work will never be completely “done” like the implementation of a particular protocol.



So my opinion is that I would be all for a general pluggable consensus project, but most likely against adding a project aiming to solely implement a particular consensus algorithm that has not been peer reviewed.


Isn't Fabric already a "general pluggable consensus project"?  If not, could you really have one that wasn't better done as enhancements to turn Fabric into that (or Sawtooth/Iroha et al)?  Or is there something generic to DLTs, in the same way the GSL proposal was generic and potentially common to all DLTs, that could be implemented?

Brian


 

Anyways, that’s just my two cents worth.  Thanks for reading, and I’d love to hear feedback on this.

 

Thanks,

Hart

 

From: hyperledger-tsc-bounces@... [mailto:hyperledger-tsc-bounces@...] On Behalf Of Brian Behlendorf via hyperledger-tsc
Sent: Wednesday, June 28, 2017 7:55 AM
To: hyperledger-tsc@...
Subject: Re: [Hyperledger Project TSC] Project Proposal: Consensus Platform

 

I am a bit more leery of viewing academic peer-review as a gating function for new projects joining Hyperledger.  It is completely appropriate IMHO for the Fabric maintainer community to state that they will only include consensus plug-ins in the Fabric install that meet some criteria for quality, and peer-reviewed consensus mechanisms may be that.  But not all possible consensus plug-ins need to be included within the Fabric project.  We could accept a new Hyperledger project to build a new/different consensus plug-in, with a different set of maintainers, who set their own standards for the proven-ness of the algorithm.  That might be the right thing for a Hashgraph plugin, especially if there are some edge-case patent concerns. 

Brian

On 06/27/2017 11:03 AM, Hart Montgomery via hyperledger-tsc wrote:

Hi Martin,

 

Is there a reason this hasn’t been submitted to a conference for peer review?  I’d highly recommend doing so—you’ll have a lot more credibility with regards to people believing the protocol if you can get it published in some conference proceedings.  If you need help, I bet there are people on this list who would be happy to point you in the right direction.

 

As it is, I’d be wary of accepting any new crypto or consensus into Hyperledger that hasn’t been published in a reputable conference or heavily reviewed and documented by outsiders.  It sets the precedent that the TSC or other members of the community would have to be academic-style crypto reviewers in order to properly assess new contributions.  While some of us may be, it puts an impossible burden on those that are not, and the appropriate place for new research to be evaluated (which new crypto or consensus is) is an academic conference anyways.  Additionally, to be truly safe, lots of time and review is needed for protocols that will be critical to systems.  For instance, the SHA-3 competition and review process took five years.  While we don’t need that much time for new consensus algorithms, I think this is still an area where it is much better to be safe than sorry.

 

Thanks for your time, and have a great day.

 

Hart

 

From: hyperledger-tsc-bounces@... [mailto:hyperledger-tsc-bounces@...] On Behalf Of Martin Arrivets via hyperledger-tsc
Sent: Tuesday, June 27, 2017 9:55 AM
To: Christian Cachin <cca@...>
Cc: Martin Arrivets via hyperledger-tsc <hyperledger-tsc@...>; Giacomo Puri Purini <giacomo@...>
Subject: Re: [Hyperledger Project TSC] Project Proposal: Consensus Platform

 

Thanks for the links,

 

Indeed the problem cannot be solved with a deterministic protocol

 

But there are ways to circumvent the FLP theorem by sacrificing determinism

cf Rabin, M.O. (1983) ‘Randomized Byzantine generals’ for example.

 

It is possible for a nondeterministic system to achieve consensus with probability

one.

 

That is what the Hashgraph does and that is what I meant when I wrote that participants 

will "eventually" (with probability one) know the exact location of a transaction in history.

 

The Hasgraph algorithm introduces periodic 'coin-rounds' where witnesses vote randomly. This

defends against attackers that control the internet and want to partition the network.

 

Martin

-----Original message-----
From: Christian Cachin
Sent: Tuesday, June 27 2017, 6:09 pm
To: Martin Arrivets
Cc: Martin Arrivets via hyperledger-tsc; David Huseby; Christopher Ferris; Giacomo Puri Purini
Subject: Re: [Hyperledger Project TSC] Project Proposal: Consensus Platform



Martin,
 
I've understood that there exists a white paper, yes.
 
But perhaps you should be aware that mathematically, in a "fully asynchronous
system" as you describe below, with < 1/3 faulty nodes (that crash or
act maliciously), or even with just one node that may crash, the "consensus
problem" cannot be solved with a deterministic protocol.
 
This is a famous result in distributed systems:
"Impossibility of Distributed Consensus with One Faulty Process" usually 
just called "FLP impossibility".
 
See more:
  https://en.wikipedia.org/wiki/Consensus_(computer_science)
  https://en.wikipedia.org/wiki/Byzantine_fault_tolerance
or a textbook I co-authored - www.distributedprogramming.net - although the
FLP result is not explained in there.
 
What you describe seems equivalent to this well-understood consensus problem.
 
In contrast to hashgraph, PBFT has been peer-reviewed and widely understood 
to achieve consensus in the appropriate model (eventual synchrony, not
asynchrony).
 
Regards,
 
      Christian
 
 
 
On 27 Jun 2017 12:03:06 +0000, Martin Arrivets <martin@...> wrote:
> Hi Christian,
> 
> The system achieves consensus in the sense that a participant will
> eventually know the exact location of a transaction in history and have
> a mathematical guarantee of finality (that this is the consensus order).
> The knowledge is not probabilistic but a mathematical guarantee.
> 
> The proofs in the whitepaper
> <http://www.swirlds.com/downloads/SWIRLDS-TR-2016-01.pdf>; make the
> standard assumptions:
> 
> More than 2/3 of the computers are "honest", which means they follow the
> algorithm correctly and are available. Although an honest computer may
> go down for a while (and stop communicating) as long as they eventually
> come back up.
> 
> Nodes can collude and are allowed to mostly control the internet . Their
> only limit on control on the internet is that if Alice repeatedly sends
> Bob messages they must eventually allow Bob to receive one.
> 
> The proofs are for a fully asynchronous system. There is no assumption
> that an honest node will always respond within a certain interval. If
> all the nodes go to sleep, then progress continues as soon as they come
> back up. In normal conditions with a small number of nodes, consensus
> can happen in less than a second.
> 
> I am not aware of any third-party reviews of the whitepaper. 
> 
> The concepts are very similar to other voting-based BFT algorithms
> except the voting is "virtual". If the security of those systems (like
> PBFT or Tendermint...) has been vetted by the schemes you mention then
> perhaps they could be applied to Hashgraph as well.
> 
> Hope this answers you questions.
> 
> Regards,
> Martin
> -----Original message-----
> From: Christian Cachin
> Sent: Tuesday, June 27 2017, 11:43 am
> To: Martin Arrivets via hyperledger-tsc
> Cc: Martin Arrivets; David Huseby; Christopher Ferris; Giacomo Puri
> Purini Subject: Re: [Hyperledger Project TSC] Project Proposal:
> Consensus Platform
> 
> 
> Martin,
> 
> Can you point to any independent analysis of the properties achieved by 
> Swirlds consensus?  In which sense does it reach "consensus"?  Are there 
> any peer-reviewed publications or other third-party endorsements
> available?
> 
> In analogy with cryptography, where such issues have been discussed at 
> large, and over many decades, the security of a system should be based 
> on well-established and widely agreed-on schemes.
> 
> (FYI - I'm a cryptographer and also working on consensus protocols...)
> 
> Regards,
> 
>     Christian
> 
> 
> --- 
> Christian Cachin                           email: cca@...
> <mailto:cca@...> IBM Research -
> Zurich                           tel: +41-44-724-8989 Säumerstrasse 4
> CH-8803 Rüschlikon, Switzerland       http://www.zurich.ibm.com/˜cca
> <http://www.zurich.ibm.com/˜cca>; 
> 
> 
> 





_______________________________________________
hyperledger-tsc mailing list
hyperledger-tsc@...
https://lists.hyperledger.org/mailman/listinfo/hyperledger-tsc

 

-- 
Brian Behlendorf
Executive Director, Hyperledger
bbehlendorf@...
Twitter: @brianbehlendorf

 

-- 
Brian Behlendorf
Executive Director, Hyperledger
bbehlendorf@...
Twitter: @brianbehlendorf

Join tsc@lists.hyperledger.org to automatically receive all group messages.