FireFly and Smart Client - overlap and convergence


David Enyeart
 

TSC members, 

I wanted to continue the FireFly discussion from the last meeting in this email thread, and make a proposal for consideration at the TSC meeting tomorrow.

Several members raised concerns about fast-tracking the FireFly project versus starting it as a Lab, including myself. Most of the concerns were fairly standard - checking on licensing issues, concern about a single organization driven project, etc. 

My concern is more fundamental. I 100% agree that a project is needed in this space, as off-chain interactions is a common pattern that we too have seen. My concern is that we have two Hyperledger contributions with common objectives moving in separate directions in this same solution space: 
- Hyperledger FireFly project proposal - https://github.com/hyperledger/hyperledger-hip/pull/3
- Hyperledger Fabric Smart Client lab - https://github.com/hyperledger-labs/fabric-smart-client

I didn't go into it during the last call, since the Smart Client lab was in the process of getting approved. Now that it is approved and code delivered, if you read the Smart Client readme https://github.com/hyperledger-labs/fabric-smart-client#readme you'll see that there is much overlap between Smart Client and FireFly. Both attempt to solve the challenges related to off-chain interactions, both have organization/node specific private data vaults, both have support for a single DLT initially for global ordering and immutability, and both are architected to extend to other DLTs in the future. The Smart Client lab is a culmination of several years of work and commit history that IBM has been incubating with real world projects that needed off-chain interactions in addition to a DLT. The fact that two similar projects came about organically is more than coincidence - it is a testament to the common gap that the contributions are trying to fill, and the need for a project in this space in the Hyperledger greenhouse.

I think we should understand how these two projects relate to each other, and attempt to converge, prior to fast-tracking one or the other into a full project. In fact, for Fabric Smart Client we debated over the last year whether to pursue as a full project (since it is architected to support other DLTs beyond Fabric in the future), pursue as a Fabric sub-project, or pursue as a Lab. We concluded that it would not be prudent to fast-track it into a full project without other organizations getting involved in a meaningful way, and without other DLT support. Rather, we thought it would be more sensible, and more aligned with the Hyperledger charter, to incubate it further as a Lab to get more traction within the community and with other organizations, before pursuing full project status. It seems the same concerns apply equally to FireFly.

The fact that one contribution came in as a Lab and one is coming in as a full project proposal is fairly arbitrary, rather than an indication that one is more mature than the other. The Fabric Smart Client actually has a larger code base, deeper commit history, and more developers (it appears the core FireFly Go contribution was written by an individual developer over the course of the last month as part of the fast-track effort, as a re-write of the typescript implementation from last fall). I am not knocking FireFly - in fact it seems that each project has value propositions that would be stronger in a joint project rather than separate.

I think the worst thing that we could do is have two competing projects that fragment the solution space. My proposal would be to start both Fabric Smart Client and FireFly as Labs, with the intention that we work to converge them and create a single joint project proposal in the coming months. I think we will be more successful collaborating in a single cross-organization project that supports multiple DLTs, rather than competing as two separate single-organization projects.

If there is still interest in fast-tracking a full project, I think the right thing to do would be to make it an "umbrella" project that gets seeded from both the Smart Client and FireFly code bases. If this is the case, a new project name should be utilized rather than picking one of the existing repo names. I would caution however that there will be less motivation for true convergence after a project is created, rather than a convergence plan being a pre-condition of the project approval. 


Dave Enyeart



Jim Zhang
 

Hi Dave,

Thank you for sharing your thoughts. We'd love to collaborate with you and Angelo on key Fabric capabilities like this one. Your note lays out an interesting idea how to bring the two communities together. 

It seems as though there is nothing stopping us from combining efforts even if it's the case of a hybrid collaboration between one project being a Hyperledger project and another being a Lab. We're very eager to see how things develop and to work together towards a common code base in Fabric-based areas.

Thanks for making fabric-smart-client code base available today in the Labs. We have done a preliminary review of the code base and documentation. 

From this review, we can see our initiatives' architectures are coming from different approaches born of solving pretty different problems.

FireFly has been multi-protocol from the get-go, with the architecture having been proven by plugin implementations for two extreme ends of the enterprise blockchain protocols space: Enterprise Ethereum (single node type, transactions are always broadcast to the whole network, JSON-RPC dev interface, account/state model) and Corda (2 node types, no shared ledger, transactions are always point-to-point, RPC over JSM for dev interface, UTXO model). The framework has been built from its inception to accommodate a Fabric plugin. With the implementation under development, this is a wonderful opportunity to collaborate with you and Angelo and the fabric-smart-client. 

One truism to keep in mind is that it can be hard to generalize from something built from one infrastructure to solve problems across multiple infrastructures, unless it was built with the multi-infrastructure in mind from the start.  Fabric-smart-client is built from a Fabric reference, born out of the goal to make consuming Fabric easier. Going from fabric-smart-client to supporting multiple protocols would be a challenging undertaking, given how each protocol is drastically different (using the spectrum visual, you would be starting from the middle and now trying to expand to support both ends of the extremes).

We are very excited about the potential for fabric-smart-client and feel it offers valuable areas for collaboration with FireFly and to provide inputs to the Fabric-related architecture. For example, the view concept for capturing member-specific business logic is interesting. We would love to collaborate with the IBM team to seek out opportunities to combine the strengths of the two. 

That said, we do feel the two projects were built with different proposed scope, different visions and architectures, especially when we look through our community's lens of addressing the industry's multi-party system challenges.

Finally, in order to make sure the TSC has a complete set of information for the meeting tomorrow, we did want to help clarify a few factual things that appear to have been points of confusion:

The FireFly code base being contributed as part of the proposal is a collection of 5 separate repositories, representing the starting point that will surely grow in number as more plugins are developed:
  - FireFly corehttps://github.com/kaleido-io/firefly, contains the current version (written in typescript, developed by 10 developers from Kaleido over the last 8 months), and the gen-2 version, which has improvements in interfaces, and pluggability, as well as being re-written in Golang (so far Peter our head of engineering is the main contributor).
  - FireFly plugin for Enterprise Ethereumhttps://github.com/kaleido-io/ethconnect, developed by 6 developers from Kaleido and 2 from customer organizations over the past 3 years, written in Golang
  - FireFly plugin for Corda OShttps://github.com/kaleido-io/firefly-cordaconnect, developed by 1 Kaleido developer over the past year, written in Java
  - FireFly CLIhttps://github.com/kaleido-io/firefly-cli, command line tools for developers building on FireFly for local environments, developed by another Kaleido developer in Golang
  - FireFly UIhttps://github.com/kaleido-io/firefly-ui, build-in dashboard/explorer, developed by 2 Kaleido developers
  - FireFly plugin for Fabric: to be implemented, preferably with the Fabric team's input and help

Best regards,
Jim

On Wed, Jun 2, 2021 at 12:57 PM David Enyeart <enyeart@...> wrote:

TSC members, 

I wanted to continue the FireFly discussion from the last meeting in this email thread, and make a proposal for consideration at the TSC meeting tomorrow.

Several members raised concerns about fast-tracking the FireFly project versus starting it as a Lab, including myself. Most of the concerns were fairly standard - checking on licensing issues, concern about a single organization driven project, etc. 

My concern is more fundamental. I 100% agree that a project is needed in this space, as off-chain interactions is a common pattern that we too have seen. My concern is that we have two Hyperledger contributions with common objectives moving in separate directions in this same solution space: 
- Hyperledger FireFly project proposal - https://github.com/hyperledger/hyperledger-hip/pull/3
- Hyperledger Fabric Smart Client lab - https://github.com/hyperledger-labs/fabric-smart-client

I didn't go into it during the last call, since the Smart Client lab was in the process of getting approved. Now that it is approved and code delivered, if you read the Smart Client readme https://github.com/hyperledger-labs/fabric-smart-client#readme you'll see that there is much overlap between Smart Client and FireFly. Both attempt to solve the challenges related to off-chain interactions, both have organization/node specific private data vaults, both have support for a single DLT initially for global ordering and immutability, and both are architected to extend to other DLTs in the future. The Smart Client lab is a culmination of several years of work and commit history that IBM has been incubating with real world projects that needed off-chain interactions in addition to a DLT. The fact that two similar projects came about organically is more than coincidence - it is a testament to the common gap that the contributions are trying to fill, and the need for a project in this space in the Hyperledger greenhouse.

I think we should understand how these two projects relate to each other, and attempt to converge, prior to fast-tracking one or the other into a full project. In fact, for Fabric Smart Client we debated over the last year whether to pursue as a full project (since it is architected to support other DLTs beyond Fabric in the future), pursue as a Fabric sub-project, or pursue as a Lab. We concluded that it would not be prudent to fast-track it into a full project without other organizations getting involved in a meaningful way, and without other DLT support. Rather, we thought it would be more sensible, and more aligned with the Hyperledger charter, to incubate it further as a Lab to get more traction within the community and with other organizations, before pursuing full project status. It seems the same concerns apply equally to FireFly.

The fact that one contribution came in as a Lab and one is coming in as a full project proposal is fairly arbitrary, rather than an indication that one is more mature than the other. The Fabric Smart Client actually has a larger code base, deeper commit history, and more developers (it appears the core FireFly Go contribution was written by an individual developer over the course of the last month as part of the fast-track effort, as a re-write of the typescript implementation from last fall). I am not knocking FireFly - in fact it seems that each project has value propositions that would be stronger in a joint project rather than separate.

I think the worst thing that we could do is have two competing projects that fragment the solution space. My proposal would be to start both Fabric Smart Client and FireFly as Labs, with the intention that we work to converge them and create a single joint project proposal in the coming months. I think we will be more successful collaborating in a single cross-organization project that supports multiple DLTs, rather than competing as two separate single-organization projects.

If there is still interest in fast-tracking a full project, I think the right thing to do would be to make it an "umbrella" project that gets seeded from both the Smart Client and FireFly code bases. If this is the case, a new project name should be utilized rather than picking one of the existing repo names. I would caution however that there will be less motivation for true convergence after a project is created, rather than a convergence plan being a pre-condition of the project approval. 


Dave Enyeart




--
Jim Zhang
Co-founder, Head of Protocol Engineering
Kaleido.io, The Blockchain Business Cloud


Angelo De Caro
 

Dear Jim,

Thanks for your technical points. I love this discussion.
Indeed, let me add to the conversation another point of view as a TSC member.

I want to start with this statement: Pluggability is as good as the abstraction behind it.
In other words, pluggability per se does not tell us anything. 
It is the abstraction that gives us confidence in the ability to support different implementations.
For example, in theory Fabric has also many pluggability points but in practice it is very hard.

The FireFly proposal does not tell us what's the abstraction used for the Blockchain, for example:
- Is the blockchain used just as time-stamping service?
- What about tokens? Does the abstraction cover both account-based and UTXO? What about privacy?
- The FireFly proposal mentions only Ethereum, Fabric (in design phase, if I understand correctly), and Corda (https://github.com/kaleido-io/firefly-cordaconnect is ~700 lines of Java code). What about Sawtooth and Iroha, for example? I ask cause the Firefly proposal says that it is an "umbrella project".

So, if I understand correctly, and I might be wrong, Firefly seems to be, at the current stage, mono blockchain (Ethereum).
The proposal does not give evidences that other blockchains can be plugged. Which means that given an application developed 
with Firefly, the blockchain can be swapped without changing the application itself (which is a great goal, by the way, if it were realizable).

The same considerations apply to all the pluggability points: Consensus, Zero Knowledge, MPC, all those named.
What's the abstraction that Firefly proposes? 

I think it is important to talk about the above technical points :)

Looking forward for additional discussions.

Thanks much, Jim.

Best,
Angelo De Caro
TSC Member

P.S.: A small comment on the Fabric Smart Client. The philosophy used there is the following: Let's use each blockchain at its maximal potential!
This is because the various DLT platforms are evolving very fast and they are also kind of becoming more specialised. It seems then reasonable to select them based on the functionality they deliver best and then use interoperability protocols to make the DLTs speak to each other. The most proximate gaol of the Fabric Smart Client is to deliver the best possible experience when using Hyperledger Fabric. 


Arun S M
 

Thanks David for introducing us to the new Hyperledger Labs project.
I wasn't aware about this, it appears to be proposed about a week back https://github.com/hyperledger-labs/hyperledger-labs.github.io/pull/169

Here are my views,

This is a great project. A ready to use blockchain client (Hyperledger Fabric in this case) such as these can save time in HL Fabric adoption. This will benefit many of the end user organizations who otherwise end up spending resources on re-doing common patterns. In addition, this project will also encourage in designing business flows better, through smart contract governance.

However, I am not clear on the design, how it will extend to other protocols? Will it be a smart client per protocol or will it make use of the SDK approach through one smart client for different protocols? Speaking from an adoption point of view, a flexibility to switch between protocols or having one enterprise integration point for multiple protocols underneath is highly beneficial.

Firefly follows the microservice based architecture to solve challenges up the stack. HL Fabric has the concept of channels and private data collections, which are useful in many ways. But implementing a similar solution on Ethereum/Sawtooth etc is not straightforward. Firefly proposes an abstract term MPC, how it is achieved in an application is left to the implementation. An organization can merely make use of Firefly for high availability, use a MQ for data exchange, and have nothing to do with blockchain at all. One of the use cases mentioned in the proposal is to guarantee the provenance through the blockchain. I could not find a complex use case/example with custom smart contract logic. But the discussion so far I had, gives an impression that application designers have to rely on events streamlined at multiple connecting points to make additional decisions. However, these are done through APIs through application designer's preferred auth mechanism instead of making them through blockchain clients. Kind of abstraction.

In the proposal Firefly promises to add support on HL Fabric, and efforts are underway. It will help if FSC can be a means of integrating HL Fabric into Firefly. I am interested to learn more about FSC in the call, and understand if it can be a Firefly connector for HL Fabric. Another point to consider is if we are opening up Hyperledger for projects that span across an application stack upwards blockchain protocol. This was a hot topic in the member summit at least.

Regards,
Arun

On Thu, Jun 3, 2021 at 1:03 PM Angelo De Caro <adc@...> wrote:
Dear Jim,

Thanks for your technical points. I love this discussion.
Indeed, let me add to the conversation another point of view as a TSC member.

I want to start with this statement: Pluggability is as good as the abstraction behind it.
In other words, pluggability per se does not tell us anything. 
It is the abstraction that gives us confidence in the ability to support different implementations.
For example, in theory Fabric has also many pluggability points but in practice it is very hard.

The FireFly proposal does not tell us what's the abstraction used for the Blockchain, for example:
- Is the blockchain used just as time-stamping service?
- What about tokens? Does the abstraction cover both account-based and UTXO? What about privacy?
- The FireFly proposal mentions only Ethereum, Fabric (in design phase, if I understand correctly), and Corda (https://github.com/kaleido-io/firefly-cordaconnect is ~700 lines of Java code). What about Sawtooth and Iroha, for example? I ask cause the Firefly proposal says that it is an "umbrella project".

So, if I understand correctly, and I might be wrong, Firefly seems to be, at the current stage, mono blockchain (Ethereum).
The proposal does not give evidences that other blockchains can be plugged. Which means that given an application developed 
with Firefly, the blockchain can be swapped without changing the application itself (which is a great goal, by the way, if it were realizable).

The same considerations apply to all the pluggability points: Consensus, Zero Knowledge, MPC, all those named.
What's the abstraction that Firefly proposes? 

I think it is important to talk about the above technical points :)

Looking forward for additional discussions.

Thanks much, Jim.

Best,
Angelo De Caro
TSC Member

P.S.: A small comment on the Fabric Smart Client. The philosophy used there is the following: Let's use each blockchain at its maximal potential!
This is because the various DLT platforms are evolving very fast and they are also kind of becoming more specialised. It seems then reasonable to select them based on the functionality they deliver best and then use interoperability protocols to make the DLTs speak to each other. The most proximate gaol of the Fabric Smart Client is to deliver the best possible experience when using Hyperledger Fabric. 


Jim Zhang
 

Thanks Angelo for your thoughtful questions, and Arun for offering your view from a customer perspective, which is extremely helpful in keeping us technology vendors on the right track.

I think Angelo you are touching on a very fundamental point about FireFly's design, the abstraction of the blockchain layer. FireFly relies on blockchain for only the most fundamental problems not solvable with any other technologies, and I expanded on this in my response to your question in the HIP pull request discussion here: https://github.com/hyperledger/hyperledger-hip/pull/3#discussion_r644772585. This abstraction also allows the pluggability of multiple protocols.

Looking forward to talking more on the TSC call soon.

Jim


On Thu, Jun 3, 2021 at 4:15 AM Arun S M <arun.s.m.cse@...> wrote:
Thanks David for introducing us to the new Hyperledger Labs project.
I wasn't aware about this, it appears to be proposed about a week back https://github.com/hyperledger-labs/hyperledger-labs.github.io/pull/169

Here are my views,

This is a great project. A ready to use blockchain client (Hyperledger Fabric in this case) such as these can save time in HL Fabric adoption. This will benefit many of the end user organizations who otherwise end up spending resources on re-doing common patterns. In addition, this project will also encourage in designing business flows better, through smart contract governance.

However, I am not clear on the design, how it will extend to other protocols? Will it be a smart client per protocol or will it make use of the SDK approach through one smart client for different protocols? Speaking from an adoption point of view, a flexibility to switch between protocols or having one enterprise integration point for multiple protocols underneath is highly beneficial.

Firefly follows the microservice based architecture to solve challenges up the stack. HL Fabric has the concept of channels and private data collections, which are useful in many ways. But implementing a similar solution on Ethereum/Sawtooth etc is not straightforward. Firefly proposes an abstract term MPC, how it is achieved in an application is left to the implementation. An organization can merely make use of Firefly for high availability, use a MQ for data exchange, and have nothing to do with blockchain at all. One of the use cases mentioned in the proposal is to guarantee the provenance through the blockchain. I could not find a complex use case/example with custom smart contract logic. But the discussion so far I had, gives an impression that application designers have to rely on events streamlined at multiple connecting points to make additional decisions. However, these are done through APIs through application designer's preferred auth mechanism instead of making them through blockchain clients. Kind of abstraction.

In the proposal Firefly promises to add support on HL Fabric, and efforts are underway. It will help if FSC can be a means of integrating HL Fabric into Firefly. I am interested to learn more about FSC in the call, and understand if it can be a Firefly connector for HL Fabric. Another point to consider is if we are opening up Hyperledger for projects that span across an application stack upwards blockchain protocol. This was a hot topic in the member summit at least.

Regards,
Arun

On Thu, Jun 3, 2021 at 1:03 PM Angelo De Caro <adc@...> wrote:
Dear Jim,

Thanks for your technical points. I love this discussion.
Indeed, let me add to the conversation another point of view as a TSC member.

I want to start with this statement: Pluggability is as good as the abstraction behind it.
In other words, pluggability per se does not tell us anything. 
It is the abstraction that gives us confidence in the ability to support different implementations.
For example, in theory Fabric has also many pluggability points but in practice it is very hard.

The FireFly proposal does not tell us what's the abstraction used for the Blockchain, for example:
- Is the blockchain used just as time-stamping service?
- What about tokens? Does the abstraction cover both account-based and UTXO? What about privacy?
- The FireFly proposal mentions only Ethereum, Fabric (in design phase, if I understand correctly), and Corda (https://github.com/kaleido-io/firefly-cordaconnect is ~700 lines of Java code). What about Sawtooth and Iroha, for example? I ask cause the Firefly proposal says that it is an "umbrella project".

So, if I understand correctly, and I might be wrong, Firefly seems to be, at the current stage, mono blockchain (Ethereum).
The proposal does not give evidences that other blockchains can be plugged. Which means that given an application developed 
with Firefly, the blockchain can be swapped without changing the application itself (which is a great goal, by the way, if it were realizable).

The same considerations apply to all the pluggability points: Consensus, Zero Knowledge, MPC, all those named.
What's the abstraction that Firefly proposes? 

I think it is important to talk about the above technical points :)

Looking forward for additional discussions.

Thanks much, Jim.

Best,
Angelo De Caro
TSC Member

P.S.: A small comment on the Fabric Smart Client. The philosophy used there is the following: Let's use each blockchain at its maximal potential!
This is because the various DLT platforms are evolving very fast and they are also kind of becoming more specialised. It seems then reasonable to select them based on the functionality they deliver best and then use interoperability protocols to make the DLTs speak to each other. The most proximate gaol of the Fabric Smart Client is to deliver the best possible experience when using Hyperledger Fabric. 



--
Jim Zhang
Co-founder, Head of Protocol Engineering
Kaleido.io, The Blockchain Business Cloud