Topics

Transact as a TLP


Brian Behlendorf
 

In Rocketchat, Shawn Amundson had this to say about the "Rollover of projects tied to a single other project" proposal:

It would be helpful to be specific about the intent in relation to existing projects. A policy that forces X project and Y project to merge may be equivalent to ejecting X project or Y project from HL. Such a move should be explicit and discussed. As it is now, it sounds like malicious maneuvering against projects that specific TSC members aren't directly invested in. TSC should focus instead on fostering collaboration instead of attacking projects. "If you don't write your project report the right way, we will immediately start to talk about 'sanctions' or removing the project." isn't collaborative.

Shawn is right. Conversations that try to address subjective disagreements with objective rulemaking can come across as attempts to legislatively solve disputes and issues that are actually more interpersonal in nature and that almost always leads to bad laws/rules. Rather than talking about new rules to try and tackle what is actually a fairly specific disagreement and possibly a niche situation, it might help to try and explain how we got here, and use that as a basis to figure out a better path out.

As I retell this, please be kind to my interpretation. I'll cite links when I can find them, but much of this covers conversations and events happening on phone calls, DMs, personal emails, etc, as well as my unavoidably subjective read of the situation. I'll try and state interpretations as plainly as I can, and avoid imputing motives, so apologies if I slip on that. If you feel I got something wrong, feel free to let me or the list know, but please assume benign ignorance of details rather than maliciousness on my part.

Transact was proposed as a top-level project at a pretty early stage in its life, likely long before any project would be accepted today. No need to revisit that decision, but to remember its rationale - that there was a strong desire across the TSC (it seemed) for a common, unified approach to allow for portability of smart contract engines across an array of different ledgers. Such an approach, of course, is never guaranteed to work - there was no small amount of optimism, risk, and hope in that goal, and it requires a ton of communication, architectural humility, and looking for common ground before writing code. Trying to define a common dividing line across systems as different in their architectures as Fabric, Sawtooth, and Iroha (let alone Besu, which hadn't shown up yet) wasn't even clear if it was possible, or if the compromises it would require wouldn't have unacceptable costs. But there was a sincere intent shown in the Transact proposal to explore this domain, and the proposed ideas seemed worth pursuing, so much so that the TSC relied upon them to make its decision to approve. You can see mention of it in the original proposal and discussion on TSC calls that led to an email-based vote. I'd say the enthusiasm for getting this right across HL projects was part of the reason people went in so early, as it's much harder to adapt once that first architecture is laid out. While I was not at the Maintainer Summit in Minnesota, I recall hearing there were very promising conversations between Sawtooth and Fabric developers and others about the possible ways for Transact to become that kind of project. So I view the TSC's approval of Transact as not just an endorsement of the principle - hey, sounds good, someone aught to do it - but a reflection of sincere intent on behalf of the other DLTs to invest time into a shared approach. That was not "we'll wait and see", but "this is good enough, let's start to work together."

But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.

Today, if Transact had been proposed, we probably would have said "let's start it in Labs" first, and given it time to make measured progress against that objective, before approving it as a TLP.  This is the path that Cactus and Avalon have gone through, successfully, and there are a lot of great ideas in Labs that haven't yet achieved their initial goals and many that won't. Had Transact started in labs, and still arrived to where it was today, my personal view is that it likely would not be accepted as a top level project, given how  core it still seems to Sawtooth. I believe we were concerned about having a production-grade DLT like Sawtooth depending upon a Labs project - which could have been solved by making it a "sawtooth-transact" repo.

That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.

To be completely fair to Transact, as an individual project it has by no means failed. I don't know if there are production systems running on it yet, but objectively there are commits and code and docs and all the trappings of a competently managed source tree. Its last release was version 0.3.1 just 12 hours ago. There are commits to the repo from more than just Bitwise employees this past year, according to the dashboard. So let's think instead about how to respect the effort those devs are putting in, an approach that helps them and Transact get closer to its original goal, or if that goal no longer applies, provides a dignified alternative path.

There are three possible fixes I see based on this history:

1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.

2) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. BUT, maintainers on other projects can't find the time or inclination to sincerely take part in that process. In that case, Transact remains a TLP, because they should not be penalized if other projects can't be bothered to meet them halfway. Even just one such DLT stepping in would be enough.

3) Transact maintainers declare that other projects are more than welcome to use the architecture they've developed, but they have their own priorities and opinions on how things should be built, take them or leave them. Many projects are successful with that perspective - including the Linux kernel! - so that's not an incorrect approach. But that seems different than how the project was originally sold as appropriate for a TLP. In this case, Transact is folded into Sawtooth as a sawtooth-transact repo, or becomes a Lab.

4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.

Comparing Transact to other tools/utilities projects like Cello, Caliper, and Explorer is difficult, because none of those projects were about so deeply about rebooting how our DLTs would work; instead they are add-ons where adaptation to other DLTs doesn't require such tight coordination with core maintainers on those DLTs. Ursa is more comparable, and Ursa now is consumed by both Indy and Iroha, who did put in the work in good faith.

There may be other proposals to put on the table. However, putting aside my "HL/LF staff objective community facilitator" hat and putting on my "experienced open source community leader" hat, the use of dependencies to argue for whether something should be a TLP inside an umbrella or not (or worse, as a basis for that status to be removed) is something I have never heard of before here, and so of course have never seen successfully applied, and my spidey sense is that aside from suggesting it (e.g. "that FabToken proposal you made, have you talked with Fabric about being a module there first? or over at Labs?") it's a poor criteria here.

Brian

-- 
Brian Behlendorf
Executive Director, Hyperledger
bbehlendorf@...
Twitter: @brianbehlendorf


Duncan Johnston-Watt
 

Brian/All

This is definitely something we should talk about next week.

My 2c is that given the variety of top level DLTs within the Hyperledger community and their very different architectures it is highly unlikely that a TLP like a Transact will be capable of supporting all of them. Therefore my simple minded approach would be to see if at least one other of the top level DLT projects is seriously interested in supporting it as a starting point.

Brian notes -

One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.

It's actually Besu not Quorum and the list includes VMware Blockchain and FISCO BCOS  too. However he is quite right that Digital Asset has invested to ensure that DAML "reaches the parts that other smart contract languages cannot reach" to misquote a Heineken advert from my youth.

Best

Duncan

On Fri, Sep 4, 2020 at 9:35 AM Brian Behlendorf <bbehlendorf@...> wrote:

In Rocketchat, Shawn Amundson had this to say about the "Rollover of projects tied to a single other project" proposal:

It would be helpful to be specific about the intent in relation to existing projects. A policy that forces X project and Y project to merge may be equivalent to ejecting X project or Y project from HL. Such a move should be explicit and discussed. As it is now, it sounds like malicious maneuvering against projects that specific TSC members aren't directly invested in. TSC should focus instead on fostering collaboration instead of attacking projects. "If you don't write your project report the right way, we will immediately start to talk about 'sanctions' or removing the project." isn't collaborative.

Shawn is right. Conversations that try to address subjective disagreements with objective rulemaking can come across as attempts to legislatively solve disputes and issues that are actually more interpersonal in nature and that almost always leads to bad laws/rules. Rather than talking about new rules to try and tackle what is actually a fairly specific disagreement and possibly a niche situation, it might help to try and explain how we got here, and use that as a basis to figure out a better path out.

As I retell this, please be kind to my interpretation. I'll cite links when I can find them, but much of this covers conversations and events happening on phone calls, DMs, personal emails, etc, as well as my unavoidably subjective read of the situation. I'll try and state interpretations as plainly as I can, and avoid imputing motives, so apologies if I slip on that. If you feel I got something wrong, feel free to let me or the list know, but please assume benign ignorance of details rather than maliciousness on my part.

Transact was proposed as a top-level project at a pretty early stage in its life, likely long before any project would be accepted today. No need to revisit that decision, but to remember its rationale - that there was a strong desire across the TSC (it seemed) for a common, unified approach to allow for portability of smart contract engines across an array of different ledgers. Such an approach, of course, is never guaranteed to work - there was no small amount of optimism, risk, and hope in that goal, and it requires a ton of communication, architectural humility, and looking for common ground before writing code. Trying to define a common dividing line across systems as different in their architectures as Fabric, Sawtooth, and Iroha (let alone Besu, which hadn't shown up yet) wasn't even clear if it was possible, or if the compromises it would require wouldn't have unacceptable costs. But there was a sincere intent shown in the Transact proposal to explore this domain, and the proposed ideas seemed worth pursuing, so much so that the TSC relied upon them to make its decision to approve. You can see mention of it in the original proposal and discussion on TSC calls that led to an email-based vote. I'd say the enthusiasm for getting this right across HL projects was part of the reason people went in so early, as it's much harder to adapt once that first architecture is laid out. While I was not at the Maintainer Summit in Minnesota, I recall hearing there were very promising conversations between Sawtooth and Fabric developers and others about the possible ways for Transact to become that kind of project. So I view the TSC's approval of Transact as not just an endorsement of the principle - hey, sounds good, someone aught to do it - but a reflection of sincere intent on behalf of the other DLTs to invest time into a shared approach. That was not "we'll wait and see", but "this is good enough, let's start to work together."

But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.

Today, if Transact had been proposed, we probably would have said "let's start it in Labs" first, and given it time to make measured progress against that objective, before approving it as a TLP.  This is the path that Cactus and Avalon have gone through, successfully, and there are a lot of great ideas in Labs that haven't yet achieved their initial goals and many that won't. Had Transact started in labs, and still arrived to where it was today, my personal view is that it likely would not be accepted as a top level project, given how  core it still seems to Sawtooth. I believe we were concerned about having a production-grade DLT like Sawtooth depending upon a Labs project - which could have been solved by making it a "sawtooth-transact" repo.

That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.

To be completely fair to Transact, as an individual project it has by no means failed. I don't know if there are production systems running on it yet, but objectively there are commits and code and docs and all the trappings of a competently managed source tree. Its last release was version 0.3.1 just 12 hours ago. There are commits to the repo from more than just Bitwise employees this past year, according to the dashboard. So let's think instead about how to respect the effort those devs are putting in, an approach that helps them and Transact get closer to its original goal, or if that goal no longer applies, provides a dignified alternative path.

There are three possible fixes I see based on this history:

1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.

2) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. BUT, maintainers on other projects can't find the time or inclination to sincerely take part in that process. In that case, Transact remains a TLP, because they should not be penalized if other projects can't be bothered to meet them halfway. Even just one such DLT stepping in would be enough.

3) Transact maintainers declare that other projects are more than welcome to use the architecture they've developed, but they have their own priorities and opinions on how things should be built, take them or leave them. Many projects are successful with that perspective - including the Linux kernel! - so that's not an incorrect approach. But that seems different than how the project was originally sold as appropriate for a TLP. In this case, Transact is folded into Sawtooth as a sawtooth-transact repo, or becomes a Lab.

4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.

Comparing Transact to other tools/utilities projects like Cello, Caliper, and Explorer is difficult, because none of those projects were about so deeply about rebooting how our DLTs would work; instead they are add-ons where adaptation to other DLTs doesn't require such tight coordination with core maintainers on those DLTs. Ursa is more comparable, and Ursa now is consumed by both Indy and Iroha, who did put in the work in good faith.

There may be other proposals to put on the table. However, putting aside my "HL/LF staff objective community facilitator" hat and putting on my "experienced open source community leader" hat, the use of dependencies to argue for whether something should be a TLP inside an umbrella or not (or worse, as a basis for that status to be removed) is something I have never heard of before here, and so of course have never seen successfully applied, and my spidey sense is that aside from suggesting it (e.g. "that FabToken proposal you made, have you talked with Fabric about being a module there first? or over at Labs?") it's a poor criteria here.

Brian

-- 
Brian Behlendorf
Executive Director, Hyperledger
bbehlendorf@...
Twitter: @brianbehlendorf



--
Duncan Johnston-Watt
CEO, BTP

Twitter: @duncanjw


Shawn Amundson
 

On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...

But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.

Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).

I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team.  The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?

...

That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.


While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.

FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
 
...

There are three possible fixes I see based on this history:

This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.

1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.

The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...

4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.

...

"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
 
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.

-Shawn


Shawn Amundson
 



On Fri, Sep 4, 2020 at 4:03 AM Duncan Johnston-Watt <duncan@...> wrote:
...
 

My 2c is that given the variety of top level DLTs within the Hyperledger community and their very different architectures it is highly unlikely that a TLP like a Transact will be capable of supporting all of them. Therefore my simple minded approach would be to see if at least one other of the top level DLT projects is seriously interested in supporting it as a starting point.


I'd personally like to see Iroha adopt Transact. But I don't think it can/should be a TSC mandate, nor do I think it should be a metric by which we judge HL's success.

Keep in mind that Transact is used in Grid, Scabbard, Sawtooth -- and because it is used in Grid, you can run Grid today on Scabbard without Sawtooth, if that's what you want to do. A "totem for our difficulties" for some, and "holy shit that's cool" for others.

-Shawn


Arnaud Le Hors
 

Hi Shawn,

While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.

I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.

Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.

Regards.
--
Arnaud  Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM




From:        "Shawn Amundson" <amundson@...>
To:        Brian Behlendorf <bbehlendorf@...>
Cc:        "hyperledger-tsc@..." <hyperledger-tsc@...>
Date:        09/04/2020 08:49 PM
Subject:        [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by:        tsc@...




On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).

I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team.  The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?

...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.

While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.

FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
 
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.

1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...

"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
 
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.

-Shawn




James Barry
 

Brian and Arnaud,

Without emotion into to it, these things take a while to gestate. I think that Transact is maturing now so that it can be used as a solid external component.  Iroha is enquiring about the use within their 2.0 and testing the framework. Sawtooth 2.0 will be using it as a core component.  Outside of Hyperledger, at this point Scabbard, the default engine for Splinter (not a Hyperledger project) is definitely using Transact.  I believe other DLT’s will use it once it gets a bit more mature and more publicity. 

These things are hard to make reusable as the TSC noted when Bill and I made our componentization pitch to the TSC last Spring.  I do think it would be a good thing for the TSC to look at common component frameworks and see if there is any uptick for the projects.  If so, perhaps a member or two might find it good for a resource to work for 3-4 months to see if it would fit into the top level frameworks.  I was on the call asking the TSC for some fairly straight forward changes so the Chinese encryption algorithm could be used by Fabric.  The pushback surprised me, especially talking about all of the issues around hardware devices etc.  Accommodating the encryption was IMHO a perfect use case for componentization, and a great one for URSA to tackle.  Yes it would take longer, but then Fabric would not be caught flat footed by the next country requiring their own proprietary encryption. So it is this hard level of reuse or accommodation that needs to be looked at, unless Hyperledger is to become a single codebase project.  Brian has some great discussions on how the DLT should all be getting closer together with clients, API’s, I/O’s etc.  Things like Quorum are in use at large Hyperledger members.  Transact had a great vision when it started and “should" be something that a Quorum project is interested in. Because Transact slowed down delivering its original promise, that should be a reason to see how to help it, not fold it in.  Same with URSA frankly.  Starting out as a Labs project might work, so long as the sponsors would be able to find two projects that would lend dedicated programmers to integrate into their project.  

Hyperledger’s TSC could define what is expected from shareable components.  URSA is another one that could use that expectation of the shared components in my opinion. 

Transact had a great vision at the beginning and is now starting to realize that vision.  Let's see how we can help it along and achieve its vision.  Its time for shareable components to emerge for the DLT’s (blockchains) as they were all islands back in 2018.  If we are to get widespread use, they need to start having some reusable pieces, not simply build it and they will come. That fractionalization really killed the overall blockchain momentum and created the “disillusionment” trough that individual walled projects have created. Through projects like Transact/Ursa etc. that sense of an entire framework, not individual projects, enterprises will find DLT’s (blockchains) more mainstream and usable within their environments.

I hope to see Transact and other projects nurtured to help each other.  I would hope that the needs of other projects are added to the architectures as they are being developed so they can be reused throughout the Hyperledger community.  We have been working with Shawn to get Sawtooth 2.x to be accommodating to our needs including putting items in the plans that were not originally in the architecture. And we are not the only ones with needs not originally contemplated by the 2.0 architecture.  Transact could use some solid PR’s or whatever process the TSC thinks will accommodate the acquiring needs of other projects. Through some strong suggestions, we can help mold Transact to be an awesome project. 

Thanks,


James Barry
Cell 303-956-8835
james@...
http://www.linkedin.com/in/jamesbarry




On Sep 4, 2020, at 2:04 PM, Arnaud Le Hors <lehors@...> wrote:

Hi Shawn,

While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.

I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.

Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.

Regards.
--
Arnaud  Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM




From:        "Shawn Amundson" <amundson@...>
To:        Brian Behlendorf <bbehlendorf@...>
Cc:        "hyperledger-tsc@..." <hyperledger-tsc@...>
Date:        09/04/2020 08:49 PM
Subject:        [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by:        tsc@...




On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).

I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team.  The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?

...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.

While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.

FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
 
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.

1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...

"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
 
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.

-Shawn





Brian Behlendorf
 

On 9/4/20 11:49 AM, Shawn Amundson wrote:
On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...

But then... that enthusiasm seemed to fizzle. Like any such collaborative process [...]

Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard),

To be clear, it wasn't a statement about Transact as a project - I went out of my way to note that there is a healthy development clip to Transact for a young project. What didn't materialize - perhaps that's a more neutral term than fizzled - was a clear process for designing Transact as a bridge specifically for the other DLTs at Hyperledger. That was what I read coming out of the Maintainer Summit, and the basis for its accelerated approval as a TLP, and that's the momentum that felt lost. Apologies that I wasn't clear.

I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team.  The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library.

Indeed, this has hobbled Fabric's potential adoption of Ursa too, and our attempts to standardize and make it easier/more foolproof to adopt modern cryptography. I still can't believe Go can't do what SWIG brought to pretty much every other programming language in like 2001. Go fans, am I wrong?

This is the kind of Technical Steering that the TSC could do, if it chose. Steering projects towards languages and architectures that encourage modularization and re-use. The selection of Go is hard to fault for having been made in 2015, and the Go client for Ethereum is clearly very successful. But It'd be really nice if the Fabric team could start thinking about whether portions of Fabric in languages other than Go would make sense going forward. Seeing Sawtooth move from Python to Rust and Iroha from C to Rust is really encouraging and I'm sure couldn't have been an easy decision. I see James made this point as well, thanks James!

Here lies the conundrum - in order to collaborate on the same code, so that we might save time/money/effort on common objectives, we have to spend more time/money/effort to get there. The calculus is challenging for sake of all the unknowns. There's a cost/benefit to engaging early in the lifecycle of something like Transact, if the architectures can adapt; versus later, when more unknowns are settled, but decisions are now more hard-coded. If there's one thing a TSC should be useful at, it's brokering the trust and conversations required to assist that calculus. Carrots, not sticks, to get to better alignment across Hyperledger. Because sticks fucking hurt.

And hey, it's 2020, and not even ICO-funded Swiss shell corps have piles of money to spend reinventing wheels and duplicating efforts. We all have to get better at building upon each other's work if we're going to survive what is clearly still a very long blockchain winter, and very few of the solutions in the market will survive it. My hope is that at least a few of the ones that do are at Hyperledger. There can be more than one :) but the case for their differentiation better be really, really strong, and they better share as much code as possible.

Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.

Not that I can see, I'm personally pretty eager to see it succeed. By the way, other TSC listers, Grid recently restarted its regular calls, and had a decently well attended meeting that sounded pretty productive, including that they use the RFCs process other projects have started to adopt. Cool.

The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work.

Note that the pitch was somewhat different - not just an abstract notion that it could work across other DLTs, but specifically that it would aim to help the ones we already have at Hyperledger. That signaling was clear in the Google doc proposal, and I'm sure the basis of Gari and Chris's support for it as well.

So for those who want to be involved in those design decisions around Transact - where do they go? Whether I go to the wiki page, the mailing list, or the RocketChat channel, I don't see those discussions, despite those being listed in the Transact README.md as the place to go. The Kanban page isn't a useful way to see design discussions, it's a way to prioritize work already agreed upon. It looks and feels like the actual discussions and work are taking place elsewhere, hidden?, with the community tools used as an after-development way to get feedback from end-users. The cheerful Q3 report doesn't give any clues either. With Grid it's the same deal - I'm quite encouraged by that community call, but again, the Wiki, the mailing list, and the RocketChat channel all leave the impression that the real work is happening elsewhere. To even find the agenda for the Grid meeting I had to go scroll up in the Rocketchat channel, and then, who knows what the results of that meeting were? I want to be wrong about all this, that I missed a memo on where these discussions are happening - what am I missing?

Working in the public is extremely challenging. When the majority of your initial maintainer cohort physically sit in the same office and work for the same company, it's a nearly unavoidable outcome. We tortured the IBM Fabric team in Raleigh by sending Dave Huseby out to sit with them for a week to find ways to both nudge and force them to work in the public, and it paid off. It builds trust, it makes it easier to help people climb the learning curve without being a burden on the maintainers, it helps end users solve their issues, it helps avoid bikeshedding... the list goes on Working across projects gets even harder - the square of how hard it is for one project, perhaps?

This is the kind of community management the TSC could do, if it chose. The quarterly reports are the ideal time to be looking at these resources and asking these questions, and I fear we might be going too quickly through them. It can be done in a way that's intended to be helpful rather than judgemental, but it has to hold the maintainers accountable to a higher standard of what an Open Source project needs to be than any ol github repo on its own. So anyone reading my paragraph above and thinking I'm taking Shawn / Transact / Grid to town, think again - this is on us, collectively, to say what's important, what our standards are, and what we'll support. Those are the objective criteria, that Arnaud is rightfully looking for, that should ultimately be the basis for accepting new projects as TLPs and keeping them there. At the same time, Shawn, I think if these projects were more public in how they work, a lot of the angst on these issues might fade, and we could get back to talking about working across projects.

Brian


-- 
Brian Behlendorf
Executive Director, Hyperledger
bbehlendorf@...
Twitter: @brianbehlendorf


Danno Ferrin
 



On Fri, Sep 4, 2020 at 8:01 PM Brian Behlendorf <bbehlendorf@...> wrote:

And hey, it's 2020, and not even ICO-funded Swiss shell corps have piles of money to spend reinventing wheels and duplicating efforts.

A number of them actually do have the "money."  The next issue they run into is engineering resources: there is a limited supply of engineers that are (a) good enough (b) willing to work in crypto/DLT/blockchain and (c) aren't total narcissists.  Even without a Code of Conduct such behavior gets in the way of successful projects. Not a dig on our CoC, we need to establish boundaries and it provides a touchpoint when boundary enforcement is required. 

Be careful where you look on crypto-twitter and various crypto reddits.  It will make your eyes bleed and our struggles in HL pale by comparison.  No one at HL has sent each other death threats (not hyperbole).  We are not perfect in the way we deal with each other but with that perspective you can see we are doing fairly well.

So for those who want to be involved in those design decisions around Transact - where do they go? Whether I go to the wiki page, the mailing list, or the RocketChat channel, I don't see those discussions, despite those being listed in the Transact README.md as the place to go. The Kanban page isn't a useful way to see design discussions, it's a way to prioritize work already agreed upon. It looks and feels like the actual discussions and work are taking place elsewhere, hidden?, with the community tools used as an after-development way to get feedback from end-users. The cheerful Q3 report doesn't give any clues either. With Grid it's the same deal - I'm quite encouraged by that community call, but again, the Wiki, the mailing list, and the RocketChat channel all leave the impression that the real work is happening elsewhere. To even find the agenda for the Grid meeting I had to go scroll up in the Rocketchat channel, and then, who knows what the results of that meeting were? I want to be wrong about all this, that I missed a memo on where these discussions are happening - what am I missing? 

This brings up an existential question: how much of this collaboration must occur in Hyperledger forums? Some part of it should but how far should we go to force it? For Besu this is not an abstract question.  Basically none of the future of ethereum is planned in Hyperledger forums, instead it's a series of discord servers, message boards, gitter rooms, and twitter threads.  Yes, at least one critical decision was made as a consequence of Twitter threads.  But for better or worse that is the way it is.  Our maintainers can't afford to focus on just HL arenas and must interact outside of the tower to keep up with our mandate of being a mainnet capable enterprise blockchain.

I don't think long term Hyperledger can afford to be a "Not Collaborated Here" venture.  That limits its relevance, scope, and impact tht Hyplereder could provice to the whole DLT/Blockchain/Distributed Computing space.  I strongly feel there needs to be room for projects that are primarily Hyperledger focused, projects that are simply Hyperledger homed, and all the gradients in between.


Duncan Johnston-Watt
 

Danno

I don't think long term Hyperledger can afford to be a "Not Collaborated Here" venture.  That limits its relevance, scope, and impact tht Hyplereder could provice to the whole DLT/Blockchain/Distributed Computing space.  I strongly feel there needs to be room for projects that are primarily Hyperledger focused, projects that are simply Hyperledger homed, and all the gradients in between.

+1

Best

Duncan

On Sat, Sep 5, 2020 at 6:19 AM Danno Ferrin <danno.ferrin@...> wrote:


On Fri, Sep 4, 2020 at 8:01 PM Brian Behlendorf <bbehlendorf@...> wrote:

And hey, it's 2020, and not even ICO-funded Swiss shell corps have piles of money to spend reinventing wheels and duplicating efforts.

A number of them actually do have the "money."  The next issue they run into is engineering resources: there is a limited supply of engineers that are (a) good enough (b) willing to work in crypto/DLT/blockchain and (c) aren't total narcissists.  Even without a Code of Conduct such behavior gets in the way of successful projects. Not a dig on our CoC, we need to establish boundaries and it provides a touchpoint when boundary enforcement is required. 

Be careful where you look on crypto-twitter and various crypto reddits.  It will make your eyes bleed and our struggles in HL pale by comparison.  No one at HL has sent each other death threats (not hyperbole).  We are not perfect in the way we deal with each other but with that perspective you can see we are doing fairly well.

So for those who want to be involved in those design decisions around Transact - where do they go? Whether I go to the wiki page, the mailing list, or the RocketChat channel, I don't see those discussions, despite those being listed in the Transact README.md as the place to go. The Kanban page isn't a useful way to see design discussions, it's a way to prioritize work already agreed upon. It looks and feels like the actual discussions and work are taking place elsewhere, hidden?, with the community tools used as an after-development way to get feedback from end-users. The cheerful Q3 report doesn't give any clues either. With Grid it's the same deal - I'm quite encouraged by that community call, but again, the Wiki, the mailing list, and the RocketChat channel all leave the impression that the real work is happening elsewhere. To even find the agenda for the Grid meeting I had to go scroll up in the Rocketchat channel, and then, who knows what the results of that meeting were? I want to be wrong about all this, that I missed a memo on where these discussions are happening - what am I missing? 

This brings up an existential question: how much of this collaboration must occur in Hyperledger forums? Some part of it should but how far should we go to force it? For Besu this is not an abstract question.  Basically none of the future of ethereum is planned in Hyperledger forums, instead it's a series of discord servers, message boards, gitter rooms, and twitter threads.  Yes, at least one critical decision was made as a consequence of Twitter threads.  But for better or worse that is the way it is.  Our maintainers can't afford to focus on just HL arenas and must interact outside of the tower to keep up with our mandate of being a mainnet capable enterprise blockchain.

I don't think long term Hyperledger can afford to be a "Not Collaborated Here" venture.  That limits its relevance, scope, and impact tht Hyplereder could provice to the whole DLT/Blockchain/Distributed Computing space.  I strongly feel there needs to be room for projects that are primarily Hyperledger focused, projects that are simply Hyperledger homed, and all the gradients in between.



--
Duncan Johnston-Watt
CEO, BTP

Twitter: @duncanjw


Duncan Johnston-Watt
 

James

+1

I can confirm that the Sawtooth 2.0 architecture conversations have been very constructive and people have been willing to adapt the design to accommodate our and others requirements.

Given the state of the world right now we should err on being supportive not adversarial/antagonistic even if that is not the intention. We are watching this very closely as the outcome of this discussion is an important bellwether for the health of Hyperledger community.

Best

Duncan

On Sat, Sep 5, 2020 at 12:17 AM James Barry <james@...> wrote:
Brian and Arnaud,

Without emotion into to it, these things take a while to gestate. I think that Transact is maturing now so that it can be used as a solid external component.  Iroha is enquiring about the use within their 2.0 and testing the framework. Sawtooth 2.0 will be using it as a core component.  Outside of Hyperledger, at this point Scabbard, the default engine for Splinter (not a Hyperledger project) is definitely using Transact.  I believe other DLT’s will use it once it gets a bit more mature and more publicity. 

These things are hard to make reusable as the TSC noted when Bill and I made our componentization pitch to the TSC last Spring.  I do think it would be a good thing for the TSC to look at common component frameworks and see if there is any uptick for the projects.  If so, perhaps a member or two might find it good for a resource to work for 3-4 months to see if it would fit into the top level frameworks.  I was on the call asking the TSC for some fairly straight forward changes so the Chinese encryption algorithm could be used by Fabric.  The pushback surprised me, especially talking about all of the issues around hardware devices etc.  Accommodating the encryption was IMHO a perfect use case for componentization, and a great one for URSA to tackle.  Yes it would take longer, but then Fabric would not be caught flat footed by the next country requiring their own proprietary encryption. So it is this hard level of reuse or accommodation that needs to be looked at, unless Hyperledger is to become a single codebase project.  Brian has some great discussions on how the DLT should all be getting closer together with clients, API’s, I/O’s etc.  Things like Quorum are in use at large Hyperledger members.  Transact had a great vision when it started and “should" be something that a Quorum project is interested in. Because Transact slowed down delivering its original promise, that should be a reason to see how to help it, not fold it in.  Same with URSA frankly.  Starting out as a Labs project might work, so long as the sponsors would be able to find two projects that would lend dedicated programmers to integrate into their project.  

Hyperledger’s TSC could define what is expected from shareable components.  URSA is another one that could use that expectation of the shared components in my opinion. 

Transact had a great vision at the beginning and is now starting to realize that vision.  Let's see how we can help it along and achieve its vision.  Its time for shareable components to emerge for the DLT’s (blockchains) as they were all islands back in 2018.  If we are to get widespread use, they need to start having some reusable pieces, not simply build it and they will come. That fractionalization really killed the overall blockchain momentum and created the “disillusionment” trough that individual walled projects have created. Through projects like Transact/Ursa etc. that sense of an entire framework, not individual projects, enterprises will find DLT’s (blockchains) more mainstream and usable within their environments.

I hope to see Transact and other projects nurtured to help each other.  I would hope that the needs of other projects are added to the architectures as they are being developed so they can be reused throughout the Hyperledger community.  We have been working with Shawn to get Sawtooth 2.x to be accommodating to our needs including putting items in the plans that were not originally in the architecture. And we are not the only ones with needs not originally contemplated by the 2.0 architecture.  Transact could use some solid PR’s or whatever process the TSC thinks will accommodate the acquiring needs of other projects. Through some strong suggestions, we can help mold Transact to be an awesome project. 

Thanks,



On Sep 4, 2020, at 2:04 PM, Arnaud Le Hors <lehors@...> wrote:

Hi Shawn,

While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.

I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.

Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.

Regards.
--
Arnaud  Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM




From:        "Shawn Amundson" <amundson@...>
To:        Brian Behlendorf <bbehlendorf@...>
Cc:        "hyperledger-tsc@..." <hyperledger-tsc@...>
Date:        09/04/2020 08:49 PM
Subject:        [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by:        tsc@...




On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).

I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team.  The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?

...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.

While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.

FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
 
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.

1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...

"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
 
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.

-Shawn






--
Duncan Johnston-Watt
CEO, BTP

Twitter: @duncanjw


Arun S M
 

Brian, Arnaud and Shawn,

I am amazed looking at the way this email conversation is going on.
Here is what I see happening in the thread. It is still the community having healthy discussion. But wrong choice of words and examples, mixing up with the emotions, is taking it to another direction.

Arnaud/TSC: Is making an effort towards defining TLP in Hyperledger. Has a proposal where possibly two top level projects are merged into one. This is up for discussion and comments / views are still welcome on it.

- Thanks for the good summary. I will leave my comments to you at the end.

Brian: Took Transact as an example, because of lack visibility into what's happening there. The reasons such as most of the maintainers are co-located, the information on feature development or the future of the project is not visible in public are influencing the thoughts. The reason for lack of visibility could also be that resources are not linked right. Used the word `unachieved` because he has not seen a result of what was initially agreed upon when the project was proposed.

- I agree to Brian's comments. We should look at the core of the problem and address it first before touching the surface. By patch decisions, things can only become more complicated in future. In this case, I see 2 possible problems which can be solved. (1) Decide when a project should get the status of incubation in TLP. (2) How to solve the issue of different TLPs linked to one. I have my comments for the issue (1) in the proposal. For the case (2), we can possibly define metrics to measure the projects. Having well defined processes and indications to the maintainers makes sense. Any decision taken by the TSC should not come as a surprise to any of the projects.

Shawn: Understands the reason for having Transact as a TLP. Is involved in discussions for multiple DLT solutions to reuse the Transact library. Went onto email emotionally because others spoke about the project without knowing these.

- You have given enough indications on usability of the project. This email should be a good record as an answer to all such speculations, that Transact is only used in one project. I know you are working on a website for Sawtooth. How about one for Transact? Or how about making any document / discussion happen in public? I understand that you're making efforts towards that. But I still feel giving out more information in public can solve most of such issues.

I don't see anybody's fault, arrogance or ignorance here.
In short, the discussion is drifting away, because this is about a particular solution instead it can talk about the actual problem and ask for more possible solutions.

Arnaud: This is my take on the issue as I always like to fix the issue to its core, I can only request you, do not take these words otherwise. There is always a high risk of negative perception due to choice of words in the virtual world. How about a dedicated TSC call on the topic of "Collaboration in Hyperledger"? You can ask maintainers of different projects to join. Let everybody give their opinion on what it means for them, what are their objectives, what is that they are looking forward to. Bring up the issue that is making you think "merge different TLP into one". Let this merger be one of the possible solutions and not the only solution to solve that bigger core issue.


Regards,
Arun

On Sat, Sep 5, 2020 at 7:18 PM Duncan Johnston-Watt <duncan@...> wrote:
James

+1

I can confirm that the Sawtooth 2.0 architecture conversations have been very constructive and people have been willing to adapt the design to accommodate our and others requirements.

Given the state of the world right now we should err on being supportive not adversarial/antagonistic even if that is not the intention. We are watching this very closely as the outcome of this discussion is an important bellwether for the health of Hyperledger community.

Best

Duncan

On Sat, Sep 5, 2020 at 12:17 AM James Barry <james@...> wrote:
Brian and Arnaud,

Without emotion into to it, these things take a while to gestate. I think that Transact is maturing now so that it can be used as a solid external component.  Iroha is enquiring about the use within their 2.0 and testing the framework. Sawtooth 2.0 will be using it as a core component.  Outside of Hyperledger, at this point Scabbard, the default engine for Splinter (not a Hyperledger project) is definitely using Transact.  I believe other DLT’s will use it once it gets a bit more mature and more publicity. 

These things are hard to make reusable as the TSC noted when Bill and I made our componentization pitch to the TSC last Spring.  I do think it would be a good thing for the TSC to look at common component frameworks and see if there is any uptick for the projects.  If so, perhaps a member or two might find it good for a resource to work for 3-4 months to see if it would fit into the top level frameworks.  I was on the call asking the TSC for some fairly straight forward changes so the Chinese encryption algorithm could be used by Fabric.  The pushback surprised me, especially talking about all of the issues around hardware devices etc.  Accommodating the encryption was IMHO a perfect use case for componentization, and a great one for URSA to tackle.  Yes it would take longer, but then Fabric would not be caught flat footed by the next country requiring their own proprietary encryption. So it is this hard level of reuse or accommodation that needs to be looked at, unless Hyperledger is to become a single codebase project.  Brian has some great discussions on how the DLT should all be getting closer together with clients, API’s, I/O’s etc.  Things like Quorum are in use at large Hyperledger members.  Transact had a great vision when it started and “should" be something that a Quorum project is interested in. Because Transact slowed down delivering its original promise, that should be a reason to see how to help it, not fold it in.  Same with URSA frankly.  Starting out as a Labs project might work, so long as the sponsors would be able to find two projects that would lend dedicated programmers to integrate into their project.  

Hyperledger’s TSC could define what is expected from shareable components.  URSA is another one that could use that expectation of the shared components in my opinion. 

Transact had a great vision at the beginning and is now starting to realize that vision.  Let's see how we can help it along and achieve its vision.  Its time for shareable components to emerge for the DLT’s (blockchains) as they were all islands back in 2018.  If we are to get widespread use, they need to start having some reusable pieces, not simply build it and they will come. That fractionalization really killed the overall blockchain momentum and created the “disillusionment” trough that individual walled projects have created. Through projects like Transact/Ursa etc. that sense of an entire framework, not individual projects, enterprises will find DLT’s (blockchains) more mainstream and usable within their environments.

I hope to see Transact and other projects nurtured to help each other.  I would hope that the needs of other projects are added to the architectures as they are being developed so they can be reused throughout the Hyperledger community.  We have been working with Shawn to get Sawtooth 2.x to be accommodating to our needs including putting items in the plans that were not originally in the architecture. And we are not the only ones with needs not originally contemplated by the 2.0 architecture.  Transact could use some solid PR’s or whatever process the TSC thinks will accommodate the acquiring needs of other projects. Through some strong suggestions, we can help mold Transact to be an awesome project. 

Thanks,



On Sep 4, 2020, at 2:04 PM, Arnaud Le Hors <lehors@...> wrote:

Hi Shawn,

While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.

I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.

Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.

Regards.
--
Arnaud  Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM




From:        "Shawn Amundson" <amundson@...>
To:        Brian Behlendorf <bbehlendorf@...>
Cc:        "hyperledger-tsc@..." <hyperledger-tsc@...>
Date:        09/04/2020 08:49 PM
Subject:        [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by:        tsc@...




On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).

I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team.  The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?

...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.

While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.

FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
 
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.

1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...

"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
 
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.

-Shawn






--
Duncan Johnston-Watt
CEO, BTP

Twitter: @duncanjw


James Barry
 

+1 on Arun entire recap

I believe the entire issue needs a start with Arun’s excellent suggestion:

 How about a dedicated TSC call on the topic of "Collaboration in Hyperledger"? You can ask maintainers of different projects to join. Let everybody give their opinion on what it means for them, what are their objectives, what is that they are looking forward to.

Frankly I am on the outside looking in.  I see a lot of projects that are each going their own way.  There is a central question as to what the maintainers think “Hyperledger.org” should be.  The discussion proposed by Arun is an excellent way too begin looking at this question. Most open source software organizations are built around helping foster community, infrastructure etc.  Hyperledger is unique in that it is part of the Linux Foundation which provides much of that, so it can concentrate on the greater blockchain community, and work across communities - that is why Brian is perfect to lead the project.. 

Right now from the outside looking in, Hyperledger is thought of as a Fabric organization with some add on’s. People are surprised when I mention we use a DLT - Sawtooth, and that tools like Caliper can be used by any blockchain. I think that the Top Level Projects that exist today have a mission that would/will greatly advance the blockchain community as a whole.  The central question is if another Hyperledger Top Level project (especially the “elephant” in the room- Fabric) don’t use a shared TLP, should they actually be a TLP?  Or should these projects be able to work and collaborate around the blockchain community as a whole while “anchoring" themselves in Hyperledger? Success would then have a different dimension to measure than simply "use by others in the Hyperledger community"? That I believe is the question that Arun would drive to in his maintainer meeting, and what Danno and Duncan mean when they mention about Hyperledger working and collaborating with other communities.  Let the maintainers discuss their motivations is an excellent first start.

My personal thought on this issue, is that Hyperledger’s structure and leadership can find, nature and bring forward projects that the entire blockchain community cannot bring forward, because Hyperledger does not have the strong monetary foundations that many of these blockchain projects were spawned with.  Hyperledger should serve as the catalyst to propose projects that will help the entire community.  Hyperledger is the only blockchain community that can do this without financial considerations clouding the direction.   

To the original thought of combining TLP’s, I think that defeats the purpose of having libraries that can be used through the entire blockchain system.  If a project is to succeed at that audacious goal to be a “core library" like Transact or Ursa it takes time, community and help that I am sure Hyperledger can provide.  The key is giving them time to mature and attract developers to their cause. To that effect Transact is getting some traction that will expand the core development base as the project matures.

Thank you for listening,

James Barry
Cell 303-956-8835
james@...
http://www.linkedin.com/in/jamesbarry




On Sep 6, 2020, at 12:29 PM, Arun S M <arun.s.m.cse@...> wrote:

Brian, Arnaud and Shawn,

I am amazed looking at the way this email conversation is going on.
Here is what I see happening in the thread. It is still the community having healthy discussion. But wrong choice of words and examples, mixing up with the emotions, is taking it to another direction.

Arnaud/TSC: Is making an effort towards defining TLP in Hyperledger. Has a proposal where possibly two top level projects are merged into one. This is up for discussion and comments / views are still welcome on it.

- Thanks for the good summary. I will leave my comments to you at the end.

Brian: Took Transact as an example, because of lack visibility into what's happening there. The reasons such as most of the maintainers are co-located, the information on feature development or the future of the project is not visible in public are influencing the thoughts. The reason for lack of visibility could also be that resources are not linked right. Used the word `unachieved` because he has not seen a result of what was initially agreed upon when the project was proposed.

- I agree to Brian's comments. We should look at the core of the problem and address it first before touching the surface. By patch decisions, things can only become more complicated in future. In this case, I see 2 possible problems which can be solved. (1) Decide when a project should get the status of incubation in TLP. (2) How to solve the issue of different TLPs linked to one. I have my comments for the issue (1) in the proposal. For the case (2), we can possibly define metrics to measure the projects. Having well defined processes and indications to the maintainers makes sense. Any decision taken by the TSC should not come as a surprise to any of the projects.

Shawn: Understands the reason for having Transact as a TLP. Is involved in discussions for multiple DLT solutions to reuse the Transact library. Went onto email emotionally because others spoke about the project without knowing these.

- You have given enough indications on usability of the project. This email should be a good record as an answer to all such speculations, that Transact is only used in one project. I know you are working on a website for Sawtooth. How about one for Transact? Or how about making any document / discussion happen in public? I understand that you're making efforts towards that. But I still feel giving out more information in public can solve most of such issues.

I don't see anybody's fault, arrogance or ignorance here.
In short, the discussion is drifting away, because this is about a particular solution instead it can talk about the actual problem and ask for more possible solutions.

Arnaud: This is my take on the issue as I always like to fix the issue to its core, I can only request you, do not take these words otherwise. There is always a high risk of negative perception due to choice of words in the virtual world. How about a dedicated TSC call on the topic of "Collaboration in Hyperledger"? You can ask maintainers of different projects to join. Let everybody give their opinion on what it means for them, what are their objectives, what is that they are looking forward to. Bring up the issue that is making you think "merge different TLP into one". Let this merger be one of the possible solutions and not the only solution to solve that bigger core issue.


Regards,
Arun

On Sat, Sep 5, 2020 at 7:18 PM Duncan Johnston-Watt <duncan@...> wrote:
James

+1

I can confirm that the Sawtooth 2.0 architecture conversations have been very constructive and people have been willing to adapt the design to accommodate our and others requirements.

Given the state of the world right now we should err on being supportive not adversarial/antagonistic even if that is not the intention. We are watching this very closely as the outcome of this discussion is an important bellwether for the health of Hyperledger community.

Best

Duncan

On Sat, Sep 5, 2020 at 12:17 AM James Barry <james@...> wrote:
Brian and Arnaud,

Without emotion into to it, these things take a while to gestate. I think that Transact is maturing now so that it can be used as a solid external component.  Iroha is enquiring about the use within their 2.0 and testing the framework. Sawtooth 2.0 will be using it as a core component.  Outside of Hyperledger, at this point Scabbard, the default engine for Splinter (not a Hyperledger project) is definitely using Transact.  I believe other DLT’s will use it once it gets a bit more mature and more publicity. 

These things are hard to make reusable as the TSC noted when Bill and I made our componentization pitch to the TSC last Spring.  I do think it would be a good thing for the TSC to look at common component frameworks and see if there is any uptick for the projects.  If so, perhaps a member or two might find it good for a resource to work for 3-4 months to see if it would fit into the top level frameworks.  I was on the call asking the TSC for some fairly straight forward changes so the Chinese encryption algorithm could be used by Fabric.  The pushback surprised me, especially talking about all of the issues around hardware devices etc.  Accommodating the encryption was IMHO a perfect use case for componentization, and a great one for URSA to tackle.  Yes it would take longer, but then Fabric would not be caught flat footed by the next country requiring their own proprietary encryption. So it is this hard level of reuse or accommodation that needs to be looked at, unless Hyperledger is to become a single codebase project.  Brian has some great discussions on how the DLT should all be getting closer together with clients, API’s, I/O’s etc.  Things like Quorum are in use at large Hyperledger members.  Transact had a great vision when it started and “should" be something that a Quorum project is interested in. Because Transact slowed down delivering its original promise, that should be a reason to see how to help it, not fold it in.  Same with URSA frankly.  Starting out as a Labs project might work, so long as the sponsors would be able to find two projects that would lend dedicated programmers to integrate into their project.  

Hyperledger’s TSC could define what is expected from shareable components.  URSA is another one that could use that expectation of the shared components in my opinion. 

Transact had a great vision at the beginning and is now starting to realize that vision.  Let's see how we can help it along and achieve its vision.  Its time for shareable components to emerge for the DLT’s (blockchains) as they were all islands back in 2018.  If we are to get widespread use, they need to start having some reusable pieces, not simply build it and they will come. That fractionalization really killed the overall blockchain momentum and created the “disillusionment” trough that individual walled projects have created. Through projects like Transact/Ursa etc. that sense of an entire framework, not individual projects, enterprises will find DLT’s (blockchains) more mainstream and usable within their environments.

I hope to see Transact and other projects nurtured to help each other.  I would hope that the needs of other projects are added to the architectures as they are being developed so they can be reused throughout the Hyperledger community.  We have been working with Shawn to get Sawtooth 2.x to be accommodating to our needs including putting items in the plans that were not originally in the architecture. And we are not the only ones with needs not originally contemplated by the 2.0 architecture.  Transact could use some solid PR’s or whatever process the TSC thinks will accommodate the acquiring needs of other projects. Through some strong suggestions, we can help mold Transact to be an awesome project. 

Thanks,



On Sep 4, 2020, at 2:04 PM, Arnaud Le Hors <lehors@...> wrote:

Hi Shawn,

While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.

I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.

Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.

Regards.
--
Arnaud  Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM




From:        "Shawn Amundson" <amundson@...>
To:        Brian Behlendorf <bbehlendorf@...>
Cc:        "hyperledger-tsc@..." <hyperledger-tsc@...>
Date:        09/04/2020 08:49 PM
Subject:        [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by:        tsc@...




On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).

I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team.  The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?

...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.

While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.

FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
 
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.

1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...

"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response. 
 
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.

-Shawn








-- 
Duncan Johnston-Watt
CEO, BTP

Twitter: @duncanjw




Arnaud Le Hors
 

Thank you all for contributing to the discussion. I think the case for Transact to continue as is has been clearly made. I will note that the quarterly project reports are the main means of communication with the TSC on the status of each project and it would have been helpful to get some of the information that has benn provided in this thread in response to the comments made on the Transact report: https://wiki.hyperledger.org/display/TSC/2020+Q3+Hyperledger+Transact

James,
I start every TSC call with a statement about the call being open to all to listen in and contribute. This is not a figure of speech and I can only encourage you to speak up if/when you think the TSC is missing something. This is of course true for everyone. We'll all be better off.

As for Arun's proposal, I'm certainly open to it. Unfortunately we've not had any maintainers meeting in a long time, in part due to the COVID situation, and generally speaking discussions between maintainers across projects has been rather limited. The maintainers list is mostly dead. There is however evidence of point to point collaboration going on and this is probably the most important part. Because of the way it is happening this isn't necessarily visible to the rest of the community unfortunately.
--
Arnaud  Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM




From:        "James Barry" <james@...>
To:        Arun S M <arun.s.m.cse@...>
Cc:        Duncan Johnston-Watt <duncan@...>, Arnaud Le Hors <lehors@...>, Shawn Amundson <amundson@...>, Brian Behlendorf <bbehlendorf@...>, "hyperledger-tsc@..." <hyperledger-tsc@...>, Hyperledger List <tsc@...>
Date:        09/07/2020 06:28 AM
Subject:        [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by:        tsc@...




+1 on Arun entire recap

I believe the entire issue needs a start with Arun’s excellent suggestion:

 How about a dedicated TSC call on the topic of "Collaboration in Hyperledger"? You can ask maintainers of different projects to join. Let everybody give their opinion on what it means for them, what are their objectives, what is that they are looking forward to.

Frankly I am on the outside looking in.  I see a lot of projects that are each going their own way.  There is a central question as to what the maintainers think “Hyperledger.org” should be.  The discussion proposed by Arun is an excellent way too begin looking at this question. Most open source software organizations are built around helping foster community, infrastructure etc.  Hyperledger is unique in that it is part of the Linux Foundation which provides much of that, so it can concentrate on the greater blockchain community, and work across communities - that is why Brian is perfect to lead the project..

Right now from the outside looking in, Hyperledger is thought of as a Fabric organization with some add on’s. People are surprised when I mention we use a DLT - Sawtooth, and that tools like Caliper can be used by any blockchain. I think that the Top Level Projects that exist today have a mission that would/will greatly advance the blockchain community as a whole.  The central question is if another Hyperledger Top Level project (especially the “elephant” in the room- Fabric) don’t use a shared TLP, should they actually be a TLP?  Or should these projects be able to work and collaborate around the blockchain community as a whole while “anchoring" themselves in Hyperledger? Success would then have a different dimension to measure than simply "use by others in the Hyperledger community"? That I believe is the question that Arun would drive to in his maintainer meeting, and what Danno and Duncan mean when they mention about Hyperledger working and collaborating with other communities.  Let the maintainers discuss their motivations is an excellent first start.

My personal thought on this issue, is that Hyperledger’s structure and leadership can find, nature and bring forward projects that the entire blockchain community cannot bring forward, because Hyperledger does not have the strong monetary foundations that many of these blockchain projects were spawned with.  Hyperledger should serve as the catalyst to propose projects that will help the entire community.  Hyperledger is the only blockchain community that can do this without financial considerations clouding the direction.  

To the original thought of combining TLP’s, I think that defeats the purpose of having libraries that can be used through the entire blockchain system.  If a project is to succeed at that audacious goal to be a “core library" like Transact or Ursa it takes time, community and help that I am sure Hyperledger can provide.  The key is giving them time to mature and attract developers to their cause. To that effect Transact is getting some traction that will expand the core development base as the project matures.

Thank you for listening,

James Barry
Cell 303-956-8835

james@...
http://www.linkedin.com/in/jamesbarry




On Sep 6, 2020, at 12:29 PM, Arun S M <arun.s.m.cse@...> wrote:

Brian, Arnaud and Shawn,

I am amazed looking at the way this email conversation is going on.
Here is what I see happening in the thread. It is still the community having healthy discussion. But wrong choice of words and examples, mixing up with the emotions, is taking it to another direction.

Arnaud/TSC:Is making an effort towards defining TLP in Hyperledger. Has a proposal where possibly two top level projects are merged into one. This is up for discussion and comments / views are still welcome on it.

- Thanks for the good summary. I will leave my comments to you at the end.

Brian: Took Transact as an example, because of lack visibility into what's happening there. The reasons such as most of the maintainers are co-located, the information on feature development or the future of the project is not visible in public are influencing the thoughts. The reason for lack of visibility could also be that resources are not linked right. Used the word `unachieved` because he has not seen a result of what was initially agreed upon when the project was proposed.

- I agree to Brian's comments. We should look at the core of the problem and address it first before touching the surface. By patch decisions, things can only become more complicated in future. In this case, I see 2 possible problems which can be solved. (1) Decide when a project should get the status of incubation in TLP. (2) How to solve the issue of different TLPs linked to one. I have my comments for the issue (1) in the proposal. For the case (2), we can possibly define metrics to measure the projects. Having well defined processes and indications to the maintainers makes sense. Any decision taken by the TSC should not come as a surprise to any of the projects.

Shawn: Understands the reason for having Transact as a TLP. Is involved in discussions for multiple DLT solutions to reuse the Transact library. Went onto email emotionally because others spoke about the project without knowing these.

- You have given enough indications on usability of the project. This email should be a good record as an answer to all such speculations, that Transact is only used in one project. I know you are working on a website for Sawtooth. How about one for Transact? Or how about making any document / discussion happen in public? I understand that you're making efforts towards that. But I still feel giving out more information in public can solve most of such issues.

I don't see anybody's fault, arrogance or ignorance here.
In short, the discussion is drifting away, because this is about a particular solution instead it can talk about the actual problem and ask for more possible solutions.

Arnaud:This is my take on the issue as I always like to fix the issue to its core, I can only request you, do not take these words otherwise. There is always a high risk of negative perception due to choice of words in the virtual world. How about a dedicated TSC call on the topic of "Collaboration in Hyperledger"? You can ask maintainers of different projects to join. Let everybody give their opinion on what it means for them, what are their objectives, what is that they are looking forward to. Bring up the issue that is making you think "merge different TLP into one". Let this merger be one of the possible solutions and not the only solution to solve that bigger core issue.


Regards,
Arun

On Sat, Sep 5, 2020 at 7:18 PM Duncan Johnston-Watt <duncan@...> wrote:
James

+1

I can confirm that the Sawtooth 2.0 architecture conversations have been very constructive and people have been willing to adapt the design to accommodate our and others requirements.

Given the state of the world right now we should err on being supportive not adversarial/antagonistic even if that is not the intention. We are watching this very closely as the outcome of this discussion is an important bellwether for the health of Hyperledger community.

Best

Duncan


On Sat, Sep 5, 2020 at 12:17 AM James Barry <james@...> wrote:
Brian and Arnaud,

Without emotion into to it, these things take a while to gestate. I think that Transact is maturing now so that it can be used as a solid external component.  Iroha is enquiring about the use within their 2.0 and testing the framework. Sawtooth 2.0 will be using it as a core component.  Outside of Hyperledger, at this point Scabbard, the default engine for Splinter (not a Hyperledger project) is definitely using Transact.  I believe other DLT’s will use it once it gets a bit more mature and more publicity.

These things are hard to make reusable as the TSC noted when Bill and I made our componentization pitch to the TSC last Spring.  I do think it would be a good thing for the TSC to look at common component frameworks and see if there is any uptick for the projects.  If so, perhaps a member or two might find it good for a resource to work for 3-4 months to see if it would fit into the top level frameworks.  I was on the call asking the TSC for some fairly straight forward changes so the Chinese encryption algorithm could be used by Fabric.  The pushback surprised me, especially talking about all of the issues around hardware devices etc.  Accommodating the encryption was IMHO a perfect use case for componentization, and a great one for URSA to tackle.  Yes it would take longer, but then Fabric would not be caught flat footed by the next country requiring their own proprietary encryption. So it is this hard level of reuse or accommodation that needs to be looked at, unless Hyperledger is to become a single codebase project.  Brian has some great discussions on how the DLT should all be getting closer together with clients, API’s, I/O’s etc.  Things like Quorum are in use at large Hyperledger members.  Transact had a great vision when it started and “should" be something that a Quorum project is interested in. Because Transact slowed down delivering its original promise, that should be a reason to see how to help it, not fold it in.  Same with URSA frankly.  Starting out as a Labs project might work, so long as the sponsors would be able to find two projects that would lend dedicated programmers to integrate into their project.  

Hyperledger’s TSC could define what is expected from shareable components.  URSA is another one that could use that expectation of the shared components in my opinion.

Transact had a great vision at the beginning and is now starting to realize that vision.  Let's see how we can help it along and achieve its vision.  Its time for shareable components to emerge for the DLT’s (blockchains) as they were all islands back in 2018.  If we are to get widespread use, they need to start having some reusable pieces, not simply build it and they will come. That fractionalization really killed the overall blockchain momentum and created the “disillusionment” trough that individual walled projects have created. Through projects like Transact/Ursa etc. that sense of an entire framework, not individual projects, enterprises will find DLT’s (blockchains) more mainstream and usable within their environments.

I hope to see Transact and other projects nurtured to help each other.  I would hope that the needs of other projects are added to the architectures as they are being developed so they can be reused throughout the Hyperledger community.  We have been working with Shawn to get Sawtooth 2.x to be accommodating to our needs including putting items in the plans that were not originally in the architecture. And we are not the only ones with needs not originally contemplated by the 2.0 architecture.  Transact could use some solid PR’s or whatever process the TSC thinks will accommodate the acquiring needs of other projects. Through some strong suggestions, we can help mold Transact to be an awesome project.

Thanks,


James Barry
Cell 303-956-8835

james@...
http://www.linkedin.com/in/jamesbarry




On Sep 4, 2020, at 2:04 PM, Arnaud Le Hors <lehors@...> wrote:

Hi Shawn,

While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.


I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.


Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.


Regards.
--
Arnaud  Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM





From:        
"Shawn Amundson" <amundson@...>
To:        
Brian Behlendorf <bbehlendorf@...>
Cc:        
"hyperledger-tsc@..." <hyperledger-tsc@...>
Date:        
09/04/2020 08:49 PM
Subject:        
[EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by:        
tsc@...




On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <
bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).


I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team.  The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?


...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.


While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.


FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.

...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.


1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...


"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.

No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.


-Shawn







--


Duncan Johnston-Watt
CEO, BTP

Twitter: @duncanjw
Mob: +44 777 190 2653
LinkedIn: https://linkedin.com/in/duncanjohnstonwatt








Shawn Amundson
 

On Fri, Sep 4, 2020 at 9:01 PM Brian Behlendorf <bbehlendorf@...> wrote:
...
So for those who want to be involved in those design decisions around Transact - where do they go? Whether I go to the wiki page, the mailing list, or the RocketChat channel, I don't see those discussions, despite those being listed in the Transact README.md as the place to go. The Kanban page isn't a useful way to see design discussions, it's a way to prioritize work already agreed upon. It looks and feels like the actual discussions and work are taking place elsewhere, hidden?, with the community tools used as an after-development way to get feedback from end-users. The cheerful Q3 report doesn't give any clues either. With Grid it's the same deal - I'm quite encouraged by that community call, but again, the Wiki, the mailing list, and the RocketChat channel all leave the impression that the real work is happening elsewhere. To even find the agenda for the Grid meeting I had to go scroll up in the Rocketchat channel, and then, who knows what the results of that meeting were? I want to be wrong about all this, that I missed a memo on where these discussions are happening - what am I missing

...

RocketChat is definitely the best place, because the maintainers are generally present there.

For Transact, it does end up being a series of very specific discussions usually between maintainers (often 1:1), and usually in the context of trying to achieve something in another project. The bulk of the discussion then turns to PR review, which is where all the maintainers get involved. In addition, recently, there has been a fair amount of Transact discussion on the Sawtooth contributors meetings because of the Sawtooth+Transact integration work currently in progress. These are really more discussions about Sawtooth than Transact, but the occasional item turns into a feature for Transact -- like a strong desire to add backward-compatible transaction processor support to Transact (as a community-driven requirement for Sawtooth 2).

Grid is much different. A lot of the engagement is centered around RFCs - for example, there are 100 comments on the Location RFC - https://github.com/hyperledger/grid-rfcs/pull/20. In the community meeting, we gave an opportunity for discussion of upcoming RFCs. There is also a lot of work going on currently to expose a lot more information through the documentation.

-Shawn
 


clive boulton
 

For Transact, Grid, and Sawtooth core. I'd like to attest that I track and follow along, and feel part of the open-source software design and development process. I keep an eye on the named RocketChat channels for upcoming design 1-2+ hour meetings. Same for Cactus who has more time sandboxed meetings of about 1 hour. Same for Fabric Private Chain Code (PDOs). It does take some effort for me to engage. I motivated in cross-pollinating or contributing to distributed resilient secure software engineering for applications (Mark S. Millers body of work on eRights: ES/JS, Node, Wasm). Which I feel welcome to do by remarking on RFCs, PRs, and ongoing design discussions.

- Clive Boulton 
Hyperledger architecture working group member. 
       

On Tue, Sep 8, 2020 at 12:11 PM Shawn Amundson <amundson@...> wrote:
On Fri, Sep 4, 2020 at 9:01 PM Brian Behlendorf <bbehlendorf@...> wrote:
...
So for those who want to be involved in those design decisions around Transact - where do they go? Whether I go to the wiki page, the mailing list, or the RocketChat channel, I don't see those discussions, despite those being listed in the Transact README.md as the place to go. The Kanban page isn't a useful way to see design discussions, it's a way to prioritize work already agreed upon. It looks and feels like the actual discussions and work are taking place elsewhere, hidden?, with the community tools used as an after-development way to get feedback from end-users. The cheerful Q3 report doesn't give any clues either. With Grid it's the same deal - I'm quite encouraged by that community call, but again, the Wiki, the mailing list, and the RocketChat channel all leave the impression that the real work is happening elsewhere. To even find the agenda for the Grid meeting I had to go scroll up in the Rocketchat channel, and then, who knows what the results of that meeting were? I want to be wrong about all this, that I missed a memo on where these discussions are happening - what am I missing

...

RocketChat is definitely the best place, because the maintainers are generally present there.

For Transact, it does end up being a series of very specific discussions usually between maintainers (often 1:1), and usually in the context of trying to achieve something in another project. The bulk of the discussion then turns to PR review, which is where all the maintainers get involved. In addition, recently, there has been a fair amount of Transact discussion on the Sawtooth contributors meetings because of the Sawtooth+Transact integration work currently in progress. These are really more discussions about Sawtooth than Transact, but the occasional item turns into a feature for Transact -- like a strong desire to add backward-compatible transaction processor support to Transact (as a community-driven requirement for Sawtooth 2).

Grid is much different. A lot of the engagement is centered around RFCs - for example, there are 100 comments on the Location RFC - https://github.com/hyperledger/grid-rfcs/pull/20. In the community meeting, we gave an opportunity for discussion of upcoming RFCs. There is also a lot of work going on currently to expose a lot more information through the documentation.

-Shawn