Transact as a TLP
In Rocketchat, Shawn Amundson had this to say about the "Rollover of projects tied to a single other project" proposal:
It would be helpful to be specific about the intent in relation to existing projects. A policy that forces X project and Y project to merge may be equivalent to ejecting X project or Y project from HL. Such a move should be explicit and discussed. As it is now, it sounds like malicious maneuvering against projects that specific TSC members aren't directly invested in. TSC should focus instead on fostering collaboration instead of attacking projects. "If you don't write your project report the right way, we will immediately start to talk about 'sanctions' or removing the project." isn't collaborative.
Shawn is right. Conversations that try to address subjective
disagreements with objective rulemaking can come across as
attempts to legislatively solve disputes and issues that are
actually more interpersonal in nature and that almost always leads
to bad laws/rules. Rather than talking about new rules to try and
tackle what is actually a fairly specific disagreement and
possibly a niche situation, it might help to try and explain how
we got here, and use that as a basis to figure out a better path
out.
As I retell this, please be kind to my interpretation. I'll cite
links when I can find them, but much of this covers conversations
and events happening on phone calls, DMs, personal emails, etc, as
well as my unavoidably subjective read of the situation. I'll try
and state interpretations as plainly as I can, and avoid imputing
motives, so apologies if I slip on that. If you feel I got
something wrong, feel free to let me or the list know, but please
assume benign ignorance of details rather than maliciousness on my
part.
Transact was proposed
as a top-level project at a pretty early stage in its life, likely
long before any project would be accepted today. No need to
revisit that decision,
but to remember its rationale - that there was a strong desire
across the TSC (it seemed) for a common, unified approach to allow
for portability of smart contract engines across an array of
different ledgers. Such an approach, of course, is never
guaranteed to work - there was no small amount of optimism, risk,
and hope in that goal, and it requires a ton of communication,
architectural humility, and looking for common ground before
writing code. Trying to define a common dividing line across
systems as different in their architectures as Fabric, Sawtooth,
and Iroha (let alone Besu, which hadn't shown up yet) wasn't even
clear if it was possible, or if the compromises it would require
wouldn't have unacceptable costs. But there was a sincere intent
shown in the Transact proposal to explore this domain, and the
proposed ideas seemed worth pursuing, so much so that the TSC
relied upon them to make its decision to approve. You can see
mention of it in the original
proposal and discussion on TSC calls that led to an
email-based vote. I'd say the enthusiasm for getting this right
across HL projects was part of the reason people went in so early,
as it's much harder to adapt once that first architecture is laid
out. While I was not at the Maintainer Summit in Minnesota, I
recall hearing there were very promising conversations between
Sawtooth and Fabric developers and others about the possible ways
for Transact to become that kind of project. So I view the TSC's
approval of Transact as not just an endorsement of the principle -
hey, sounds good, someone aught to do it - but a reflection of
sincere intent on behalf of the other DLTs to invest time into a
shared approach. That was not "we'll wait and see", but "this is
good enough, let's start to work together."
But then... that enthusiasm seemed to fizzle. Like any such
collaborative process, it can be awfully hard to try and figure
out why projects don't make that leap from great initial hopes to
continued participation. Should the Fabric team have bent their
plans for their 2.0 architecture more? Should Transact's initial
architecture have changed more? Were Iroha even asked to
participate? Even if we couldn't figure it out in 2019, could we
figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc
timeframe? I imagine there'll be as many different reasons why
things went south as there were people engaged. There's really no
need to assign blame here - we've long believed strongly that
maintainers should set the technical course for their projects,
unless they royally screw up - but we should acknowledge that
collectively this effort hasn't lived up to that promise, and that
promise was the basis for the vote in favor of Transact as a TLP.
Collaboration is hard and it requires all sides to make extra
effort beyond scratching their own itches (which is a metaphor for
open source I've long despised!). It's a challenge to prioritize
that over customer-paying work, let alone the usual technical
differences of opinion on The Way To Build A DLT. It requires
people realizing they have more to gain from a common platform
than lose from that extra work. That requires a leap of faith and
trust between individuals. Sometimes, that just can't be willed
into existence. Or maybe everyone just realized one API to rule
them all was too hard, asked for too many compromises to be
useful. One sign of this is that Digital Asset went to some
lengths to port their DAML engine across Fabric, Sawtooth, Corda
and I think Quorum, and while that wasn't cheap I'm sure, it
suggested it wasn't a show-stopper either.
Today, if Transact had been proposed, we probably would have said
"let's start it in Labs" first, and given it time to make measured
progress against that objective, before approving it as a TLP.
This is the path that Cactus and Avalon have gone through,
successfully, and there are a lot of great ideas in Labs that
haven't yet achieved their initial goals and many that won't. Had
Transact started in labs, and still arrived to where it was today,
my personal view is that it likely would not be accepted as a top
level project, given how core it still seems to Sawtooth. I
believe we were concerned about having a production-grade DLT like
Sawtooth depending upon a Labs project - which could have been
solved by making it a "sawtooth-transact" repo.
That's what, I believe, is at the heart of the issue for several
people on the TSC and in the community - a history that is more
personal than can be fixed by a rule like "when to combine
projects", which is really "when to combine projects against the
will of their maintainers". It doesn't sound like the maintainers
of either Sawtooth or Transact want them to be combined. But at
the same time, the existence of Transact as a top level project is
a reminder of that failed effort, a totem for our difficulties in
collaborating across projects.
To be completely fair to Transact, as an individual project it
has by no means failed. I don't know if there are production
systems running on it yet, but objectively there are commits and
code and docs and all the trappings of a competently managed
source tree. Its last release was version 0.3.1 just 12 hours ago.
There are commits to the repo from more than just Bitwise
employees this past year, according
to the dashboard. So let's think instead about how to
respect the effort those devs are putting in, an approach that
helps them and Transact get closer to its original goal, or if
that goal no longer applies, provides a dignified alternative
path.
There are three possible fixes I see based on this history:
1) Transact maintainers declare that they're still interested in
that original goal, and are willing to put in the work to adapt
their current architectural plans to meet the needs of others.
Maintainers on other projects also commit to further discussions
and efforts to use Transact on their platforms as appropriate.
Meetings to discuss how each DLT takes steps in their next major
release to allow them to use Transact, or at least narrow the gap.
Transact might still take a few years to achieve that goal, but it
remains a top level project so long as those attempts continue.
2) Transact maintainers declare that they're still interested in
that original goal, and are willing to put in the work to adapt
their current architectural plans to meet the needs of others.
BUT, maintainers on other projects can't find the time or
inclination to sincerely take part in that process. In that case,
Transact remains a TLP, because they should not be penalized if
other projects can't be bothered to meet them halfway. Even just
one such DLT stepping in would be enough.
3) Transact maintainers declare that other projects are more than welcome to use the architecture they've developed, but they have their own priorities and opinions on how things should be built, take them or leave them. Many projects are successful with that perspective - including the Linux kernel! - so that's not an incorrect approach. But that seems different than how the project was originally sold as appropriate for a TLP. In this case, Transact is folded into Sawtooth as a sawtooth-transact repo, or becomes a Lab.
4) Ditch all this, let proposal goal bygones be bygones, and
devise strictly objective metrics for demoting or removing TLPs
based on community health metrics, not based on
what-depends-upon-what. Someone would need to define those health
metrics and the sequence of steps the TSC does to help projects
meet those metrics before taking such a harsh measure.
Comparing Transact to other tools/utilities projects like Cello,
Caliper, and Explorer is difficult, because none of those projects
were about so deeply about rebooting how our DLTs would work;
instead they are add-ons where adaptation to other DLTs doesn't
require such tight coordination with core maintainers on those
DLTs. Ursa is more comparable, and Ursa now is consumed by both
Indy and Iroha, who did put in the work in good faith.
There may be other proposals to put on the table. However,
putting aside my "HL/LF staff objective community facilitator" hat
and putting on my "experienced open source community leader" hat,
the use of dependencies to argue for whether something should be a
TLP inside an umbrella or not (or worse, as a basis for that
status to be removed) is something I have never heard of before
here, and so of course have never seen successfully applied, and
my spidey sense is that aside from suggesting it (e.g. "that
FabToken proposal you made, have you talked with Fabric about
being a module there first? or over at Labs?") it's a poor
criteria here.
Brian
-- Brian Behlendorf Executive Director, Hyperledger bbehlendorf@... Twitter: @brianbehlendorf
One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
In Rocketchat, Shawn Amundson had this to say about the "Rollover of projects tied to a single other project" proposal:
It would be helpful to be specific about the intent in relation to existing projects. A policy that forces X project and Y project to merge may be equivalent to ejecting X project or Y project from HL. Such a move should be explicit and discussed. As it is now, it sounds like malicious maneuvering against projects that specific TSC members aren't directly invested in. TSC should focus instead on fostering collaboration instead of attacking projects. "If you don't write your project report the right way, we will immediately start to talk about 'sanctions' or removing the project." isn't collaborative.
Shawn is right. Conversations that try to address subjective disagreements with objective rulemaking can come across as attempts to legislatively solve disputes and issues that are actually more interpersonal in nature and that almost always leads to bad laws/rules. Rather than talking about new rules to try and tackle what is actually a fairly specific disagreement and possibly a niche situation, it might help to try and explain how we got here, and use that as a basis to figure out a better path out.
As I retell this, please be kind to my interpretation. I'll cite links when I can find them, but much of this covers conversations and events happening on phone calls, DMs, personal emails, etc, as well as my unavoidably subjective read of the situation. I'll try and state interpretations as plainly as I can, and avoid imputing motives, so apologies if I slip on that. If you feel I got something wrong, feel free to let me or the list know, but please assume benign ignorance of details rather than maliciousness on my part.
Transact was proposed as a top-level project at a pretty early stage in its life, likely long before any project would be accepted today. No need to revisit that decision, but to remember its rationale - that there was a strong desire across the TSC (it seemed) for a common, unified approach to allow for portability of smart contract engines across an array of different ledgers. Such an approach, of course, is never guaranteed to work - there was no small amount of optimism, risk, and hope in that goal, and it requires a ton of communication, architectural humility, and looking for common ground before writing code. Trying to define a common dividing line across systems as different in their architectures as Fabric, Sawtooth, and Iroha (let alone Besu, which hadn't shown up yet) wasn't even clear if it was possible, or if the compromises it would require wouldn't have unacceptable costs. But there was a sincere intent shown in the Transact proposal to explore this domain, and the proposed ideas seemed worth pursuing, so much so that the TSC relied upon them to make its decision to approve. You can see mention of it in the original proposal and discussion on TSC calls that led to an email-based vote. I'd say the enthusiasm for getting this right across HL projects was part of the reason people went in so early, as it's much harder to adapt once that first architecture is laid out. While I was not at the Maintainer Summit in Minnesota, I recall hearing there were very promising conversations between Sawtooth and Fabric developers and others about the possible ways for Transact to become that kind of project. So I view the TSC's approval of Transact as not just an endorsement of the principle - hey, sounds good, someone aught to do it - but a reflection of sincere intent on behalf of the other DLTs to invest time into a shared approach. That was not "we'll wait and see", but "this is good enough, let's start to work together."
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Today, if Transact had been proposed, we probably would have said "let's start it in Labs" first, and given it time to make measured progress against that objective, before approving it as a TLP. This is the path that Cactus and Avalon have gone through, successfully, and there are a lot of great ideas in Labs that haven't yet achieved their initial goals and many that won't. Had Transact started in labs, and still arrived to where it was today, my personal view is that it likely would not be accepted as a top level project, given how core it still seems to Sawtooth. I believe we were concerned about having a production-grade DLT like Sawtooth depending upon a Labs project - which could have been solved by making it a "sawtooth-transact" repo.
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.To be completely fair to Transact, as an individual project it has by no means failed. I don't know if there are production systems running on it yet, but objectively there are commits and code and docs and all the trappings of a competently managed source tree. Its last release was version 0.3.1 just 12 hours ago. There are commits to the repo from more than just Bitwise employees this past year, according to the dashboard. So let's think instead about how to respect the effort those devs are putting in, an approach that helps them and Transact get closer to its original goal, or if that goal no longer applies, provides a dignified alternative path.
There are three possible fixes I see based on this history:
1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
2) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. BUT, maintainers on other projects can't find the time or inclination to sincerely take part in that process. In that case, Transact remains a TLP, because they should not be penalized if other projects can't be bothered to meet them halfway. Even just one such DLT stepping in would be enough.
3) Transact maintainers declare that other projects are more than welcome to use the architecture they've developed, but they have their own priorities and opinions on how things should be built, take them or leave them. Many projects are successful with that perspective - including the Linux kernel! - so that's not an incorrect approach. But that seems different than how the project was originally sold as appropriate for a TLP. In this case, Transact is folded into Sawtooth as a sawtooth-transact repo, or becomes a Lab.
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
Comparing Transact to other tools/utilities projects like Cello, Caliper, and Explorer is difficult, because none of those projects were about so deeply about rebooting how our DLTs would work; instead they are add-ons where adaptation to other DLTs doesn't require such tight coordination with core maintainers on those DLTs. Ursa is more comparable, and Ursa now is consumed by both Indy and Iroha, who did put in the work in good faith.
There may be other proposals to put on the table. However, putting aside my "HL/LF staff objective community facilitator" hat and putting on my "experienced open source community leader" hat, the use of dependencies to argue for whether something should be a TLP inside an umbrella or not (or worse, as a basis for that status to be removed) is something I have never heard of before here, and so of course have never seen successfully applied, and my spidey sense is that aside from suggesting it (e.g. "that FabToken proposal you made, have you talked with Fabric about being a module there first? or over at Labs?") it's a poor criteria here.
Brian
-- Brian Behlendorf Executive Director, Hyperledger bbehlendorf@... Twitter: @brianbehlendorf
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.
...
There are three possible fixes I see based on this history:
1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
My 2c is that given the variety of top level DLTs within the Hyperledger community and their very different architectures it is highly unlikely that a TLP like a Transact will be capable of supporting all of them. Therefore my simple minded approach would be to see if at least one other of the top level DLT projects is seriously interested in supporting it as a starting point.
While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.
I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.
Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.
Regards.
--
Arnaud Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM
From: "Shawn Amundson" <amundson@...>
To: Brian Behlendorf <bbehlendorf@...>
Cc: "hyperledger-tsc@..." <hyperledger-tsc@...>
Date: 09/04/2020 08:49 PM
Subject: [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by: tsc@...
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).
I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team. The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?
...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.
While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.
FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.
1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...
"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.
-Shawn
On Sep 4, 2020, at 2:04 PM, Arnaud Le Hors <lehors@...> wrote:Hi Shawn,
While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.
I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.
Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.
Regards.
--
Arnaud Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM
From: "Shawn Amundson" <amundson@...>
To: Brian Behlendorf <bbehlendorf@...>
Cc: "hyperledger-tsc@..." <hyperledger-tsc@...>
Date: 09/04/2020 08:49 PM
Subject: [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by: tsc@...
On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).
I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team. The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?
...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.
While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.
FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.
1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...
"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.
-Shawn
On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process [...]
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard),
To be clear, it wasn't a statement about Transact as a project -
I went out of my way to note that there is a healthy development
clip to Transact for a young project. What didn't materialize -
perhaps that's a more neutral term than fizzled - was a clear
process for designing Transact as a bridge specifically for the
other DLTs at Hyperledger. That was what I read coming out of the
Maintainer Summit, and the basis for its accelerated approval as a
TLP, and that's the momentum that felt lost. Apologies that I
wasn't clear.
I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team. The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library.
Indeed, this has hobbled Fabric's potential adoption of Ursa too,
and our attempts to standardize and make it easier/more foolproof
to adopt modern cryptography. I still can't believe Go can't do
what SWIG brought to pretty much every other programming language
in like 2001. Go fans, am I wrong?
This is the kind of Technical Steering that the TSC could
do, if it chose. Steering projects towards languages and
architectures that encourage modularization and re-use. The
selection of Go is hard to fault for having been made in 2015, and
the Go client for Ethereum is clearly very successful. But It'd be
really nice if the Fabric team could start thinking about whether
portions of Fabric in languages other than Go would make sense
going forward. Seeing Sawtooth move from Python to Rust and Iroha
from C to Rust is really encouraging and I'm sure couldn't have
been an easy decision. I see James made this point as well, thanks
James!
Here lies the conundrum - in order to collaborate on the same
code, so that we might save time/money/effort on common
objectives, we have to spend more time/money/effort to get there.
The calculus is challenging for sake of all the unknowns. There's
a cost/benefit to engaging early in the lifecycle of something
like Transact, if the architectures can adapt; versus later, when
more unknowns are settled, but decisions are now more hard-coded.
If there's one thing a TSC should be useful at, it's brokering the
trust and conversations required to assist that calculus. Carrots,
not sticks, to get to better alignment across Hyperledger. Because
sticks fucking hurt.
And hey, it's 2020, and not even ICO-funded Swiss shell corps
have piles of money to spend reinventing wheels and duplicating
efforts. We all have to get better at building upon each other's
work if we're going to survive what is clearly still a very long
blockchain winter, and very few of the solutions in the market
will survive it. My hope is that at least a few of the ones that
do are at Hyperledger. There can be more than one :) but the case
for their differentiation better be really, really strong, and
they better share as much code as possible.
Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
Not that I can see, I'm personally pretty eager to see it
succeed. By the way, other TSC listers, Grid recently restarted
its regular calls, and had a decently well
attended meeting that sounded pretty productive, including
that they use the RFCs
process other projects have started to adopt. Cool.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work.
Note that the pitch was somewhat different - not just an abstract
notion that it could work across other DLTs, but specifically that
it would aim to help the ones we already have at Hyperledger. That
signaling was clear in the Google doc proposal, and I'm sure the
basis of Gari and Chris's support for it as well.
So for those who want to be involved in those design decisions around Transact - where do they go? Whether I go to the wiki page, the mailing list, or the RocketChat channel, I don't see those discussions, despite those being listed in the Transact README.md as the place to go. The Kanban page isn't a useful way to see design discussions, it's a way to prioritize work already agreed upon. It looks and feels like the actual discussions and work are taking place elsewhere, hidden?, with the community tools used as an after-development way to get feedback from end-users. The cheerful Q3 report doesn't give any clues either. With Grid it's the same deal - I'm quite encouraged by that community call, but again, the Wiki, the mailing list, and the RocketChat channel all leave the impression that the real work is happening elsewhere. To even find the agenda for the Grid meeting I had to go scroll up in the Rocketchat channel, and then, who knows what the results of that meeting were? I want to be wrong about all this, that I missed a memo on where these discussions are happening - what am I missing?
Working in the public is extremely challenging. When the majority
of your initial maintainer cohort physically sit in the same
office and work for the same company, it's a nearly unavoidable
outcome. We tortured the IBM Fabric team in Raleigh by sending
Dave Huseby out to sit with them for a week to find ways to both
nudge and force them to work in the public, and it paid off. It
builds trust, it makes it easier to help people climb the learning
curve without being a burden on the maintainers, it helps end
users solve their issues, it helps avoid bikeshedding... the list
goes on Working across projects gets even harder - the square of
how hard it is for one project, perhaps?
This is the kind of community management the TSC could do,
if it chose. The quarterly reports are the ideal time to be
looking at these resources and asking these questions, and I fear
we might be going too quickly through them. It can be done in a
way that's intended to be helpful rather than judgemental, but it
has to hold the maintainers accountable to a higher standard of
what an Open Source project needs to be than any ol github repo on
its own. So anyone reading my paragraph above and thinking I'm
taking Shawn / Transact / Grid to town, think again - this is on
us, collectively, to say what's important, what our standards are,
and what we'll support. Those are the objective criteria, that
Arnaud is rightfully looking for, that should ultimately be the
basis for accepting new projects as TLPs and keeping them there.
At the same time, Shawn, I think if these projects were more
public in how they work, a lot of the angst on these issues might
fade, and we could get back to talking about working across
projects.
Brian
-- Brian Behlendorf Executive Director, Hyperledger bbehlendorf@... Twitter: @brianbehlendorf
And hey, it's 2020, and not even ICO-funded Swiss shell corps have piles of money to spend reinventing wheels and duplicating efforts.
So for those who want to be involved in those design decisions around Transact - where do they go? Whether I go to the wiki page, the mailing list, or the RocketChat channel, I don't see those discussions, despite those being listed in the Transact README.md as the place to go. The Kanban page isn't a useful way to see design discussions, it's a way to prioritize work already agreed upon. It looks and feels like the actual discussions and work are taking place elsewhere, hidden?, with the community tools used as an after-development way to get feedback from end-users. The cheerful Q3 report doesn't give any clues either. With Grid it's the same deal - I'm quite encouraged by that community call, but again, the Wiki, the mailing list, and the RocketChat channel all leave the impression that the real work is happening elsewhere. To even find the agenda for the Grid meeting I had to go scroll up in the Rocketchat channel, and then, who knows what the results of that meeting were? I want to be wrong about all this, that I missed a memo on where these discussions are happening - what am I missing?
I don't think long term Hyperledger can afford to be a "Not Collaborated Here" venture. That limits its relevance, scope, and impact tht Hyplereder could provice to the whole DLT/Blockchain/Distributed Computing space. I strongly feel there needs to be room for projects that are primarily Hyperledger focused, projects that are simply Hyperledger homed, and all the gradients in between.
On Fri, Sep 4, 2020 at 8:01 PM Brian Behlendorf <bbehlendorf@...> wrote:And hey, it's 2020, and not even ICO-funded Swiss shell corps have piles of money to spend reinventing wheels and duplicating efforts.
A number of them actually do have the "money." The next issue they run into is engineering resources: there is a limited supply of engineers that are (a) good enough (b) willing to work in crypto/DLT/blockchain and (c) aren't total narcissists. Even without a Code of Conduct such behavior gets in the way of successful projects. Not a dig on our CoC, we need to establish boundaries and it provides a touchpoint when boundary enforcement is required.Be careful where you look on crypto-twitter and various crypto reddits. It will make your eyes bleed and our struggles in HL pale by comparison. No one at HL has sent each other death threats (not hyperbole). We are not perfect in the way we deal with each other but with that perspective you can see we are doing fairly well.So for those who want to be involved in those design decisions around Transact - where do they go? Whether I go to the wiki page, the mailing list, or the RocketChat channel, I don't see those discussions, despite those being listed in the Transact README.md as the place to go. The Kanban page isn't a useful way to see design discussions, it's a way to prioritize work already agreed upon. It looks and feels like the actual discussions and work are taking place elsewhere, hidden?, with the community tools used as an after-development way to get feedback from end-users. The cheerful Q3 report doesn't give any clues either. With Grid it's the same deal - I'm quite encouraged by that community call, but again, the Wiki, the mailing list, and the RocketChat channel all leave the impression that the real work is happening elsewhere. To even find the agenda for the Grid meeting I had to go scroll up in the Rocketchat channel, and then, who knows what the results of that meeting were? I want to be wrong about all this, that I missed a memo on where these discussions are happening - what am I missing?
This brings up an existential question: how much of this collaboration must occur in Hyperledger forums? Some part of it should but how far should we go to force it? For Besu this is not an abstract question. Basically none of the future of ethereum is planned in Hyperledger forums, instead it's a series of discord servers, message boards, gitter rooms, and twitter threads. Yes, at least one critical decision was made as a consequence of Twitter threads. But for better or worse that is the way it is. Our maintainers can't afford to focus on just HL arenas and must interact outside of the tower to keep up with our mandate of being a mainnet capable enterprise blockchain.I don't think long term Hyperledger can afford to be a "Not Collaborated Here" venture. That limits its relevance, scope, and impact tht Hyplereder could provice to the whole DLT/Blockchain/Distributed Computing space. I strongly feel there needs to be room for projects that are primarily Hyperledger focused, projects that are simply Hyperledger homed, and all the gradients in between.
Brian and Arnaud,Without emotion into to it, these things take a while to gestate. I think that Transact is maturing now so that it can be used as a solid external component. Iroha is enquiring about the use within their 2.0 and testing the framework. Sawtooth 2.0 will be using it as a core component. Outside of Hyperledger, at this point Scabbard, the default engine for Splinter (not a Hyperledger project) is definitely using Transact. I believe other DLT’s will use it once it gets a bit more mature and more publicity.These things are hard to make reusable as the TSC noted when Bill and I made our componentization pitch to the TSC last Spring. I do think it would be a good thing for the TSC to look at common component frameworks and see if there is any uptick for the projects. If so, perhaps a member or two might find it good for a resource to work for 3-4 months to see if it would fit into the top level frameworks. I was on the call asking the TSC for some fairly straight forward changes so the Chinese encryption algorithm could be used by Fabric. The pushback surprised me, especially talking about all of the issues around hardware devices etc. Accommodating the encryption was IMHO a perfect use case for componentization, and a great one for URSA to tackle. Yes it would take longer, but then Fabric would not be caught flat footed by the next country requiring their own proprietary encryption. So it is this hard level of reuse or accommodation that needs to be looked at, unless Hyperledger is to become a single codebase project. Brian has some great discussions on how the DLT should all be getting closer together with clients, API’s, I/O’s etc. Things like Quorum are in use at large Hyperledger members. Transact had a great vision when it started and “should" be something that a Quorum project is interested in. Because Transact slowed down delivering its original promise, that should be a reason to see how to help it, not fold it in. Same with URSA frankly. Starting out as a Labs project might work, so long as the sponsors would be able to find two projects that would lend dedicated programmers to integrate into their project.Hyperledger’s TSC could define what is expected from shareable components. URSA is another one that could use that expectation of the shared components in my opinion.Transact had a great vision at the beginning and is now starting to realize that vision. Let's see how we can help it along and achieve its vision. Its time for shareable components to emerge for the DLT’s (blockchains) as they were all islands back in 2018. If we are to get widespread use, they need to start having some reusable pieces, not simply build it and they will come. That fractionalization really killed the overall blockchain momentum and created the “disillusionment” trough that individual walled projects have created. Through projects like Transact/Ursa etc. that sense of an entire framework, not individual projects, enterprises will find DLT’s (blockchains) more mainstream and usable within their environments.I hope to see Transact and other projects nurtured to help each other. I would hope that the needs of other projects are added to the architectures as they are being developed so they can be reused throughout the Hyperledger community. We have been working with Shawn to get Sawtooth 2.x to be accommodating to our needs including putting items in the plans that were not originally in the architecture. And we are not the only ones with needs not originally contemplated by the 2.0 architecture. Transact could use some solid PR’s or whatever process the TSC thinks will accommodate the acquiring needs of other projects. Through some strong suggestions, we can help mold Transact to be an awesome project.Thanks,On Sep 4, 2020, at 2:04 PM, Arnaud Le Hors <lehors@...> wrote:Hi Shawn,
While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.
I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.
Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.
Regards.
--
Arnaud Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM
From: "Shawn Amundson" <amundson@...>
To: Brian Behlendorf <bbehlendorf@...>
Cc: "hyperledger-tsc@..." <hyperledger-tsc@...>
Date: 09/04/2020 08:49 PM
Subject: [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by: tsc@...
On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).
I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team. The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?
...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.
While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.
FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.
1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...
"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.
-Shawn
James+1I can confirm that the Sawtooth 2.0 architecture conversations have been very constructive and people have been willing to adapt the design to accommodate our and others requirements.Given the state of the world right now we should err on being supportive not adversarial/antagonistic even if that is not the intention. We are watching this very closely as the outcome of this discussion is an important bellwether for the health of Hyperledger community.BestDuncanOn Sat, Sep 5, 2020 at 12:17 AM James Barry <james@...> wrote:Brian and Arnaud,Without emotion into to it, these things take a while to gestate. I think that Transact is maturing now so that it can be used as a solid external component. Iroha is enquiring about the use within their 2.0 and testing the framework. Sawtooth 2.0 will be using it as a core component. Outside of Hyperledger, at this point Scabbard, the default engine for Splinter (not a Hyperledger project) is definitely using Transact. I believe other DLT’s will use it once it gets a bit more mature and more publicity.These things are hard to make reusable as the TSC noted when Bill and I made our componentization pitch to the TSC last Spring. I do think it would be a good thing for the TSC to look at common component frameworks and see if there is any uptick for the projects. If so, perhaps a member or two might find it good for a resource to work for 3-4 months to see if it would fit into the top level frameworks. I was on the call asking the TSC for some fairly straight forward changes so the Chinese encryption algorithm could be used by Fabric. The pushback surprised me, especially talking about all of the issues around hardware devices etc. Accommodating the encryption was IMHO a perfect use case for componentization, and a great one for URSA to tackle. Yes it would take longer, but then Fabric would not be caught flat footed by the next country requiring their own proprietary encryption. So it is this hard level of reuse or accommodation that needs to be looked at, unless Hyperledger is to become a single codebase project. Brian has some great discussions on how the DLT should all be getting closer together with clients, API’s, I/O’s etc. Things like Quorum are in use at large Hyperledger members. Transact had a great vision when it started and “should" be something that a Quorum project is interested in. Because Transact slowed down delivering its original promise, that should be a reason to see how to help it, not fold it in. Same with URSA frankly. Starting out as a Labs project might work, so long as the sponsors would be able to find two projects that would lend dedicated programmers to integrate into their project.Hyperledger’s TSC could define what is expected from shareable components. URSA is another one that could use that expectation of the shared components in my opinion.Transact had a great vision at the beginning and is now starting to realize that vision. Let's see how we can help it along and achieve its vision. Its time for shareable components to emerge for the DLT’s (blockchains) as they were all islands back in 2018. If we are to get widespread use, they need to start having some reusable pieces, not simply build it and they will come. That fractionalization really killed the overall blockchain momentum and created the “disillusionment” trough that individual walled projects have created. Through projects like Transact/Ursa etc. that sense of an entire framework, not individual projects, enterprises will find DLT’s (blockchains) more mainstream and usable within their environments.I hope to see Transact and other projects nurtured to help each other. I would hope that the needs of other projects are added to the architectures as they are being developed so they can be reused throughout the Hyperledger community. We have been working with Shawn to get Sawtooth 2.x to be accommodating to our needs including putting items in the plans that were not originally in the architecture. And we are not the only ones with needs not originally contemplated by the 2.0 architecture. Transact could use some solid PR’s or whatever process the TSC thinks will accommodate the acquiring needs of other projects. Through some strong suggestions, we can help mold Transact to be an awesome project.Thanks,On Sep 4, 2020, at 2:04 PM, Arnaud Le Hors <lehors@...> wrote:Hi Shawn,
While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.
I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.
Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.
Regards.
--
Arnaud Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM
From: "Shawn Amundson" <amundson@...>
To: Brian Behlendorf <bbehlendorf@...>
Cc: "hyperledger-tsc@..." <hyperledger-tsc@...>
Date: 09/04/2020 08:49 PM
Subject: [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by: tsc@...
On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).
I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team. The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?
...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.
While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.
FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.
1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...
"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.
-Shawn
--
How about a dedicated TSC call on the topic of "Collaboration in Hyperledger"? You can ask maintainers of different projects to join. Let everybody give their opinion on what it means for them, what are their objectives, what is that they are looking forward to.
On Sep 6, 2020, at 12:29 PM, Arun S M <arun.s.m.cse@...> wrote:Brian, Arnaud and Shawn,I am amazed looking at the way this email conversation is going on.Here is what I see happening in the thread. It is still the community having healthy discussion. But wrong choice of words and examples, mixing up with the emotions, is taking it to another direction.Arnaud/TSC: Is making an effort towards defining TLP in Hyperledger. Has a proposal where possibly two top level projects are merged into one. This is up for discussion and comments / views are still welcome on it.- Thanks for the good summary. I will leave my comments to you at the end.Brian: Took Transact as an example, because of lack visibility into what's happening there. The reasons such as most of the maintainers are co-located, the information on feature development or the future of the project is not visible in public are influencing the thoughts. The reason for lack of visibility could also be that resources are not linked right. Used the word `unachieved` because he has not seen a result of what was initially agreed upon when the project was proposed.- I agree to Brian's comments. We should look at the core of the problem and address it first before touching the surface. By patch decisions, things can only become more complicated in future. In this case, I see 2 possible problems which can be solved. (1) Decide when a project should get the status of incubation in TLP. (2) How to solve the issue of different TLPs linked to one. I have my comments for the issue (1) in the proposal. For the case (2), we can possibly define metrics to measure the projects. Having well defined processes and indications to the maintainers makes sense. Any decision taken by the TSC should not come as a surprise to any of the projects.Shawn: Understands the reason for having Transact as a TLP. Is involved in discussions for multiple DLT solutions to reuse the Transact library. Went onto email emotionally because others spoke about the project without knowing these.- You have given enough indications on usability of the project. This email should be a good record as an answer to all such speculations, that Transact is only used in one project. I know you are working on a website for Sawtooth. How about one for Transact? Or how about making any document / discussion happen in public? I understand that you're making efforts towards that. But I still feel giving out more information in public can solve most of such issues.I don't see anybody's fault, arrogance or ignorance here.In short, the discussion is drifting away, because this is about a particular solution instead it can talk about the actual problem and ask for more possible solutions.Arnaud: This is my take on the issue as I always like to fix the issue to its core, I can only request you, do not take these words otherwise. There is always a high risk of negative perception due to choice of words in the virtual world. How about a dedicated TSC call on the topic of "Collaboration in Hyperledger"? You can ask maintainers of different projects to join. Let everybody give their opinion on what it means for them, what are their objectives, what is that they are looking forward to. Bring up the issue that is making you think "merge different TLP into one". Let this merger be one of the possible solutions and not the only solution to solve that bigger core issue.Regards,ArunOn Sat, Sep 5, 2020 at 7:18 PM Duncan Johnston-Watt <duncan@...> wrote:James+1I can confirm that the Sawtooth 2.0 architecture conversations have been very constructive and people have been willing to adapt the design to accommodate our and others requirements.Given the state of the world right now we should err on being supportive not adversarial/antagonistic even if that is not the intention. We are watching this very closely as the outcome of this discussion is an important bellwether for the health of Hyperledger community.BestDuncanOn Sat, Sep 5, 2020 at 12:17 AM James Barry <james@...> wrote:Brian and Arnaud,Without emotion into to it, these things take a while to gestate. I think that Transact is maturing now so that it can be used as a solid external component. Iroha is enquiring about the use within their 2.0 and testing the framework. Sawtooth 2.0 will be using it as a core component. Outside of Hyperledger, at this point Scabbard, the default engine for Splinter (not a Hyperledger project) is definitely using Transact. I believe other DLT’s will use it once it gets a bit more mature and more publicity.These things are hard to make reusable as the TSC noted when Bill and I made our componentization pitch to the TSC last Spring. I do think it would be a good thing for the TSC to look at common component frameworks and see if there is any uptick for the projects. If so, perhaps a member or two might find it good for a resource to work for 3-4 months to see if it would fit into the top level frameworks. I was on the call asking the TSC for some fairly straight forward changes so the Chinese encryption algorithm could be used by Fabric. The pushback surprised me, especially talking about all of the issues around hardware devices etc. Accommodating the encryption was IMHO a perfect use case for componentization, and a great one for URSA to tackle. Yes it would take longer, but then Fabric would not be caught flat footed by the next country requiring their own proprietary encryption. So it is this hard level of reuse or accommodation that needs to be looked at, unless Hyperledger is to become a single codebase project. Brian has some great discussions on how the DLT should all be getting closer together with clients, API’s, I/O’s etc. Things like Quorum are in use at large Hyperledger members. Transact had a great vision when it started and “should" be something that a Quorum project is interested in. Because Transact slowed down delivering its original promise, that should be a reason to see how to help it, not fold it in. Same with URSA frankly. Starting out as a Labs project might work, so long as the sponsors would be able to find two projects that would lend dedicated programmers to integrate into their project.Hyperledger’s TSC could define what is expected from shareable components. URSA is another one that could use that expectation of the shared components in my opinion.Transact had a great vision at the beginning and is now starting to realize that vision. Let's see how we can help it along and achieve its vision. Its time for shareable components to emerge for the DLT’s (blockchains) as they were all islands back in 2018. If we are to get widespread use, they need to start having some reusable pieces, not simply build it and they will come. That fractionalization really killed the overall blockchain momentum and created the “disillusionment” trough that individual walled projects have created. Through projects like Transact/Ursa etc. that sense of an entire framework, not individual projects, enterprises will find DLT’s (blockchains) more mainstream and usable within their environments.I hope to see Transact and other projects nurtured to help each other. I would hope that the needs of other projects are added to the architectures as they are being developed so they can be reused throughout the Hyperledger community. We have been working with Shawn to get Sawtooth 2.x to be accommodating to our needs including putting items in the plans that were not originally in the architecture. And we are not the only ones with needs not originally contemplated by the 2.0 architecture. Transact could use some solid PR’s or whatever process the TSC thinks will accommodate the acquiring needs of other projects. Through some strong suggestions, we can help mold Transact to be an awesome project.Thanks,On Sep 4, 2020, at 2:04 PM, Arnaud Le Hors <lehors@...> wrote:Hi Shawn,
While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.
I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.
Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.
Regards.
--
Arnaud Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM
From: "Shawn Amundson" <amundson@...>
To: Brian Behlendorf <bbehlendorf@...>
Cc: "hyperledger-tsc@..." <hyperledger-tsc@...>
Date: 09/04/2020 08:49 PM
Subject: [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by: tsc@...
On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).
I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team. The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?
...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.
While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.
FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.
1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...
"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.
-Shawn--
James,
I start every TSC call with a statement about the call being open to all to listen in and contribute. This is not a figure of speech and I can only encourage you to speak up if/when you think the TSC is missing something. This is of course true for everyone. We'll all be better off.
As for Arun's proposal, I'm certainly open to it. Unfortunately we've not had any maintainers meeting in a long time, in part due to the COVID situation, and generally speaking discussions between maintainers across projects has been rather limited. The maintainers list is mostly dead. There is however evidence of point to point collaboration going on and this is probably the most important part. Because of the way it is happening this isn't necessarily visible to the rest of the community unfortunately.
--
Arnaud Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM
From: "James Barry" <james@...>
To: Arun S M <arun.s.m.cse@...>
Cc: Duncan Johnston-Watt <duncan@...>, Arnaud Le Hors <lehors@...>, Shawn Amundson <amundson@...>, Brian Behlendorf <bbehlendorf@...>, "hyperledger-tsc@..." <hyperledger-tsc@...>, Hyperledger List <tsc@...>
Date: 09/07/2020 06:28 AM
Subject: [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by: tsc@...
+1 on Arun entire recap
I believe the entire issue needs a start with Arun’s excellent suggestion:
How about a dedicated TSC call on the topic of "Collaboration in Hyperledger"? You can ask maintainers of different projects to join. Let everybody give their opinion on what it means for them, what are their objectives, what is that they are looking forward to.
Frankly I am on the outside looking in. I see a lot of projects that are each going their own way. There is a central question as to what the maintainers think “Hyperledger.org” should be. The discussion proposed by Arun is an excellent way too begin looking at this question. Most open source software organizations are built around helping foster community, infrastructure etc. Hyperledger is unique in that it is part of the Linux Foundation which provides much of that, so it can concentrate on the greater blockchain community, and work across communities - that is why Brian is perfect to lead the project..
Right now from the outside looking in, Hyperledger is thought of as a Fabric organization with some add on’s. People are surprised when I mention we use a DLT - Sawtooth, and that tools like Caliper can be used by any blockchain. I think that the Top Level Projects that exist today have a mission that would/will greatly advance the blockchain community as a whole. The central question is if another Hyperledger Top Level project (especially the “elephant” in the room- Fabric) don’t use a shared TLP, should they actually be a TLP? Or should these projects be able to work and collaborate around the blockchain community as a whole while “anchoring" themselves in Hyperledger? Success would then have a different dimension to measure than simply "use by others in the Hyperledger community"? That I believe is the question that Arun would drive to in his maintainer meeting, and what Danno and Duncan mean when they mention about Hyperledger working and collaborating with other communities. Let the maintainers discuss their motivations is an excellent first start.
My personal thought on this issue, is that Hyperledger’s structure and leadership can find, nature and bring forward projects that the entire blockchain community cannot bring forward, because Hyperledger does not have the strong monetary foundations that many of these blockchain projects were spawned with. Hyperledger should serve as the catalyst to propose projects that will help the entire community. Hyperledger is the only blockchain community that can do this without financial considerations clouding the direction.
To the original thought of combining TLP’s, I think that defeats the purpose of having libraries that can be used through the entire blockchain system. If a project is to succeed at that audacious goal to be a “core library" like Transact or Ursa it takes time, community and help that I am sure Hyperledger can provide. The key is giving them time to mature and attract developers to their cause. To that effect Transact is getting some traction that will expand the core development base as the project matures.
Thank you for listening,
James Barry
Cell 303-956-8835
james@...
http://www.linkedin.com/in/jamesbarry
On Sep 6, 2020, at 12:29 PM, Arun S M <arun.s.m.cse@...> wrote:
Brian, Arnaud and Shawn,
I am amazed looking at the way this email conversation is going on.
Here is what I see happening in the thread. It is still the community having healthy discussion. But wrong choice of words and examples, mixing up with the emotions, is taking it to another direction.
Arnaud/TSC:Is making an effort towards defining TLP in Hyperledger. Has a proposal where possibly two top level projects are merged into one. This is up for discussion and comments / views are still welcome on it.
- Thanks for the good summary. I will leave my comments to you at the end.
Brian: Took Transact as an example, because of lack visibility into what's happening there. The reasons such as most of the maintainers are co-located, the information on feature development or the future of the project is not visible in public are influencing the thoughts. The reason for lack of visibility could also be that resources are not linked right. Used the word `unachieved` because he has not seen a result of what was initially agreed upon when the project was proposed.
- I agree to Brian's comments. We should look at the core of the problem and address it first before touching the surface. By patch decisions, things can only become more complicated in future. In this case, I see 2 possible problems which can be solved. (1) Decide when a project should get the status of incubation in TLP. (2) How to solve the issue of different TLPs linked to one. I have my comments for the issue (1) in the proposal. For the case (2), we can possibly define metrics to measure the projects. Having well defined processes and indications to the maintainers makes sense. Any decision taken by the TSC should not come as a surprise to any of the projects.
Shawn: Understands the reason for having Transact as a TLP. Is involved in discussions for multiple DLT solutions to reuse the Transact library. Went onto email emotionally because others spoke about the project without knowing these.
- You have given enough indications on usability of the project. This email should be a good record as an answer to all such speculations, that Transact is only used in one project. I know you are working on a website for Sawtooth. How about one for Transact? Or how about making any document / discussion happen in public? I understand that you're making efforts towards that. But I still feel giving out more information in public can solve most of such issues.
I don't see anybody's fault, arrogance or ignorance here.
In short, the discussion is drifting away, because this is about a particular solution instead it can talk about the actual problem and ask for more possible solutions.
Arnaud:This is my take on the issue as I always like to fix the issue to its core, I can only request you, do not take these words otherwise. There is always a high risk of negative perception due to choice of words in the virtual world. How about a dedicated TSC call on the topic of "Collaboration in Hyperledger"? You can ask maintainers of different projects to join. Let everybody give their opinion on what it means for them, what are their objectives, what is that they are looking forward to. Bring up the issue that is making you think "merge different TLP into one". Let this merger be one of the possible solutions and not the only solution to solve that bigger core issue.
Regards,
Arun
On Sat, Sep 5, 2020 at 7:18 PM Duncan Johnston-Watt <duncan@...> wrote:
James
+1
I can confirm that the Sawtooth 2.0 architecture conversations have been very constructive and people have been willing to adapt the design to accommodate our and others requirements.
Given the state of the world right now we should err on being supportive not adversarial/antagonistic even if that is not the intention. We are watching this very closely as the outcome of this discussion is an important bellwether for the health of Hyperledger community.
Best
Duncan
On Sat, Sep 5, 2020 at 12:17 AM James Barry <james@...> wrote:
Brian and Arnaud,
Without emotion into to it, these things take a while to gestate. I think that Transact is maturing now so that it can be used as a solid external component. Iroha is enquiring about the use within their 2.0 and testing the framework. Sawtooth 2.0 will be using it as a core component. Outside of Hyperledger, at this point Scabbard, the default engine for Splinter (not a Hyperledger project) is definitely using Transact. I believe other DLT’s will use it once it gets a bit more mature and more publicity.
These things are hard to make reusable as the TSC noted when Bill and I made our componentization pitch to the TSC last Spring. I do think it would be a good thing for the TSC to look at common component frameworks and see if there is any uptick for the projects. If so, perhaps a member or two might find it good for a resource to work for 3-4 months to see if it would fit into the top level frameworks. I was on the call asking the TSC for some fairly straight forward changes so the Chinese encryption algorithm could be used by Fabric. The pushback surprised me, especially talking about all of the issues around hardware devices etc. Accommodating the encryption was IMHO a perfect use case for componentization, and a great one for URSA to tackle. Yes it would take longer, but then Fabric would not be caught flat footed by the next country requiring their own proprietary encryption. So it is this hard level of reuse or accommodation that needs to be looked at, unless Hyperledger is to become a single codebase project. Brian has some great discussions on how the DLT should all be getting closer together with clients, API’s, I/O’s etc. Things like Quorum are in use at large Hyperledger members. Transact had a great vision when it started and “should" be something that a Quorum project is interested in. Because Transact slowed down delivering its original promise, that should be a reason to see how to help it, not fold it in. Same with URSA frankly. Starting out as a Labs project might work, so long as the sponsors would be able to find two projects that would lend dedicated programmers to integrate into their project.
Hyperledger’s TSC could define what is expected from shareable components. URSA is another one that could use that expectation of the shared components in my opinion.
Transact had a great vision at the beginning and is now starting to realize that vision. Let's see how we can help it along and achieve its vision. Its time for shareable components to emerge for the DLT’s (blockchains) as they were all islands back in 2018. If we are to get widespread use, they need to start having some reusable pieces, not simply build it and they will come. That fractionalization really killed the overall blockchain momentum and created the “disillusionment” trough that individual walled projects have created. Through projects like Transact/Ursa etc. that sense of an entire framework, not individual projects, enterprises will find DLT’s (blockchains) more mainstream and usable within their environments.
I hope to see Transact and other projects nurtured to help each other. I would hope that the needs of other projects are added to the architectures as they are being developed so they can be reused throughout the Hyperledger community. We have been working with Shawn to get Sawtooth 2.x to be accommodating to our needs including putting items in the plans that were not originally in the architecture. And we are not the only ones with needs not originally contemplated by the 2.0 architecture. Transact could use some solid PR’s or whatever process the TSC thinks will accommodate the acquiring needs of other projects. Through some strong suggestions, we can help mold Transact to be an awesome project.
Thanks,
James Barry
Cell 303-956-8835
james@...
http://www.linkedin.com/in/jamesbarry
On Sep 4, 2020, at 2:04 PM, Arnaud Le Hors <lehors@...> wrote:
Hi Shawn,
While I understand your emotional response I think it's important to keep it to facts. In what way has the TSC been disruptive to date?? The TSC isn't trying to kill anything. At best it is trying to get some order into a house that by at least some measure is a bit more chaotic than one might like (as Hart pointed out this may be more of a marketing issue than anything else). Despite what you're claiming this is definitely not a Fabric vs Transact endeavor because when I asked whether this issue should be pursued or not the only two people who said no where Dan from Intel and Chris from IBM.
I'm still uncertain as to whether the rollover idea is something we should do and what I drafted is the way we should go at it but I don't think it helps to distort what is intended or claim that the TSC is only trying to get in the way. I think the track record clearly shows that the TSC is trying to help. And, as chair, I'm only trying to facilitate discussion and helping us, as a community, progress on whatever issue is raised. If the community thinks the TSC is going in the wrong direction I have no doubts the TSC will listen and react accordingly.
Let's please stop speculating on other people's intention and try to be constructive. For that matter, if Transact is already used in more than one project or about to, this whole issue shouldn't worry you at all. So, why is this even of a concern to you? Brian made it a Transact specific issue starting this thread but this surely wasn't my intent because I think it'd be wrong to treat Transact any differently from other projects.
Regards.
--
Arnaud Le Hors - Senior Technical Staff Member, Blockchain & Web Open Technologies - IBM
From: "Shawn Amundson" <amundson@...>
To: Brian Behlendorf <bbehlendorf@...>
Cc: "hyperledger-tsc@..." <hyperledger-tsc@...>
Date: 09/04/2020 08:49 PM
Subject: [EXTERNAL] Re: [Hyperledger TSC] Transact as a TLP
Sent by: tsc@...
On Fri, Sep 4, 2020 at 3:35 AM Brian Behlendorf <bbehlendorf@...> wrote:
...
But then... that enthusiasm seemed to fizzle. Like any such collaborative process, it can be awfully hard to try and figure out why projects don't make that leap from great initial hopes to continued participation. Should the Fabric team have bent their plans for their 2.0 architecture more? Should Transact's initial architecture have changed more? Were Iroha even asked to participate? Even if we couldn't figure it out in 2019, could we figure it out in the Sawtooth 2.0 / Fabric 3.0 / Iroha-next / etc timeframe? I imagine there'll be as many different reasons why things went south as there were people engaged. There's really no need to assign blame here - we've long believed strongly that maintainers should set the technical course for their projects, unless they royally screw up - but we should acknowledge that collectively this effort hasn't lived up to that promise, and that promise was the basis for the vote in favor of Transact as a TLP. Collaboration is hard and it requires all sides to make extra effort beyond scratching their own itches (which is a metaphor for open source I've long despised!). It's a challenge to prioritize that over customer-paying work, let alone the usual technical differences of opinion on The Way To Build A DLT. It requires people realizing they have more to gain from a common platform than lose from that extra work. That requires a leap of faith and trust between individuals. Sometimes, that just can't be willed into existence. Or maybe everyone just realized one API to rule them all was too hard, asked for too many compromises to be useful. One sign of this is that Digital Asset went to some lengths to port their DAML engine across Fabric, Sawtooth, Corda and I think Quorum, and while that wasn't cheap I'm sure, it suggested it wasn't a show-stopper either.
Enthusiasm among those involved in Transact hasn't fizzled. Quite the opposite, since Sawtooth is at warp-speed right now integrating it (Sawtooth will be the 2nd DLT to use Transact, after Scabbard), and we just learned the future of Iroha will be Rust (very enthusiastic about that, and also, very likely future collaborations - if the TSC can stay out of the way).
I challenge the assertion that the success of Transact should be defined by the participation of the Fabric team. The Fabric team loses out on extending its overall engineering capability by maintaining a walled-garden approach. At the end of the day though, it is a hard sell - Fabric is a Go project and we have a Rust library. Transact covers about 3/4 of what the endorser does, and no one likes to throw away their existing code on a whim. Does it still make sense to collaborate? Probably. Academically, I'm quite interested in a from-scratch Rust implementation of the endorser. Could be a parallel effort with the current implementation, maybe a new labs project. Do I have time to implement it? ... not in isolation, but we haven't really explored whether there is a sufficient Rust community within the Fabric community. Maybe?
...
That's what, I believe, is at the heart of the issue for several people on the TSC and in the community - a history that is more personal than can be fixed by a rule like "when to combine projects", which is really "when to combine projects against the will of their maintainers". It doesn't sound like the maintainers of either Sawtooth or Transact want them to be combined. But at the same time, the existence of Transact as a top level project is a reminder of that failed effort, a totem for our difficulties in collaborating across projects.
While we continue to put up PRs and do real engineering work on the project, please do elaborate on this: "a reminder of that failed effort, a totem for our difficulties" -- My initial reaction was like WTF, but I'm sure I'm misunderstanding you... I'm sure you meant the opportunities are ahead of us as we continue to foster collaboration across projects, but it is totally not how it initially reads.
FWIW, the only negativity currently is generated from the TSC and this surrounding discussion fallout. The TSC itself is mostly disruptive, not productive. There is a big difference between "how should we get projects to collaborate more" and "we should kill this project we aren't involved with". Transact is a fundamental piece of keeping Grid DLT-agnostic, so I assume this is an anti-Grid play overall, and everyone else is just sucked into the "objective" discussion.
...
There are three possible fixes I see based on this history:
This would be a fun exercise across projects. "1) Fabric and Sawtooth maintainers declare that they are still interested in distributed non-centralized ledgers." etc.
1) Transact maintainers declare that they're still interested in that original goal, and are willing to put in the work to adapt their current architectural plans to meet the needs of others. Maintainers on other projects also commit to further discussions and efforts to use Transact on their platforms as appropriate. Meetings to discuss how each DLT takes steps in their next major release to allow them to use Transact, or at least narrow the gap. Transact might still take a few years to achieve that goal, but it remains a top level project so long as those attempts continue.
The original goal of creating a Rust library that can be used across DLTs remains. You are suggesting we aren't factoring these things in already, but you are incorrect as it is a normal part of design discussions to consider how different DLTs work. AFAIK the gaps you are suggesting aren't as big as you are suggesting, and the majority of the work here is in cleaning up DLT code enough to draw the boundary around transaction execution -- a super non-trivial task, to be sure, since the functionality provided by Transact is specific but also substantial.
...
4) Ditch all this, let proposal goal bygones be bygones, and devise strictly objective metrics for demoting or removing TLPs based on community health metrics, not based on what-depends-upon-what. Someone would need to define those health metrics and the sequence of steps the TSC does to help projects meet those metrics before taking such a harsh measure.
...
"Ditch all this" -- probably a good approach for those not involved in the project. TBH, I don't see how engineering work continuing to go into HL projects gets this type of negative response.
No one is asking, but Transact's goals are likely to expand as we go forward into a more full solution around the current capability -- so that if you adopt it, it comes with more stuff "for free" -- which will make it more compelling over time.
-Shawn
--
Duncan Johnston-Watt
CEO, BTP
Twitter: @duncanjw
Mob: +44 777 190 2653
LinkedIn: https://linkedin.com/in/duncanjohnstonwatt
So for those who want to be involved in those design decisions around Transact - where do they go? Whether I go to the wiki page, the mailing list, or the RocketChat channel, I don't see those discussions, despite those being listed in the Transact README.md as the place to go. The Kanban page isn't a useful way to see design discussions, it's a way to prioritize work already agreed upon. It looks and feels like the actual discussions and work are taking place elsewhere, hidden?, with the community tools used as an after-development way to get feedback from end-users. The cheerful Q3 report doesn't give any clues either. With Grid it's the same deal - I'm quite encouraged by that community call, but again, the Wiki, the mailing list, and the RocketChat channel all leave the impression that the real work is happening elsewhere. To even find the agenda for the Grid meeting I had to go scroll up in the Rocketchat channel, and then, who knows what the results of that meeting were? I want to be wrong about all this, that I missed a memo on where these discussions are happening - what am I missing
For Transact, it does end up being a series of very specific discussions usually between maintainers (often 1:1), and usually in the context of trying to achieve something in another project. The bulk of the discussion then turns to PR review, which is where all the maintainers get involved. In addition, recently, there has been a fair amount of Transact discussion on the Sawtooth contributors meetings because of the Sawtooth+Transact integration work currently in progress. These are really more discussions about Sawtooth than Transact, but the occasional item turns into a feature for Transact -- like a strong desire to add backward-compatible transaction processor support to Transact (as a community-driven requirement for Sawtooth 2).
-Shawn
On Fri, Sep 4, 2020 at 9:01 PM Brian Behlendorf <bbehlendorf@...> wrote:...So for those who want to be involved in those design decisions around Transact - where do they go? Whether I go to the wiki page, the mailing list, or the RocketChat channel, I don't see those discussions, despite those being listed in the Transact README.md as the place to go. The Kanban page isn't a useful way to see design discussions, it's a way to prioritize work already agreed upon. It looks and feels like the actual discussions and work are taking place elsewhere, hidden?, with the community tools used as an after-development way to get feedback from end-users. The cheerful Q3 report doesn't give any clues either. With Grid it's the same deal - I'm quite encouraged by that community call, but again, the Wiki, the mailing list, and the RocketChat channel all leave the impression that the real work is happening elsewhere. To even find the agenda for the Grid meeting I had to go scroll up in the Rocketchat channel, and then, who knows what the results of that meeting were? I want to be wrong about all this, that I missed a memo on where these discussions are happening - what am I missing...RocketChat is definitely the best place, because the maintainers are generally present there.
For Transact, it does end up being a series of very specific discussions usually between maintainers (often 1:1), and usually in the context of trying to achieve something in another project. The bulk of the discussion then turns to PR review, which is where all the maintainers get involved. In addition, recently, there has been a fair amount of Transact discussion on the Sawtooth contributors meetings because of the Sawtooth+Transact integration work currently in progress. These are really more discussions about Sawtooth than Transact, but the occasional item turns into a feature for Transact -- like a strong desire to add backward-compatible transaction processor support to Transact (as a community-driven requirement for Sawtooth 2).Grid is much different. A lot of the engagement is centered around RFCs - for example, there are 100 comments on the Location RFC - https://github.com/hyperledger/grid-rfcs/pull/20. In the community meeting, we gave an opportunity for discussion of upcoming RFCs. There is also a lot of work going on currently to expose a lot more information through the documentation.
-Shawn