Date   

Re: Maintainer nominations

Christopher Ferris
 

I agree that these would be two good additions to the docs maintainership. I approve.

Chris


On Mon, Nov 25, 2019 at 5:26 PM Pam Andrejko <pama@...> wrote:

All,

I would like to nominate two new Documentation maintainers for the Hyperledger Fabric project. They are Chris Gabriel (Hyperchain Labs) and Joe Alewine (IBM).

Chris has been an instrumental member of the Documentation community workgroup for several years now. In addition to being a regular content reviewer and contributor, he is a consumer of Fabric in his own Hyperchain Labs business that he founded. The insights, perspective, and content that he's provided based on his experience have been invaluable to the documentation work group and Fabric community as a whole.

Joe has been providing important Fabric documentation for over two and half years, is recognized as an expert on the ordering service, the Fabric upgrade process and channel capabilities, and was recently recognized as a Hyperledger significant contributor.

Adding two more Documentation Maintainers will greatly facilitate the addition and approval of Fabric documentation content going forward.

I have opened a separate PR for each nomination:

Chris Gabriel - https://github.com/hyperledger/fabric/pull/317
Joe Alewine - https://github.com/hyperledger/fabric/pull/316

I'm requesting that existing maintainers review the nominations and indicate whether they agree with a comment in the PR. Other’s are welcome to provide their input.

Warm regards,
Pam Andrejko


Re: Maintainer nominations

Yacov
 

The SDK documentation and CA is only documentation about APIs, it's not documentation about concepts, ideas, and tutorials.

I don't think that extracting the fabric documentation files out of the fabric repository would harm contribution - why would it ?

You can just link to the document repository from the readme.md of the Fabric repository and everyone would find it just as they find the files inside the docs repository.

> The repositories were not separated because they had different maintainer pools
In the past, the fabric-CA and fabric-node-SDK repositories inherited the maintainers of the fabric core, and that was a bad thing, because the only people that could merge code were those in the intersection of understanding the code in the corresponding repositories, and that were also fabric core maintainers.
As time went by, they were given their own maintainers and then progress was quicker.




From:        "Matthew Sykes" <matthew.sykes@...>
To:        fabric@...
Date:        11/26/2019 04:29 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by:        fabric@...




For the record, I am not in favor of extracting the documentation. I strongly feel that the doc and the code should live together where feasible. My perspective is that lowering the barriers to contribute is far better than segmenting the ecosystem into a number of fiefdoms.

If we want doc changes to only require a single doc review, we should state that's the desired process and put a technology plan in place to make that happen.

With that, a few comments to Dave's note:

> As Fabric has grown the sets of maintainers has naturally diverged across the fabric-* repos. The core Fabric maintainers are naturally different from the Fabric CA maintainers, the Java SDK maintainers, the Node.js SDK maintainers, etc. 

The repositories were not separated because they had different maintainer pools; they were separated because, in most cases, they use fundamentally different implementation and CI processes. I'll also note that each of these contain their own documentation tree. (There's a docs folder in CA and API documentation is part of the code in node and Java.) Are you proposing we split all of these out as well?

> It has been challenging to get a majority of core Fabric maintainers to weigh in on decisions, since only about half of the maintainers are active day-to-day. This becomes more of an issue with the recent introduction of the fabric-rfcs process, which requires a majority of Fabric maintainers to agree before taking on new enhancements.

This is a problem of our own making. We have far too much process in place to make rapid forward movement. It's our choice to require a "majority" of maintainers and "2 mandatory reviews" to integrate things.


On Tue, Nov 26, 2019 at 7:11 AM David Enyeart <enyeart@...> wrote:
The intent of my proposal was exactly as Yacov says - doc maintainers would be comprised of the core Fabric maintainers in addition to heavy doc contributors/reviewers.

I do understand Brian's point, and we have been hesitant to split fabric code and fabric docs repos/maintainers up until now for the reasons Brian mentions. However in my opinion the time has come to split them for the following reasons:

As Fabric has grown the sets of maintainers has naturally diverged across the fabric-* repos. The core Fabric maintainers are naturally different from the Fabric CA maintainers, the Java SDK maintainers, the Node.js SDK maintainers, etc. We've had good results with different maintainers for each of these repos.

It has been challenging to get a majority of core Fabric maintainers to weigh in on decisions, since only about half of the maintainers are active day-to-day. This becomes more of an issue with the recent introduction of the fabric-rfcs process, which requires a majority of Fabric maintainers to agree before taking on new enhancements.

With the transition to Azure Pipelines, there is not (yet) an easy way to suppress CI for doc-only PRs.

With the transition to GitHub, we would like different branch protection rules for Fabric code and Fabric docs.


Dave Enyeart

"Yacov" ---11/26/2019 01:49:37 AM---I think the best solution for these 2 seemingly conflicting ideas is: All maintainers of code reposi

From:
"Yacov" <yacovm@...>
To:
"Brian Behlendorf" <bbehlendorf@...>
Cc:
fabric@...
Date:
11/26/2019 01:49 AM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by:
fabric@...




I think the best solution for these 2 seemingly conflicting ideas is:
  • All maintainers of code repositories should be ableto merge documentation contributions
  • Maintainers of the documentation should not be able to merge code contributions.



From:
"Brian Behlendorf" <bbehlendorf@...>
To:
fabric@...
Date:
11/26/2019 07:36 AM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by:
fabric@...




On 11/25/19 6:09 PM, David Enyeart wrote:
I'd suggest that we identify the top documentation contributors and reviewers to seed the fabric docs repository maintainer list in the coming weeks (including Chris and Joe), rather than pulling the trigger in the Fabric repository this week.

Are there separate maintainer pools for different fabric-* repos?

If so, I can understand the argument, coming from a world where the precautionary principle would apply, and where prior version control systems (and even earlier versions of git) allowed for a dangerous degree of repository damage if someone made the wrong set of changes. I also of course get the point for different maintainers for different Hyperledger projects, e.g. Fabric vs Sawtooth.

But for the same project, which is arguably the same "community" of contributors and representatives of end-users, you may want to consider a single pool of maintainers across all fabric-* repos. The easy case is to argue for the ability for anyone who's a maintainer on the main code repos to also be able to merge in changes into docs-related repos. The harder case is for people who come in as solid contributors to docs, but aren't (yet!) code contributors. There, I'd still argue for it - the boundary between docs and code is rarely so hard, as changes to error messages/logging or admin/user interfaces often are driven by a docs-level view of what would be easier to explain. And, I've seen great core developers on projects come in first through a docs-related role. Also, just for simplicity: I'll counter your precautionary principle with an Occam's Razor, which makes understanding the project easier for newcomers. I bet reversing any mistaken merges is a lower cost than the value of contributions you might not otherwise see.

Up to you all,

Brian

--
Brian Behlendorf
Executive Director, Hyperledger

bbehlendorf@...
Twitter: @brianbehlendorf








--
Matthew Sykes
matthew.sykes@...




Re: Maintainer nominations

Matthew Sykes
 

For the record, I am not in favor of extracting the documentation. I strongly feel that the doc and the code should live together where feasible. My perspective is that lowering the barriers to contribute is far better than segmenting the ecosystem into a number of fiefdoms.

If we want doc changes to only require a single doc review, we should state that's the desired process and put a technology plan in place to make that happen.

With that, a few comments to Dave's note:

> As Fabric has grown the sets of maintainers has naturally diverged across the fabric-* repos. The core Fabric maintainers are naturally different from the Fabric CA maintainers, the Java SDK maintainers, the Node.js SDK maintainers, etc. 

The repositories were not separated because they had different maintainer pools; they were separated because, in most cases, they use fundamentally different implementation and CI processes. I'll also note that each of these contain their own documentation tree. (There's a docs folder in CA and API documentation is part of the code in node and Java.) Are you proposing we split all of these out as well?

> It has been challenging to get a majority of core Fabric maintainers to weigh in on decisions, since only about half of the maintainers are active day-to-day. This becomes more of an issue with the recent introduction of the fabric-rfcs process, which requires a majority of Fabric maintainers to agree before taking on new enhancements.

This is a problem of our own making. We have far too much process in place to make rapid forward movement. It's our choice to require a "majority" of maintainers and "2 mandatory reviews" to integrate things.


On Tue, Nov 26, 2019 at 7:11 AM David Enyeart <enyeart@...> wrote:

The intent of my proposal was exactly as Yacov says - doc maintainers would be comprised of the core Fabric maintainers in addition to heavy doc contributors/reviewers.

I do understand Brian's point, and we have been hesitant to split fabric code and fabric docs repos/maintainers up until now for the reasons Brian mentions. However in my opinion the time has come to split them for the following reasons:

As Fabric has grown the sets of maintainers has naturally diverged across the fabric-* repos. The core Fabric maintainers are naturally different from the Fabric CA maintainers, the Java SDK maintainers, the Node.js SDK maintainers, etc. We've had good results with different maintainers for each of these repos.

It has been challenging to get a majority of core Fabric maintainers to weigh in on decisions, since only about half of the maintainers are active day-to-day. This becomes more of an issue with the recent introduction of the fabric-rfcs process, which requires a majority of Fabric maintainers to agree before taking on new enhancements.

With the transition to Azure Pipelines, there is not (yet) an easy way to suppress CI for doc-only PRs.

With the transition to GitHub, we would like different branch protection rules for Fabric code and Fabric docs.


Dave Enyeart

"Yacov" ---11/26/2019 01:49:37 AM---I think the best solution for these 2 seemingly conflicting ideas is: All maintainers of code reposi

From: "Yacov" <yacovm@...>
To: "Brian Behlendorf" <bbehlendorf@...>
Cc: fabric@...
Date: 11/26/2019 01:49 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by: fabric@...





I think the best solution for these 2 seemingly conflicting ideas is:
    • All maintainers of code repositories should be able to merge documentation contributions
    • Maintainers of the documentation should not be able to merge code contributions.



From:
"Brian Behlendorf" <bbehlendorf@...>
To:
fabric@...
Date:
11/26/2019 07:36 AM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by:
fabric@...




On 11/25/19 6:09 PM, David Enyeart wrote:
I'd suggest that we identify the top documentation contributors and reviewers to seed the fabric docs repository maintainer list in the coming weeks (including Chris and Joe), rather than pulling the trigger in the Fabric repository this week.

Are there separate maintainer pools for different fabric-* repos?

If so, I can understand the argument, coming from a world where the precautionary principle would apply, and where prior version control systems (and even earlier versions of git) allowed for a dangerous degree of repository damage if someone made the wrong set of changes. I also of course get the point for different maintainers for different Hyperledger projects, e.g. Fabric vs Sawtooth.

But for the same project, which is arguably the same "community" of contributors and representatives of end-users, you may want to consider a single pool of maintainers across all fabric-* repos. The easy case is to argue for the ability for anyone who's a maintainer on the main code repos to also be able to merge in changes into docs-related repos. The harder case is for people who come in as solid contributors to docs, but aren't (yet!) code contributors. There, I'd still argue for it - the boundary between docs and code is rarely so hard, as changes to error messages/logging or admin/user interfaces often are driven by a docs-level view of what would be easier to explain. And, I've seen great core developers on projects come in first through a docs-related role. Also, just for simplicity: I'll counter your precautionary principle with an Occam's Razor, which makes understanding the project easier for newcomers. I bet reversing any mistaken merges is a lower cost than the value of contributions you might not otherwise see.

Up to you all,

Brian

--
Brian Behlendorf
Executive Director, Hyperledger

bbehlendorf@...
Twitter: @brianbehlendorf









--
Matthew Sykes
matthew.sykes@...


回复: [Hyperledger Fabric] Maintainer nominations

david liu <david-khala@...>
 

+1 for Jay's suggestion

发件人: fabric@... <fabric@...> 代表 Jay Guo <guojiannan1101@...>
发送时间: 2019年11月26日 22:11
收件人: Jay Guo <guojiannan1101@...>
抄送: David Enyeart <enyeart@...>; fabric <fabric@...>
主题: Re: [Hyperledger Fabric] Maintainer nominations
 
Just in case i wasn't stating my opinion clearly, i'd suggest to separate docs from code, and have multiple languages dirs within it, e.g.

content
   |- en
   |- zh
   |- de


- J

On Tue, Nov 26, 2019 at 9:25 PM Jay Guo via Lists.Hyperledger.Org <guojiannan1101=gmail.com@...> wrote:

from translation point of view, i've seen some popular projects like k8s and tensorflow that separate docs from code, where i18n is first class citizen within the dir structure, for example [1] and [2].

we, technical working group china, has been translating docs to Chinese via [3] and serve it from our own readthedocs domain. This has been giving us problems in marketing the material, leveraging community tools, and incentivizing translators.

With CODEOWNERS, i think we will be able to have fine-grained maintainership across different languages


On Tue, Nov 26, 2019 at 8:11 PM David Enyeart <enyeart@...> wrote:

The intent of my proposal was exactly as Yacov says - doc maintainers would be comprised of the core Fabric maintainers in addition to heavy doc contributors/reviewers.

I do understand Brian's point, and we have been hesitant to split fabric code and fabric docs repos/maintainers up until now for the reasons Brian mentions. However in my opinion the time has come to split them for the following reasons:

As Fabric has grown the sets of maintainers has naturally diverged across the fabric-* repos. The core Fabric maintainers are naturally different from the Fabric CA maintainers, the Java SDK maintainers, the Node.js SDK maintainers, etc. We've had good results with different maintainers for each of these repos.

It has been challenging to get a majority of core Fabric maintainers to weigh in on decisions, since only about half of the maintainers are active day-to-day. This becomes more of an issue with the recent introduction of the fabric-rfcs process, which requires a majority of Fabric maintainers to agree before taking on new enhancements.

With the transition to Azure Pipelines, there is not (yet) an easy way to suppress CI for doc-only PRs.

With the transition to GitHub, we would like different branch protection rules for Fabric code and Fabric docs.


Dave Enyeart

"Yacov" ---11/26/2019 01:49:37 AM---I think the best solution for these 2 seemingly conflicting ideas is: All maintainers of code reposi

From: "Yacov" <yacovm@...>
To: "Brian Behlendorf" <bbehlendorf@...>
Cc: fabric@...
Date: 11/26/2019 01:49 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by: fabric@...





I think the best solution for these 2 seemingly conflicting ideas is:
    • All maintainers of code repositories should be able to merge documentation contributions
    • Maintainers of the documentation should not be able to merge code contributions.



From:
"Brian Behlendorf" <bbehlendorf@...>
To:
fabric@...
Date:
11/26/2019 07:36 AM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by:
fabric@...




On 11/25/19 6:09 PM, David Enyeart wrote:
I'd suggest that we identify the top documentation contributors and reviewers to seed the fabric docs repository maintainer list in the coming weeks (including Chris and Joe), rather than pulling the trigger in the Fabric repository this week.

Are there separate maintainer pools for different fabric-* repos?

If so, I can understand the argument, coming from a world where the precautionary principle would apply, and where prior version control systems (and even earlier versions of git) allowed for a dangerous degree of repository damage if someone made the wrong set of changes. I also of course get the point for different maintainers for different Hyperledger projects, e.g. Fabric vs Sawtooth.

But for the same project, which is arguably the same "community" of contributors and representatives of end-users, you may want to consider a single pool of maintainers across all fabric-* repos. The easy case is to argue for the ability for anyone who's a maintainer on the main code repos to also be able to merge in changes into docs-related repos. The harder case is for people who come in as solid contributors to docs, but aren't (yet!) code contributors. There, I'd still argue for it - the boundary between docs and code is rarely so hard, as changes to error messages/logging or admin/user interfaces often are driven by a docs-level view of what would be easier to explain. And, I've seen great core developers on projects come in first through a docs-related role. Also, just for simplicity: I'll counter your precautionary principle with an Occam's Razor, which makes understanding the project easier for newcomers. I bet reversing any mistaken merges is a lower cost than the value of contributions you might not otherwise see.

Up to you all,

Brian

--
Brian Behlendorf
Executive Director, Hyperledger

bbehlendorf@...
Twitter: @brianbehlendorf








Re: Maintainer nominations

Jay Guo
 

Just in case i wasn't stating my opinion clearly, i'd suggest to separate docs from code, and have multiple languages dirs within it, e.g.

content
   |- en
   |- zh
   |- de


- J

On Tue, Nov 26, 2019 at 9:25 PM Jay Guo via Lists.Hyperledger.Org <guojiannan1101=gmail.com@...> wrote:
from translation point of view, i've seen some popular projects like k8s and tensorflow that separate docs from code, where i18n is first class citizen within the dir structure, for example [1] and [2].

we, technical working group china, has been translating docs to Chinese via [3] and serve it from our own readthedocs domain. This has been giving us problems in marketing the material, leveraging community tools, and incentivizing translators.

With CODEOWNERS, i think we will be able to have fine-grained maintainership across different languages


On Tue, Nov 26, 2019 at 8:11 PM David Enyeart <enyeart@...> wrote:

The intent of my proposal was exactly as Yacov says - doc maintainers would be comprised of the core Fabric maintainers in addition to heavy doc contributors/reviewers.

I do understand Brian's point, and we have been hesitant to split fabric code and fabric docs repos/maintainers up until now for the reasons Brian mentions. However in my opinion the time has come to split them for the following reasons:

As Fabric has grown the sets of maintainers has naturally diverged across the fabric-* repos. The core Fabric maintainers are naturally different from the Fabric CA maintainers, the Java SDK maintainers, the Node.js SDK maintainers, etc. We've had good results with different maintainers for each of these repos.

It has been challenging to get a majority of core Fabric maintainers to weigh in on decisions, since only about half of the maintainers are active day-to-day. This becomes more of an issue with the recent introduction of the fabric-rfcs process, which requires a majority of Fabric maintainers to agree before taking on new enhancements.

With the transition to Azure Pipelines, there is not (yet) an easy way to suppress CI for doc-only PRs.

With the transition to GitHub, we would like different branch protection rules for Fabric code and Fabric docs.


Dave Enyeart

"Yacov" ---11/26/2019 01:49:37 AM---I think the best solution for these 2 seemingly conflicting ideas is: All maintainers of code reposi

From: "Yacov" <yacovm@...>
To: "Brian Behlendorf" <bbehlendorf@...>
Cc: fabric@...
Date: 11/26/2019 01:49 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by: fabric@...





I think the best solution for these 2 seemingly conflicting ideas is:
    • All maintainers of code repositories should be able to merge documentation contributions
    • Maintainers of the documentation should not be able to merge code contributions.



From:
"Brian Behlendorf" <bbehlendorf@...>
To:
fabric@...
Date:
11/26/2019 07:36 AM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by:
fabric@...




On 11/25/19 6:09 PM, David Enyeart wrote:
I'd suggest that we identify the top documentation contributors and reviewers to seed the fabric docs repository maintainer list in the coming weeks (including Chris and Joe), rather than pulling the trigger in the Fabric repository this week.

Are there separate maintainer pools for different fabric-* repos?

If so, I can understand the argument, coming from a world where the precautionary principle would apply, and where prior version control systems (and even earlier versions of git) allowed for a dangerous degree of repository damage if someone made the wrong set of changes. I also of course get the point for different maintainers for different Hyperledger projects, e.g. Fabric vs Sawtooth.

But for the same project, which is arguably the same "community" of contributors and representatives of end-users, you may want to consider a single pool of maintainers across all fabric-* repos. The easy case is to argue for the ability for anyone who's a maintainer on the main code repos to also be able to merge in changes into docs-related repos. The harder case is for people who come in as solid contributors to docs, but aren't (yet!) code contributors. There, I'd still argue for it - the boundary between docs and code is rarely so hard, as changes to error messages/logging or admin/user interfaces often are driven by a docs-level view of what would be easier to explain. And, I've seen great core developers on projects come in first through a docs-related role. Also, just for simplicity: I'll counter your precautionary principle with an Occam's Razor, which makes understanding the project easier for newcomers. I bet reversing any mistaken merges is a lower cost than the value of contributions you might not otherwise see.

Up to you all,

Brian

--
Brian Behlendorf
Executive Director, Hyperledger

bbehlendorf@...
Twitter: @brianbehlendorf








Re: Maintainer nominations

Jay Guo
 

from translation point of view, i've seen some popular projects like k8s and tensorflow that separate docs from code, where i18n is first class citizen within the dir structure, for example [1] and [2].

we, technical working group china, has been translating docs to Chinese via [3] and serve it from our own readthedocs domain. This has been giving us problems in marketing the material, leveraging community tools, and incentivizing translators.

With CODEOWNERS, i think we will be able to have fine-grained maintainership across different languages


On Tue, Nov 26, 2019 at 8:11 PM David Enyeart <enyeart@...> wrote:

The intent of my proposal was exactly as Yacov says - doc maintainers would be comprised of the core Fabric maintainers in addition to heavy doc contributors/reviewers.

I do understand Brian's point, and we have been hesitant to split fabric code and fabric docs repos/maintainers up until now for the reasons Brian mentions. However in my opinion the time has come to split them for the following reasons:

As Fabric has grown the sets of maintainers has naturally diverged across the fabric-* repos. The core Fabric maintainers are naturally different from the Fabric CA maintainers, the Java SDK maintainers, the Node.js SDK maintainers, etc. We've had good results with different maintainers for each of these repos.

It has been challenging to get a majority of core Fabric maintainers to weigh in on decisions, since only about half of the maintainers are active day-to-day. This becomes more of an issue with the recent introduction of the fabric-rfcs process, which requires a majority of Fabric maintainers to agree before taking on new enhancements.

With the transition to Azure Pipelines, there is not (yet) an easy way to suppress CI for doc-only PRs.

With the transition to GitHub, we would like different branch protection rules for Fabric code and Fabric docs.


Dave Enyeart

"Yacov" ---11/26/2019 01:49:37 AM---I think the best solution for these 2 seemingly conflicting ideas is: All maintainers of code reposi

From: "Yacov" <yacovm@...>
To: "Brian Behlendorf" <bbehlendorf@...>
Cc: fabric@...
Date: 11/26/2019 01:49 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by: fabric@...





I think the best solution for these 2 seemingly conflicting ideas is:
    • All maintainers of code repositories should be able to merge documentation contributions
    • Maintainers of the documentation should not be able to merge code contributions.



From:
"Brian Behlendorf" <bbehlendorf@...>
To:
fabric@...
Date:
11/26/2019 07:36 AM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by:
fabric@...




On 11/25/19 6:09 PM, David Enyeart wrote:
I'd suggest that we identify the top documentation contributors and reviewers to seed the fabric docs repository maintainer list in the coming weeks (including Chris and Joe), rather than pulling the trigger in the Fabric repository this week.

Are there separate maintainer pools for different fabric-* repos?

If so, I can understand the argument, coming from a world where the precautionary principle would apply, and where prior version control systems (and even earlier versions of git) allowed for a dangerous degree of repository damage if someone made the wrong set of changes. I also of course get the point for different maintainers for different Hyperledger projects, e.g. Fabric vs Sawtooth.

But for the same project, which is arguably the same "community" of contributors and representatives of end-users, you may want to consider a single pool of maintainers across all fabric-* repos. The easy case is to argue for the ability for anyone who's a maintainer on the main code repos to also be able to merge in changes into docs-related repos. The harder case is for people who come in as solid contributors to docs, but aren't (yet!) code contributors. There, I'd still argue for it - the boundary between docs and code is rarely so hard, as changes to error messages/logging or admin/user interfaces often are driven by a docs-level view of what would be easier to explain. And, I've seen great core developers on projects come in first through a docs-related role. Also, just for simplicity: I'll counter your precautionary principle with an Occam's Razor, which makes understanding the project easier for newcomers. I bet reversing any mistaken merges is a lower cost than the value of contributions you might not otherwise see.

Up to you all,

Brian

--
Brian Behlendorf
Executive Director, Hyperledger

bbehlendorf@...
Twitter: @brianbehlendorf








Re: Maintainer nominations

David Enyeart
 

The intent of my proposal was exactly as Yacov says - doc maintainers would be comprised of the core Fabric maintainers in addition to heavy doc contributors/reviewers.

I do understand Brian's point, and we have been hesitant to split fabric code and fabric docs repos/maintainers up until now for the reasons Brian mentions. However in my opinion the time has come to split them for the following reasons:

As Fabric has grown the sets of maintainers has naturally diverged across the fabric-* repos. The core Fabric maintainers are naturally different from the Fabric CA maintainers, the Java SDK maintainers, the Node.js SDK maintainers, etc. We've had good results with different maintainers for each of these repos.

It has been challenging to get a majority of core Fabric maintainers to weigh in on decisions, since only about half of the maintainers are active day-to-day. This becomes more of an issue with the recent introduction of the fabric-rfcs process, which requires a majority of Fabric maintainers to agree before taking on new enhancements.

With the transition to Azure Pipelines, there is not (yet) an easy way to suppress CI for doc-only PRs.

With the transition to GitHub, we would like different branch protection rules for Fabric code and Fabric docs.


Dave Enyeart

"Yacov" ---11/26/2019 01:49:37 AM---I think the best solution for these 2 seemingly conflicting ideas is: All maintainers of code reposi

From: "Yacov" <yacovm@...>
To: "Brian Behlendorf" <bbehlendorf@...>
Cc: fabric@...
Date: 11/26/2019 01:49 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by: fabric@...





I think the best solution for these 2 seemingly conflicting ideas is:
    • All maintainers of code repositories should be able to merge documentation contributions
    • Maintainers of the documentation should not be able to merge code contributions.



From:
"Brian Behlendorf" <bbehlendorf@...>
To:
fabric@...
Date:
11/26/2019 07:36 AM
Subject:
[EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by:
fabric@...




On 11/25/19 6:09 PM, David Enyeart wrote:
I'd suggest that we identify the top documentation contributors and reviewers to seed the fabric docs repository maintainer list in the coming weeks (including Chris and Joe), rather than pulling the trigger in the Fabric repository this week.

Are there separate maintainer pools for different fabric-* repos?

If so, I can understand the argument, coming from a world where the precautionary principle would apply, and where prior version control systems (and even earlier versions of git) allowed for a dangerous degree of repository damage if someone made the wrong set of changes. I also of course get the point for different maintainers for different Hyperledger projects, e.g. Fabric vs Sawtooth.

But for the same project, which is arguably the same "community" of contributors and representatives of end-users, you may want to consider a single pool of maintainers across all fabric-* repos. The easy case is to argue for the ability for anyone who's a maintainer on the main code repos to also be able to merge in changes into docs-related repos. The harder case is for people who come in as solid contributors to docs, but aren't (yet!) code contributors. There, I'd still argue for it - the boundary between docs and code is rarely so hard, as changes to error messages/logging or admin/user interfaces often are driven by a docs-level view of what would be easier to explain. And, I've seen great core developers on projects come in first through a docs-related role. Also, just for simplicity: I'll counter your precautionary principle with an Occam's Razor, which makes understanding the project easier for newcomers. I bet reversing any mistaken merges is a lower cost than the value of contributions you might not otherwise see.

Up to you all,

Brian

--
Brian Behlendorf
Executive Director, Hyperledger

bbehlendorf@...
Twitter: @brianbehlendorf








Re: #fabric Certificates and Keys generation and sharing #fabric

Jean-Gaël Dominé <jgdomine@...>
 

Something I forgot to mention, I'm also going investigate the use of a HSM.

Maybe this could make things easier as apparently peers and orderers can connect to the HSM. If it enabled the retrieval of their certificate and private key, this would be great.


#fabric Certificates and Keys generation and sharing #fabric

Jean-Gaël Dominé <jgdomine@...>
 

Hi all,

I would like to expose some ideas to have your feelings and ideas about the generation and sharing of the artifacts because I'm not sure about which path to take.
In my network (deployed in Kubernetes) I have a batch that generates all the certificates and keys (TLS included) of the admins, peers, orderers, genesis block, ... by connecting to the CA using fabric-ca-client.
Then I export all these artifacts as secrets in K8S so that the components have access to them.
This works fine but it does not look like production mode to me.

So I was trying to think of how this process would be handled in production. Here are some ideas:

1) Peers and orderers enroll themselves at startup:
  • by installing fabric-ca-client on them so that they can register and enroll to the CA
  • by exposing some endpoints on an API using the SDK that they would call
This would work if we add peers and orderers to the existing the organizations as normally the root certificates are already present in the genesis block and thus known by everybody.
But an issue I foresee is the management in case the component restarts, we must avoid going through the registration/enrollment again since it was already done. How can this be achieved?
Also the LCM of the certificates could be an issue

Besides this would become more complex to add a new organization.
In case a new organization is added, I don't see how to automate it since the system channel configuration must be updated...

So if anyone has a better idea on how to handle this part of Fabric, I'd be happy to learn about it :)

Thanks

JG


Re: Proposal : Hyperledger Fabric block archiving

Gari Singh <garis@...>
 

Hi Atsushi -

Thanks for sharing your efforts to date.

Overall, I like the idea of providing a utility to do this as generally we tell people that they can do this but don't provide any tools for doing so.

I do, however, have concerns about integrating any part of this functionality into the actual peer binary itself. I don't actually think you need to do that.
I think running a separate "archive client" without modifying the peer is the way to go. It keeps this functionality clean and separate from the peer and allows it to progress on its own.

It seems the only thing you really wanted to use the peer for was to propagate information to other peers within the same organization. My take here is that you can more easily do something such as having the archive client write its status to a file in the "archiver repository". This way other archiver clients within the same organization can simply periodically poll this file for status. Additionally, you can also use this same file repository to maintain some type of process lock file such that you'll only have one archiver client actively performing the archival.

-- G

-----------------------------------------
Gari Singh
Distinguished Engineer, CTO - IBM Blockchain
IBM Middleware
550 King St
Littleton, MA 01460
Cell: 978-846-7499
garis@...
-----------------------------------------

-----fabric@... wrote: -----
To: "Manish" <manish.sethi@...>
From: "Yacov"
Sent by: fabric@...
Date: 11/25/2019 04:21PM
Cc: nekia <atsushin@...>, "fabric@..." <fabric@...>
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Proposal : Hyperledger Fabric block archiving

Hey Atsushi,

one thing I noticed while skimming the code, is that while you send the ArchivedBlockFileMsg via gossip, you are not ensuring it is eventually propagated to peers successfully.

This means that if a peer didn't get the message, it won't archive your file.

I suggest that you think of a more robust mechanism, like periodically comparing digests of ranges.

The code in https://github.com/hyperledger-labs/fabric-block-archiving/blob/master/gossip/gossip/pull/pullstore.go is a generic pull mechanism based on digests. You might want to give it a look.


- Yacov.



From: "Manish" <manish.sethi@...>
To: nekia <atsushin@...>
Cc: "fabric@..." <fabric@...>
Date: 11/25/2019 10:50 PM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Proposal : Hyperledger Fabric block archiving
Sent by: fabric@...



Hi Atsushi,

Thanks for your proposal and at high level the objective makes sense to me and below is my high level observations that you may want to consider.

First, the fundamental assumption that you make is that all the block files are same across peers is incorrect. The block files are not guaranteed to contain same number of blocks across peers. This is because a block file is bounded by the file size and not by the number of blocks. Further, the size of a given block may vary slightly on each peer. Though the header and the data section of a blocks are of same size across peers but this difference in overall size could be caused by the metadata section which contains concenter signatures. In addition, on some of the peers, the metadata may also include block commit hash as an additional data. So, either you have to operate at the block numbers (i.e., during purging an archiver client on a peer deals a file that should be purged partially based on where in the file the target block is located) or if you want to deal at the files level the archiver client could just consider files up to previous file.

Second, there are certain kind of queries for which a peer assumes the presence of block files in the current way. This primarily includes history queries, blocks related queries, and txid related queries. These queries may start failing or lead to crashes or unexpected results if you simply delete the files. I did not see any details in your design how you plan to handle these. The potential solutions may range from simply denying these kind of queries to more sophisticated solution such as serving the queries by involving the achiever repository. However, in either of these the challenge would be to know that the desired block/ transaction has been purged from the local peer (e.g., consider blockByHash or transactionByTxid kind of queries.)

Third, somewhat similar to the second point above, peer has a feature wherein it rebuilds the statedb and historydb if they are dropped and peer is simply restarted. For this feature as well it relies on the percense of blockfiles.

Fourth, I am not sure if gossip is the right communication mechanism that you want to employ for this communication. An archiver client perhaps can simply poll (or register for updates with) the archiver repository.

Finally, I would like to understand in more details what are the benefits of having a separate repository? Why not simply let the files be there on the anchor peer and purge from other peers? If the answer is compression, then ideally we should explore a choise of writing the data in blockfiles in compressed format.


Hope this helps.

Thanks,
Manish

On Thu, Nov 14, 2019 at 10:26 PM nekia <atsushin@...> wrote:
Hello everybody,

I’d like to propose a new feature ‘block archiving’ for Hyperledger Fabric. We are working on this block archiving project which is listed under Hyperledger Labs repository. Our current main efforts are focused on improvement of reliability. If we could get some feedback on our proposed feature from members involved in Hyperledger Fabric implementation, it’ll be quite useful for further improvement of UX.

- Hyperledger Fabric Block Archiving
https://github.com/hyperledger-labs/fabric-block-archiving

This enhancement for Hyperledger Fabric is aiming to:

- Reduce the total amount of storage space required for an organisation to operate a Hyperledger Fabric network by archiving block data into repository.
- For organisations, operate a Hyperledger Fabric network with low resourced nodes, such as a IoT edge devices for example.

- Our proposal
https://github.com/hyperledger-labs/hyperledger-labs.github.io/blob/master/labs/fabric-block-archiving.md

- Technical overview
https://github.com/nekia/fabric-block-archiving/blob/techoverview/BlockVault%20-%20Technical%20Overview.pdf


Kind regards,
Atsushi Neki
RocketChat: nekia

Atsushi Neki
Senior Software Development Engineer

Fujitsu Australia Software Technology Pty Ltd
14 Rodborough Road, Frenchs Forest NSW 2086, Australia
T +61 2 9452 9036 M +61 428 223 387
AtsushiN@...
fastware.com.au



Disclaimer
The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.

Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.

If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe@...


Re: Maintainer nominations

Yacov
 

I think the best solution for these 2 seemingly conflicting ideas is:
  • All maintainers of code repositories should be able to merge documentation contributions
  • Maintainers of the documentation should not be able to merge code contributions.



From:        "Brian Behlendorf" <bbehlendorf@...>
To:        fabric@...
Date:        11/26/2019 07:36 AM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Maintainer nominations
Sent by:        fabric@...




On 11/25/19 6:09 PM, David Enyeart wrote:
I'd suggest that we identify the top documentation contributors and reviewers to seed the fabric docs repository maintainer list in the coming weeks (including Chris and Joe), rather than pulling the trigger in the Fabric repository this week.

Are there separate maintainer pools for different fabric-* repos? 

If so, I can understand the argument, coming from a world where the precautionary principle would apply, and where prior version control systems (and even earlier versions of git) allowed for a dangerous degree of repository damage if someone made the wrong set of changes. I also of course get the point for different maintainers for different Hyperledger projects, e.g. Fabric vs Sawtooth. 

But for the same project, which is arguably the same "community" of contributors and representatives of end-users, you may want to consider a single pool of maintainers across all fabric-* repos.  The easy case is to argue for the ability for anyone who's a maintainer on the main code repos to also be able to merge in changes into docs-related repos.  The harder case is for people who come in as solid contributors to docs, but aren't (yet!) code contributors.  There, I'd still argue for it - the boundary between docs and code is rarely so hard, as changes to error messages/logging or admin/user interfaces often are driven by a docs-level view of what would be easier to explain.  And, I've seen great core developers on projects come in first through a docs-related role.  Also, just for simplicity: I'll counter your precautionary principle with an Occam's Razor, which makes understanding the project easier for newcomers.  I bet reversing any mistaken merges is a lower cost than the value of contributions you might not otherwise see.

Up to you all,

Brian

--
Brian Behlendorf
Executive Director, Hyperledger
bbehlendorf@...
Twitter: @brianbehlendorf





Re: Maintainer nominations

Anthony O'Dowd <a_o-dowd@...>
 

Thank you Pam.

I'm very much in favour of these two nominations, and personally, I'd like to see more maintainers and contributors to Hyperledger Fabric in general, and documentation in particular.

On a related note, an additional request would be to support the work that Rich Zhao started on Chinese language docs. With our move to GitHub, and the soon creation of a separate docs repo, two of the items that Rich requires to make progress are overcome.  A final requirement would be a Chinese language docs maintainer.  I'd be happy to discuss this topic and the introduction of other languages on the next Documentation workgroup calls -- it's probably most relevant on the Eastern hemi call. I'll add an item to both agendas.

Thanks, Anthony.


Re: #fabric #raft Orderers and organization, how to organize them? #fabric #raft

Yueming Xu
 

I do not think you’d benefit by using different root CA’s for orderer and peer of the same org. Each org already need 2 root CA’s, one for signing cert, the other for TLS. If you double that, it’s just too much management overhead for not much gain, I think. 

Yueming Xu


Re: Maintainer nominations

Brian Behlendorf <bbehlendorf@...>
 

On 11/25/19 6:09 PM, David Enyeart wrote:
I'd suggest that we identify the top documentation contributors and reviewers to seed the fabric docs repository maintainer list in the coming weeks (including Chris and Joe), rather than pulling the trigger in the Fabric repository this week.

Are there separate maintainer pools for different fabric-* repos? 

If so, I can understand the argument, coming from a world where the precautionary principle would apply, and where prior version control systems (and even earlier versions of git) allowed for a dangerous degree of repository damage if someone made the wrong set of changes. I also of course get the point for different maintainers for different Hyperledger projects, e.g. Fabric vs Sawtooth. 

But for the same project, which is arguably the same "community" of contributors and representatives of end-users, you may want to consider a single pool of maintainers across all fabric-* repos.  The easy case is to argue for the ability for anyone who's a maintainer on the main code repos to also be able to merge in changes into docs-related repos.  The harder case is for people who come in as solid contributors to docs, but aren't (yet!) code contributors.  There, I'd still argue for it - the boundary between docs and code is rarely so hard, as changes to error messages/logging or admin/user interfaces often are driven by a docs-level view of what would be easier to explain.  And, I've seen great core developers on projects come in first through a docs-related role.  Also, just for simplicity: I'll counter your precautionary principle with an Occam's Razor, which makes understanding the project easier for newcomers.  I bet reversing any mistaken merges is a lower cost than the value of contributions you might not otherwise see.

Up to you all,

Brian


-- 
Brian Behlendorf
Executive Director, Hyperledger
bbehlendorf@...
Twitter: @brianbehlendorf


Re: Maintainer nominations

David Enyeart
 

Our intent has been to split Fabric docs into its own repository after the transition to GitHub and Azure Pipelines. We only have one more repository to switch over (fabric-test), then will proceed with the documentation split. If anybody would like to drive the documentation split in parallel, that would certainly expedite things.

I'd suggest that we identify the top documentation contributors and reviewers to seed the fabric docs repository maintainer list in the coming weeks (including Chris and Joe), rather than pulling the trigger in the Fabric repository this week.


Dave Enyeart

"Pam Andrejko" ---11/25/2019 05:26:48 PM---All, I would like to nominate two new Documentation maintainers for the Hyperledger Fabric project.

From: "Pam Andrejko" <pama@...>
To: fabric@...
Date: 11/25/2019 05:26 PM
Subject: [EXTERNAL] [Hyperledger Fabric] Maintainer nominations
Sent by: fabric@...





All,

I would like to nominate two new Documentation maintainers for the Hyperledger Fabric project. They are Chris Gabriel (Hyperchain Labs) and Joe Alewine (IBM).


Chris has been an instrumental member of the Documentation community workgroup for several years now. In addition to being a regular content reviewer and contributor, he is a consumer of Fabric in his own Hyperchain Labs business that he founded. The insights, perspective, and content that he's provided based on his experience have been invaluable to the documentation work group and Fabric community as a whole.

Joe has been providing important Fabric documentation for over two and half years, is recognized as an expert on the ordering service, the Fabric upgrade process and channel capabilities, and was recently recognized as a
Hyperledger significant contributor.

Adding two more Documentation Maintainers will greatly facilitate the addition and approval of Fabric documentation content going forward.

I have opened a separate PR for each nomination:

Chris Gabriel - https://github.com/hyperledger/fabric/pull/317
Joe Alewine -
https://github.com/hyperledger/fabric/pull/316

I'm requesting that existing maintainers review the nominations and indicate whether they agree with a comment in the PR. Other’s are welcome to provide their input.

Warm regards,
Pam Andrejko





Re: Hyperledger Fabric GitHub Migration

David Enyeart
 

The transition to GitHub for source control management, and Azure Pipelines for CI, is now complete for all Fabric development repositories. The final repository to shift will be fabric-test in early December.

You can open pull requests at https://github.com/hyperledger/fabric/pulls.

We use the standard GitHub fork and branch PR workflow. If you need a refresher on the workflow, please see the instructions at https://hyperledger-fabric.readthedocs.io/en/latest/github/github.html.

Thanks to Brett Logan for making the transition a smooth one!


Dave Enyeart

"Brett T Logan" ---11/21/2019 01:11:31 AM---Hello Contributors,

From: "Brett T Logan" <brett.t.logan@...>
To: fabric@...
Date: 11/21/2019 01:11 AM
Subject: [EXTERNAL] [Hyperledger Fabric] Hyperledger Fabric GitHub Migration
Sent by: fabric@...





Hello Contributors,

The time has finally come. The Hyperledger Fabric maintainers are planning for a migration of the core Fabric repository to GitHub this Friday morning Eastern Standard Time.

We are asking that effective immediately, all contributors stop pushing changes to Gerrit. Instead contributors can open their changes as pull requests using the Hyperledger Fabric repository in Github https://github.com/hyperledger/fabric. We have already configured CI to run against new pull requests using Azure Pipelines. This will allow the maintainers time merge as many Gerrit CR's as they can before the cutover.

Any changes that don't make it in before the Friday cutover will be abandoned and contributors will need to reopen them in GitHub. If you feel it's unlikely your change will make it in by Friday morning, we ask that you move it to GitHub now, and close your CR so maintainers can focus on changes that will get merged in the next day.

We will be publishing updated documentation about best practices, but in the meantime a few reminders about GitHub contributions:
    • Commits should be focused and small
    • Commit messages should include the Jira number on their first line and the body should include meaningful information on the change
    • Your signature must be included in your commit message, you can do this using the "-s" flag when running the "git commit" command
    • When opening a pull request it must come from your fork of the Fabric repository
    • When opening the pull request, your pull request message should include a meaningful title and provide the reasoning around the change, this will help maintainers understand what you are attempting to achieve (we will be publishing an automatic template yet this week, once that happens you should fill out the template accordingly)
With this migration, Hyperledger will have migrated all of its development repositories off of Gerrit and Jenkins. Contributions are welcome to any of the Hyperledger projects through GitHub moving forward. It is our hope that by adopting this industry standard we can lower the barrier of entry for new contributors and attract even more of the community to participate in contributing.

As always, thank you for your contributions!

Brett Logan
Software Engineer, IBM Blockchain
Phone: 1-984-242-6890
E-mail: brett.t.logan@...







Re: Maintainer nominations

Yacov
 

>  and was recently recognized as a Hyperledger significant contributor.


Just for the record.... only IBMers can access this link....        



From:        "Pam Andrejko" <pama@...>
To:        fabric@...
Date:        11/26/2019 12:26 AM
Subject:        [EXTERNAL] [Hyperledger Fabric] Maintainer nominations
Sent by:        fabric@...




All,

I would like to nominate two new Documentation maintainers for the Hyperledger Fabric project. They are Chris Gabriel (Hyperchain Labs) and Joe Alewine (IBM).

Chris has been an instrumental member of the Documentation community workgroup for several years now. In addition to being a regular content reviewer and contributor, he is a consumer of Fabric in his own Hyperchain Labs business that he founded. The insights, perspective, and content that he's provided based on his experience have been invaluable to the documentation work group and Fabric community as a whole.

Joe has been providing important Fabric documentation for over two and half years, is recognized as an expert on the ordering service, the Fabric upgrade process and channel capabilities, and was recently recognized as a Hyperledger significant contributor.

Adding two more Documentation Maintainers will greatly facilitate the addition and approval of Fabric documentation content going forward.

I have opened a separate PR for each nomination:

Chris Gabriel - https://github.com/hyperledger/fabric/pull/317
Joe Alewine - https://github.com/hyperledger/fabric/pull/316

I'm requesting that existing maintainers review the nominations and indicate whether they agree with a comment in the PR. Other’s are welcome to provide their input.

Warm regards,
Pam Andrejko





Maintainer nominations

Pam Andrejko
 

All,

I would like to nominate two new Documentation maintainers for the Hyperledger Fabric project. They are Chris Gabriel (Hyperchain Labs) and Joe Alewine (IBM).

Chris has been an instrumental member of the Documentation community workgroup for several years now. In addition to being a regular content reviewer and contributor, he is a consumer of Fabric in his own Hyperchain Labs business that he founded. The insights, perspective, and content that he's provided based on his experience have been invaluable to the documentation work group and Fabric community as a whole.

Joe has been providing important Fabric documentation for over two and half years, is recognized as an expert on the ordering service, the Fabric upgrade process and channel capabilities, and was recently recognized as a Hyperledger significant contributor.

Adding two more Documentation Maintainers will greatly facilitate the addition and approval of Fabric documentation content going forward.

I have opened a separate PR for each nomination:

Chris Gabriel - https://github.com/hyperledger/fabric/pull/317
Joe Alewine - https://github.com/hyperledger/fabric/pull/316

I'm requesting that existing maintainers review the nominations and indicate whether they agree with a comment in the PR. Other’s are welcome to provide their input.

Warm regards,
Pam Andrejko


Re: Proposal : Hyperledger Fabric block archiving

Yacov
 

Hey Atsushi,

one thing I noticed while skimming the code, is that while you send the ArchivedBlockFileMsg via gossip, you are not ensuring it is eventually propagated to peers successfully.

This means that if a peer didn't get the message, it won't archive your file.

I suggest that you think of a more robust mechanism, like periodically comparing digests of ranges.

The code in https://github.com/hyperledger-labs/fabric-block-archiving/blob/master/gossip/gossip/pull/pullstore.go is a generic pull mechanism based on digests.  You might want to give it a look.


- Yacov.



From:        "Manish" <manish.sethi@...>
To:        nekia <atsushin@...>
Cc:        "fabric@..." <fabric@...>
Date:        11/25/2019 10:50 PM
Subject:        [EXTERNAL] Re: [Hyperledger Fabric] Proposal : Hyperledger Fabric block archiving
Sent by:        fabric@...




Hi Atsushi,

Thanks for your proposal and at high level the objective makes sense to me and below is my high level observations that you may want to consider. 

First, the fundamental assumption that you make is that all the block files are same across peers is incorrect. The block files are not guaranteed to contain same number of blocks across peers. This is because a block file is bounded by the file size and not by the  number of blocks. Further, the size of a given block may vary slightly on each peer. Though the header and the data section of a blocks are of same size across peers but this difference in overall size could be caused by the metadata section which contains concenter signatures. In addition, on some of the peers, the metadata may also include block commit hash as an additional data. So, either you have to operate at the block numbers (i.e., during purging an archiver client on a peer deals a file that should be purged partially based on where in the file the target block is located) or if you want to deal at the files level the archiver client could just consider files up to previous file.

Second, there are certain kind of queries for which a peer assumes the presence of block files in the current way. This primarily includes history queries, blocks related queries, and txid related queries. These queries may start failing or lead to crashes or unexpected results if you simply delete the files. I did not see any details in your design how you plan to handle these. The potential solutions may range from simply denying these kind of queries to more sophisticated solution such as serving the queries by involving the  achiever repository. However, in either of these the challenge would be to know that the desired block/ transaction has been purged from the local peer (e.g., consider blockByHash or transactionByTxid kind of queries.)

Third, somewhat similar to the second point above, peer has a feature wherein it rebuilds the statedb and historydb if they are dropped and peer is simply restarted. For this feature as well it relies on the percense of blockfiles.

Fourth, I am not sure if gossip is the right communication mechanism that you want to employ for this communication. An archiver client perhaps can simply poll (or register for updates with) the archiver repository.

Finally, I would like to understand in more details what are the benefits of having a separate repository? Why not simply let the files be there on the anchor peer and purge from other peers? If the answer is compression, then ideally we should explore a choise of writing the data in blockfiles in compressed format.


Hope this helps.

Thanks,
Manish


On Thu, Nov 14, 2019 at 10:26 PM nekia <atsushin@...> wrote:
Hello everybody,

 

 

I’d like to propose a new feature ‘block archiving’ for Hyperledger Fabric. We are working on this block archiving project which is listed under Hyperledger Labs repository. Our current main efforts are focused on improvement of reliability. If we could get some feedback on our proposed feature from members involved in Hyperledger Fabric implementation, it’ll be quite useful for further improvement of UX.

 

- Hyperledger Fabric Block Archiving

    https://github.com/hyperledger-labs/fabric-block-archiving

 

    This enhancement for Hyperledger Fabric is aiming to:

 

        - Reduce the total amount of storage space required for an organisation to operate a Hyperledger Fabric network by archiving block data into repository.

        - For organisations, operate a Hyperledger Fabric network with low resourced nodes, such as a IoT edge devices for example.

 

- Our proposal

    https://github.com/hyperledger-labs/hyperledger-labs.github.io/blob/master/labs/fabric-block-archiving.md

 

- Technical overview

    https://github.com/nekia/fabric-block-archiving/blob/techoverview/BlockVault%20-%20Technical%20Overview.pdf

 

 

Kind regards,

Atsushi Neki

RocketChat:  nekia

 

Atsushi Neki
Senior Software Development Engineer

Fujitsu Australia Software Technology Pty Ltd

14 Rodborough Road, Frenchs Forest NSW 2086, Australia
T
+61 2 9452 9036 M +61 428 223 387

AtsushiN@...
fastware.com.au


Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.

Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.

If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe@...





Re: Proposal : Hyperledger Fabric block archiving

Manish
 

Hi Atsushi,

Thanks for your proposal and at high level the objective makes sense to me and below is my high level observations that you may want to consider. 

First, the fundamental assumption that you make is that all the block files are same across peers is incorrect. The block files are not guaranteed to contain same number of blocks across peers. This is because a block file is bounded by the file size and not by the  number of blocks. Further, the size of a given block may vary slightly on each peer. Though the header and the data section of a blocks are of same size across peers but this difference in overall size could be caused by the metadata section which contains concenter signatures. In addition, on some of the peers, the metadata may also include block commit hash as an additional data. So, either you have to operate at the block numbers (i.e., during purging an archiver client on a peer deals a file that should be purged partially based on where in the file the target block is located) or if you want to deal at the files level the archiver client could just consider files up to previous file.

Second, there are certain kind of queries for which a peer assumes the presence of block files in the current way. This primarily includes history queries, blocks related queries, and txid related queries. These queries may start failing or lead to crashes or unexpected results if you simply delete the files. I did not see any details in your design how you plan to handle these. The potential solutions may range from simply denying these kind of queries to more sophisticated solution such as serving the queries by involving the  achiever repository. However, in either of these the challenge would be to know that the desired block/ transaction has been purged from the local peer (e.g., consider blockByHash or transactionByTxid kind of queries.)

Third, somewhat similar to the second point above, peer has a feature wherein it rebuilds the statedb and historydb if they are dropped and peer is simply restarted. For this feature as well it relies on the percense of blockfiles.

Fourth, I am not sure if gossip is the right communication mechanism that you want to employ for this communication. An archiver client perhaps can simply poll (or register for updates with) the archiver repository.

Finally, I would like to understand in more details what are the benefits of having a separate repository? Why not simply let the files be there on the anchor peer and purge from other peers? If the answer is compression, then ideally we should explore a choise of writing the data in blockfiles in compressed format.


Hope this helps.

Thanks,
Manish


On Thu, Nov 14, 2019 at 10:26 PM nekia <atsushin@...> wrote:

Hello everybody,

 

 

I’d like to propose a new feature ‘block archiving’ for Hyperledger Fabric. We are working on this block archiving project which is listed under Hyperledger Labs repository. Our current main efforts are focused on improvement of reliability. If we could get some feedback on our proposed feature from members involved in Hyperledger Fabric implementation, it’ll be quite useful for further improvement of UX.

 

- Hyperledger Fabric Block Archiving

    https://github.com/hyperledger-labs/fabric-block-archiving

 

    This enhancement for Hyperledger Fabric is aiming to:

 

        - Reduce the total amount of storage space required for an organisation to operate a Hyperledger Fabric network by archiving block data into repository.

        - For organisations, operate a Hyperledger Fabric network with low resourced nodes, such as a IoT edge devices for example.

 

- Our proposal

    https://github.com/hyperledger-labs/hyperledger-labs.github.io/blob/master/labs/fabric-block-archiving.md

 

- Technical overview

    https://github.com/nekia/fabric-block-archiving/blob/techoverview/BlockVault%20-%20Technical%20Overview.pdf

 

 

Kind regards,

Atsushi Neki

RocketChat:  nekia

 

Atsushi Neki
Senior Software Development Engineer

Fujitsu Australia Software Technology Pty Ltd

14 Rodborough Road, Frenchs Forest NSW 2086, Australia
T +61 2 9452 9036 M +61 428 223 387
AtsushiN@...
fastware.com.au


Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.


Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.


If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe@...

4281 - 4300 of 11525