Date
1 - 9 of 9
Does Raft orderer work across multiple Kubernetes clusters?
Jay Guo
Nye,
It would be nice if you could share some technical details of k8s that harm p2p network. I believe it could benefit both communities (HL and CNCF)
cheers,
- J
On Tue, Nov 19, 2019 at 3:35 PM Nye Liu <nye@...> wrote:
Not to stray too far off topic, but k8s emerged from a client/server microservices/stack deployment requirement, not a p2p requirement.Its popularity for that purpose has (unfortunately, IMO) bled into places it isn’t really appropriate for because there are more fullstack devs than p2p devs.Just my 2 cents, I know it is an unpopular opinion.On Nov 19, 2019, at 2:31 PM, Hakan Eryargi <hakan.eryargi@...> wrote:I guess K8S is the preferred way to deploy many things not only Fabric ;) and we will go production in a few months with Fabric in K8S.We are not mixing K8S clusters for now, as it's not a requirement ATM, but will eventually be the case, mixing K8S with other K8S or on premises or whatever. That time possibly my preferred way of doing it will be using hostAliases to point to other nodes.Check Raft orderer section in our repo for how we do it:
https://github.com/APGGroeiFabriek/PIVT#scaled-up-raft-networkOn Tue, Nov 19, 2019 at 11:23 PM Yueming Xu <yxucolo@...> wrote:So, in the case of a network across 2 k8s clusters, I have to make all orderer nodes accessible outside of k8s, and put the public host names and ports in the genesis block. Correct? K8s is still a preferred way to deploy the fabric network as I can see it, so such scenario will occur in production. Thanks.
Nye Liu <nye@...>
Not to stray too far off topic, but k8s emerged from a client/server microservices/stack deployment requirement, not a p2p requirement.
toggle quoted message
Show quoted text
Its popularity for that purpose has (unfortunately, IMO) bled into places it isn’t really appropriate for because there are more fullstack devs than p2p devs.
Just my 2 cents, I know it is an unpopular opinion.
On Nov 19, 2019, at 2:31 PM, Hakan Eryargi <hakan.eryargi@...> wrote:I guess K8S is the preferred way to deploy many things not only Fabric ;) and we will go production in a few months with Fabric in K8S.We are not mixing K8S clusters for now, as it's not a requirement ATM, but will eventually be the case, mixing K8S with other K8S or on premises or whatever. That time possibly my preferred way of doing it will be using hostAliases to point to other nodes.Check Raft orderer section in our repo for how we do it:
https://github.com/APGGroeiFabriek/PIVT#scaled-up-raft-networkOn Tue, Nov 19, 2019 at 11:23 PM Yueming Xu <yxucolo@...> wrote:So, in the case of a network across 2 k8s clusters, I have to make all orderer nodes accessible outside of k8s, and put the public host names and ports in the genesis block. Correct? K8s is still a preferred way to deploy the fabric network as I can see it, so such scenario will occur in production. Thanks.
I guess K8S is the preferred way to deploy many things not only Fabric ;) and we will go production in a few months with Fabric in K8S.
We are not mixing K8S clusters for now, as it's not a requirement ATM, but will eventually be the case, mixing K8S with other K8S or on premises or whatever. That time possibly my preferred way of doing it will be using hostAliases to point to other nodes.
Check Raft orderer section in our repo for how we do it:
https://github.com/APGGroeiFabriek/PIVT#scaled-up-raft-network
https://github.com/APGGroeiFabriek/PIVT#scaled-up-raft-network
On Tue, Nov 19, 2019 at 11:23 PM Yueming Xu <yxucolo@...> wrote:
So, in the case of a network across 2 k8s clusters, I have to make all orderer nodes accessible outside of k8s, and put the public host names and ports in the genesis block. Correct? K8s is still a preferred way to deploy the fabric network as I can see it, so such scenario will occur in production. Thanks.
Alexandre Pauwels
I've said it before on this mailing list and I guess I'll say it again, I don't understand this hate with k8s and Fabric.
Using nginx and coredns you can achieve nearly any sort of routing combination you want. The ONLY port that needs to be exposed to the outside world is 443, any endpoint on any internal pod can be defined as a subdomain and routed to the correct pod internally using clusterIP services. CoreDNS rewrites are leveraged to avoid hairpinning traffic (which I don't see as hack...it's just using DNS for what it was meant to do, interpreting a hostname and routing it to the appropriate IP, no more no less).
To answer the original question: yes, RAFT orderers on multiple clusters absolutely CAN communicate with each other AND identify each other using TLS certificate pinning, and without exposing anything other than the default port 443. Select an ingress which supports TLS passthrough/L4 routing (nginx supports this just fine). Enable passthrough on your ingress resource. Define your ingress to route a particular subdomain to a specific port on your orderer's service.
I think putting down leveraging k8s to host Fabric is harmful to Fabric adoption as a whole. K8s is an apt host of Fabric services.
Alex
On Tue, Nov 19, 2019, 17:08 Nye Liu <nye@...> wrote:
This is precisely why k8s and p2p are never a good match. K8s simply breaks far too many networking rfcs. It is suitable for asymmetric client/server microservices or stacks, and even then a tiny subset of networking protocols.On Tue, Nov 19, 2019, 1:47 PM Yueming Xu <yxucolo@...> wrote:If I have 2 orgs contributing orderer nodes using etcd raft consensus, and each org runs its nodes in its own Kubernetes cluster. How can I make Raft work across these 2 orgs? I guess that every orderer node will have to make some ports open to outside of its K8s cluster via e.g., ELB, but I am not sure which port and whether all orderer nodes must be open to public. Is there a concept like the anchor peer, where each org need to make only the anchor peer open to external world?
Or, should I simply make all orderer nodes live in the same Kubernetes cluster as a best practice? Thanks.
Yueming Xu
So, in the case of a network across 2 k8s clusters, I have to make all orderer nodes accessible outside of k8s, and put the public host names and ports in the genesis block. Correct? K8s is still a preferred way to deploy the fabric network as I can see it, so such scenario will occur in production. Thanks.
Nye Liu <nye@...>
This is precisely why k8s and p2p are never a good match. K8s simply breaks far too many networking rfcs. It is suitable for asymmetric client/server microservices or stacks, and even then a tiny subset of networking protocols.
On Tue, Nov 19, 2019, 1:47 PM Yueming Xu <yxucolo@...> wrote:
If I have 2 orgs contributing orderer nodes using etcd raft consensus, and each org runs its nodes in its own Kubernetes cluster. How can I make Raft work across these 2 orgs? I guess that every orderer node will have to make some ports open to outside of its K8s cluster via e.g., ELB, but I am not sure which port and whether all orderer nodes must be open to public. Is there a concept like the anchor peer, where each org need to make only the anchor peer open to external world?
Or, should I simply make all orderer nodes live in the same Kubernetes cluster as a best practice? Thanks.
The idea of anchor peer is providing information about other peers in the same organization. There is no need for a peer to be completely public but it should be at least accessible by other peers somehow. VPN, white listing or whatever. If only anchor peer is accessible by outer world/other peers, what is the point of providing information about some peers which are not accessible? ;)
Regarding your question, each Raft orderer node should be access to each other. port is by default 7050 and the host/port is actually defined in configtx.yaml and encoded in the genesis block.
Best,
Hakan
On Tue, Nov 19, 2019 at 10:47 PM Yueming Xu <yxucolo@...> wrote:
If I have 2 orgs contributing orderer nodes using etcd raft consensus, and each org runs its nodes in its own Kubernetes cluster. How can I make Raft work across these 2 orgs? I guess that every orderer node will have to make some ports open to outside of its K8s cluster via e.g., ELB, but I am not sure which port and whether all orderer nodes must be open to public. Is there a concept like the anchor peer, where each org need to make only the anchor peer open to external world?
Or, should I simply make all orderer nodes live in the same Kubernetes cluster as a best practice? Thanks.
Yueming Xu
If I have 2 orgs contributing orderer nodes using etcd raft consensus, and each org runs its nodes in its own Kubernetes cluster. How can I make Raft work across these 2 orgs? I guess that every orderer node will have to make some ports open to outside of its K8s cluster via e.g., ELB, but I am not sure which port and whether all orderer nodes must be open to public. Is there a concept like the anchor peer, where each org need to make only the anchor peer open to external world?
Or, should I simply make all orderer nodes live in the same Kubernetes cluster as a best practice? Thanks.
Or, should I simply make all orderer nodes live in the same Kubernetes cluster as a best practice? Thanks.