Hyperledger setup on multiple AKS clusters #fabric-kubernetes #kubernetes


Vijaya Bhaskar
 

Hello,

This is regarding the solutions available to setup HLF on multiple kubernetes clusters.
the solution I have is as below, It would be helpful if anybody can suggest a better solution which is scalable.

We have 2 K8s  clusters in CentralIndia and eastUS
the certs and TLS are generated using service names( ex: orderer1-service etc)
the issue is in connecting the clusters enabling communication.
what we are using now: skupper a 3rd party tool which exposes services from one cluster to another

but if we need to expand, this is not an ideal solution.
In AKS, we have vnet peering which is not working as we expected, hence we resorted to use skupper.

If anybody has seen or setup a multi cluster HLF network, can you provide some pointers for us.

Thanks,
Vijay


jkneubuh@...
 

Hi Vijaya,

Please consider using the fabric-operator to help manage your deployment across multiple Kubernetes clusters.  The operator creates ingress points for your services, allowing you to set up routes based on DNS wildcard domains.  Traffic can be directed on the public internet, or on a dedicated circuit if desired.

Here is a brief example of using fabric-operator to emulate the sample "test network," using a local KIND cluster and deploying the channel across three separate namespaces: fabric-kube-test-network.   When running on a local / single system, the network and ingress has been configured to respond at the special *.localho.st domain, resolving all traffic to the local kubernetes cluster.

This example can also be extended to run across multiple clusters - see the feature/multi-cluster branch for how the *.localho.st domain can be replaced with a wildcard DNS hosts to serve as the ingress points for your network nodes.  The main update in this configuration are that the configtx.yaml / channel configuration must be specified with the external host names, rather than the Kubernetes DNS host names to route traffic between nodes on the internal network.

Setting up ingress can be slightly different based on the cloud provider - A good alternative is to "bring your own Nginx" ingress, mapping a public DNS wildcard domain to the ingress controller port.  (There are a couple of examples of setting this up for IKS and EKS in the fabric-operator project, it will be similar for AKS.)  In this scenario you will need access to a public wildcard DNS entry for your clusters.

The setup above works well on a single node, but the "scripting" for the multi-cluster scenario requires a little manual intervention for the coordination and exchange of channel MSP / certificates.  Despite a couple of rough edges in the scripting, it will achieve the scenario of a multi-org network spanning multiple Kubernetes clusters.  If you have feedback on the approach or need some general guidance on the multi-cluster setup, please use the discord #fabric-operator or #fabric-kubernetes channels for additional details : https://discord.gg/hyperledger

-josh


Hakan Eryargi
 

Hi Vijaya,

Have a look at our cross cluster sample:

Best,
Hakan

On Sat, Nov 26, 2022 at 11:30 AM Vijaya Bhaskar <acvbhaskar94@...> wrote:
Hello,

This is regarding the solutions available to setup HLF on multiple kubernetes clusters.
the solution I have is as below, It would be helpful if anybody can suggest a better solution which is scalable.

We have 2 K8s  clusters in CentralIndia and eastUS
the certs and TLS are generated using service names( ex: orderer1-service etc)
the issue is in connecting the clusters enabling communication.
what we are using now: skupper a 3rd party tool which exposes services from one cluster to another

but if we need to expand, this is not an ideal solution.
In AKS, we have vnet peering which is not working as we expected, hence we resorted to use skupper.

If anybody has seen or setup a multi cluster HLF network, can you provide some pointers for us.

Thanks,
Vijay