what's the recommened number of Kafkas and Zookeepers if I want to have around 5 organizations, with each organization to have 5-6 peer nodes?

From a performance standpoint, Kafka and Zookeeper (ZK) are not going to be the bottleneck.
So how many ZK nodes and Kafka brokers you deploy will depend on your resiliency model / requirements.

Zookeeper's fault tolerance is based on a 2f+1 model where f represents the number of faulty nodes you are willing to tolerate.
If you are willing to tolerate a single failure (f=1) then you'd need 3 ZK nodes (2*1 + 1). Typically, most people set up ZK clusters to support 1-5 faulty nodes (so 3,5,7,9 or 11 nodes).
I'd suggest starting out with 3 or 5 nodes.

For Kafka, the number of brokers you need will also be based on your tolerance for failures and for resiliency. From a Fabric perspective, it's best to use a minimum of 4 Kafka brokers.
This will give you a replication factor of 3 and allow you to tolerate a single node going down and still maintain full operations.



-----------------------------------------
Gari Singh
Distinguished Engineer, CTO - IBM Blockchain
IBM Middleware
550 King St
Littleton, MA 01460
Cell: 978-846-7499
garis@...
-----------------------------------------

-----fabric@... wrote: -----
To: fabric@...
From: "Simon Yau"
Sent by: fabric@...
Date: 07/16/2018 12:39AM
Subject: [Hyperledger Fabric] what's the recommended number of kafkas and zookeepers? #fabric #fabric-questions

what's the recommened number of Kafkas and Zookeepers if I want to have around 5 organizations, with each organization to have 5-6 peer nodes?

Hi Gari, 

what if i would like to allow 2 nodes failure, so the number of ZK is 2*2+1 =5

while the number of K is 3(replication factor) +2 = 5?

Hey Simon -

On the ZK side, that's correct ... it's a 2f+1 model so f=2 means you need 5 ZK nodes.


Kafka is a little trickier in terms of availability and data loss.
For data loss, Kafka supports N-1 server failures where N is the number of replicates. So if you want to support 2 servers going down without data loss, then you need N=3 (replication factor set to 3).
For availability, you need to make sure that you have N brokers available (where again N is the replication factor). So given we want a replication factor of 3 (N=3) and you want to support the loss of 2 brokers while still being available, then 5 brokers is correct.

Hope this helps.


-- G


-----------------------------------------
Gari Singh
Distinguished Engineer, CTO - IBM Blockchain
IBM Middleware
550 King St
Littleton, MA 01460
Cell: 978-846-7499
garis@...
-----------------------------------------

-----fabric@... wrote: -----
To: fabric@...
From: "Simon Yau"
Sent by: fabric@...
Date: 07/16/2018 04:51AM
Subject: Re: [Hyperledger Fabric] what's the recommended number of kafkas and zookeepers? #fabric #fabric-questions

Hi Gari,

what if i would like to allow 2 nodes failure, so the number of ZK is 2*2+1 =5
while the number of K is 3(replication factor) +2 = 5?