Hello HyperLedger Fabric community,
I am working on implementing an application that receives chaincode events of interest (via a regular expression match on the event name) and then places those events onto a Kafka Topic--we'll call this the eventreceiver. Downstream applications that are interested in the events can then subscribe to the Kafka Topic.
We are using Kafka because this is a technology that is already widely used within the organization and we're also trying to limit how many internal systems directly access the blockchain network. We also have an API layer for submitting transactions and running queries, but that isn't important for this line of questions.
There is a requirement that events not be lost as they move from the peer to the Kafka Topic. Per , events are ephemeral; so if the event listener goes down, events will be missed. So, if the eventreceiver goes down, then it must figure out what was missed and specifically request events associated with the blocks that were committed while it was down. It suggests a couple of ways of doing this. Both approaches amount to implementing what is effectively a Kafka ConsumerGroup-like construct (only for HyperLedger Fabric events). Doing that is a lot more complicated than the original functionality we are trying to implement.
Along similar lines, if we want to run multiple instances of this eventreceiver for HA, then we have to deal with a message deduping process before writing the events to the Kafka Topic. Again, this seems like something that a ConsumerGroup like construct in HyperLedger Fabric SDKs would help solve. I am using node.js and fabric-node-sdk.
Are there plans to implement anything like this? Or, does something exist already that I have simply missed?
Thank you in advance for your time.