You can add a chaincode / contract listener instead of a block listener and receive only the chaincode events. Use addContractListener() method on your Contract object:

https://hyperledger.github.io/fabric-gateway-java/release-2.2/org/hyperledger/fabric/gateway/Contract.html

If using Fabric v2.4 (or later) the Fabric Gateway client API provides an equivalent getChaincodeEvents() method on the Network object:

https://hyperledger.github.io/fabric-gateway/main/api/java/org/hyperledger/fabric/client/Network.html