Re: memory used by peer went up horribly

Matthew Sykes

Fabric uses protocol buffers for messages and persistence and gRPC+protobuf for inter-process communication. When a proposal message is sent to the peer, several copies of the proposal message end up in memory. Some of these may be eligible for garbage collection before endorsement processing is complete but many are not.

In terms of copy, in general we get:

0. The bytes representing the signed proposal as received by the gRPC transport
1. The decompressed signed proposal message received by the gRPC server from the client.
2. The unmarshalled signed proposal containing the proposal bytes that is passed to the endorser
3. The proposal instance unmarshaled from from the signed proposal bytes that contains the proposal payload
4. The chaincode proposal payload that contains the serialized chaincode invocation spec
5. The chaincode invocation spec that contains the chaincode spec and a copy of the chaincode arguments
6. A copy of the chaincode proposal payload that contains the serizlied chaincode invocation spec and omits the transient map
7. A serialized version of the chaincode invocation spec that omits the transient map
8. A marshaled copy of the chaincode input that includes another copy of the chaincode args
9. A marshaled chaincode message that includes a copy of the chaincode args that is sent to the chaincode

Assuming the arguments provided to the chaincode are going to be stored in the ledger,

1. The bytes received from by the gRPC transport for the put state
2. The chaincode message containing the marshaled put state message
3. The unmarshalled put state message that includes a copy of the state value that is then held in the tx simulator rw set
4. A marshaled copy of the payload in the write set portion of the simulated response
5. A marshaled chaincode action that includes the simulated response which includes the write set
6. A mashaled proposal response payload that includes a copy of the marshaled chaincode action
7. A marshaled proposal response that includes a copy of the marshaled proposal response payload
8. A compressed copy of the response to send through the transport

Within Fabric, the multiple copies are mostly related to the enveloping structure of the messages. While what's listed above may not be perfect, it gives you an idea of how many copies of chaincode input get carried around in the peer while invoking chaincode. Assuming we're looking at around 20 copies, that still only accounts for about 850MB. It's possible there are more places where some or all of the input message is duplicated.

Neither gRPC nor Fabric are designed to handle large messages. For gRPC, the recommendation is to keep messages to under 1MB and the default message size limit is set to 4MB.

Finally, go memory management strategies change from release to release and platform to platform. In some scenarios, go will release unused memory back to the OS if it has not used it after a few minutes. If you believe there's a memory leak, pprof heap profiles may help with debugging.

On Tue, Oct 20, 2020 at 4:36 AM Ye, Qing <ye_qing.pfu@...> wrote:

escription was not very correct before and need to be modified.
The 42.5MB byte[] was passed to chaincode through go-sdk.
And then PutState was called in chaincode to save byte[].

Best wishes
Qing, Ye

Matthew Sykes

Join to automatically receive all group messages.