Your first log snippet showed CouchDB queries timing out after 35 seconds each. This is the core.yaml ledger.state.couchDBConfig.requestTimeout of 35s. It means that CouchDB is still processing the query when the peer client gives up on it.
Thanks a lot David for your response. So digging deeper into the logs, we found that peer was returning below error:
[chaincode] BuildQueryResponse -> ERRO Failed to get query result from iterator [chaincode] HandleTransaction -> ERRO Failed to handle QUERY_STATE_NEXT. error: net/http: request canceled (Client.Timeout exceeded while reading body) error reading response body
I wanted to understand the "Client.Timeout" meaning here. I am understanding that the "Client" could be either the peer to Couch DB connection or it could be the Fabric SDK client to peer connection.
IMO, this looks like the peer to Couch DB connection, where the error is getting logged in peer logs within <60s (from the timeout query request comes to the time it errors out), even when CORE_CHAINCODE_EXECUTETIMEOUT is set to 300s. So, this does not look like a timeout issue. Couch DB is returning error much before the set timeout config on the peer. Health of Couch DB and peer containers seem to be good as well.