Hi Tomás,
I had seen something similar in a test flake in the past and the issue was a race condition between the closing of the levelDB and acquiring an iterator. I am not sure if that's app;icable here still, to clear any doubts, can you check your leveldb logs (located at {peer.fileSystemPath in core.yaml}/ledgersData/stateLeveldb/LOG near the time of this panic. Do you see something like this - "db@close"? If not, sharing the log may help further.
Thanks, Manish
I was reading about similar issues in fabric version 1.4.2 (the images I'm using on my containers are 1.4.1, so I'm not sure if it is related at all or not), and how it was related to a certain timeout when one of the containers stops (from an issue on Jira
from August 2019,
https://jira.hyperledger.org/browse/FAB-16292). Don't know if that's what is happening. Last execution, it stopped after approximately 15 minutes, but I've seen it crash sooner and later.
The full error message is as follows:
De: David Enyeart <enyeart@...>
Enviado: quarta-feira, 10 de fevereiro de 2021 19:31
Para: Tomás Peixinho <tom.peixinho@...>
Cc: fabric@... <fabric@...>
Assunto: Re: [Hyperledger Fabric] Weird "panic: runtime error" after multiple transactions
You would need to provide the full stack trace from the panic message in order to troubleshoot.
Dave Enyeart
 "Tomás
Peixinho" ---02/10/2021 01:34:13 PM---Good afternoon fellow colleagues, I'm developing an application using Hyperledger Fabric and I've fo
From: "Tomás Peixinho" <tom.peixinho@...>
To: "fabric@..." <fabric@...>
Date: 02/10/2021 01:34 PM
Subject: [EXTERNAL] [Hyperledger Fabric] Weird "panic: runtime error" after multiple transactions
Sent by: fabric@...
Good afternoon fellow colleagues, I'm developing an application...
| This Message Is From an External Sender |
| This message came from outside your organization. |
|
|
|
Good afternoon fellow colleagues,
I'm developing an application using Hyperledger Fabric and I've found a weird error that I'm not sure how to fix.
I'm running everything on just one machine, with two orgs, one peer on each, and one orderer. My application receives a file with multiple lines (right now, around 5000), which correspond to one transaction each (technically, two transactions
each, because I have to do a query by partial key, and then a separate transaction for the actual update, as per recommendation of Dave Enyart, from a previous question of mine). So, 5000 lines, around 10000 transactions. They are being handled concurrently
using threads. And the peers are being alternated, so that one doesn't happen to take all the load, while the other does nothing.
My problem now is that it usually ends with one of the peers randomly crashing with an Exit Code 2, and the logs from that peer saying "panic: runtime error: invalid memory address or nil pointer dereference". I say randomly because it
can happen after a 1000 lines (2000 transactions) or 4000 lines (8000 transactions), which is what I find weird!
I've read that it could be because of problems with certificates or with misconfigured paths for them, but I feel that if that were the case, it would happen much sooner, or actually right at the start.
My question now is (and I know that I haven't provided much information or any logs, and that it's difficult to find the problem like this) if there is any obvious situation in which this might happen? Or if there are a few cases that
might lead to this error? Something that might point me in the right direction of what I need to be looking at to see if I have any configuration problems. Or could it simply be a problem of my physical machine? Could it be just a memory problem, which crashes
the peer if the traffic gets too heavy?
These errors are really not obvious sometimes, and I was just wondering if someone might have some insight as to what the problem might be, based on previous experiences or something.
Thanks in advance
Tomás
|
