-->
Page 1 of 1

Better crash dump reporting

PostPosted: Sat Apr 04, 2020 6:50 pm
by cmarrin
I have been using the ESP RTOS SDK but I recently switched to using the Arduino SDK. One of the best things about the RTOS SDK is its crash logging. It has something called IDF Monitor which is a tool that acts as a serial console, but has some special behavior when it sees that the device has crashed. The raw crash log looks like this:

Code: Select allGuru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC      : 0x402234de  PS      : 0x00000033  A0      : 0x402234db  A1      : 0x3ffef640 

A2      : 0x00000001  A3      : 0x3ffea65c  A4      : 0xffffffff  A5      : 0x00000000 
A6      : 0x3fff4838  A7      : 0x3fff4830  A8      : 0xbfff4880  A9      : 0x0000012c 
A10     : 0x00000006  A11     : 0x00000002  A12     : 0x00000000  A13     : 0x00000000 
A14     : 0x00000000  A15     : 0x3ffef680  SAR     : 0x0000001f  EXCCAUSE: 0x0000001c 

Backtrace: 0x402234de:0x3ffef640 0x40225e88:0x3ffef680 0x40226389:0x3ffef6a0 0x4025e978:0x3ffef6c0 0x4025ef89:0x3ffef6e0 0x40260454:0x3ffef700 0x40260514:0x3ffef760 0x402600bb:0x3ffef770


but IDF Monitor decorates it to make it look like this:

Code: Select allGuru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC      : 0x402234de  PS      : 0x00000033  A0      : 0x402234db  A1      : 0x3ffef640 
0x402234de: xQueueGenericSend at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/freertos/freertos/queue.c:2332

0x402234db: xQueueGenericSend at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/freertos/freertos/queue.c:2332

A2      : 0x00000001  A3      : 0x3ffea65c  A4      : 0xffffffff  A5      : 0x00000000 
A6      : 0x3fff4838  A7      : 0x3fff4830  A8      : 0xbfff4880  A9      : 0x0000012c 
A10     : 0x00000006  A11     : 0x00000002  A12     : 0x00000000  A13     : 0x00000000 
A14     : 0x00000000  A15     : 0x3ffef680  SAR     : 0x0000001f  EXCCAUSE: 0x0000001c 

Backtrace: 0x402234de:0x3ffef640 0x40225e88:0x3ffef680 0x40226389:0x3ffef6a0 0x4025e978:0x3ffef6c0 0x4025ef89:0x3ffef6e0 0x40260454:0x3ffef700 0x40260514:0x3ffef760 0x402600bb:0x3ffef770
0x402234de: xQueueGenericSend at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/freertos/freertos/queue.c:2332

0x40225e88: sys_mbox_post at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/lwip/port/esp8266/freertos/sys_arch.c:408

0x40226389: tcpip_api_call at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/lwip/lwip/src/api/tcpip.c:408

0x4025e978: tcpip_adapter_start_dhcp at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/tcpip_adapter/tcpip_adapter_lwip.c:1220

0x4025ef89: tcpip_adapter_dhcpc_start at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/tcpip_adapter/tcpip_adapter_lwip.c:1220

0x40260454: system_event_sta_connected_handle_default at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/esp8266/source/event_default_handlers.c:126

0x40260514: esp_event_process_default at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/esp8266/source/event_default_handlers.c:308

0x402600bb: esp_event_loop_task at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/esp8266/source/event_loop.c:92


All those code pointers use ASCII colors so they stand out better.

You give IDFMonitor a pointer to the elf file and it uses addr2line to find any hex values that are in code space and prints the line in a source file. It makes it super easy to see where the crash occurred, or at least near where it crashed.

You can use IDF Monitor as the console for Arduino, but it doesn't give nearly as useful info. It might give you the line of code for the current PC, but that's usually an abort or assert of something. The problem is that the crash dump generated by Arduino doesn't format it's hex numbers with a 0x at the start and it doesn't seem to dump the entire stack trace.

Is there a similar mechanism to make crash debugging easier on Arduino or is there a replacement crash dump output that is more useful?

Re: Better crash dump reporting

PostPosted: Sat Apr 04, 2020 9:37 pm
by rudy
I have not used these yet so I can't say how useful they are, but worth looking at.

https://github.com/brainelectronics/EspSaveCrashSpiffs

https://github.com/krzychb/EspSaveCrash