Debugging a LoadProhibited during a PSRAM alloc

[raphaelhetzel]

Hey all,
I'm currently struggling with a LoadProhibited during a larger PSRAM alloc (should be 1_114_112 Bytes) on an ESP32-S3.
So far i was unable to reproduce this in a simple example as this only seems to happen while the alloc is embedded deep into the program (it does however happen every time). I assume this is due to some fragmentation i was not able to reproduce yet.

From previous discussions I gathered that LoadProhibited might be related to the stack, however I don't think this is the case here (A1=0x3fccb980 < 0x3fcdc700 (start of the .dram2_uninit section which is the one after the stack).

I would assume that a non-fit would result in a rust exception instead of that. For some reason it also seems to load EXCVADDR=0xffffffff, which sounds like some math has been done on a null pointer.

Do you have any suggestions on how to properly debug this?

Versions used:
esp_alloc = 0.7
esp_hal = 1.0.0-beta.0

HEAP INFO
Size: 2195456
Current usage: 65932
Max usage: 69164
Total freed: 52564
Total allocated: 118496
Memory Layout:
External | ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | Used: 1% (Used 23872 of 2097152, free: 2073280)
Internal | ██████████████████████░░░░░░░░░░░░░ | Used: 64% (Used 42060 of 65536, free: 23476)
Internal | ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | Used: 0% (Used 0 of 32768, free: 32768)




Exception occurred 'LoadProhibited'
Context
PC=0x420359a4       PS=0x00060735
0x420359a4 - linked_list_allocator::hole::Cursor::split_current
    at /home/edgeless/.cargo/registry/src/index.crates.io-6f17d22bba15001f/linked_list_allocator-0.10.5/src/hole.rs:69
A0=0x82035e4e       A1=0x3fccb980       A2=0x3fccb9f0       A3=0x3c1f4f70       A4=0x3c1f4ff0
A5=0x3c1f4f6f       A6=0xffffffff       A7=0x3c1f4f70       A8=0xffffffff       A9=0x3c1f4f73
A10=0x00000001      A11=0x00000003      A12=0x3c1f4f70      A13=0x00000080      A14=0xfffffffc
A15=0x3fccbb80
SAR=00000018
EXCCAUSE=0x0000001c EXCVADDR=0xffffffff
LBEG=0x400570e8     LEND=0x400570f3     LCOUNT=0x00000000
THREADPTR=0x00000000
SCOMPARE1=0x00000100
BR=0x00000000
ACCLO=0x00000000    ACCHI=0x00000000
M0=0x00000000       M1=0x00000000       M2=0x00000000       M3=0x00000000
F64R_LO=0x00000000  F64R_HI=0x00000000  F64S=0x00000001
FCR=0x00000000      FSR=0x00000000
F0=0x00000000       F1=0x00000000       F2=0x00000000       F3=0x00000000       F4=0x00000000
F5=0x00000000       F6=0x00000000       F7=0x00000000       F8=0x00000000       F9=0x00000000
F10=0x00000000      F11=0x00000000      F12=0x00000000      F13=0x00000000      F14=0x00000000
F15=0x00000000
0x420355bf
esp_alloc::EspHeap::alloc_caps::{{closure}}
    at /home/edgeless/.cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-alloc-0.7.0/src/lib.rs:509
0x420357c3
<esp_alloc::EspHeap as core::alloc::global::GlobalAlloc>::alloc
    at /home/edgeless/.cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-alloc-0.7.0/src/lib.rs:540
0x4208f1bd
core::result::Result<T,E>::map_err
    at /home/edgeless/.rustup/toolchains/esp/lib/rustlib/src/rust/library/core/src/result.rs:854
0x4208f6ea
<core::result::Result<T,E> as core::ops::try_trait::Try>::branch
    at /home/edgeless/.rustup/toolchains/esp/lib/rustlib/src/rust/library/core/src/result.rs:1996
0x4208f36f
alloc::raw_vec::RawVecInner<A>::grow_one
    at /home/edgeless/.rustup/toolchains/esp/lib/rustlib/src/rust/library/alloc/src/raw_vec.rs:568
0x420b29a5
alloc::vec::Vec<T,A>::push
    at ??:??
0x420055e1
wasmi::memory::Memory::new
    at /home/edgeless/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmi-0.40.0/src/memory/mod.rs:397
0x4201e8b6
<core::result::Result<T,E> as core::ops::try_trait::Try>::branch
    at /home/edgeless/.rustup/toolchains/esp/lib/rustlib/src/rust/library/core/src/result.rs:1996
0x42015fd9
wasmi::linker::Linker<T>::instantiate
    at /home/edgeless/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmi-0.40.0/src/linker.rs:595
0x4201c5b9
core::result::Result<T,E>::map_err
    at /home/edgeless/.rustup/toolchains/esp/lib/rustlib/src/rust/library/core/src/result.rs:854

[bugadani]

Are you using an ESP32-S2 by any chance?

[raphaelhetzel]

Sorry, forgot that part: It is an ESP32-S3 (older N8R8).
The second core is active but should be idle.
Is there a bug with the S2 that might be releated?

[bugadani]

There was a recent fix on the S2, but that is a very different chip.

It would be nice to know more about your firmware. We are testing simple PSRAM cases on an S3 - I'm not sure exactly which one, but one with octal PSRAM like the N8R8 is.

[raphaelhetzel]

It's a very experimental/dirty POC of a serverless runtime for embedded devices:
https://github.com/edgeless-project/nextless/tree/psram_bug/edgeless_embedded_esp32

Essentially it combines esp_hal, embassy, a wasm runtime (the component causing the issue), cbor and coap-lite.
The issue is triggered when the wasm runtime attempts to allocate its store after the wifi is up (allocations on the internal section) and some messages have been parsed / deserialzed (with allocs on the external section).
The app is quite memory intensive and I am alway close to th system being out of memory, but in that current example I don't think I'm actually out of stack space / memory.

The code is not really useable as a repro, that's why i did not file this as a bug.
I will try to simplify it down to a repro as soon as possible.
Was just hoping that a second set of eyes might have suggestions from the stacktrace above / that there is a known bug.

[bugadani]

One thing I see is you're not configuring PSRAM for octal mode. Try ESP_HAL_CONFIG_PSRAM_MODE = "octal"

[raphaelhetzel]

Thanks so much, that was the solution (wasted more than a day on this bug).
Actually this is the second time i tripped over this with the S3....

[bjoernQ]

We probably should test the PSRAM after mapping it by default - would still not work without setting the cfg-option but at least the problem would be a bit easier to guess (maybe)