-->
Page 2 of 14

Re: Replacing LWIP

PostPosted: Thu Mar 21, 2019 12:21 pm
by d-a-v
I just noticed the link in my previous-previous post is not well rendered in this forum.
(to moderator: please ensure links are displayed properly in this post)

lwip2 step towards support on esp-open-sdk:
https://github.com/d-a-v/esp82xx-nonos- ... er/pull/29

ongoing discussion for esp-open-sdk support (and by extension native nonos-sdk)
https://github.com/pfalcon/esp-open-sdk ... -466613304

Re: Replacing LWIP

PostPosted: Thu Mar 21, 2019 7:24 pm
by davydnorris
Yeah I have also recompiled lwIP from the supplied source in the SDK and it's just like d-a-v specified.

I am very keen to try and use his lwIP2 code to attempt a merge with the SDK version, and then push it to them.

Right now I also want to try and expose the DNS TTL values in some way because I've set up a local DNS cache that uses the RTC memory so it survives deep sleep and I'm faking the TTL info, which is a bit dangerous.

Basically I have an LRU cache that stores the IP and TTL against a crc8 of the requested hostname, which means I avoid a DNS query for my SNTP servers, map server REST API, and IoT platform server every time I come back from deep sleep.

Re: Replacing LWIP

PostPosted: Fri Mar 22, 2019 3:16 am
by eriksl
d-a-v wrote:> Can anyone tell me what's the current overall status?
lwip2 status is:
- it is working for over a year in arduino-esp8266, which is using nonos-sdk (v2)
- following the github link above will show you the lwip2 status about esp-open-sdk (and by extension nonos-sdk)

For me that boils down to "proven to be working". Now the challenge will be to integrate it into my project. I think that won't be trivial. But still very useful information. If it wasn't proven to be feasible, I wouldn't even start ;-) So I guess it's worth the effort. I was under the impression this was still very experimental work.

d-a-v wrote:> Can anyone tell me what's the current overall status?
Compiling lwip (v1.4-espressif-patched) can be done like this:
$ cd third_party/lwip
then
$ make
(or)
$ make COMPILE=gcc

output:
...
xtensa-lx106-elf-ar: creating .output/eagle/debug/lib/liblwip.a

That's wat in the doc yes. It seems a bit oversimplified, I can hardly believe it works this way. But I'll try. That's why I asked for people actually having done this.

Re: Replacing LWIP

PostPosted: Mon Mar 25, 2019 1:33 pm
by eriksl
Just like I thought, it just DOESN'T work if you're not using 100% the Espressif Makefile and directory lay-out (which is not really what I'd describe as sensible).

The lwip source is full of Makefiles that do almost nothing other than including the Makefile in the directory parent. So in the end your own Makefile is included and that doesn't necessarily work well. And is completely hopelessly complex.

But I got it working, after having had my teeth really into it (Dutch paraphrase ;-)) It may help others, so I'll report here what I did to get it working.

1) the Makefiles. Less than helpful. I removed all of them.
2) copied all of the files under third_party/lwip and third_party/include under my own project so I could manage them
3) added all object files (*.o) from each lwip subdirectory to an object group (app, api, core, netif).
4) add a dependency on lib_<object_group>.a to the related objects with a Makefile rule that creates .a file from the .o files using ar ru (you don't even need to use the lx106 version of ar here)
5) now all of the lwip files are compiled using the compile flags I also use for my own c sources, which is good
6) compilation was successful after I made some 10 "warning=error" constructs into warnings
7) after that, IRAM was overflowed by about 1k, while previously I had 900 bytes spare. Checked ALL lwip source files and add the ICACHE_FLASH_ATTR (well in fact not that, but my own #define which is called "iram" and is, imho, somewhat more clear) to EVERY function I found.
8) for most of the sources I fixed, the IRAM usage did not go down. I found out why: for some of the functions, the ICACHE_FLASH_ATTR is applied to the functions themselves, for some of the functions it's applied to the function declaration (in the header files) and for some of the both. I added the attribute to many functions in the sources where it was already applied to the declaration in the header file (which I didn't spot at that time).
9) after I did ALL functions, IRAM usage was back at 900 bytes spare I had before. The question is how that comes, that I manually need to add "IRAM" attributes to functions, while the precompiled library from Espressif has that to all functions already. It could be that Espressif is using a different source internally, with all functions, marked as IRAM, but I found something else is going on. One of the recent updates of the SDK adds a line like to this the load script:
Code: Select all*liblwip.a:(.literal.* .text.*)
which makes all functions from lwip go to IRAM regardless their section attribute. Now I don't have a "liblwip.a" but three lib*.a files, this line wasn't working anymore. I added explicit lines for all of them in the load script and removed all ICACHE_FLASH_ATTR mentionings from source files (which Espressif should have done themselves) and all is well now.
10) removed all source files / obj entries that are not required to build the project, that was quite a lot (like the whole /api tree). Also removed the whole app/espconn tree, because I am not using the espconn interface anymore. This saved me piles of errors while compiling (from the usual bad Espressif coding quality).
11) tried to fixed the rest of the source files from compiler warnings. Most of them I could easily fix, some of them would mean a complete rewrite, which I didn't feel like, so I left them in (not using const * where it should have been). It's interesting to see that most of the errors/warnings not related to const abuse where obviously caused by code introduced by Espressif. They're such lame coders. Some of them are real bugs, like at three places in a function returning a value and omitting returning a value at the fourth place (-> undefined return value).

Now it works really well. I don't see any improvement, but also no regression as well (even with all code running from flash). The image has shrunk somewhat now the espconn code is no longer in. So I'm happy.