Advanced Users can post their questions and comments here for the not so Newbie crowd.
I have had a crap day trying to get the newer MQTT libs working with lwIP - have you got sockets turned on in your lib? The code as written for the 8266 RTOS and ESP32 uses a couple of socket calls and they're not enabled in the default Espressif lib.
I have to say, the new MQTT library is a much cleaner implementation - the original author tuanpm is obviously a big contributor, but the whole thing has been rewritten and it's really neat. But it uses lwIP and mbedtls directly, whereas the old libs used espconn.
If I could get the new lib working it would also fix a whole lot of stuff I had to do by hand and I could ditch espconn - but I need it fast. I may just end up hacking the old code to implement an 'outbox' for pending messages (they've done this in the new libs and it's much neater)
My interface to LWIP is very basic as to keep overhead to an absolute minimum. Data packets of up to 4k can be sent either using udp or tcp. Using udp I do the fragmention on the application layer (IP fragmentation turned off in LWIP), optionally terminating a data packet with a zero-length datagram (yes, that's allowed and works). For tcp I send whatever amount of data can be sent immediately and the rest of the packet gets sent on a "data sent" notification, just as long as it takes to send the whole data packet. After that, the data packet is unlocked for re-use.
Reception of data likewise, data is collected in small enough chunks and when the end marker has been received, processed as a whole.
That way I don't need fragmentation, which is not really possible on devices low on memory. Strictly the process on tcp is not called fragmentation but segmentation, but it's almost the same and the same objections apply.
The good news is the socket api does not need to have any Espressif-specific code, so I guess it will just work.
BTW I think the socket api is great for desktops and servers, I've been using it since 1988 (...) but it's far too heavy for us.
The new MQTT code uses setsockoptions() to set up a receive time out and then select() to enable it - I think I'll just have to hack the existing library at the moment but it's apparently working fine in the RTOS SDK
BTW I thought of you last night - I was trawling for ESP8266 lwIP repos and found one that had got SLIP working in the native lwIP. Don't know if you're interested?