ESP8266 Community Forum

eriksl wrote:I didn't mean "poll your server subsecond", I meant that if you poll your server frequently (e.g. once every ten minutes) the precision will be quite acceptable, i.e. subsecond.

That's pretty much what I do and yes you can get down to a good precision. I do a lot of sleeping though so I have to poll a lot more than I normally would like. With the unit powered up all the time, the main clock is really quite good at keeping time and is only out about a second every day. If you correct every 30 mins as per the NTP best practice you can get millisecond time keeping.

eriksl wrote:In the meantime I have succeed in making the pbuf allocation from LWIP completely static, with minimal unnecessary overhead. End result is 5200 bytes to be thrown in and completely lost, but LWIP will never ask for more, at any point and it will be able to send and receive payloads of 4k over tcp and udp, not bad imho.

That will make the work on the LWIP (2) migration easier (less dependence on Espressif code). BTW I've left the code for mem_alloc/pvPortAlloc etc. stuff in, it's selectable using the lwip/opt.h file.

That's excellent! Can't wait to dig into that sometime soon.

I found the clock to be more than one second off a day when not corrected, more like 30 seconds. Maybe your modules have better crystals than mine, who knows

The LWIP upgrade will have to wait now for a bit. I am running very tight on memory now, so I need to free up some first. I have one big spender and that's a binary representation of the text config stored in flash sector. The binary represention is for quick lookup. The downside is that it wastes a LOT of memory and at the same time it limits the total amount of config items and it unnecessary limits their lengths (variable name + value). So what I am making now is using the generic flash sector buffer (which I already am using for other stuff) and use it as cache for the config sector. The lookup will have to be kind of smart because it will have to be run quite often. I think I nailed that part. The other challenge is implement the deletion and change of a variable. As the sector buffer is used on many places, I can't always leave the data in there, so it needs to be updated in place and then written back immediately. I am still thinking about the details to implement that best.

When I nailed the whole thing, I will have another > 8000 bytes extra to spend. I think that will be enough to run LWIP flawlessly. The small amount I have left now (~3k) gives issues every now and then, either from LWIP or SDK, I am not completely sure, more probably SDK I think.

eriksl wrote:I found the clock to be more than one second off a day when not corrected, more like 30 seconds. Maybe your modules have better crystals than mine, who knows

If you use the RTC for your time keeping it will be off. Using the hardware CPU clock you should get better accuracy than that for sure. You do need to calibrate the system at least though, but once you know what a unit is doing it's normally very good.

eriksl wrote:...
I have one big spender and that's a binary representation of the text config stored in flash sector. The binary representation is for quick lookup.
...
So what I am making now is using the generic flash sector buffer (which I already am using for other stuff) and use it as cache for the config sector. The lookup will have to be kind of smart because it will have to be run quite often. I think I nailed that part. The other challenge is implement the deletion and change of a variable. As the sector buffer is used on many places, I can't always leave the data in there, so it needs to be updated in place and then written back immediately. I am still thinking about the details to implement that best.

I've built a key/value storage system for my persistent config that uses a journal log approach to write entries, and a round robin of sectors to prevent flash fatigue. On start up it creates a linked list of valid entries, and then only retrieves the full data when requested. It's more than fast enough reading and writing for my needs

I'll post up the code for you

That is very similar to what I already had. I already finished my new approach. The complete sector is cached in a chunk of memory I also use for other purposes (like flashing from OTA and writing sectors for the sequencer). Very rarely the cached sector is flushed and reused for other purposes.

This way the config takes netto 0,0 kB of DRAM.

I have an esp here that I use for testing, many sectors get overwritten > 10 times a day, for over a year now (before that, I used another unit). It must have been written thousands of times now, maybe even tens of thousands. The flash memory is specified for 100.000 cycles, so I am not really concerned about wear of certain sectors. Especially not the config, cause it's only written very seldomly. As soon as a sector becomes faulty, I'll happily replace the flash chip, it'll cost me about $0.50.

So, I don't do the "wear" protection thing.