ESP8266 Community Forum

- Sun Mar 08, 2015 2:06 am #11567 EDIT: There is now a simplified version of this script with even better memory performance. I have attached it to this post (flashmod.zip). See my post several posts down for usage and information.


EDIT2: Added oledv2.zip using the new flashmod approach. See my post on page 2 for details.

It seems like the primary heap usage on my apps so far is the functions themselves. It seems like that is the primary drawback of lua compared to native C apps. C functions can be executed directly from flash, while lua functions must be loaded into the heap to run. So, I decided to create a module to fix that.

Enter flashmodule.lua. Here's how it works. You define 3 tables in your module: public, private, and flash. Then you put data and functions on public and private, and functions only on flash. The private data/functions will remain in memory, but only be available in the environment of your public and flash functions. The public data/functions will remain in memory, but your module will return public, so these functions will also be available to users. The flash table is where the magic happens. These functions will be serialized to a file on flash, named <modName>_<funcName>.lc. Then, through the magic of metatables, flashmodule makes these methods appear to be on the public table, except they are dynamically loaded from flash at the moment they are called.

Note: local variables in your module file are only available on the closure of private and public functions. Flash functions can access private, public, and global data, but if they try to access local data, it represents an upvalue that makes the function not serializable, and will not work.

flashmodule.lua is written as a generic file that can be executed as:

Code: Select allreturn loadfile("flashmodule.lc")("oled", public, private, flash)


Fill in your 3 tables, place this at the bottom of the file, and your done.

Then you can require your module, and treat it just like any other module, with the only difference between flash and public functions being flash functions require a little more time to call. So, place time-critical functions or functions which are called a lot in public or private. But if your app is not time-critical but is memory-constrained, put all the functions in flash and you will see a huge savings on heap.

One other note: you can separate flash functions into their own file, and call flashmodule like this:

Code: Select allloadfile("flashmodule.lc")("oled", {}, {}, flash)


Then, in the actual module file that you will require in, call it like this:

Code: Select allreturn loadfile("flashmodule.lc")("oled", public, private, {})


By placing them all in the same file, it uses a lot of heap during the require... most of that heap can be immediately garbage collected back, but if you are memory-constrained even at require time, then this technique can help. You can run the flash file once, while your main app is not running, then reset and run your main app and the flash functions will still be there waiting to be called. You can even separate your flash functions across multiple files if you are having trouble running a large module. This can give you the illusion of having a large module with an extensive api without hitting the memory ceiling so quickly.

I'll refactor my lhttpd implementation to use this approach next, to provide a more generally useful example. SSD1306 just provided a nice simple project that people have already been having memory issues with. If I modify this example slightly to use all flash functions and read both font sizes from flash instead of keeping the small font in memory, I can get heap usage down near 2k. It's still under 5k with a sensible selection of in-memory functions, where it was over 9k with the standard module approach.

Attachments

oledv2.zip

Version 2 of oled flashmodule using simpler flashMod approach.
(4.67 KiB) Downloaded 711 times

flashmod.zip

(2.14 KiB) Downloaded 721 times

oled.zip

Example flash module for SSD1306 SPI OLED
(4.66 KiB) Downloaded 597 times

Last edited by dpwhittaker on Wed Mar 11, 2015 11:43 pm, edited 2 times in total.

- Sun Mar 08, 2015 2:34 pm #11592 Hi!

I'm testing your solution since today morning and I found it very useful, but I cannot get in working in a case, where there's a function which will be passed as a callback, for example:

Code: Select alllocal flash = {}

 function flash.ListAP(t)
    local res = ""
    if t then
      for k,v in pairs(t) do
        res = res .. '"' .. k .. '"' .. ",\n"
      end
     if file.open("aplist.txt","w+") then
          file.write(res..'null')
          file.close()
          print("accesspoint list : " .. res .. " has ben written to aplist.txt file")
     end
    end
    collectgarbage("collect")
   end

  function flash.GetAPList()
     local list = ""
     if file.open("aplist.txt","r") then
          list = file.read()
          file.close()
     end
     return list
  end

return loadfile("flashmodule.lc")("config", {}, {}, flash)


If I use above code for prepare a list of accesspoints (which will be passed as a json to a browser):

Code: Select allconfig = require "config"
config.ConfigModule()
wifi.sta.getap(config.ListAP)

heap usage (after collectgarbage) is significant (17520 vs. 20336 when this function is not called)

Is it possible also to optimize callbacks?

Thanks,
Patriko

- Sun Mar 08, 2015 3:08 pm #11595 The function is loaded into memory when the .FuncName is evaluated, and garbage collected after all the references are released, but the caller holds a reference to the callback function, so I think wrapping it with a tiny function that can then load the big function at callback time should do the trick.

Code: Select allwifi.sta.getap(function(t) config.ListAP(t) end)


- Sun Mar 08, 2015 3:20 pm #11598 Still that same
I've even added config=nil

Code: Select all  wifi.sta.getap(function(t)
               config.ListAP(t)
               config = nil
               end)

but still it doesn't get optimized...