As it happens, I've build a very similar framework to yours -- no upvalue dynamically loaded functions using an _index setmetatable binding, and which are able to be Garbage Collected (becuse I came across yours after I'd done my own). One difference is that I don't bother with HTTP but just use a raw stripped down TCP based RPC protocol with a server side command-line tool to do all of the work.
OK, the 20-odd K RAM on the current chipsets is a real problem with the nodeMCU Lua implementation: if you set LUAL_BUFFERSIZE to 1K then you get burnt by trying to manipulate long strings; if you set it to 4K then you have more wiggle room but run out of C stack if you glance sideways. But for my usecase (and because I've done a lot of old school embedded work) your (and my) general approach works fine and allow rapid embedded app development.
But I don't expect be able to get this generation of ESP8266 with this amount of RAM running a decent webserver, etc. If ESP bring out a new chip in 12 months time with 128K or 256K RAM then this might be different story, but I'll be using an RPi with a properly configured and secure OS and Apache service as my server to drive these devices. I might even switch to MQTT as my transport once this is stable, but our (shared) approach is really good for this type of embedded application. My embedded devices are largely stateless with any state being maintained on the server, so the SPIF write cycle lifetime isn't really an issue for me.
What I am trying to say is: don't do yourself down; for a whole class of applications, this is a good way to go.