- Fri Aug 26, 2016 8:30 pm
#53881
I've been working with the SD WebServer code as a starting point for one of the projects I'm working upon and have made some improvements to pieces of it that I thought others might find interesting and useful. I'll post the parts so people can pick and choose what they find useful, but they slot into the SD Webserver example for ESP8266 Arduino code.
The first is an enhancement I did to the loadFromSdCard where I added a few more types and added support for including a '/gz/' directory on the root of the SD card with sub directories for various types of files. I haven't bothered with supporting long filenames so I just replaced whatever the file's original extension was with .gz. So for example, a compressed .js file would be file.gz rather than file.gz.js. Just place the file into /gz/js and this code will serve it up as type 'application/javascript' and the server code will then see it as compressed and add the appropriate encoding type so the browser knows it is gzipped.
One of the things I did before creating more code (that it will be necessary to do for some of the other examples to follow) is I added two global variables for 'serverError' and another for 'serverMessage' that I use with sub-functions that exist for utility purposes only so they can set either return messages or errors to be reported by their parent functions. I then reset these any time I check them as well as in the main loop() function.
These then use modified return functions (also included here).
I then added a utility function that will read a 'dir' parameter for some of the other methods which checks if the directory actually exists, setting errors if necessary before returning the parameter as a string:
I use this further down to modify the upload function. But first, I modified the PrintDirectory to use a sub function to actually read the directory that uses the ArduinoJson library to create the Json output. This not only makes it easier to build the Json string, but allows me to pass the object reference recursively to descend directory structures if desired. (with a boolean parameter in the function) The function works the same way but uses the getDirArg function to get and check the dir parameter now and sets descend to false to produce basically the same result, except I also added 'size' to the values returned for type=file.
I then added a second function called printFS that will scan the entire file system and cache it to a .jsn jason file in the root directory of the server. (You can force a re-scan by setting a ?scan=1 parameter when you call it.)
Note, this code requires adding an #include <ArduinoJson.h> and a global variable for JsonBuffer
Finally, I modified the file upload to use the printFS function's output to allow you to upload to any directory. This requires adding an extra handler that I created with the url "/listall" in the server setup. I added it after the existing "/list" handler.
The new upload code is:
To use this function with the error reporting, modify the handler code in the webserver setup as follows (replace the existing line(s) for handleFileUpload):
One thing I figured out real quick is that the Webserver back-end code doesn't parse the Post parameters from a form when there's a file upload. (int server.args == 0) But it will parse URL parameters. So I'll include the html of my flat file that I put in the root of the SD card which has bare-bones ajax code and updates the form tag's "action" parameter based on the directory you select.
/upload.htm
The first is an enhancement I did to the loadFromSdCard where I added a few more types and added support for including a '/gz/' directory on the root of the SD card with sub directories for various types of files. I haven't bothered with supporting long filenames so I just replaced whatever the file's original extension was with .gz. So for example, a compressed .js file would be file.gz rather than file.gz.js. Just place the file into /gz/js and this code will serve it up as type 'application/javascript' and the server code will then see it as compressed and add the appropriate encoding type so the browser knows it is gzipped.
Code: Select all
bool loadFromSdCard(String path) {
String dataType = "text/plain";
if (path.endsWith("/")) path += "index.htm";
if (path.endsWith(".src")) path = path.substring(0, path.lastIndexOf("."));
else if (path.endsWith(".htm")) dataType = "text/html";
else if (path.endsWith(".csv")) dataType = "text/csv";
else if (path.endsWith(".css")) dataType = "text/css";
else if (path.endsWith(".xml")) dataType = "text/xml";
else if (path.endsWith(".png")) dataType = "image/png";
else if (path.endsWith(".gif")) dataType = "image/gif";
else if (path.endsWith(".jpg")) dataType = "image/jpeg";
else if (path.endsWith(".ico")) dataType = "image/x-icon";
else if (path.endsWith(".svg")) dataType = "image/svg+xml";
else if (path.endsWith(".ico")) dataType = "image/x-icon";
else if (path.endsWith(".js")) dataType = "application/javascript";
else if (path.endsWith(".pdf")) dataType = "application/pdf";
else if (path.endsWith(".zip")) dataType = "application/zip";
else if (path.endsWith(".gz")) {
if (path.startsWith("/gz/htm")) dataType = "text/html";
else if (path.startsWith("/gz/css")) dataType = "text/css";
else if (path.startsWith("/gz/csv")) dataType = "text/csv";
else if (path.startsWith("/gz/xml")) dataType = "text/xml";
else if (path.startsWith("/gz/js")) dataType = "application/javascript";
else if (path.startsWith("/gz/svg")) dataType = "image/svg+xml";
else dataType = "application/x-gzip";
}
File dataFile = SD.open(path.c_str());
if (dataFile.isDirectory()) {
path += "/index.htm";
dataType = "text/html";
dataFile = SD.open(path.c_str());
}
if (!dataFile)
return false;
if (server.hasArg("download")) dataType = "application/octet-stream";
if (server.streamFile(dataFile, dataType) != dataFile.size()) {
DBG_OUTPUT_PORT.println("Sent less data than expected!");
}
dataFile.close();
return true;
}
One of the things I did before creating more code (that it will be necessary to do for some of the other examples to follow) is I added two global variables for 'serverError' and another for 'serverMessage' that I use with sub-functions that exist for utility purposes only so they can set either return messages or errors to be reported by their parent functions. I then reset these any time I check them as well as in the main loop() function.
These then use modified return functions (also included here).
Code: Select all
String serverError = ""; // use in utility (sub) routines to denote errors
String serverMessage = ""; // use in utility (sub) routines to supply a return status message
void loop(void) {
serverError = "";
serverMessage = "";
server.handleClient();
}
void returnOK() {
server.send(200, "text/plain", "");
}
void returnMsg(String msg) {
server.send(200, "text/plain", msg + " successful\r\n");
}
void returnFail(String msg) {
server.send(500, "text/plain", msg + "\r\n");
}
void returnFailJSON(String msg) {
server.send(500, "application/json", "{serverError:\"" + msg + "\"}");
}
void returnJSON(String jsonString) {
server.send(200, "application/json", jsonString);
}
I then added a utility function that will read a 'dir' parameter for some of the other methods which checks if the directory actually exists, setting errors if necessary before returning the parameter as a string:
Code: Select all
String getDirArg() {
int sargs = server.args();
DBG_OUTPUT_PORT.println(sargs);
DBG_OUTPUT_PORT.println(server.hasArg("dir"));
DBG_OUTPUT_PORT.println(server.argName(0));
String d = (server.hasArg("dir")) ? server.arg("dir") : "";
d.trim();
DBG_OUTPUT_PORT.print("d trimmed: ");
DBG_OUTPUT_PORT.println(d);
if ((d != "/") && (d != "")) {
if (SD.exists((char *)d.c_str())) {
File df = SD.open((char *)d.c_str());
if (df.isDirectory()) {
if (!(d.endsWith("/"))) d += "/"; // add trailing slash if needed
} else {
serverError = "GETPATH: PATH NOT DIR";
}
df.close();
} else {
serverError = "GETPATH: PATH NOT EXIST";
}
}
DBG_OUTPUT_PORT.print("returning: ");
DBG_OUTPUT_PORT.println(d);
return d;
}
I use this further down to modify the upload function. But first, I modified the PrintDirectory to use a sub function to actually read the directory that uses the ArduinoJson library to create the Json output. This not only makes it easier to build the Json string, but allows me to pass the object reference recursively to descend directory structures if desired. (with a boolean parameter in the function) The function works the same way but uses the getDirArg function to get and check the dir parameter now and sets descend to false to produce basically the same result, except I also added 'size' to the values returned for type=file.
I then added a second function called printFS that will scan the entire file system and cache it to a .jsn jason file in the root directory of the server. (You can force a re-scan by setting a ?scan=1 parameter when you call it.)
Note, this code requires adding an #include <ArduinoJson.h> and a global variable for JsonBuffer
Code: Select all
#include <ArduinoJson.h>
DynamicJsonBuffer jsonBuffer;
// utility function - should be called from print methods below
void listDirJSON(JsonArray& jArr, String path, boolean descend) {
File dir = SD.open((char *)path.c_str());
dir.rewindDirectory();
for (int cnt = 0; true; ++cnt) {
File entry = dir.openNextFile();
if (!entry)
break;
String filename = entry.name();
// skip dot files
if (filename.startsWith("."))
break;
JsonObject& item = jArr.createNestedObject();
item["name"] = filename;
if (entry.isDirectory()) {
item["type"] = "dir";
if (descend == true) {
JsonArray& subcont = item.createNestedArray("content");
String fullpath = path + filename + "/";
DBG_OUTPUT_PORT.println("descending path: " + fullpath);
listDirJSON(subcont, fullpath, false);
}
} else {
item["type"] = "file";
item["size"] = String(entry.size(), DEC);
}
entry.close();
}
dir.close();
}
void printFS() {
String cacheFn = "/fstree.jsn";
if (server.hasArg("scan") || !SD.exists((char *)cacheFn.c_str())) {
JsonObject& root = jsonBuffer.createObject();
root["name"] = "/";
root["type"] = "dir";
JsonArray& content = root.createNestedArray("content");
listDirJSON(content, "/", true);
File dataFile = SD.open(cacheFn, FILE_WRITE);
root.printTo(dataFile);
dataFile.close();
}
loadFromSdCard(cacheFn);
}
void printDirectory() {
String path = getDirArg();
if (serverError != "") {
returnFail(serverError);
serverError = "";
return;
}
DBG_OUTPUT_PORT.println("scanning directory" + path);
JsonObject& root = jsonBuffer.createObject();
root["name"] = path;
root["type"] = "dir";
JsonArray& content = root.createNestedArray("content");
listDirJSON(content, path, false);
String jsonOut;
root.printTo(jsonOut);
returnJSON(jsonOut);
}
Finally, I modified the file upload to use the printFS function's output to allow you to upload to any directory. This requires adding an extra handler that I created with the url "/listall" in the server setup. I added it after the existing "/list" handler.
Code: Select all
server.on("/list", HTTP_GET, printDirectory);
server.on("/listall", HTTP_GET, printFS);
The new upload code is:
Code: Select all
void handleFileUpload() {
if (server.uri() != "/edit") return;
HTTPUpload& upload = server.upload();
if (upload.status == UPLOAD_FILE_START) {
String fDir = getDirArg();
String fn = fDir + upload.filename;
if (serverError == "") {
serverMessage = "uploading";
DBG_OUTPUT_PORT.print("Upload: START, filename: ");
DBG_OUTPUT_PORT.println(fn);
if (SD.exists((char *)fn.c_str())) {
serverMessage += " over";
DBG_OUTPUT_PORT.println("file exists, deleting!");
// todo check to make sure file isn't a directory
SD.remove((char *)fn.c_str());
}
serverMessage += " dir: " + fDir + " fn: " + upload.filename;
uploadFile = SD.open((char *)fn.c_str(), FILE_WRITE);
DBG_OUTPUT_PORT.println("opened!");
}
} else if ((upload.status == UPLOAD_FILE_WRITE) && (serverError == "")) {
if (uploadFile) uploadFile.write(upload.buf, upload.currentSize);
DBG_OUTPUT_PORT.print("Upload: WRITE, Bytes: ");
DBG_OUTPUT_PORT.println(upload.currentSize);
} else if ((upload.status == UPLOAD_FILE_END) && (serverError == "")) {
if (uploadFile) uploadFile.close();
DBG_OUTPUT_PORT.print("Upload: END, Size: ");
DBG_OUTPUT_PORT.println(upload.totalSize);
}
}
To use this function with the error reporting, modify the handler code in the webserver setup as follows (replace the existing line(s) for handleFileUpload):
Code: Select all
server.on("/edit", HTTP_POST, []() {
if (serverError != "") {
returnFail(serverError);
} else if (serverMessage != "") {
// uncomment to refresh back to form
//server.sendHeader("Refresh", "3; url=/upload.htm");
returnMsg(serverMessage);
} else {
returnMsg("upload");
}
serverMessage = "";
serverError = "";
}, handleFileUpload);
One thing I figured out real quick is that the Webserver back-end code doesn't parse the Post parameters from a form when there's a file upload. (int server.args == 0) But it will parse URL parameters. So I'll include the html of my flat file that I put in the root of the SD card which has bare-bones ajax code and updates the form tag's "action" parameter based on the directory you select.
/upload.htm
Code: Select all
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
<title>file upload</title>
<script type="application/javascript" language="JavaScript">
var baseUrl = "";
var dirSel;
function removeOptions(selectbox) {
for(var i = selectbox.options.length - 1 ; i >= 0 ; i--)
selectbox.remove(i);
}
function createOption(val,txt) {
txt = (typeof(txt) == "undefined") ? val : txt;
var newOpt = document.createElement('option');
newOpt.value = val;
newOpt.innerHTML = txt;
return newOpt;
}
function setVisibility(obj,vis) {
//var mainCont = document.getElementById("mainContent");
vis = (typeof(vis) == 'undefined') ? true : vis;
if(vis) {
//mainCont.style.visibility = "hidden";
obj.style.visibility = "visible";
} else {
//mainCont.style.visibility = "hidden";
obj.style.visibility = "visible";
}
}
function jsonHelper() {
// set proper request mode based on browser capabilities
var xhttp = (window.XMLHttpRequest) ? new XMLHttpRequest() : new ActiveXObject("Microsoft.XMLHTTP");
// staring point for building on relative urls
this.baseUrl = baseUrl;
this.serialize = function( obj ) {
return '?'+Object.keys(obj).reduce(function(a,k){a.push(k+'='+encodeURIComponent(obj[k]));return a},[]).join('&')
}
/**
*
* @param {string} uri uri
* @param {string|object} inp input parameters
* @param {object} obj object to modify
* @param {function} cbf callback function
*/
this.getData = function(uri,inp,obj,cbf) {
setLoading();
this.getAjax(
uri, inp,
function(d) {
//if(obj.hasOwnProperty('populate') && (typeof(obj.populate) == 'function')) obj.populate(d);
(typeof cbf == "function") && cbf();
}
);
};
this.getAjax = function(url,inp,cbf) {
inp = (typeof(inp) == 'object') ? this.serialize(inp) : inp;
url = ((typeof(inp) == 'string') && (inp != '')) ? url + "?" + inp : url;
xhttp.onreadystatechange=function()
{
if (xhttp.readyState==4 && xhttp.status==200)
{
var rjson = JSON.parse(xhttp.responseText);
var res = (typeof(rjson) == 'object') ? rjson : xhttp.responseText;
(typeof(cbf) == "function") && cbf(res);
};
};
xhttp.open("GET",url,true);
xhttp.send();
};
};
var actionPrefix = "/edit";
function setAction() {
var formTag = document.getElementById('uploadForm');
var dirText = dirSel[dirSel.selectedIndex].innerHTML;
formTag.action = actionPrefix + "?dir=" + dirText.replace(/\//g, "%2F");
return true;
}
function populateDirlist(data) {
//console.log({json:data});
removeOptions(dirSel);
dirSel.add(createOption("/","/"));
parseDir("/",data.content);
}
function parseDir (path,data) {
for(var i in data) {
if(data[i].type == "dir") {
var fullPath = path+data[i].name+"/";
dirSel.add(createOption(data[i].name,fullPath));
parseDir(fullPath, data[i].content);
}
}
setAction();
};
dirTree.prototype = new jsonHelper();
function dirTree() {
this.dirtree = {};
this.uri = "/listall";
this.getVals = function() {
var self = this;
this.getAjax(
this.uri, {},
function(d) {
populateDirlist(d);
}
);
}
}
var sdDirs = new dirTree();
function onLoaded() {
dirSel = document.getElementById("uploadDir");
sdDirs.getVals();
}
</script>
</head>
<body onload="onLoaded();" class=" >
<form id="uploadForm" action="/edit" method="post" enctype="multipart/form-data">
<div>
<label for="uploadDir">directory:</label>
<!-- you can define initial directories here and disable the ajax or to use as a fallback -->
<select name="dir" id="uploadDir" onChange="setAction();">
<option value="/">/</option>
<option value="/images">/images/</option>
<option value="/css">/css/</option>
<option value="/js">/js/</option>
<option value="/gz/css">/gz/css/</option>
<option value="/gz/js">/gz/js/</option>
<option value="/gz/htm">/gz/htm/</option>
<option value="/gz/svg">/gz/svg/</option>
<option value="/gz/xml">/gz/xml/</option>
<option value="/fonts">/fonts</option>
</select>
</div>
<br>
<label for="fileUpload">Select image to upload:</label>
<input type="file" name="upload" id="fileUpload">
<br>
<input type="submit" value="Upload File" name="submit">
</form>
</body>
</html>
Last edited by treii28 on Sat Aug 27, 2016 2:20 am, edited 3 times in total.
