ESP8266 Community Forum

- Fri Nov 04, 2016 12:39 pm #57664 All, please be gentle with me... this is my first post.

I am trying to upload a 150KB file from a NodeMCU to a remote server. The file has been uploaded to the module using the IDE SPIFFS uploader, where it is read by my program and I attempt to upload it to the remote server with 1 http.write command.

It seems that there is a packet limitation for http.write - somewhere around 3K in size? However, I've seen example code where people have suggested sending many hundreds of bytes is possible in fractions of a second (see https://github.com/esp8266/Arduino/issues/1853 ). When I split my code into numerous http.write instructions, it's very very slow. I'm communicating with the server successfully, and the headers I send are received properly. I just can't seem to find a way to send the binary file quickly.

Can anyone advise what I'm doing wrong in the code? A "conceptual" version of my code is shown below. (I've taken the specific server calls out to protect my client etc.) I'm using v2.3.0 of the library.

Thanks.

Code: Select all#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <WiFiUdp.h>
#include <SPI.h>
#include "FS.h"

#include "string.h"

#define USE_SERIAL Serial

char ssid[] = "ssid";         //  your network SSID (name)
char pass[] = "Pass";             // your network password
WiFiClient client;

void setup() {

  USE_SERIAL.begin(230400);
  USE_SERIAL.printf("Connecting to WiFi");
  WiFi.begin(ssid, pass);
  WiFi.mode(WIFI_STA);
}

void loop()
{
  // wait for WiFi connection
  if (WiFi.status() == WL_CONNECTED)
  {
    const char* host = "host.com";
    unsigned int content_length=0;

    if (client.connect(host, 80))
    {
      client.setNoDelay(true);
      char send_buf_header[400], send_buf_body[1000];
      USE_SERIAL.println("Connected!");

      memset(send_buf_header, 0, 400);
      memset(send_buf_body, 0, 1000);

      char *BOUNDARYSTR = "**BOUNDARYSTR**";
      char BOUNDARY[100];
      sprintf(BOUNDARY, "--%s\r\n", BOUNDARYSTR);
      char ENDBOUNDARY[100];
      sprintf(ENDBOUNDARY, "--%s--\r\n\r\n", BOUNDARYSTR);
 
      // create http header
      strcat(send_buf_header, "POST /v1/identify HTTP/1.1\r\n");
      strcat(send_buf_header, "Host: host.com\r\n");
      char tmp_buf[200];
      sprintf(tmp_buf, "Content-Type: multipart/form-data; boundary=%s\r\n", BOUNDARYSTR);
      strcat(send_buf_header, tmp_buf);

      // create http body
      int content_body_len = 0;

      char *signature = "abcdefgh=";
      sprintf(tmp_buf, "%sContent-Disposition: form-data; name=\"%s\"\r\n\r\n%s\r\n", BOUNDARY,"signature",signature);
      memcpy(send_buf_body+content_body_len, tmp_buf, strlen(tmp_buf));
      content_body_len += strlen(tmp_buf);

      // more headers etc are defined here, which I have taken out to aid readability. I know this part of the code is ok.
     
      sprintf(tmp_buf, "Content-Length: %d\r\n\r\n", content_body_len);
      strcat(send_buf_header, tmp_buf);

      USE_SERIAL.println("Sending>>>>>>");
      USE_SERIAL.println("(header)");
      client.print(send_buf_header);
      USE_SERIAL.println("(body)");
      client.print(send_buf_body);
      USE_SERIAL.println("(file)");

      //**** This is where my problems begin!!! ****

      /* This file upload technique works but is very very very slow!
      #define MTU_SIZE 2*1460
      for (int i=0; i<150000; i=i+MTU_SIZE) client.write(&bigfile[i],MTU_SIZE);   
      client.println();
      */

      // This is what I'm trying to get to work....
      SPIFFS.begin();
      File spiFile = SPIFFS.open("bigfile.bin", "r");
      if (!spiFile) USE_SERIAL.println("file open failed");
      else             USE_SERIAL.println("wav file opened");
      client.write((uint8_t)spiFile); // only sends 1 byte, throws an error unless I cast it????
      //client.write(spiFile);
      spiFile.close();
      client.println();

      sprintf(tmp_buf, "%sContent-Disposition: form-data; name=\r\n\r\n\r\n", BOUNDARY);
      memcpy(send_buf_body+content_body_len, tmp_buf, strlen(tmp_buf));
      content_body_len += strlen(tmp_buf);
      client.println();
     
      client.print(ENDBOUNDARY);
      USE_SERIAL.println(">>>>>>Complete");           
     
      if(!eRcv());
      client.stop();

      while (client.connected())
      {
        if ( client.available() )
        {
          char str = client.read();
          Serial.print(str);
        }
      }
    }
  }

  delay(10000);
}

byte eRcv()
{
  byte respCode;
  byte thisByte;
  while(!client.available()) delay(1);
  respCode = client.peek();
  while(client.available())
  { 
    thisByte = client.read();   
    Serial.write(thisByte);
  }
  return 1;
}

- Fri Nov 04, 2016 2:51 pm #57668

Code: Select allclient.write((uint8_t)spiFile); // only sends 1 byte, throws an error unless I cast it????

You are sending a single byte with the value of "spiFile" file handle, not the content of the file.

You need to do something like that :

Code: Select all    char buf[1024];
    int siz = spiFile.size();
    while(siz > 0) {
      size_t len = std::min((int)(sizeof(buf) - 1), siz);
      spiFile.read((uint8_t *)buf, len);
      client.write((const char*)buf, len);
      siz -= len;
    }
    spiFile.close();


- Fri Nov 04, 2016 4:07 pm #57672 (I'm posting this again as my first attempt didn't seem to get posted.....)

Hi Martin. Thank you for your comments. Is your proposal different to my "slow" method:

/* This file upload technique works but is very very very slow!
#define MTU_SIZE 2*1460
for (int i=0; i<150000; i=i+MTU_SIZE) client.write(&bigfile[i],MTU_SIZE);
client.println();
*/

It seems to me that multiple client.write calls are what slows the transfer down. Isn't there a way to transfer more bytes in one call?

- Sat Nov 05, 2016 9:56 am #57699 Why are you concluded that this code is very slow ?

It could not be faster, even if you were able to pass all data in a single client.write(), under the hood, it will still be transmitted using MTU chunk size of 1460, therefore it won't be faster.

If you still have slowness, the problem could be elsewhere.