- Tue Mar 15, 2016 1:46 pm
#43237
User
Dave S
Description
This project has been designed as a general-purpose framework which can be customized for unique applications. The system performs as a 3-protocol server. A common command set is served when receiving either http GET requests or a subscribed MQTT message. The third protocol supported addresses communication with an Arduino or any other processor connected to the ESP8266 serial port. In addition, unsolicited MQTT publications are made from a completed link received over the serial link.
To make this as flexible as possible, network, MQTT, serial, and FOTA parameters are configurable using a web interface. The non-volatile values are stored in the ESP8266 EEPROM.
The software was built using the Arduino IDE. While not as rich and flexible as the EspressIf SDK, the Arduino IDE has a wider appeal, making this application framework useful to a larger audience. The design does, however, exploit some of the preferred SDK feature not naively supported by the Arduino IDE.
The circuit was built around classic AI-Thinker bare ESP8266-12 module. Through experience, a minimal set of basic interface circuitry was assembled to ensure reliable, reset free performance.
see 1_systemblockdiagram.jpg
Externally exposed to the world through the WIFI connection, and to an Arduino via the serial port, the system supports identical incoming http and Mqtt request as well as Arduino MQTT publication requests. The http and Mqtt requests can also be directed at controlling and reading Arduino sensors.
see 2_projectimage1.jpg
see 3_projectimage2.jpg
Supported Requests
While this application currently only supports a small group of requests, it can easily be expanded as needed. In order to leverage the same internal code for both http and mqtt requests, the http domain suffix and an mqtt request message payload are parsed the same. Here are the currently supported requests.
see 4_supportedrequests.jpg
In addition to server request received externally from the http or MQTT, the ESP8266 will respond to unsolicited requests from the Arduino serial link. Any time a serial line of data is received from the Arduino, it is passed on as the payload for an MQTT message. A demo Arduino program has been loaded on the Arduino Nano to support the last two rows in the above table.
The ESP8266-Arduino-MQTT broker system presented here is on line now for demonstration. All of the requests listed above can be tested. The MQTT topics are:
mqtt_tx_18fe34a26629 Topic for Messages From ESP8266 mqtt_rx_18fe34a26629 Topic for Messages To ESP8266
“username” and “password” or “mqtt” and “broker” are valid username/password credentials for the mqtt broker used in this project. The domain is “internetofhomethings.duckdns.org” and the mqtt port is 11883.
Web Page Configuration
The application's configuration page is accessed using the URL:
<ESP IP>:<ESP PORT>/config
The default for this project IP=192.168.0.132 and Port-8266, with an URL:
192.168.0.132:8266/config
And here is how it appears:
see 5_webconfiguration.jpg
With this page, it is possible to set all the network, MQTT and serial parameters without modification to the ESP8266 code. Note that the MQTT password and Arduino Server can be disabled.
This is especially useful since MQTT initialization is different for password protected and password-free brokers. As a configurable parameter, you can switch brokers without modifying the ESP8266 code.
Most developers find it useful to use the serial port for debugging code. When the Arduino server is disabled, a number of serial messages are enabled to facility developmental troubleshooting.
Note that the Serial baud is set way down to the snails pace of 1200 baud. This was necessary for this application as a low-end Arduino Nano was used to test the serial interface. Using a “software serial” port on the Nano's digital pins 10 and 11. reliable communication is not possible with baud rates higher than 1200.
But if a more powerful Arduino is used, such as the Mega, with multiple hardware serial ports available, the baud rate could be increased significantly. A simple matter of changing sand saving the desired baud rate on the wen configuration page.
A reset button is also provided on this configuration page. Modified parameters that are saved (by clicking on the save button) do not become effective until the ESP8266 is reset. The button on this page will take care of that requirement.
Know Issue:
While the configuration page renders as intended from a computer on the same local network as the ESP8266, it will not appear from when using an external domain that is “port forwarded” to the ESP8266. After some investigation and testing, it was observed that the rendering from an external domain stops for pages requiring more that 1927 bytes, including http headers, html, css and JavaScript. The configuration page currently sends 6767 bytes to a browser for page rendering. It is not a CORS issue as the correct header is used and it does work for pages less than 1927 bytes. Any ideas as to how to resolve this would be appreciated.
Hardware Circuit
While this project emphasizes on the web/mqtt Server features, it also supports data acquisition. The DAQ code provided with this design simply monitors 8 external analog inputs. But how is that possible. The ESP8266-12 only has 1 analog input? Simple, as you can see in the following schematic, an 8-1 analog multiplexer funnels 8 inputs into the 1 ESP8266 analog input. Three ESP8266 digital channels are used to select the input.
see 6_schematic.jpg
The remaining components in the schematic are used to achieve a stable and reliable hardware platform. Note that during operation, the ESP8266 serial port's Rx and Tx lines are disconnected from the USB interface and instead connected to the Arduino Nano's software serial pins (10 and 11).
My initial circuit assembly setup used a copper plated breadboard. The problem was the ESP contact centers did not match the standard breadboard pitch. So the ESP8266 module had to be mounted to the board using thin (28 AWG) bare wires. These wires were soldered between the ESP8266 through-holes and the breadboard pads. While this soldered circuit operated reliably, the tiny interface wires between the ESP8266 SoC and the breadboard required a lot of time to initially assemble.
But then I found an adapter plate that simplified assembly.
see 7_adapterplate.jpg
Yes, assembly was much easier after I stumbled upon a clean and inexpensive adapter for the ESP8266-7/12 modules. For just 28 cents (AliExpress), you get a PWB plate with per-soldered mounting pads and a 10k pull-up connected to the chip enable (CH_PD).
A few design considerations
Notice the 0.1uF decoupling capacitor added close to the ESP8266 assembly. Positioning this essential capacitor there…directly across the ESP8266 VCC and Gnd pins, is absolutely critical for stable operation.
see 8_decouplingcap.jpg
The header pins were mounted backwards so the longer posts are facing upwards. This will be useful should it be necessary to connect female jumpers directly to any ESP8266 pin.
The USB to serial adapter cable connections are split between two terminal sections. The bottom two terminals connect USB 5V and Ground. The top two terminals, serial transmit and receive, are used when flashing the firmware. Once deployed, the top two terminals are disconnected and the terminal contacts are redirected to the Arduino serial port pins. The 3rd terminal is used to provide a ground connection between the Arduino and the ESP8266 circuit.
With the ESP8266 Tx/Rx contacts removed, only 5V and ground connections are attached to a USB cable. This supports power sourced from a USB wall or car adapter as well as a USB battery.
The ESP serial receive pin has a 100 ohm series resistor included from the serial transmit source. In addition, a 3.3V zener diode clamps the input voltage into the ESP8266. Many ESP8266 users omit these parts. This is not advisable. Exceeding the specified input range will likely degrade the system performance and possibly introduce hard to find latent defects! For long-term reliable operation, these common, readily available components should be included in every interface circuit.
see 9_serialportinterface.jpg
An AMS1117 voltage regulator converts the 5V to 3.3V for the ESP8266 supply. This is mounted on the breadboard copper side of the breadboard so the heat sink built into the regulator can be soldered to the board.
see 10_voltageregulator.jpg
To minimize instabilities from voltage ripple, two 470 uF capacitors are used, one across the 5V source and one across the 3.3V ESP8266 supply.
Software Design Features
The software design supports the capabilities as noted. The code includes the following key features, which can be used in many ESP8266 projects:
1. NIST time: For use in http headers
2. URL decoding: Of http $_GET url parameters for configuration strings
3. SDK Parameter Decoding: Makes it very easy to parse server request parameters
4. SDK Web Server: Using the same callback structure for event driven design
5. Web Browser configuration: change run-time settings without changing sketch
6. MQTT Password Support: Support provided for connections with and without password credentials
7. Unique MQTT Topics: ChipID(MAC) embedded in default MQTT Topics
8. Expandable Server Requests: Design makes it easy to add additional custom requests
9. DAQ: One sensor is read every 2.5 seconds, served via http/mqtt on demand
10. WIFI Monitor: Connection check and reconnect performed every loop
11. FOTA: Simple structure embedded in sketch for wireless firmware updates
The Arduino Sketch
A simple sketch is provided with this project to test the serial interface. It was used with an Arduino Nano, although any Arduino will likely work with this simple sketch. It is called ArduinoHomeAutomation.
Project Installation
The zip file accompanying this document includes a README.md file with step-by-step installation instructions. The project was built using Arduino IDE 1.6.7 and ESP Board Support Package 2.1.0
MQTT Broker
While any MQTT broker can be used, a local broker connected to the same WIFI network as the ESP8266 has been used to test this design. It should be noted that the broker need not be local, however. The local broker was constructed because it was both free and supports password connection. Prior to this, the open source test.mosquitto.org broker was used. But that broker does not support password connections. So another broker was needed to test that capability of this design.
There are many options for setting up a MQTT broker. A simple Linux USB memory stick installation was chosen.
With a little research, you can find all the information needed to set up your own MQTT broker, including step-by-step instructions provided on my blog. No link provided here in strict adherence to the rules.
Parts
Links
Video
Images
11_voltagesmoothingcapacitors.jpg (10.51 KiB) Viewed 4408 times
9_serialportinterface.jpg (10.25 KiB) Viewed 4408 times
Dave S
Description
This project has been designed as a general-purpose framework which can be customized for unique applications. The system performs as a 3-protocol server. A common command set is served when receiving either http GET requests or a subscribed MQTT message. The third protocol supported addresses communication with an Arduino or any other processor connected to the ESP8266 serial port. In addition, unsolicited MQTT publications are made from a completed link received over the serial link.
To make this as flexible as possible, network, MQTT, serial, and FOTA parameters are configurable using a web interface. The non-volatile values are stored in the ESP8266 EEPROM.
The software was built using the Arduino IDE. While not as rich and flexible as the EspressIf SDK, the Arduino IDE has a wider appeal, making this application framework useful to a larger audience. The design does, however, exploit some of the preferred SDK feature not naively supported by the Arduino IDE.
The circuit was built around classic AI-Thinker bare ESP8266-12 module. Through experience, a minimal set of basic interface circuitry was assembled to ensure reliable, reset free performance.
see 1_systemblockdiagram.jpg
Externally exposed to the world through the WIFI connection, and to an Arduino via the serial port, the system supports identical incoming http and Mqtt request as well as Arduino MQTT publication requests. The http and Mqtt requests can also be directed at controlling and reading Arduino sensors.
see 2_projectimage1.jpg
see 3_projectimage2.jpg
Supported Requests
While this application currently only supports a small group of requests, it can easily be expanded as needed. In order to leverage the same internal code for both http and mqtt requests, the http domain suffix and an mqtt request message payload are parsed the same. Here are the currently supported requests.
see 4_supportedrequests.jpg
In addition to server request received externally from the http or MQTT, the ESP8266 will respond to unsolicited requests from the Arduino serial link. Any time a serial line of data is received from the Arduino, it is passed on as the payload for an MQTT message. A demo Arduino program has been loaded on the Arduino Nano to support the last two rows in the above table.
The ESP8266-Arduino-MQTT broker system presented here is on line now for demonstration. All of the requests listed above can be tested. The MQTT topics are:
mqtt_tx_18fe34a26629 Topic for Messages From ESP8266 mqtt_rx_18fe34a26629 Topic for Messages To ESP8266
“username” and “password” or “mqtt” and “broker” are valid username/password credentials for the mqtt broker used in this project. The domain is “internetofhomethings.duckdns.org” and the mqtt port is 11883.
Web Page Configuration
The application's configuration page is accessed using the URL:
<ESP IP>:<ESP PORT>/config
The default for this project IP=192.168.0.132 and Port-8266, with an URL:
192.168.0.132:8266/config
And here is how it appears:
see 5_webconfiguration.jpg
With this page, it is possible to set all the network, MQTT and serial parameters without modification to the ESP8266 code. Note that the MQTT password and Arduino Server can be disabled.
This is especially useful since MQTT initialization is different for password protected and password-free brokers. As a configurable parameter, you can switch brokers without modifying the ESP8266 code.
Most developers find it useful to use the serial port for debugging code. When the Arduino server is disabled, a number of serial messages are enabled to facility developmental troubleshooting.
Note that the Serial baud is set way down to the snails pace of 1200 baud. This was necessary for this application as a low-end Arduino Nano was used to test the serial interface. Using a “software serial” port on the Nano's digital pins 10 and 11. reliable communication is not possible with baud rates higher than 1200.
But if a more powerful Arduino is used, such as the Mega, with multiple hardware serial ports available, the baud rate could be increased significantly. A simple matter of changing sand saving the desired baud rate on the wen configuration page.
A reset button is also provided on this configuration page. Modified parameters that are saved (by clicking on the save button) do not become effective until the ESP8266 is reset. The button on this page will take care of that requirement.
Know Issue:
While the configuration page renders as intended from a computer on the same local network as the ESP8266, it will not appear from when using an external domain that is “port forwarded” to the ESP8266. After some investigation and testing, it was observed that the rendering from an external domain stops for pages requiring more that 1927 bytes, including http headers, html, css and JavaScript. The configuration page currently sends 6767 bytes to a browser for page rendering. It is not a CORS issue as the correct header is used and it does work for pages less than 1927 bytes. Any ideas as to how to resolve this would be appreciated.
Hardware Circuit
While this project emphasizes on the web/mqtt Server features, it also supports data acquisition. The DAQ code provided with this design simply monitors 8 external analog inputs. But how is that possible. The ESP8266-12 only has 1 analog input? Simple, as you can see in the following schematic, an 8-1 analog multiplexer funnels 8 inputs into the 1 ESP8266 analog input. Three ESP8266 digital channels are used to select the input.
see 6_schematic.jpg
The remaining components in the schematic are used to achieve a stable and reliable hardware platform. Note that during operation, the ESP8266 serial port's Rx and Tx lines are disconnected from the USB interface and instead connected to the Arduino Nano's software serial pins (10 and 11).
My initial circuit assembly setup used a copper plated breadboard. The problem was the ESP contact centers did not match the standard breadboard pitch. So the ESP8266 module had to be mounted to the board using thin (28 AWG) bare wires. These wires were soldered between the ESP8266 through-holes and the breadboard pads. While this soldered circuit operated reliably, the tiny interface wires between the ESP8266 SoC and the breadboard required a lot of time to initially assemble.
But then I found an adapter plate that simplified assembly.
see 7_adapterplate.jpg
Yes, assembly was much easier after I stumbled upon a clean and inexpensive adapter for the ESP8266-7/12 modules. For just 28 cents (AliExpress), you get a PWB plate with per-soldered mounting pads and a 10k pull-up connected to the chip enable (CH_PD).
A few design considerations
Notice the 0.1uF decoupling capacitor added close to the ESP8266 assembly. Positioning this essential capacitor there…directly across the ESP8266 VCC and Gnd pins, is absolutely critical for stable operation.
see 8_decouplingcap.jpg
The header pins were mounted backwards so the longer posts are facing upwards. This will be useful should it be necessary to connect female jumpers directly to any ESP8266 pin.
The USB to serial adapter cable connections are split between two terminal sections. The bottom two terminals connect USB 5V and Ground. The top two terminals, serial transmit and receive, are used when flashing the firmware. Once deployed, the top two terminals are disconnected and the terminal contacts are redirected to the Arduino serial port pins. The 3rd terminal is used to provide a ground connection between the Arduino and the ESP8266 circuit.
With the ESP8266 Tx/Rx contacts removed, only 5V and ground connections are attached to a USB cable. This supports power sourced from a USB wall or car adapter as well as a USB battery.
The ESP serial receive pin has a 100 ohm series resistor included from the serial transmit source. In addition, a 3.3V zener diode clamps the input voltage into the ESP8266. Many ESP8266 users omit these parts. This is not advisable. Exceeding the specified input range will likely degrade the system performance and possibly introduce hard to find latent defects! For long-term reliable operation, these common, readily available components should be included in every interface circuit.
see 9_serialportinterface.jpg
An AMS1117 voltage regulator converts the 5V to 3.3V for the ESP8266 supply. This is mounted on the breadboard copper side of the breadboard so the heat sink built into the regulator can be soldered to the board.
see 10_voltageregulator.jpg
To minimize instabilities from voltage ripple, two 470 uF capacitors are used, one across the 5V source and one across the 3.3V ESP8266 supply.
Software Design Features
The software design supports the capabilities as noted. The code includes the following key features, which can be used in many ESP8266 projects:
1. NIST time: For use in http headers
2. URL decoding: Of http $_GET url parameters for configuration strings
3. SDK Parameter Decoding: Makes it very easy to parse server request parameters
4. SDK Web Server: Using the same callback structure for event driven design
5. Web Browser configuration: change run-time settings without changing sketch
6. MQTT Password Support: Support provided for connections with and without password credentials
7. Unique MQTT Topics: ChipID(MAC) embedded in default MQTT Topics
8. Expandable Server Requests: Design makes it easy to add additional custom requests
9. DAQ: One sensor is read every 2.5 seconds, served via http/mqtt on demand
10. WIFI Monitor: Connection check and reconnect performed every loop
11. FOTA: Simple structure embedded in sketch for wireless firmware updates
The Arduino Sketch
A simple sketch is provided with this project to test the serial interface. It was used with an Arduino Nano, although any Arduino will likely work with this simple sketch. It is called ArduinoHomeAutomation.
Project Installation
The zip file accompanying this document includes a README.md file with step-by-step installation instructions. The project was built using Arduino IDE 1.6.7 and ESP Board Support Package 2.1.0
MQTT Broker
While any MQTT broker can be used, a local broker connected to the same WIFI network as the ESP8266 has been used to test this design. It should be noted that the broker need not be local, however. The local broker was constructed because it was both free and supports password connection. Prior to this, the open source test.mosquitto.org broker was used. But that broker does not support password connections. So another broker was needed to test that capability of this design.
There are many options for setting up a MQTT broker. A simple Linux USB memory stick installation was chosen.
With a little research, you can find all the information needed to set up your own MQTT broker, including step-by-step instructions provided on my blog. No link provided here in strict adherence to the rules.
Parts
Links
Video
Images
Attachments
(27.99 KiB) Downloaded 351 times
(1.23 MiB) Downloaded 885 times
(28.02 KiB) Downloaded 311 times
Surplus parts
https://epartsconnect.com
Live realtime stock market prediction website. https://www.stocksignalslive.com my latest creation. Uses AI and machine learning.
New site featuring ESP8266 and ESP32 projects. tutorials, and news.
http://EverythingESP.com
ESP8266.com Founder and Administrator
The Mind Factory (get more from your 3D printer)
Home of the ESP8266, ESP8285, and ESP32 microcontrollers....
https://epartsconnect.com
Live realtime stock market prediction website. https://www.stocksignalslive.com my latest creation. Uses AI and machine learning.
New site featuring ESP8266 and ESP32 projects. tutorials, and news.
http://EverythingESP.com
ESP8266.com Founder and Administrator
The Mind Factory (get more from your 3D printer)
Home of the ESP8266, ESP8285, and ESP32 microcontrollers....
