ESP8266 Community Forum

- Tue Sep 29, 2020 10:18 am #88948 So what you're saying, in very short, is, what I already was starting to suspect: there is no standardisation on the content of the messages. So I can't make code that sends the names of all detected sensors with their readings, because the recipients (ignoring brokers here) require a specific formatting and will not understand my format. If I would adapt it to one service, it would still not work on any other service.

So the only way to get it working in a generic way, is to have a library and to have the "customers" write the code the fill the messages, right?

- Tue Sep 29, 2020 9:47 pm #88953 As mentioned, almost every cloud broker has some sort of transform capability, so you can send in multiple formats and transform into a single format, or vice versa - send in one format and transform to multiple.

Even if you do it yourself, it's trivial to write a client that listens on one topic, transforms the messages and publishes on another. Then you just run that as a background app or even as a serverless function (another thing that all cloud providers have)

This is effectively what I do - I send my payloads in JSON like this:
event data
pwr {"V":"1.23","I":"347","ts":"1601432881"}
env {"T":"23.6","P":"101623","Rho":"34.55","ts":"1601432765"}
snd {"dBA":"48.9","ts":"1601432761"}

the MQTT client name is the sensor name, so I get that for free, and the events are the MQTT message types. The MQTT broker handles all connect, disconnect, heartbeats and health checks so it tells me if a sensor goes offline or doesn't check in within its promised time range.

The MQTT broker receives the events and timestamps them when they are received (which may be different from when the data was collected). Then I have a background task that takes the data, splits it and transforms it, and does some stats with it, then stores it in a NoSQL database with tables for each sensor category and a primary key made up of <sensor id>:<ts>, which makes it easy to query as a time series

It also publishes the processed data on another MQTT queue, where my web app listens for it and displays it if a user's connected.

- Fri Oct 02, 2020 3:01 am #88978 So if I would this format (which seems quite logical to me) at least multiple services would be able to process it (either directly or with a translation step)?

Then it would be interesting.

My basis is a little bit different then yours I think, in that the concept is that you can attach any amount of I2C sensors as long as they don't have conflicting addresses and can be successfully probed/detected. Using a bus multiplexer, that can be quite a lot and the list will differ with each instance.

So at least the "name" would be the sensor's identification (e.g. TMP75) and added either a sequence or the address+bus branch to the name. This might be a bit complex at the processing side to handle, as they're always different from instance (ESP) to instance, according to what and how one connects the sensors.

- Fri Oct 02, 2020 8:12 am #88993 I actually think we are the same. I have I2C, I2S, and UART devices and they can be added and removed and changed in my sensors.

So I have a layered architecture where, just like you, the device is found on the bus, but when a chip is probed and registered by a device driver, it then gets registered in a sensor class, along with what sensor channels (measurements) it supplies, and it's precision etc. This means only the device driver cares what chip I have - everything else works on sensor types and channels. It also means nothing changes in my system when I upgrade a device for a better one.

A good example of this is my old sensor board used a BME280 to get Temp, Pressure and Humidity. I've now got a BMP388 and an SHT35 instead. I also changed the CCS811 air quality sensor for an SGP30. The only thing I had to do is write device drivers

When my sensor is initialised, it records the list of classes and channels and at each boot it compares what is found and warns me if the list changes - if I've just added new sensors then I can tell it to update the list, if not I can do maintenance.

This whole idea is based around the Industrial I/O (IIO) subsystem approach