ESP8266 Community Forum

- Sun Sep 27, 2020 3:30 am #88921 The clearify once more, I don't have the intention to use a broker or receiver in the cloud myself. If I am going to use MQTT to push my sensors' reading, it will be towards a receiver in my own network and eventually, after assessing the protocol implementation is correct, I will make my own receiver. But it should be compatible to those cloud services and local receivers for other people to use.

So the question remains: people that are using cloud services or ready-made receivers, what services/receivers are you using and how are the messages formatted (content).

If it turns out every cloud service or receiver define their own content format, I think it's a futile excercition to try to make a "generic" interface to them. Then indeed you'd need just a library and the end user needs to write the code to insert the content. Or use something like Arduino, not really my kind of thing.

Also if no standard (either real or defacto) exists for sending sensor readings, then I struggle a bit to see the added value for the extra burden of the MQTT encapsulation, one could almost just as good put the content into a raw UDP datagram. Or use TCP for more reliability but then you'd have to create a record structure with a length field, kind of MQTT does, as TCP has no concept of records/boundaries.

Just brainstorming here...

- Sun Sep 27, 2020 8:33 pm #88925

The clearify once more, I don't have the intention to use a broker or receiver in the cloud myself. If I am going to use MQTT to push my sensors' reading, it will be towards a receiver in my own network and eventually, after assessing the protocol implementation is correct, I will make my own receiver. But it should be compatible to those cloud services and local receivers for other people to use.

So the question remains: people that are using cloud services or ready-made receivers, what services/receivers are you using and how are the messages formatted (content).

If it turns out every cloud service or receiver define their own content format, I think it's a futile excercition to try to make a "generic" interface to them. Then indeed you'd need just a library and the end user needs to write the code to insert the content. Or use something like Arduino, not really my kind of thing.

Also if no standard (either real or defacto) exists for sending sensor readings, then I struggle a bit to see the added value for the extra burden of the MQTT encapsulation, one could almost just as good put the content into a raw UDP datagram. Or use TCP for more reliability but then you'd have to create a record structure with a length field, kind of MQTT does, as TCP has no concept of records/boundaries.

Just brainstorming here...

MQTT is a publish/subscribe messaging protocol, it's not point to point - if you're going point to point then I wouldn't use MQTT, I'd just keep to what you're doing as that's going to work the best for you.

If you do decide you want to use MQTT and a pub/sub approach then look at Mosquitto as an open source MQTT broker - don't invent your own.

Because MQTT is pub/sub, the broker only has to understand the messaging protocol not the content - your implementation is completely your own and you're free to design the payload however you wish.

BUT... all the cloud providers out there have implemented their systems so that they unpack JSON directly if your message is sent like that. In addition, they have almost all implemented the idea of a logical interface where you can either define the payload structure via a UI and the provider will unpack it to JSON, or supply a middleware function to manually unmarshall your messages into a JSON document.

The logical interface concept also works with the JSON content too - for instance, suppose your system has three different makes of sensor that all measure temperature, but one measures in F and the other two in C, and they all have different precision. They also all have different names for the field that contains the temp. You then define a single logical interface that you map to each of the physical interfaces so that you get a single name for the measurement, and have consistent units and precision

- Mon Sep 28, 2020 10:15 am #88933 Ok that clears up things already a lot.

But in practise. Guys that do use MQTT to send their sensor readings to the cloud, how do they do it? Is there a more or less standard procedure/interaction, or do all of them define the interaction at both the sending and receiving end? If that is the case, I don't think adding MQTT to my firmware will have much added value, because one would have to adapt the code and compile the modified firmware themselves.

- Mon Sep 28, 2020 7:34 pm #88941 Regardless of whether the broker is in the cloud or not, and in fact regardless of whether you're even using MQTT specifically, pub/sub is an architectural pattern that offers a number of capabilities:

- standard connect and disconnect to brokers based on the protocol, not the specific implementation
- receiving only the information you want, through subscription to named topics
- topic subscription filtering so you can subscribe to the same topic with multiple filters that route info different ways
- topics are persistent at the broker, making the protocol fault tolerant (subscribers can retrieve message backlogs if desired)
- different Quality of Service (QoS) levels for your messages, both sending and receiving. QoS0 is send once with receipt not guaranteed (aka "at most once"), QoS1 is send and guarantee receipt by sending multiple times if you aren't acknowledged (aka "at least once"), and QoS2 is send it once and make sure they got it, do not repeat unless you have to (aka "exactly once")

The one thing that all pub/sub architectures do not specify is the nature and format of the payload. They can sometimes specify whether it's compressed or not, but basically it's like sending snail mail to a PO box at a post office (the broker is the postal worker)
- they don't care who sends the mail, they just put it where they're told
- they don't care who collects the mail, as long as they have the right PO box number
- they can redirect the mail to other PO boxes based on rules and the address on the front, but they won't open your mail
- they guarantee that whatever you put in the envelope will be delivered to the PO Box as is and they don't look at the content - if the content's in Eskimo it's up to you to find a translator

So when you decide to use pub/sub as an architectural pattern, the first thing you need to do is define the message content and structure and publish that for someone to use. It's exactly the same as I2C in many ways - you publish or fetch from registers and you use the spec to understand the data.

Cloud platforms have been developed in the same way - they provide an MQTT broker, and they provide management systems that are simply MQTT clients. Then they give you their spec sheet that says what topics they will publish command messages to, and the format for their commands, and also how you can tell them what topics to listen for, and how to read your messages. They typically also provide a generic JSON reader that will turn JSON into objects that their platform can then use to put on graphs and dashboards etc. However, you can choose to build your own application client completely yourself and ignore theirs - the MQTT broker doesn't care.

If you are interested, I would be very happy to web call with you and show you through my set up in the IBM Cloud sometime