Datagraphy

Datagraphy


Musing on technology and a few other topics

Share


Tags


IoT experiment with MQTT

In this post, I will describe a small experiment I conducted to collect environmental data from a device and post it to a REST API on a remote server. The purpose of this experiment is to understand more concretely how to work with IoT devices and collect their information to create web applications. I have split this post in five different parts below before concluding and listing potential next steps I will conduct later.

Overall setup

I tried to keep the setup simple but not too simple in order to ensure that I cover enough components and learn something along the way.

There are three components in the setup I will describe, as illustrated in the figure below.

Components of the MQTT experiment

There are three components in the setup:

There is actually a fourth component, a REST API server. But given that I didn't set it up, I don't count it. Though, to ensure the architecture is complete, if you are interested in reproducing this setup, you would have to take this into account and setup a REST API server.

The flow of information is illustrated in the following figure.

Information flows

The flow of information is the following:

  1. The sensors on the device measure three parameters: temperature, pressure and ambient humidity.
  2. The device sends the measurements for each parameter to the MQTT broker through its publication to three different topics on the broker.
  3. The broker receives the messages on each topic and makes them available to subscribers on the same three different topics.
  4. The Python client which has subscribed to the three topics will receive the measurements
  5. The Python client will send the measurements to the REST API server.

As mentioned above, I have decided to make all communications between the components encrypted and authenticated. In order to do so, it is necessary to use the TLS protocol. TLS is a security protocol that provides the following security mechanisms:

I will describe the TLS setup in the next section as it is used in all the components before describing the components themselves.

Security

As I described in the diagram above, I have decided to encrypt all communications between the components. To do so, I needed to create X.509 certificates for the device, the broker and the client. X.509 is an ITU (the telecommunication standardization body) standard which defines the format of public key certificates used in asymmetric cryptography.

X.509 certificates are delivered by a Certificate Authority (CA) which acts as a trusted entity for all parties. Given that all entities in this case are under my control, I have decided to create my own CA. If that were not the case, I would have had to rely on an external CA such as Let's Encrypt for example.

To create my own certificate authority, I used the ubiquitous OpenSSL library. There are other libraries out there (Network Security Services from the Mozilla Project and GnuTLS are two that come to mind), but I don't know them as well as OpenSSL.

Creation of the CA Certificate

The creation of both the keys and the certificate for the CA can be done in one line with OpenSSL. To do that, simply call:

$ openssl req -new -x509 -days 365 -extensions v3_ca -keyout my-ca.key -out my-ca.crt
Generating a RSA private key
........+++++
....................................................................+++++
writing new private key to 'my-ca.key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:FR
State or Province Name (full name) [Some-State]:IdF
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Datagraphy
Organizational Unit Name (eg, section) []:Certificate Authority
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:

Of course, you should make sure you protect your key using a passphrase. This is standard security stuff.

With the command above, the certificate of your CA will be valid for one year (thanks to -days 365). And -extensions v3_ca will generate a X.509 version 3 certificate.

Theoretically, if I wanted to do something more robust, I should have created two CA, a root one and a secondary one. The root one would sign the certificate request of the secondary one which would then sign the certificate requests of the device and client.

Let's now move to the broker certificate.

Creation of the Broker Certificate

To create the certificate for the broker, I first generate its key pair using the openssl genrsa command.

$ openssl genrsa -out mosquitto.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
......................+++++
...................+++++
e is 65537 (0x010001)

I can then create the certificate request that will be sent to the CA for signature using the openssl req command:

$ openssl req -out mosquitto.csr -key mosquitto.key -new
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:FR
State or Province Name (full name) [Some-State]:IdF
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Datagraphy
Organizational Unit Name (eg, section) []:MQTT broker
Common Name (e.g. server FQDN or YOUR name) []:mosquitto
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

I have called the broker mosquitto, a name that will become clear later. In the X.509 linguo, this name is called the Common Name (CN).

The file mosquitto.csr is our certificate request that needs to be signed by the CA. If I had decided to use an external CA, this is the file I would have to provide the CA for signature (and probably with some documents proving that this is my device and that I am who I pretend I am... But I digress).

I can now use our CA certificate and key to sign the certificate requests for the broker. To do that, ket's use OpenSSL again:

$ openssl x509 -req -in mosquitto.csr -CA my-ca.crt -CAkey my-ca.key -CAcreateserial -out mosquitto.crt -days 180
Signature ok
subject=C = FR, ST = IdF, O = Datagraphy, OU = MQTT broker, CN = mosquitto
Getting CA Private Key
Enter pass phrase for my-ca.key:

I have defined a duration of 180 days for the certificate of the broker to be valid. That means that after expiration, a new certificate will need to be emitted by the CA, using the same process as above.

I now have my certificate for the broker: mosquitto.crt. To operate properly, the MQTT broker will need both the certificate and the key file mosquitto.key. So make sure you have both before proceeding.

Creation of the device and client certificates

To create the device and the client certificates, simply follow the same steps as for the broker and adjust the CN in the certificate requests and the file names.

MQTT Broker

The MQTT broker is the central piece in the setup. The IoT device will publish messages on topics that are managed by the MQTT broker and the client will subscribe to those topics to receive those messages.

To implement the MQTT broker, I chose to use Mosquitto from the Eclipse project. Mosquitto is a free and open source MQTT broker that implements the MQTT protocol version 5.0, 3.1.1 and 3.1.

The installation of Mosquitto is straightforward. I chose to implement the broker on a cloud server provided by Linode. I chose an Linux Ubuntu 19.10 machine as it is readily available and easy to manage.

After the initial setup, the Mosquitto broker can be installed using the following command:

sudo apt install mosquitto

Simple, isn't it?

I then need to configure the broker. It's easy to check that it is already running:

$ systemctl status mosquitto
● mosquitto.service - Mosquitto MQTT v3.1/v3.1.1 Broker
   Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-04-05 17:15:41 UTC; 4min 48s ago
     Docs: man:mosquitto.conf(5)
           man:mosquitto(8)
 Main PID: 642 (mosquitto)
    Tasks: 1 (limit: 1078)
   Memory: 1.7M
   CGroup: /system.slice/mosquitto.service
           └─642 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

The configuration file is located in /etc/mosquitto/mosquitto.conf, and the content of the file is the following:

# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example

pid_file /var/run/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

So I will place my configuration file into /etc/mosquitto/conf.d/. I have placed the mosquitto.conf file corresponding to the one I have on the server on my Github Repo, specifically in the broker directory.

The important lines in the configuration file are described below, starting with:

allow_anonymous false

This line forbids anonymous connections to the broker. All subscribers and publishers have to be authenticated. Given that I don't provide any password file, this means the subscribers and publishers will have to be authenticated in a different way. That's why I created the certificates earlier.

To ensure the broker has the necessary information regarding the CA, its own key and its certificate, I have put the following lines in the configuration file:

# TLS encryption and authentication information
# Directory to store Certificate authority PEM encoded certificate files
capath /etc/mosquitto/ca_certificates/

# Broker PEM encoded certificate and key files
certfile /etc/mosquitto/certs/mosquitto.pem
keyfile /etc/mosquitto/certs/mosquitto.key

I have provided a path for the CA certificate, and the broker will trust all certificates in the path. I have only stored the my-ca.crt that I renamed my-ca.pem, and I did the same with the broker certificate.

The last important line in the file is the one requiring subscibers and publishers to provide a valid and trusted certificate to the broker. By trusted, I simply mean emitted by a trusted CA. The corresponding line is the following:

require_certificate true

With this configuration, our broker is ready to run its communication over TLS. Don't forget to store the certificates and the keys as mentionned in the configuration file. To ensure the key is protected, I need to only allow the mosquitto user to be able to read it. This is done using:

$ sudo chown mosquitto /etc/mosquitto/certs/mosquitto.key
$ sudo chmod 400 /etc/mosquitto/certs/mosquitto.key

I can now restart the broker using:

$ sudo systemctl restart mosquitto

With this done, let's now move to the IoT device.

IoT Device

For the device, I went for something that would get me up and running quickly without going to deep into the electronics. That's why I chose the pycom fipy module. The reason I chose a pycom device are:

There are obviously plenty of other possibilities when it comes to device choice, both in terms of microcontrollers and languages.

I won't go into the detail of the basic setup of the pycom device here, given that the documentation at docs.pycom.io provides all the information necessary as to how to go through the setup.

So, after the initial setup, the device is ready to be programmed. In MicroPython, the main code needs to be placed in the file main.py. There is also a boot.py file but I won't use it here. In theory, given that boot.py is run before main.py, I should use the boot.py file for everything related to device setup such as network connections, etc.

Since this is an experiment, I went for the easy route and store everythin in main.py.

The code I used is available in my GitHub repo in the device directory.

This is how I proceeded:

There are several files to upload to the device before writing the code for the pycom module. Those files are:

To upload those files, the easiest way is to connect to the pycom module with an FTP program using passive mode (type passive at the ftp> prompt). On all pycom modules, the default user for FTP is micro and the password is python.
Upon connection, you will arrive at the root directory and you need to place the certificates in the /flash/certs/ directory and the library files in the /flash/lib/ directory.

When this is done, you can now program the device using the Pymakr plugins provided by pycom.

The key elements in the code are the following:

ps = Pysense()
sensor = SI7006A20(ps)
sensor2 = MPL3115A2(ps)
pycom.heartbeat(False)

The last line simply turns off the onboard LED on the device.

net = WLAN(mode=WLAN.STA)
net.connect(ssid=SSID, auth=(WLAN.WPA2, WLAN_PASSWD))

where SSID should be defined previously as the SSID if your WiFi network and WLAN_PASSWD as the password for your network.

PARAMS = {'keyfile': 'cert/device.key', 
            'certfile': 'cert/device.pem', 
            'cert_reqs': ssl.CERT_REQUIRED, 
            'ca_certs': 'cert/my-ca.pem'}

client = MQTTClient("device", MQTT_BROKER, port=1883, ssl=True, ssl_params = PARAMS)

Here PARAMS defines different parameters for the TLS authentication and encryption mechanism. To create the client, I simply call the MQTTClient() class that I previously imported from the mqtt.py module. Since I have not defined a specific port on the broker previously, I can use the default MQTT port of 1883. MQTT_BROKER is the IP address or hostnmane that I have defined previously in my code (see the whole code in the repo for that).

print("Connecting to MQTT broker...")
try:
    client.connect()
    client.subscribe(topic=MQTT_TOPIC)
    client.subscribe(topic=MQTT_TOPIC2)
    client.subscribe(topic=MQTT_TOPIC3)
    print("Done")
    CONNECT = True
except OSError:
    print("Cannot connect to MQTT broker...")
    CONNECT = False

The code is self explanatory, but I need to subscribe the client to each MQTT topic that I define (three topics, one for each measure).

while CONNECT:
    temp = sensor.temperature()
    humid_ambient = sensor.humid_ambient(temp)
    pressure = sensor2.pressure()

Again, the code is simple, I just call the right method on the instances of the sensors I created above. I need to measure temperature first given that it is necessary to measure ambient humidity.

    client.publish(topic=MQTT_TOPIC, msg="{:.1f}".format(temp))
    client.publish(topic=MQTT_TOPIC2, msg="{:.2f}".format(pressure))
    client.publish(topic=MQTT_TOPIC3, msg="{:.1f}".format(humid_ambient))

Data is published as a string, so I format it before sending it. MicroPython is not compatible with Python 3.6 so I couldn't resort to f-strings here and use the .format() way of formating strings.

There are also some other instructions in the code, such as a subscribe callback, but I couldn't make it work with the TLS based implementation (it worked with a user / password based authentication), so I left it aside for now.

Before putting everything together, the last step is to create the Python client that will subscribe to the broker to get the data published by the device on the different topics.

Python client

The Python client is the last component of the architecture. Its role is to subscribe to the topics on the MQTT broker to receive the data published by the device and to forward it to a REST API server.

Again, the code of the Python client is on my GitHub repo and I will explain the overall flow here.

As I did for the device, I describe below the key elements in the client code:

import paho.mqtt.client as mqtt

I decided to us the Paho MQTT client from the Eclipse Project, just like Mosquitto. It has the advantage of offering clients in a wide range of languages, including Python. It is also quite easy to use.

def callback_on_message(client, userdata, msg):
    print(f"In topic: {msg.topic}, received payload {msg.payload.decode('utf-8')}.")

    data = prepare_data(msg)

    post_data(data, TOPIC_DATA.get(msg.topic).get('url'))

The callback is the code that will be called when the client receives messages on the topics it has subscribed. This code will be run each time a message is received. Aside from printing on the screen the topic on which the message was received and the content of the message, it will prepare a dictionary through the function prepare_data() that will be sent as JSON with the post_data() function.

def prepare_data(msg):
    date = datetime.datetime.now().astimezone(tz=pytz.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    data = {"id": TOPIC_DATA.get(msg.topic).get("id"),
            "name": TOPIC_DATA.get(msg.topic).get("name"),
            "values": [
                {"date_time": date,
                 "value": float(msg.payload.decode('utf-8'))}
            ]}
    return data

The MQTT message is formated into a dictionary and timestamped with the current date in the UTC timezone. The function simply returns the dictionary.

def post_data(data, url):
    r = requests.put(url, json=data)
    r.raise_for_status()

This code is quite simple, and it's using on the requests library. The only trick is to use the keyword argument json= and not data=. I initially used data= and it didn't work. I found a workaround by importing the json package and used json.dumps(data) to pass the data to the PUT request until I realized the simplest way is the one above.

if __name__ == '__main__':
    print('Creating Client')
    client = mqtt.Client("Python Client")
    client.on_message = callback_on_message
    print('Connecting to broker...')
    client.tls_set(ca_certs="certs/my-ca.pem", certfile="certs/client.pem",
                   keyfile="certs/client.key", cert_reqs=ssl.CERT_REQUIRED)

    # to be removed at a later stage!
    client.tls_insecure_set(True)

    # Connect to broker
    client.connect(MQTT_BROKER)

The client is created through a mqtt.Client() call which instanciate the Client() class. I then pass the callback I defined previously which will be run on each received message.
Then, I pass the different certificates and key that will be used for the connection before connecting to the broker. Of course, the certificates and the key need to be stored in the appropriate directory.

    # Start loop and subscribe to topics
    client.loop_start()
    print(f'Subscribing to topic {MQTT_TOPIC}, {MQTT_TOPIC2} and {MQTT_TOPIC3}')
    client.subscribe(MQTT_TOPIC)
    client.subscribe(MQTT_TOPIC2)
    client.subscribe(MQTT_TOPIC3)

    # Go for 1000 iterations
    i = 0
    while i < 1000:
        time.sleep(1)
        i += 1

    client.loop_stop()
    print('Exiting...')

The three key elements here are the start of the loop through client.loop_start(), the subscription to the different topics through client.subscribe() and the end of the loop through client.loop_stop().

If I want an infinite loop, I have to call client.loop_forever() instead of starting the loop as above. Of course, by looping forever, I don't need to stop the loop.
I don't need to check for messages in this loop because this is taken care of by the callback I defined previously.

With that done, it's time to test the whole set-up!

Putting it all together

Well... Everything is ready now!

First, I ensure the Mosquitto broker is running and I monitor the logs to see the connections (using sudo tail -f /var/log/mosquitto/mosquitto.log).

I then plug my pycom device and using Pymakr, I upload the code to the device if this has not been done already. The device should start publishing data shortly after. Then I start the Python client and see if the data is received.

Finally I can check on the REST API server whether data has been correctly published.

When I plug the device, I get the following output in the Pymakr console:

Connecting to /dev/tty.usbmodemPy3434341...
Trying to connect to network...
Trying to connect to network...
Trying to connect to network...
Trying to connect to network...
Trying to connect to network...
Connected!
('192.168.1.20', '255.255.255.0', '192.168.1.1', '192.168.1.1')
Creating MQTT client...
Connecting to MQTT broker...
Done
Sending Data to MQTT broker
Sending Data to MQTT broker

So from the device point of view, it works.

When I start the Python client, I get the following output:

Creating Client
Connecting to broker...
Subscribing to topic topic/TEMP, topic/PRESSURE and topic/HUMID
In topic: topic/TEMP, received payload 24.7.
In topic: topic/PRESSURE, received payload 101522.00.
In topic: topic/HUMID, received payload 55.4.
In topic: topic/TEMP, received payload 24.7.
In topic: topic/PRESSURE, received payload 101521.50.
In topic: topic/HUMID, received payload 55.2.

And I can see that I received the data posted to the REST API server:

Temperature data posted on REST API server

The log file of the Mosquitto broker shows me the connections of both the device and the Python client:

1586369286: mosquitto version 1.6.6 starting
1586369286: Config loaded from /etc/mosquitto/mosquitto.conf.
1586369286: Opening ipv4 listen socket on port 1883.
1586369286: Opening ipv6 listen socket on port 1883.
1586369783: New connection from XX.XX.XX.XX on port 1883.
1586369786: New client connected from XX.XX.XX.XX as device (p2, c1, k0, u'pycom-device').
1586369786: device 0 topic/TEMP
1586369786: device 0 topic/PRESSURE
1586369786: device 0 topic/HUMID
1586370414: New connection from XX.XX.XX.XX on port 1883.
1586370417: Client device already connected, closing old connection.
1586370417: New client connected from XX.XX.XX.XX as device (p2, c1, k0, u'pycom-device').
1586370417: device 0 topic/TEMP
1586370417: device 0 topic/PRESSURE
1586370417: device 0 topic/HUMID
1586370430: New connection from XX.XX.XX.XX on port 1883.
1586370430: New client connected from XX.XX.XX.XX as Python Client (p2, c1, k60, u'python-client').
1586370430: Python Client 0 topic/TEMP
1586370430: Python Client 0 topic/PRESSURE
1586370430: Python Client 0 topic/HUMID
1586370451: Socket error on client Python Client, disconnecting.
1586371087: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.

Since I aborted the client before the end of the 1000 iterations, the second to last line is normal. Same thing with the two connections from the device, since I unplugged it after the first run. The log file shows to which topics both the client and the device subscribe / publish.

Conclusion and next steps

This experiment with IoT and MQTT looks simpler now that it was. I struggled to make the setup works at several occasions, but overall this proves that the best way of learning is really by doing. Nothing beats hitting a wall and trying to get over it.

Hopefully there is enough information here for anyone interested to get a similar setup running from start to finish in a shorter time than it took me to do it.

So now, what's next? Honestly, this is very basic! There are a lot of things to explore with this basic setup and also a lot of things to improve. In no specific order, here are the list of additional topics and questions I had in mind:

So, many questions and no answers yet, but that gives me the basis for further exploration and additional posts!

Until then, take care and stay safe... Yes, that's Day 23 of confinement here, time for me to write a confinement post after this one!

View Comments