MQTT Connector Configuration

• Connector configuration: mqtt.json
  • Section “broker”
    • Subsection “security”
  • Section “mapping”
    • Subsection “converter”
  • Section “connectRequests”
  • Section “disconnectRequest”
  • Section “attributeRequests”
  • Section “attributeUpdates”
  • Server side RPC commands
• Next steps

This guide will help you to get familiar with MQTT Connector configuration for ThingsBoard IoT Gateway. Use general configuration to enable this Connector. The purpose of this Connector is to connect to an external MQTT broker and subscribe to data feed from devices. The connector is also able to push data to MQTT brokers based on the updates/commands from ThingsBoard.

This connector is useful when you have local MQTT broker in your facility or corporate network and you would like to push data from this broker to ThingsBoard.

We will describe connector configuration file below.

Connector configuration: mqtt.json

Connector configuration is a JSON file that contains information about how to connect to external MQTT broker, what topics to use when subscribing to data feed and how to process the data. Let’s review the format of the configuration file using the example below.

Example of MQTT Connector config file.

The example listed below will connect to MQTT broker in a local network deployed on server with IP 192.168.1.100. Connector will use basic MQTT auth using username and password. Then, connector will subscribe to a list of topics using topic filters from the mapping section. See more info in the description below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
{
  "broker": {
    "name":"Default Local Broker",
    "host":"192.168.1.100",
    "port":1883,
    "security": {
      "type": "basic",
      "username": "user",
      "password": "password"
    }
  },
  "mapping": [
    {
      "topicFilter": "sensor/data",
      "converter": {
        "type": "json",
        "deviceNameJsonExpression": "${serialNumber}",
        "deviceTypeJsonExpression": "${sensorType}",
        "timeout": 60000,
        "attributes": [
          {
            "type": "string",
            "key": "model",
            "value": "${sensorModel}"
          },
          {
            "type": "string",
            "key": "${sensorModel}",
            "value": "on"
          }
        ],
        "timeseries": [
          {
            "type": "integer",
            "key": "temperature",
            "value": "${temp}"
          },
          {
            "type": "integer",
            "key": "humidity",
            "value": "${hum}"
          },
          {
            "type": "string",
            "key": "combine",
            "value": "${hum}:${temp}"
          }
        ]
      }
    },
    {
      "topicFilter": "sensor/+/data",
      "converter": {
        "type": "json",
        "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/data)",
        "deviceTypeTopicExpression": "Thermometer",
        "timeout": 60000,
        "attributes": [
          {
            "type": "string",
            "key": "model",
            "value": "${sensorModel}"
          }
        ],
        "timeseries": [
          {
            "type": "integer",
            "key": "temperature",
            "value": "${temp}"
          },
          {
            "type": "integer",
            "key": "humidity",
            "value": "${hum}"
          }
        ]
      }
    },
    {
      "topicFilter": "sensor/raw_data",
      "converter": {
        "type": "bytes",
        "deviceNameExpression": "[0:4]",
        "deviceTypeExpression": "default",
        "timeout": 60000,
        "attributes": [
          {
            "type": "raw",
            "key": "rawData",
            "value": "[:]"
          }
        ],
        "timeseries": [
          {
            "type": "raw",
            "key": "temp",
            "value": "[4:]"
          }
        ]
      }
    },
    {
      "topicFilter": "custom/sensors/+",
      "converter": {
        "type": "custom",
        "extension": "CustomMqttUplinkConverter",
        "extension-config": {
            "temperatureBytes" : 2,
            "humidityBytes" :  2,
            "batteryLevelBytes" : 1
        }
      }
    }
  ],
  "connectRequests": [
    {
      "topicFilter": "sensor/connect",
      "deviceNameJsonExpression": "${serialNumber}"
    },
    {
      "topicFilter": "sensor/+/connect",
      "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/connect)"
    }
  ],
  "disconnectRequests": [
    {
      "topicFilter": "sensor/disconnect",
      "deviceNameJsonExpression": "${serialNumber}"
    },
    {
      "topicFilter": "sensor/+/disconnect",
      "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/disconnect)"
    }
  ],
  "attributeRequests": [
    {
      "retain": false,
      "topicFilter": "v1/devices/me/attributes/request",
      "topicExpression": "${serialNumber}",
      "valueExpression": "${sensorModel}"
    }
  ],
  "attributeUpdates": [
    {
      "retain": false,
      "deviceNameFilter": ".*",
      "attributeFilter": "firmwareVersion",
      "topicExpression": "sensor/${deviceName}/${attributeKey}",
      "valueExpression": "{\"${attributeKey}\":\"${attributeValue}\"}"
    }
  ],
  "serverSideRpc": [
    {
      "deviceNameFilter": ".*",
      "methodFilter": "echo",
      "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "responseTopicExpression": "sensor/${deviceName}/response/${methodName}/${requestId}",
      "responseTimeout": 10000,
      "valueExpression": "${params}"
    },
    {
      "deviceNameFilter": ".*",
      "methodFilter": "no-reply",
      "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "valueExpression": "${params.hum}::${params.temp}"
    }
  ]
}

Section “broker”

Subsection “security”

Subsection “security” provides configuration for client authorization at Mqtt Broker.

1
2
3
4
5
    "security": {
      "type": "basic",
      "username": "username",
      "password": "password"
    }

1
2
3
    "security": {
      "type": "anonymous"
    }

1
2
3
4
5
6
"security":{
    "type": "certificates",
    "caCert": "/etc/thingsboard-gateway/ca.pem",
    "privateKey": "/etc/thingsboard-gateway/privateKey.pem",
    "cert": "/etc/thingsboard-gateway/certificate.pem"
}

Section “mapping”

This configuration section contains an array of topics that the gateway will subscribe to after connecting to the broker, along with settings about processing incoming messages (converter)..

The topicFilter supports special symbols: ‘#’ and ‘+’, allowing to subscribe to multiple topics.

Also, MQTT connector supports shared subscriptions. To create shared subscription you need to add “$share/” as a prefix for topic filter and shared subscription group name. For example to subscribe to the my-shared-topic in group my-group-name you can set the topic filter to “$share/my-group-name/my-shared-topic”.

Let’s assume we would like to subscribe and process the following data from Thermometer devices:

In this case the following messages are valid:

Example 1:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/data" -m '{"serialNumber": "SN-001", "sensorType": "Thermometer", "sensorModel": "T1000", "temp": 42, "hum": 58}'

Example 2:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/SN-001/data" -m '{"sensorType": "Thermometer", "sensorModel": "T1000", "temp": 42, "hum": 58}'

Now let’s review how we can configure JSON converter to parse this data

Subsection “converter”

This subsection contains configurations for processing incoming messages.

The types of MQTT converters are as follows:

1. json – Default converter
2. raw – Raw default converter
3. custom – Custom converter (You can write it yourself, and it will be used to convert incoming data from the broker.)

jsonRecommended if json will be received in response

bytesRecommended if bytes will be received in response

customRecommended if bytes or anything else will be received in response

Json converter is the default converter, it looks for ‘deviceName’, ‘deviceType’, attributes and telemetry in the incoming message from the broker, following the rules described in this subsection: Parameter Default value Description type json Provides information to connector that default converter is to be used for converting data from topic. deviceNameJsonExpression ${serialNumber} Simple JSON expression, is used for looking up device name in the incoming message (parameter “serialNumber” will be used as the device name). deviceTypeJsonExpression ${sensorType} Simple JSON expression, is used for looking up device type in the incoming message (parameter “sensorType” will be used as the device type). timeout 60000 Timeout for triggering “Device Disconnected” event attributes:   This subsection contains parameters of the incoming message, to be interpreted as attributes for the device. … type string Type of incoming data for a current attribute. … key model Attribute name, to be sent to ThingsBoard instance. … value ${sensorModel} Simple JSON expression, is used for looking up value in the incoming message, to be sent to ThingsBoard instance as the value of key parameter.       … type string Type of incoming data for a current attribute. … key ${sensorModel} Simple JSON expression, is used for looking up value in the incoming message, to be used as attribute name. … value on Attribute value, to be sent to ThingsBoard instance. timeseries:   This subsection contains parameters of the incoming message, to be interpreted as telemetry for the device. … type integer Type of incoming data for a current telemetry. … key temperature Telemetry name, to be sent to ThingsBoard instance. … value ${temp} Simple JSON expression, is used for looking up value in the incoming message, to be sent to ThingsBoard instance as the value of key parameter.       … type integer Type of incoming data for a current telemetry. … key humidity Telemetry name, to be sent to ThingsBoard instance. … value ${hum} Simple JSON expression, is used for looking up value in the incoming message, to be sent to ThingsBoard instance as the value of key parameter. Parameters in attributes and telemetry section may differ from those presented above, but will follow the same structure. Note: The device profile is set when the device is created. Changing the device profile using a Gateway is not supported. Mapping subsection for Example 1 will look like: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 { "topicFilter": "sensor/data", "converter": { "type": "json", "deviceNameJsonExpression": "${serialNumber}", "deviceTypeJsonExpression": "${sensorType}", "timeout": 60000, "attributes": [ { "type": "string", "key": "model", "value": "${sensorModel}" }, { "type": "string", "key": "${sensorModel}", "value": "on" } ], "timeseries": [ { "type": "integer", "key": "temperature", "value": "${temp}" }, { "type": "integer", "key": "humidity", "value": "${hum}" } ] } } Mapping for Example 2 will look like: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 { "topicFilter": "sensor/+/data", "converter": { "type": "json", "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/data)", "deviceTypeTopicExpression": "Thermometer", "timeout": 60000, "attributes": [ { "type": "string", "key": "model", "value": "${sensorModel}" } ], "timeseries": [ { "type": "integer", "key": "temperature", "value": "${temp}" }, { "type": "integer", "key": "humidity", "value": "${hum}" } ] } }

Bytes converter is the default converter, it looks for ‘deviceName’, ‘deviceType’, attributes and telemetry in the incoming message from the broker, following the rules described in this subsection: Parameter Default value Description type bytes Provides information to connector that default converter is to be used for converting data from topic. deviceNameExpression [0:4] The expression that is used to find device name in the incoming message. deviceTypeExpression [1:3] The expression that is used to find device type in the incoming message. timeout 60000 Timeout for triggering “Device Disconnected” event. attributes:   This subsection contains parameters of the incoming message, to be interpreted as attributes for the device. … type raw Type of incoming data for a current attribute. … key temp Attribute name, to be sent to ThingsBoard instance. … value [:] Final view of data that will be sent to ThingsBoard, [:] - will replace to device data using python slice rules. timeseries:   This subsection contains parameters of the incoming message, to be interpreted as telemetry for the device. … type raw Type of incoming data for a current telemetry. … key rawData Telemetry name, to be sent to ThingsBoard instance. … value [4:] Final view of data that will be sent to ThingsBoard, [:] - will replace to device data using python slice rules. Parameters in attributes and telemetry section may differ from those presented above, but will have the same structure. Note: The device profile is set when the device is created. Changing the device profile using a Gateway is not supported. Mapping subsection will look like: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 { "topicFilter": "sensor/raw_data", "converter": { "type": "bytes", "deviceNameExpression": "[0:4]", "deviceTypeExpression": "default", "timeout": 60000, "attributes": [ { "type": "raw", "key": "rawData", "value": "[:]" } ], "timeseries": [ { "type": "raw", "key": "temp", "value": "[4:]" } ] } }

A custom converter is converter written for some device: Parameter Default value Description type custom Provides information to connector that a custom converter will be used for converting data from the topic. extension CustomMqttUplinkConverter Name of custom converter class. extension-config   This subsection is a configuration for the custom converter. In default example it contains the number of bytes and keys for telemetry. temperatureBytes 2 In the default example, first 2 bytes from received message will be interpreted as temperature key of telemetry (Substring “Bytes” will remove if exists). humidityBytes 2 In the default example, the second and third byte from received message will be interpreted as humidity key of telemetry (Substring “Bytes” will remove if exists). batteryLevelBytes 1 In the default example, the fifth byte from received message will be interpreted as batteryLevel key of telemetry (Substring “Bytes” will removed if exists). All parameters from this subsection and topic will be transferred as dictionary during initialization to the converter object. Converter subsection in the configuration will look like: 1 2 3 4 5 6 7 8 9 "converter": { "type": "custom", "extension": "CustomMqttUplinkConverter", "extension-config": { "temperatureBytes" : 2, "humidityBytes" : 2, "batteryLevelBytes" : 1 } }

Note: You can specify multiple mapping objects inside the array.

Also, you can combine values from MQTT message in attributes, telemetry and serverSideRpc section, for example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
{
  {
      "topicFilter": "sensor/data",
      "converter": {
        "type": "json",
        "deviceNameJsonExpression": "${serialNumber}",
        "deviceTypeJsonExpression": "${sensorType}",
        "timeout": 60000,
        "attributes": [],
        "timeseries": [
          {
            "type": "integer",
            "key": "temperature",
            "value": "${temp}"
          },
          {
            "type": "integer",
            "key": "humidity",
            "value": "${hum}"
          },
          {
            "type": "string",
            "key": "combine",
            "value": "${hum}:${temp}"
          }
        ]
      }
    }
}

Mapping process subscribes to the MQTT topics using topicFilter parameter of the mapping object. Each message that is published to this topic by other devices or applications is analyzed to extract device name, type and data (attributes and/or timeseries values). By default, gateway uses Json converter, but it is possible to provide custom converter. See examples in the source code.

Connector won’t pass the ‘None’ value from the converter

Now let’s review an example of sending data from “SN-001” thermometer device.

Let’s assume MQTT broker is installed locally on your server.

Use terminal to simulate sending message from the device to the MQTT broker:

1
mosquitto_pub -h 127.0.0.1 -p 1883 -t "sensor/data" -m '{"serialNumber": "SN-001", "sensorType": "Thermometer", "sensorModel": "T1000", "temp": 42, "hum": 58}'

The device will be created and displayed in ThingsBoard based on the passed parameters.

Section “connectRequests”

ThingsBoard allows sending RPC commands and notifications about device attribute updates to the device. But in order to send them, the platform needs to know if the target device is connected and what gateway or session is used to connect the device at the moment. If your device is constantly sending telemetry data - ThingsBoard already knows how to push notifications. If your device just connects to MQTT broker and waits for commands/updates, you need to send a message to the Gateway and inform that device is connected to the broker.

1. Name in a message from broker:

2. Name in topic address:

This section in configuration looks like:

1
2
3
4
5
6
7
8
9
10
  "connectRequests": [
    {
      "topicFilter": "sensor/connect",
      "deviceNameJsonExpression": "${serialNumber}"
    },
    {
      "topicFilter": "sensor/+/connect",
      "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/connect)"
    }
  ]

In this case the following messages are valid:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/connect" -m '{"serialNumber":"SN-001"}'

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/SN-001/connect" -m ''

Now let’s review an example.

Use a terminal to simulate sending a message from the device to the MQTT broker:

1
mosquitto_pub -h 127.0.0.1 -p 1883 -t "sensor/connect" -m '{"serialNumber": "SN-001"}'

Your ThingsBoard instance will get information from the broker about last connecting time of the device. You can see this information under the “Server attributes” scope in the “Attributes” tab.

Section “disconnectRequest”

This configuration section is optional.
Configuration, provided in this section will be used to get information from the broker about disconnecting device.
If your device just disconnects from MQTT broker and waits for commands/updates, you need to send a message to the Gateway and inform it that device is disconnected from the broker.

1. Name in a message from broker:

2. Name in topic address:

This section in configuration file looks like:

1
2
3
4
5
6
7
8
9
10
  "disconnectRequests": [
    {
      "topicFilter": "sensor/disconnect",
      "deviceNameJsonExpression": "${serialNumber}"
    },
    {
      "topicFilter": "sensor/+/disconnect",
      "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/disconnect)"
    }
  ]

In this case the following messages are valid:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/disconnect" -m '{"serialNumber":"SN-001"}'

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/SN-001/disconnect" -m ''

Now let’s review an example.

Use a terminal to simulate sending a message from the device to MQTT broker:

1
mosquitto_pub -h 127.0.0.1 -p 1883 -t "sensor/disconnect" -m '{"serialNumber": "SN-001"}'

Your ThingsBoard instance will get information from the broker about last disconnecting time of the device. You can see this information under the “Server attributes” scope in the “Attributes” tab.

Section “attributeRequests”

This configuration section is optional.

In order to request client-side or shared device attributes to ThingsBoard server node, Gateway allows sending attribute requests.

This section in configuration file looks like:

1
2
3
4
5
6
7
8
9
10
"attributeRequests": [
  {
    "retain": false,
    "topicFilter": "v1/devices/me/attributes/request",
    "deviceNameJsonExpression": "${serialNumber}",
    "attributeNameJsonExpression": "${versionAttribute}",
    "topicExpression": "devices/${deviceName}/attrs",
    "valueExpression": "${attributeKey}: ${attributeValue}"
  }
]

Also, you can request multiple attributes at once. Simply add one more JSON-path to attributeNameExpression parameter. For example, we want to request two shared attributes in one request, our config will look like:

1
2
3
4
5
6
7
8
9
10
"attributeRequests": [
  {
    "retain": false,
    "topicFilter": "v1/devices/me/attributes/request",
    "deviceNameJsonExpression": "${serialNumber}",
    "attributeNameJsonExpression": "${versionAttribute}, ${pduAttribute}",
    "topicExpression": "devices/${deviceName}/attrs",
    "valueExpression": "${attributeKey}: ${attributeValue}"
  }
]

Section “attributeUpdates”

This configuration section is optional.
ThingsBoard allows to provision device attributes and fetch some of them from the device application. You can treat this as a remote configuration for devices. Your devices are able to request shared attributes from ThingsBoard. See user guide for more details.

The “attributeUpdates” configuration allows configuring the format of the corresponding attribute request and response messages.

This section in configuration file looks like:

1
2
3
4
5
6
7
8
9
  "attributeUpdates": [
    {
      "retain": false,
      "deviceNameFilter": ".*",
      "attributeFilter": "firmwareVersion",
      "topicExpression": "sensor/${deviceName}/${attributeKey}",
      "valueExpression": "{\"${attributeKey}\":\"${attributeValue}\"}"
    }
  ]

Let’s look at an example.

Run the command below to start the mosquitto_sub client, subscribing to the topic “sensor/SN-001/firmwareVersion” of the local broker. Start waiting for new messages from ThingsBoard server to broker.

1
mosquitto_sub -h 127.0.0.1 -p 1883 -t sensor/SN-001/firmwareVersion

Update device attribute value on the ThingsBoard server following these steps:

• Open the “Devices” page;
• Click on your device and navigate to the “Attributes” tab;
• Choose “Shared attributes” scope and click on the “pencil” icon next to “firmwareVersion” attribute.

• Change firmware version value from “1.1” to “1.2”. Then click “Update” button.

The firmware version has been updated to “1.2”.

Broker received new message from the ThingsBoard server about updating attribute “FirmwareVersion” to “1.2”.

Server side RPC commands

ThingsBoard allows sending RPC commands to the device that is connected to ThingsBoard directly or via Gateway.

Configuration, provided in this section is used for sending RPC requests from ThingsBoard to device.

There are 2 options for RPC request:

1. With a response – If the configuration includes a responseTopicExpression, the gateway will attempt to subscribe to it and wait for a response.
2. Without a response – If the configuration does not include a responseTopicExpression, the gateway will simply send the message without waiting for a response.

This section in configuration file looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
  "serverSideRpc": [
    {
      "deviceNameFilter": ".*",
      "methodFilter": "echo",
      "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "responseTopicExpression": "sensor/${deviceName}/response/${methodName}/${requestId}",
      "responseTimeout": 10000,
      "valueExpression": "${params}"
    },
    {
      "deviceNameFilter": ".*",
      "methodFilter": "no-reply",
      "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "valueExpression": "${params.hum}::${params.temp}"
    }
  ]

You can use deviceNameFilter and methodFilter to apply different mapping rules for various devices/methods. Once Gateway receives RPC request from the server to the device, it will publish the corresponding message based on requestTopicExpression and valueExpression. In case you expect a reply to the request from the device, you should also specify responseTopicExpression and responseTimeout. The Gateway will subscribe to the “response” topic and wait for a device reply until “responseTimeout” is reached (in milliseconds).

Here is an example of an RPC request (rpc-request.json) that needs to be sent from the server:

1
2
3
4
5
6
{
  "method": "echo",
  "params": {
    "message": "Hello!"
  }
}

Also, every telemetry and attribute parameter has built-in GET and SET RPC methods out of the box, so you don’t need to configure it manually. To use them, make sure you set all the required parameters (in the case of MQTT Connector, these are the following: requestTopicExpression, responseTopicExpression, responseTimeout, valueExpression). See the guide.

Next steps

Explore guides related to main ThingsBoard features:

• Data Visualization - how to visualize collected data.
• Device attributes - how to use device attributes.
• Telemetry data collection - how to collect telemetry data.
• Using RPC capabilities - how to send commands to/from devices.
• Rule Engine - how to use rule engine to analyze data from devices.