MQTT Connector Configuration

This configuration section contains settings of the connection to broker, such as:

• Host - MQTT broker hostname or ip address;
• Port - MQTT port on the broker;
• MQTT version - MQTT protocol version;
• Client ID - this is the client ID. It must be unique for each session;
• Security - configuration for client authorization at MQTT Broker (anonymous, basic, or certificates).

This configuration section contains settings of the connection to broker, such as:

Example:

1
2
3
4
5
6
7
8
9
10
"broker": {
    "name": "Default Local Broker",
    "host": "127.0.0.1",
    "port": 1883,
    "version": 5,
    "clientId": "tb_gw_li06e",
    "security": {
      "type": "anonymous"
    }
},

Now select the security type:

Now select the security type:

1
2
3
    "security": {
      "type": "anonymous"
    }

1
2
3
4
5
    "security": {
      "type": "basic",
      "username": "username",
      "password": "password"
    }

1
2
3
4
5
6
"security":{
    "type": "certificates",
    "pathToCACert": "/etc/thingsboard-gateway/ca.pem",
    "pathToPrivateKey": "/etc/thingsboard-gateway/privateKey.pem",
    "pathToClientCert": "/etc/thingsboard-gateway/certificate.pem"
}

For adding new data mapping, click the “plus” icon:

Opened modal window provide the following fields: topic filter, QoS, payload type:

The Topic filter - topic address for subscribing. The Topic filter supports special symbols: ‘#’ and ‘+’, allowing to subscribe to multiple topics.

Also, MQTT connector supports shared subscriptions. To create shared subscription you need to add “$share/” as a prefix for topic filter and shared subscription group name. For example to subscribe to the my-shared-topic in group my-group-name you can set the topic filter to “$share/my-group-name/my-shared-topic”.

MQTT Quality of Service (QoS) is an agreement between the message sender and receiver that defines the level of delivery guarantee for a specific message. (0-At most once, 1-At least once, 2-Exactly once)

The topicFilter supports special symbols: ‘#’ and ‘+’, allowing to subscribe to multiple topics.

Also, MQTT connector supports shared subscriptions. To create shared subscription you need to add “$share/” as a prefix for topic filter and shared subscription group name. For example to subscribe to the my-shared-topic in group my-group-name you can set the topic filter to “$share/my-group-name/my-shared-topic”.

Now select the payload type:

JSONRecommended if json will be received in response

BytesRecommended if bytes will be received in response

CustomRecommended if bytes or anything else will be received in response

Converter for this payload type processes MQTT messages in JSON format. It uses JSON Path expressions to extract vital details such as device names, device profile names, attributes, and time series from the message. And regular expressions to get device details from topics. Select “JSON” payload type, and fill in the fields in the “Device” section. Then click “pencil” icon of the “Attributes” section to add new attribute key; In the opened window click “Add attribute”. Enter the key name, select the type and enter the value of the key. Click “Apply”; Now click on the “pencil” icon of the “Time series” section to add new time series key; In the opened window click “Add time series”. Enter the key name, select the type and enter the value of the key. Click “Apply”; Click “Add”.

Converter for this payload type designed for binary MQTT payloads, this converter directly interprets binary data to retrieve device names and device profile names, along with attributes and time series, using specific byte positions for data extraction. Select “Bytes” payload type, and fill in the fields in the “Device” section. Then click “pencil” icon of the “Attributes” section to add new attribute key; In the opened window click “Add attribute”. Enter the key name, select the type and enter the value of the key. Click “Apply”; Now click on the “pencil” icon of the “Time series” section to add new time series key; In the opened window click “Add time series”. Enter the key name, select the type and enter the value of the key. Click “Apply”; Click “Add”.

This option allows you to use a custom converter for specific data tasks. You need to add your custom converter to the extension folder and enter its class name in the UI settings. Any keys you provide will be sent as configuration to your custom converter. Select “Custom” payload type, and fill in the fields in the “Device” section. Then click “pencil” icon of the “Keys” section to add new key; In the opened window click “Add key”. Enter the key name and its value. Click “Apply”; Click “Add”.

Now select the payload type:

JSONRecommended if json will be received in response

BytesRecommended if bytes will be received in response

CustomRecommended if bytes or anything else will be received in response

Json converter is the default converter, it looks for ‘deviceName’, ‘deviceType’, attributes and telemetry in the incoming message from the broker, following the rules described in this subsection: Parameter Default value Description type json Provides information to connector that default converter is to be used for converting data from topic. deviceNameExpression ${serialNumber} Simple JSON expression, is used for looking up device name in the incoming message (parameter “serialNumber” will be used as the device name). deviceProfileExpression ${sensorType} Simple JSON expression, is used for looking up device type in the incoming message (parameter “sensorType” will be used as the device type). timeout 60000 Timeout for triggering “Device Disconnected” event attributes:   This subsection contains parameters of the incoming message, to be interpreted as attributes for the device. … type string Type of incoming data for a current attribute. … key model Attribute name, to be sent to ThingsBoard instance. … value ${sensorModel} Simple JSON expression, is used for looking up value in the incoming message, to be sent to ThingsBoard instance as the value of key parameter.       … type string Type of incoming data for a current attribute. … key ${sensorModel} Simple JSON expression, is used for looking up value in the incoming message, to be used as attribute name. … value on Attribute value, to be sent to ThingsBoard instance. timeseries:   This subsection contains parameters of the incoming message, to be interpreted as telemetry for the device. … type integer Type of incoming data for a current telemetry. … key temperature Telemetry name, to be sent to ThingsBoard instance. … value ${temp} Simple JSON expression, is used for looking up value in the incoming message, to be sent to ThingsBoard instance as the value of key parameter.       … type integer Type of incoming data for a current telemetry. … key humidity Telemetry name, to be sent to ThingsBoard instance. … value ${hum} Simple JSON expression, is used for looking up value in the incoming message, to be sent to ThingsBoard instance as the value of key parameter. Parameters in attributes and telemetry section may differ from those presented above, but will follow the same structure. Note: The device profile is set when the device is created. Changing the device profile using a Gateway is not supported. Mapping subsection for Example 1 will look like: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 { "topicFilter": "sensor/data", "subscriptionQos": 1, "converter": { "type": "json", "deviceInfo": { "deviceNameExpressionSource": "message", "deviceNameExpression": "${serialNumber}", "deviceProfileExpressionSource": "message", "deviceProfileExpression": "${sensorType}" }, "sendDataOnlyOnChange": false, "timeout": 60000, "attributes": [ { "type": "string", "key": "model", "value": "${sensorModel}" }, { "type": "string", "key": "${sensorModel}", "value": "on" } ], "timeseries": [ { "type": "string", "key": "temperature", "value": "${temp}" }, { "type": "double", "key": "humidity", "value": "${hum}" }, { "type": "string", "key": "combine", "value": "${hum}:${temp}" } ] } } Mapping for Example 2 will look like: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 "topicFilter": "sensor/+/data", "subscriptionQos": 1, "converter": { "type": "json", "deviceInfo": { "deviceNameExpressionSource": "topic", "deviceNameExpression": "(?<=sensor/)(.*?)(?=/data)", "deviceProfileExpressionSource": "constant", "deviceProfileExpression": "Thermometer" }, "sendDataOnlyOnChange": false, "timeout": 60000, "attributes": [ { "type": "string", "key": "model", "value": "${sensorModel}" } ], "timeseries": [ { "type": "double", "key": "temperature", "value": "${temp}" }, { "type": "string", "key": "humidity", "value": "${hum}" } ] } }

Bytes converter is the default converter, it looks for ‘deviceName’, ‘deviceType’, attributes and telemetry in the incoming message from the broker, following the rules described in this subsection: Parameter Default value Description type bytes Provides information to connector that default converter is to be used for converting data from topic. deviceNameExpression [0:4] The expression that is used to find device name in the incoming message. deviceProfileExpression [1:3] The expression that is used to find device type in the incoming message. timeout 60000 Timeout for triggering “Device Disconnected” event. attributes:   This subsection contains parameters of the incoming message, to be interpreted as attributes for the device. … type raw Type of incoming data for a current attribute. … key temp Attribute name, to be sent to ThingsBoard instance. … value [:] Final view of data that will be sent to ThingsBoard, [:] - will replace to device data using python slice rules. timeseries:   This subsection contains parameters of the incoming message, to be interpreted as telemetry for the device. … type raw Type of incoming data for a current telemetry. … key rawData Telemetry name, to be sent to ThingsBoard instance. … value [4:] Final view of data that will be sent to ThingsBoard, [:] - will replace to device data using python slice rules. Parameters in attributes and telemetry section may differ from those presented above, but will have the same structure. Note: The device profile is set when the device is created. Changing the device profile using a Gateway is not supported. Mapping subsection will look like: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 { "topicFilter": "sensor/raw_data", "subscriptionQos": 1, "converter": { "type": "bytes", "deviceInfo": { "deviceNameExpressionSource": "message", "deviceNameExpression": "[0:4]", "deviceProfileExpressionSource": "constant", "deviceProfileExpression": "default" }, "sendDataOnlyOnChange": false, "timeout": 60000, "attributes": [ { "type": "raw", "key": "rawData", "value": "[:]" } ], "timeseries": [ { "type": "raw", "key": "temp", "value": "[4:]" } ] } }

A custom converter is converter written for some device: Parameter Default value Description type custom Provides information to connector that a custom converter will be used for converting data from the topic. extension CustomMqttUplinkConverter Name of custom converter class. extension-config   This subsection is a configuration for the custom converter. In default example it contains the number of bytes and keys for telemetry. temperatureBytes 2 In the default example, first 2 bytes from received message will be interpreted as temperature key of telemetry (Substring “Bytes” will remove if exists). humidityBytes 2 In the default example, the second and third byte from received message will be interpreted as humidity key of telemetry (Substring “Bytes” will remove if exists). batteryLevelBytes 1 In the default example, the fifth byte from received message will be interpreted as batteryLevel key of telemetry (Substring “Bytes” will removed if exists). All parameters from this subsection and topic will be transferred as dictionary during initialization to the converter object. Converter subsection in the configuration will look like: 1 2 3 4 5 6 7 8 9 10 11 12 "topicFilter": "custom/sensors/+", "subscriptionQos": 1, "converter": { "type": "custom", "extension": "CustomMqttUplinkConverter", "cached": true, "extensionConfig": { "temperature": 2, "humidity": 2, "batteryLevel": 1 } }

Note: You can specify multiple mapping objects inside the array.

Also, you can combine values from MQTT message in attributes, telemetry and serverSideRpc section, for example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{
    {
        "topicFilter": "sensor/data",
        "subscriptionQos": 1,
        "converter": {
            "type": "json",
            "deviceInfo": {
            "deviceNameExpressionSource": "message",
            "deviceNameExpression": "${serialNumber}",
            "deviceProfileExpressionSource": "message",
            "deviceProfileExpression": "${sensorType}"
            "timeout": 60000,
            "attributes": [],
            "timeseries": [
                {
                    "type": "integer",
                    "key": "temperature",
                    "value": "${temp}"
                },
                {
                    "type": "integer",
                    "key": "humidity",
                    "value": "${hum}"
                },
                {
                    "type": "string",
                    "key": "combine",
                    "value": "${hum}:${temp}"
                }
            ]
        }
    }
}

Mapping process subscribes to the MQTT topics using topicFilter parameter of the mapping object. Each message that is published to this topic by other devices or applications is analyzed to extract device name, type and data (attributes and/or timeseries values). By default, gateway uses Json converter, but it is possible to provide custom converter. See examples in the source code.

Connector won’t pass the ‘None’ value from the converter

For adding new requests mapping, click “plus” icon:

Modal window will open. Select request type, set a topic filter, fill in the entities fields of the “Device” section. Then, click “Add”.

To adding new requests mapping, navigate to the “Requests mapping” tab and click the “plus” icon. In the open modal window, select the “Connect request” type, set a topic filter, and fill in the “Name” and “Profile name” fields of the “Device” section. Then, click “Add”.

Device name from the message body:

In this case the following messages are valid:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/connect" -m '{"serialNumber":"SN-001"}'

Device name from the topic:

In this case the following messages are valid:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/SN-001/connect" -m ''

This section in configuration looks like:

1
2
3
4
5
6
7
8
9
10
11
"connectRequests": [
  {
    "topicFilter": "sensor/connect",
    "deviceInfo": {
      "deviceNameExpressionSource": "message",
      "deviceNameExpression": "${serialNumber}",
      "deviceProfileExpressionSource": "constant",
      "deviceProfileExpression": "Thermometer"
    }
  }        
]

Device name from the message body:

In this case the following messages are valid:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/connect" -m '{"serialNumber":"SN-001"}'

Device name from the topic:

In this case the following messages are valid:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/SN-001/connect" -m ''

To adding new requests mapping, navigate to the “Requests mapping” tab and click the “plus” icon. In the open modal window, select the “Disconnect request” type, set a topic filter, and fill in the “Name” field of the “Device” section. Then, click “Add”.

Name in a message from broker:

In this case the following messages are valid:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/disconnect" -m '{"serialNumber":"SN-001"}'

Name in topic address:

In this case the following messages are valid:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/SN-001/disconnect" -m ''

This section in configuration file looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
"disconnectRequests": [
  {
    "topicFilter": "sensor/disconnect",
    "deviceInfo": {
      "deviceNameExpressionSource": "message",
      "deviceNameExpression": "${serialNumber}"
  }
},
  {
    "topicFilter": "sensor/+/disconnect",
    "deviceInfo": {
      "deviceNameExpressionSource": "topic",
      "deviceNameExpression": "(?<=sensor/)(.*?)(?=/connect)"
      }
  }
]

Name in a message from broker:

In this case the following messages are valid:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/disconnect" -m '{"serialNumber":"SN-001"}'

Name in topic address:

In this case the following messages are valid:

1
mosquitto_pub -h YOUR_MQTT_BROKER_HOST -p YOUR_MQTT_BROKER_PORT -t "sensor/SN-001/disconnect" -m ''

To adding new requests mapping, navigate to the “Requests mapping” tab and click the “plus” icon. In the open modal window, select the “Attribute request” type, set a topic filter. Fill in the fields of the “Input request parsing” and “Output request parsing” sections. Then, click “Add”.

This section in configuration file looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
"attributeRequests": [
  {
    "retain": false, 
    "topicFilter": "v1/devices/me/attributes/request", 
    "deviceInfo": {
      "deviceNameExpressionSource": "message",
      "deviceNameExpression": "${serialNumber}"
    },
    "attributeNameExpressionSource": "message",
    "attributeNameExpression": "${versionAttribute}",
    "topicExpression": "devices/${deviceName}/attrs",
    "valueExpression": "${attributeKey}: ${attributeValue}"
  }
]

Also, you can request multiple attributes at once. Simply add one more JSON-path to attributeNameExpression parameter. For example, we want to request two shared attributes in one request, our config will look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
"attributeRequests": [
  {
    "retain": false,
    "topicFilter": "v1/devices/me/attributes/request",
    "deviceInfo": {
      "deviceNameExpressionSource": "message",
      "deviceNameExpression": "${serialNumber}"
    },
    "attributeNameExpressionSource": "message",
    "attributeNameExpression": "${versionAttribute}, ${pduAttribute}",
    "topicExpression": "devices/${deviceName}/attrs",
    "valueExpression": "${attributeKey}: ${attributeValue}"
  }
]

To adding new requests mapping, navigate to the “Requests mapping” tab and click the “plus” icon. In the open modal window, select the “Attribute update” type, and set a device name filter, attribute filter, response value expression, and response topic expression. Then, click “Add”.

This section in configuration file looks like:

1
2
3
4
5
6
7
8
9
  "attributeUpdates": [
    {
      "retain": false,
      "deviceNameFilter": ".*",
      "attributeFilter": "firmwareVersion",
      "topicExpression": "sensor/${deviceName}/${attributeKey}",
      "valueExpression": "{\"${attributeKey}\":\"${attributeValue}\"}"
    }
  ]

There are 2 options for RPC request: With a response – If the configuration includes a responseTopicExpression, the gateway will attempt to subscribe to it and wait for a response.

To adding new requests mapping, navigate to the “Requests mapping” tab and click the “plus” icon. In the open modal window, select the “RPC command” request type, and set a device name filter, method filter, request topic expression, value expression, response topic expression, response topic QoS and response timeout. Then, click “Add”.

Without a response – If the configuration does not include a responseTopicExpression, the gateway will simply send the message without waiting for a response.

To adding new requests mapping, navigate to the “Requests mapping” tab and click the “plus” icon. In the open modal window, select the “RPC command” request type, toggle “Without response” option, and set a device name filter, method filter, and value expression. Then, click “Add”.

You can use device name filter and method filter to apply different mapping rules for various devices/methods. Once Gateway receives RPC request from the server to the device, it will publish the corresponding message based on request topic expression and value expression. In case you expect a reply to the request from the device, you should also specify response topic expression and response timeout. The Gateway will subscribe to the “response” topic and wait for a device reply until “responseTimeout” is reached (in milliseconds).

Here is an example of an RPC request (rpc-request.json) that needs to be sent from the server:

1
2
3
4
5
6
{
  "method": "echo",
  "params": {
    "message": "Hello!"
  }
}

Also, every telemetry and attribute parameter has built-in GET and SET RPC methods out of the box, so you don’t need to configure it manually. To use them, make sure you set all the required parameters (in the case of MQTT Connector, these are the following: request topic expression, response topic expression, response timeout, value expression). See the guide.

This section in configuration file looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
"serverSideRpc": [
   {
      "type": "twoWay",
      "deviceNameFilter": ".*",
      "methodFilter": "echo",
      "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "responseTopicExpression": "sensor/${deviceName}/response/${methodName}/${requestId}",
      "responseTopicQoS": 1,
      "responseTimeout": 10000,
      "valueExpression": "${params}"
   },
   {
      "type": "oneWay",
      "deviceNameFilter": ".*",
      "methodFilter": "no-reply",
      "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "valueExpression": "${params}"
   }
]

There are 2 options for RPC request:

1. With a response – If the configuration includes a responseTopicExpression, the gateway will attempt to subscribe to it and wait for a response.
2. Without a response – If the configuration does not include a responseTopicExpression, the gateway will simply send the message without waiting for a response.

You can use deviceNameFilter and methodFilter to apply different mapping rules for various devices/methods. Once Gateway receives RPC request from the server to the device, it will publish the corresponding message based on requestTopicExpression and valueExpression. In case you expect a reply to the request from the device, you should also specify responseTopicExpression and responseTimeout. The Gateway will subscribe to the “response” topic and wait for a device reply until “responseTimeout” is reached (in milliseconds).

Here is an example of an RPC request (rpc-request.json) that needs to be sent from the server:

1
2
3
4
5
6
{
  "method": "echo",
  "params": {
    "message": "Hello!"
  }
}

Also, every telemetry and attribute parameter has built-in GET and SET RPC methods out of the box, so you don’t need to configure it manually. To use them, make sure you set all the required parameters (in the case of MQTT Connector, these are the following: requestTopicExpression, responseTopicExpression, responseTimeout, valueExpression). See the guide.

Max number of workers is a maximal number of workers thread for converters (amount of workers changes dynamically, depending on load). Recommended amount 50-100.

Max messages queue per worker is a maximal messages count that will be in queue for each converter worker.

• maxNumberOfWorkers is a maximal number of workers thread for converters (amount of workers changes dynamically, depending on load). Recommended amount 50-100;
• maxMessageNumberPerWorker is a maximal messages count that will be in queue for each converter worker.