Working with telemetry data

Data points
Time-series data upload API
Data visualization
Data storage
Data retention
Data durability
Rule engine
Data Query REST API
WebSocket API
- Example

ThingsBoard provides a rich set of features related to time-series data:

Collect data from devices using various protocols and integrations;
Store time series data in SQL (PostgreSQL) or NoSQL (Cassandra or Timescale) databases;
Query the latest time series data values or all data within the specified time range with flexible aggregation;
Subscribe to data updates using WebSockets for visualization or real-time analytics;
Visualize time series data using configurable and highly customizable widgets and dashboards;
Filter and analyze data using flexible Rule Engine;
Generate alarms based on collected data;
Forward data to external systems using External Rule Nodes (e.g. Kafka or RabbitMQ Rule Nodes).

This guide provides an overview of the features listed above, and some useful links to get more details.

Data points

ThingsBoard internally treats time-series data as timestamped key-value pairs. We call single timestamped key-value pair a data point. Flexibility and simplicity of the key-value format allow easy and seamless integration with almost any IoT device on the market. Key is always a string and is basically a data point key name, while the value can be either string, boolean, double, integer or JSON.

The following JSON contains 5 data points: temperature (double), humidity (integer), hvacEnabled (boolean), hvacState (string) and configuration (JSON):

{
 "temperature": 42.2, 
 "humidity": 70,
 "hvacEnabled": true,
 "hvacState": "IDLE",
 "configuration": {
    "someNumber": 42,
    "someArray": [1,2,3],
    "someNestedObject": {"key": "value"}
 }
}

You may notice that the JSON listed above does not have a timestamp information. In such case, ThingsBoard uses current server timestamp. However, you may include timestamp information into the message. See example below:

{
  "ts": 1527863043000,
  "values": {
    "temperature": 42.2,
    "humidity": 70
  }
}

Time-series data upload API

You may use built-in transport protocol implementations:

Most of the protocols above support JSON, Protobuf or own data format. For other protocols, please review “How to connect your device?” guide.

Data visualization

We assume you have already pushed time-series data to ThingsBoard. Now you may use it in your dashboards. We recommend dashboards overview to get started. Once you are familiar how to create dashboards and configure data sources, you may use widgets to visualize either latest values or real-time changes and historical values. Good examples of widgets that visualize latest values are digital and analog gauges, or cards. Charts are used to visualize historical and real-time values and maps to visualize movement of devices and assets.

You may also use input widgets to allow dashboard users to input new time-series values using the dashboards.

Data storage

System administrator is able to configure ThingsBoard to store time-series data in SQL (PostgreSQL) or NoSQL (Cassandra or Timescale) databases. Using SQL storage is recommended for small environments with less than 5000 data points per second. Storing data in Cassandra makes sense when you have either high throughput or high availability requirements for your solution.

See SQL vs NoSQL vs Hybrid for more information.

Data retention

Data retention policy and configuration depends on the chosen storage.

Cassandra supports time-to-live(TTL) parameter for each inserted row. That is why, you may configure default TTL parameter on a system level, using ‘TS_KV_TTL’ environment variable. You may overwrite the default value in the “Save Timeseries” rule node or using “TTL” metadata field of your message. This allows you to optimize storage consumption.

PostgreSQL and Timescale does not support time-to-live(TTL) parameter for each inserted row. That is why, you may only configure periodic time-series data cleanup routine using ‘SQL_TTL_*’ environment variables.

Data durability

The device that sends message with time-series data to ThingsBoard will receive confirmation once the message is successfully stored into the Rule Engine Queue that is configured for particular device profile.

As a tenant administrator, you may configure processing strategy for the queue. You may configure the queue either to reprocess or ignore the failures of the message processing. This allows granular control on the level of durability for the time-series data and all other messages processed by the rule engine.

Rule engine

The Rule Engine is responsible for processing all sorts of incoming data and event. You may find most popular scenarios of using attributes within rule engine below:

Generate alarms based on the logical expressions against time-series values

Use alarm rules to configure most common alarm conditions via UI or use filter nodes to configure more specific use cases via custom JS functions.

Modify incoming time-series data before they are stored in the database

Use message type switch rule node to filter messages that contain “Post telemetry” request. Then, use transformation rule nodes to modify a particular message.

Calculate delta between previous and current time-series value

Use calculate delta rule node to calculate power, water and other consumption based on smart-meter readings.

Fetch previous time-series values to analyze incoming telemetry from device

Use originator telemetry rule node to enrich incoming time-series data message with previous time-series data of the device.

Fetch attribute values to analyze incoming telemetry from device

Use enrichment rule nodes to enrich incoming telemetry message with attributes of the device, related asset, customer or tenant. This is extremely powerful technique that allows to modify processing logic and parameters based on settings stored in the attributes.

Use analytics rule nodes to aggregate data for related assets

Use analytics rule nodes to aggregate data from multiple devices or assets.

Useful to calculate total water consumption for the building/district based on data from multiple water meters.

Data Query REST API

ThingsBoard provides following REST API to fetch entity data:

Get time-series data keys for specific entity

You can fetch list of all time-series data keys for particular entity type and entity id using GET request to the following URL

http(s)://host:port/api/plugins/telemetry/{entityType}/{entityId}/keys/timeseries

get-telemetry-keys.sh
get-telemetry-keys-result.json

curl -v -X GET http://localhost:8080/api/plugins/telemetry/DEVICE/ac8e6020-ae99-11e6-b9bd-2b15845ada4e/keys/timeseries \
--header "Content-Type:application/json" \
--header "X-Authorization: $JWT_TOKEN"

["gas","temperature"]

Supported entity types are: TENANT, CUSTOMER, USER, DASHBOARD, ASSET, DEVICE, ALARM, ENTITY_VIEW

Get latest time-series data values for specific entity

You can fetch list of latest values for particular entity type and entity id using GET request to the following URL

http(s)://host:port/api/plugins/telemetry/{entityType}/{entityId}/values/timeseries?keys=key1,key2,key3

get-latest-telemetry-values.sh
get-latest-telemetry-values-result.json

curl -v -X GET http://localhost:8080/api/plugins/telemetry/DEVICE/ac8e6020-ae99-11e6-b9bd-2b15845ada4e/values/timeseries?keys=gas,temperature \
--header "Content-Type:application/json" \
--header "X-Authorization: $JWT_TOKEN"

{
  "gas": [
    {
      "ts": 1479735870786,
      "value": "1"
    }
  ],
  "temperature": [
    {
      "ts": 1479735870786,
      "value": "3"
    }
  ]
}

Supported entity types are: TENANT, CUSTOMER, USER, DASHBOARD, ASSET, DEVICE, ALARM, ENTITY_VIEW

Get historical time-series data values for specific entity

You can also fetch list of historical values for particular entity type and entity id using GET request to the following URL

http(s)://host:port/api/plugins/telemetry/{entityType}/{entityId}/values/timeseries?keys=key1,key2,key3&startTs=1479735870785&endTs=1479735871858&interval=60000&limit=100&agg=AVG

The supported parameters are described below:

keys - comma-separated list of telemetry keys to fetch.
startTs - Unix timestamp that identifies the start of the interval in milliseconds.
endTs - Unix timestamp that identifies the end of the interval in milliseconds.
interval - the aggregation interval, in milliseconds.
agg - the aggregation function. One of MIN, MAX, AVG, SUM, COUNT, NONE.
limit - the max amount of data points to return or intervals to process.

ThingsBoard will use startTs, endTs, and interval to identify aggregation partitions or sub-queries and execute asynchronous queries to DB that leverage built-in aggregation functions.

get-telemetry-values.sh
get-telemetry-values-result.json

curl -v -X GET "http://localhost:8080/api/plugins/telemetry/DEVICE/ac8e6020-ae99-11e6-b9bd-2b15845ada4e/values/timeseries?keys=gas,temperature&startTs=1479735870785&endTs=1479735871858&interval=60000&limit=100&agg=AVG" \
--header "Content-Type:application/json" \
--header "X-Authorization: $JWT_TOKEN"

{
  "gas": [
    {
      "ts": 1479735870786,
      "value": "1"
    },
    {
      "ts": 1479735871857,
      "value": "2"
    }
  ],
  "temperature": [
    {
      "ts": 1479735870786,
      "value": "3"
    },
    {
      "ts": 1479735871857,
      "value": "4"
    }
  ]
}

Supported entity types are: TENANT, CUSTOMER, USER, DASHBOARD, ASSET, DEVICE, ALARM, ENTITY_VIEW

WebSocket API

WebSockets are actively used by Thingsboard Web UI. WebSocket API duplicates REST API functionality and provides the ability to subscribe to device data changes. You can open a WebSocket connection to a telemetry service using the following URL

ws(s)://host:port/api/ws

Once opened, you need to authenticate the session within 10 seconds with auth command:

{
  "authCmd": {
    "cmdId": 0,
    "token": "$JWT_TOKEN"
  }
}

Then you can send subscription commands and receive subscription updates:

where

cmdId - unique command id (within corresponding WebSocket connection)
entityType - unique entity type. Supported entity types are: TENANT, CUSTOMER, USER, DASHBOARD, ASSET, DEVICE, ALARM
entityId - unique entity identifier
keys - a comma-separated list of data keys
timeWindow - fetch interval for time series subscriptions, in milliseconds. Data will be fetch within following interval [now()-timeWindow, now()]
startTs - start time of fetch interval for historical data query, in milliseconds.
endTs - end time of fetch interval for historical data query, in milliseconds.

Example

Change values of the following variables :

token - to the JWT token which you can get using the following link.
entityId - to your device id.

In case of live-demo server :

replace host:port with demo.thingsboard.io and choose secure connection - wss://

In case of local installation :

replace host:port with 127.0.0.1:8080 and choose ws://

web-socket.html

<!DOCTYPE HTML>
<html>
<head>

    <script type="text/javascript">
        function WebSocketAPIExample() {
            var token = "YOUR_JWT_TOKEN";
            var entityId = "YOUR_DEVICE_ID";
            var webSocket = new WebSocket("ws(s)://host:port/api/ws");

            if (entityId === "YOUR_DEVICE_ID") {
                alert("Invalid device id!");
                webSocket.close();
            }

            if (token === "YOUR_JWT_TOKEN") {
                alert("Invalid JWT token!");
                webSocket.close();
            }

            webSocket.onopen = function () {
                var object = {
                    authCmd: {
                        cmdId: 0,
                        token: token
                    },
                    cmds: [
                        {
                            entityType: "DEVICE",
                            entityId: entityId,
                            scope: "LATEST_TELEMETRY",
                            cmdId: 10,
                            type: "TIMESERIES"
                        }
                    ]
                };
                var data = JSON.stringify(object);
                webSocket.send(data);
                alert("Message is sent: " + data);
            };

            webSocket.onmessage = function (event) {
                var received_msg = event.data;
                alert("Message is received: " + received_msg);
            };

            webSocket.onclose = function (event) {
                alert("Connection is closed!");
            };
        }
    </script>

</head>
<body>

<div id="sse">
    <a href="javascript:WebSocketAPIExample()">Run WebSocket</a>
</div>

</body>
</html>