- Prerequisites
- Step 1. Clone ThingsBoard CE K8S scripts repository
- Step 2. Define environment variables
- Step 3. Configure and create AKS cluster
- Step 4. Update the context of kubectl
- Step 5. Provision Databases
- Step 6. Azure Cache for Redis
- Step 7. Installation
- Step 8. Starting
- Step 9. Configure Load Balancers
- Step 10. Using
- Upgrading
- Next steps
This guide will help you to setup ThingsBoard in microservices mode in Azure AKS.
Prerequisites
Install and configure tools
To deploy ThingsBoard on EKS cluster you’ll need to install kubectl
,
az
tools.
After installation is done you need to login in cli using az login command.
1
az login
Step 1. Clone ThingsBoard CE K8S scripts repository
1
2
git clone -b release-3.8.1 https://github.com/thingsboard/thingsboard-ce-k8s.git
cd thingsboard-ce-k8s/azure/microservices
Step 2. Define environment variables
Define environment variables that you will use in various commands later in this guide.
We assume you are using Linux. Execute the following command:
1
2
3
4
5
6
7
8
export AKS_RESOURCE_GROUP=ThingsBoardResources
export AKS_LOCATION=eastus
export AKS_GATEWAY=tb-gateway
export TB_CLUSTER_NAME=tb-cluster
export TB_DATABASE_NAME=tb-db
export TB_REDIS_NAME=tb-redis
echo "You variables ready to create resource group $AKS_RESOURCE_GROUP in location $AKS_LOCATION
and cluster in it $TB_CLUSTER_NAME with database $TB_DATABASE_NAME"
where:
- myResourceGroup - a logical group in which Azure resources are deployed and managed. We will refer to it later in this guide using AKS_RESOURCE_GROUP ;
- eastus - is location where you want to create resource group. We will refer to it later in this guide using AKS_LOCATION; You can see all locations list executing az account list-locations;
- tb-gateway - the name of Azure application gateway;
- tb-cluster - cluster name. We will refer to it later in this guide using TB_CLUSTER_NAME ;
- tb-db is the name of your database server. You may input a different name. We will refer to it later in this guide using TB_DATABASE_NAME;
Step 3. Configure and create AKS cluster
Before creation AKS cluster we need to create Azure Resource Group, we will use Azure CLI for this
1
az group create --name $AKS_RESOURCE_GROUP --location $AKS_LOCATION
To see more info about az group
please you can use this.
After we created Resource group we can create AKS cluster in it using:
1
2
3
4
5
6
7
8
az aks create --resource-group $AKS_RESOURCE_GROUP \
--name $TB_CLUSTER_NAME \
--generate-ssh-keys \
--enable-addons ingress-appgw \
--appgw-name $AKS_GATEWAY \
--appgw-subnet-cidr "10.2.0.0/16" \
--node-vm-size Standard_DS3_v2 \
--node-count 3
az aks create
has two required parameters - name
and resource-group
(we use variable that we set earlier)
and a lot not required parameters(will use defaults values) few of them are:
- node-count - Number of nodes in the Kubernetes node pool. After creating a cluster, you can change the size of its node pool with az aks scale (default value is 3);
- enable-addons - Enable the Kubernetes addons in a comma-separated list (use az aks addon list to get avaliable addons list );
- node-osdisk-size - OS disk type to be used for machines in a given agent pool: Ephemeral or Managed. Defaults to ‘Ephemeral’ when possible in conjunction with VM size and OS disk size. May not be changed for this pool after creation;
- node-vm-size (or -s) - Size of Virtual Machines to create as Kubernetes nodes (default value is Standard_DS2_v2);
- generate-ssh-keys - Generate SSH public and private key files if missing. The keys will be stored in the ~/.ssh directory.
From the command above we add AKS addon for ApplicationGateway. We will use this gateway as Path-Based Load Balancer for Thingsboard Infrastructure
Full list af az aks create
options you can see here
Alternatively, you may use this guide for custom cluster setup.
Step 4. Update the context of kubectl
When claster created we can connect kubectl to it using
1
az aks get-credentials --resource-group $AKS_RESOURCE_GROUP --name $TB_CLUSTER_NAME
Fot validation you can execute command
1
kubectl get nodes
You should see cluster`s nodes list
Step 5. Provision Databases
5.3. Create Azure Database for PostgreSQL servers
You’ll need to set up PostgreSQL on Azure. ThingsBoard will use it as a main database to store devices, dashboards, rule chains and device telemetry.
You may follow this guide, but take into account the following requirements:
- Keep your postgresql password in a safe place. We will refer to it later in this guide using YOUR_RDS_PASSWORD;
- Make sure your Azure Database for PostgreSQL version is latest 16.x;
- Make sure your Azure Database for PostgreSQL instance is accessible from the ThingsBoard cluster;
- Make sure you use “thingsboard” as initial database name;
and recomendations:
- Use “High availability” enabled. It enables a lot of useful settings by default;
Another way by which you can create Azure Database for PostgreSQL is use az tool (don’t forget to replace ‘POSTGRESS_USER’ and ‘POSTGRESS_PASS’ with your username and password):
1
2
3
4
az postgres flexible-server create --location $AKS_LOCATION --resource-group $AKS_RESOURCE_GROUP \
--name $TB_DATABASE_NAME --admin-user POSTGRESS_USER --admin-password POSTGRESS_PASS \
--public-access 0.0.0.0 --storage-size 32 \
--version 12 -d thingsboard
az postgres flexible-server create
has a lot of parameters few of them are:
- location - Location. Values from: az account list-locations;
- resource-group (or -g) - Name of resource group;
- name - Name of the server. The name can contain only lowercase letters, numbers, and the hyphen (-) character. Minimum 3 characters and maximum 63 characters;
- admin-user - Administrator username for the server. Once set, it cannot be changed;
- admin-password - The password of the administrator. Minimum 8 characters and maximum 128 characters. Password must contain characters from three of the following categories: English uppercase letters, English lowercase letters, numbers, and non-alphanumeric characters;
- public-access - Determines the public access. Enter single or range of IP addresses to be included in the allowed list of IPs. IP address ranges must be dash-separated and not contain any spaces. Specifying 0.0.0.0 allows public access from any resources deployed within Azure to access your server. Setting it to “None” sets the server in public access mode but does not create a firewall rule;
- storage-size - The storage capacity of the server. Minimum is 32 GiB and max is 16 TiB;
- version - Server major version.
- high-availability - nable or disable high availability feature. Default value is Disabled. High availability can only be set during flexible server create time (accepted values: Disabled, Enabled. default value: Disabled);
- database-name (or -d) - The name of the database to be created when provisioning the database server.
You can see full parameters list here.
Example of response:
1
2
3
4
5
6
7
8
9
10
11
12
13
{
"connectionString": "postgresql://postgres:postgres@$tb-db.postgres.database.azure.com/postgres?sslmode=require",
"databaseName": "thingsboard",
"firewallName": "AllowAllAzureServicesAndResourcesWithinAzureIps_2021-11-17_15-45-6",
"host": "tb-db.postgres.database.azure.com",
"id": "/subscriptions/daff3288-1d5d-47c7-abf0-bfb7b738a18c/resourceGroups/myResourceGroup/providers/Microsoft.DBforPostgreSQL/flexibleServers/thingsboard",
"location": "East US",
"password": "postgres",
"resourceGroup": "myResourceGroup",
"skuname": "Standard_D2s_v3",
"username": "postgres",
"version": "12"
}
Note the value of host from the command output (tb-db.postgres.database.azure.com in our case). Also note username and password (postgres) from the command.
Edit the database settings file and replace YOUR_AZURE_POSTGRES_ENDPOINT_URL with the host value, YOUR_AZURE_POSTGRES_USER and YOUR_AZURE_POSTGRES_PASSWORD with the correct values:
1
nano tb-node-db-configmap.yml
5.2. Cassandra
Using Cassandra is an optional step. We recommend to use Cassandra if you plan to insert more than 5K data points per second or would like to optimize storage space.
Provision additional node pools
Provision additional node pools that will be hosting Cassandra instances. You may change the machine type. At least 4 vCPUs and 16GB of RAM is recommended.
We will create 3 separate node pools with 1 node per zone. Since we plan to use zonal disks, we don’t want k8s to launch a pod on the node where the corresponding disk is not available. Those zones will have the same node label. We will use this label to target deployment of our stateful set.
So, define 3 node pools in three zones for your location:
1
2
3
az aks nodepool add --resource-group $AKS_RESOURCE_GROUP --cluster-name $TB_CLUSTER_NAME --name tbcassandra1 --node-count 1 --zones 1 --labels role=cassandra
az aks nodepool add --resource-group $AKS_RESOURCE_GROUP --cluster-name $TB_CLUSTER_NAME --name tbcassandra2 --node-count 1 --zones 2 --labels role=cassandra
az aks nodepool add --resource-group $AKS_RESOURCE_GROUP --cluster-name $TB_CLUSTER_NAME --name tbcassandra3 --node-count 1 --zones 3 --labels role=cassandra
Deploy Cassandra stateful set
Create ThingsBoard namespace:
1
2
kubectl apply -f tb-namespace.yml
kubectl config set-context $(kubectl config current-context) --namespace=thingsboard
Deploy Cassandra to new node groups:
1
kubectl apply -f receipts/cassandra.yml
The startup of Cassandra cluster may take few minutes. You may monitor the process using:
1
kubectl get pods
Update DB settings
Edit the ThingsBoard DB settings file and add Cassandra settings:
1
2
3
echo " DATABASE_TS_TYPE: cassandra" >> tb-node-db-configmap.yml
echo " CASSANDRA_URL: cassandra:9042" >> tb-node-db-configmap.yml
echo " CASSANDRA_LOCAL_DATACENTER: dc1" >> tb-node-db-configmap.yml
Check that the settings are updated:
1
cat tb-node-db-configmap.yml | grep DATABASE_TS_TYPE
Expected output:
1
DATABASE_TS_TYPE: cassandra
Create keyspace
Create thingsboard keyspace using following command:
1
2
3
4
5
6
kubectl exec -it cassandra-0 -- bash -c "cqlsh -e \
\"CREATE KEYSPACE IF NOT EXISTS thingsboard \
WITH replication = { \
'class' : 'NetworkTopologyStrategy', \
'dc1' : '3' \
};\""
Step 6. Azure Cache for Redis
You’ll need to set up Azure Cache for Redis. ThingsBoard uses cache to improve performance and avoid frequent DB reads.
You can do this via this guide
or using az cli tools
1
az redis create --name $TB_REDIS_NAME --location $AKS_LOCATION --resource-group $AKS_RESOURCE_GROUP --sku Basic --vm-size c0 --enable-non-ssl-port
As like az postgres az redis create has a lot of options and few of them are required like:
- name (or -n) - Name of the Redis cache;
- resource-group - Name of resource group;
- sku - Type of Redis cache(accepted values: Basic, Premium, Standard;
- vm-size - Size of Redis cache to deploy. Basic and Standard Cache sizes start with C. Premium Cache sizes start with P (accepted values: c0, c1, c2, c3, c4, c5, c6, p1, p2, p3, p4, p5);
- location (or -l) - Location. Values from: az account list-locations.
To see the full list of parameters please see
Example of response:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
{
"accessKeys": null,
"enableNonSslPort": true,
"hostName": "tb-redis.redis.cache.windows.net",
"id": "/subscriptions/daff3288-1d5d-47c7-abf0-bfb7b738a18c/resourceGroups/myResourceGroup/providers/Microsoft.Cache/Redis/tb-redis",
"instances": [
{
"isMaster": false,
"isPrimary": false,
"nonSslPort": 13000,
"shardId": null,
"sslPort": 15000,
"zone": null
}
],
"linkedServers": [],
"location": "East US",
"minimumTlsVersion": null,
"name": "tb-redis",
"port": 6379,
"privateEndpointConnections": null,
"provisioningState": "Creating",
"publicNetworkAccess": "Enabled",
"redisConfiguration": {
"maxclients": "256",
"maxfragmentationmemory-reserved": "12",
"maxmemory-delta": "2",
"maxmemory-reserved": "2"
},
"redisVersion": "4.0.14",
"replicasPerMaster": null,
"replicasPerPrimary": null,
"resourceGroup": "myResourceGroup",
"shardCount": null,
"sku": {
"capacity": 0,
"family": "C",
"name": "Basic"
},
"sslPort": 6380,
"staticIp": null,
"subnetId": null,
"tags": {},
"tenantSettings": {},
"type": "Microsoft.Cache/Redis",
"zones": null
}
We need to take hostName
parameter and replace YOUR_REDIS_ENDPOINT_URL_WITHOUT_PORT in file tb-redis-configmap.yml
After this we need to get redis keys for connection, for this we need to execute:
1
az redis list-keys --name $TB_REDIS_NAME --resource-group $AKS_RESOURCE_GROUP
after took “primary” and paste into tb-redis-configmap.yml file replacing YOU_REDIS_PASS
Step 7. Installation
Execute the following command to run the initial setup of the database. This command will launch short-living ThingsBoard pod to provision necessary DB tables, indexes, etc
1
./k8s-install-tb.sh --loadDemo
Where:
--loadDemo
- optional argument. Whether to load additional demo data.
After this command finish you should see the next line in the console:
1
Installation finished successfully!
Otherwise, please check if you set the PostgreSQL URL in the tb-node-db-configmap.yml
correctly.
Step 8. Starting
Execute the following command to deploy ThingsBoard services:
1
./k8s-deploy-resources.sh
After few minutes you may call kubectl get pods
. If everything went fine, you should be able to see tb-node-0
pod in the READY
state.
After few minutes you may call kubectl get pods. If everything went fine, you should be able to see tb-node-0 pod in the READY state.
You should also deploy the transport microservices. Omit the protocols that you don’t use in order to save resources:
HTTP Transport (optional)
1
kubectl apply -f transports/tb-http-transport.yml
MQTT transport (optional)
1
kubectl apply -f transports/tb-mqtt-transport.yml
CoAP transport (optional)
1
kubectl apply -f transports/tb-coap-transport.yml
LwM2M transport (optional)
1
kubectl apply -f transports/tb-lwm2m-transport.yml
SNMP transport (optional)
1
kubectl apply -f transports/tb-snmp-transport.yml
Step 9. Configure Load Balancers
9.1. Configure HTTP(S) Load Balancer
Configure HTTP(S) Load Balancer to access web interface of your ThingsBoard instance. Basically you have 3 possible options of configuration:
- http - Load Balancer without HTTPS support. Recommended for development. The only advantage is simple configuration and minimum costs. May be good option for development server but definitely not suitable for production.
- https - Load Balancer with HTTPS support. Recommended for production. Acts as an SSL termination point. You may easily configure it to issue and maintain a valid SSL certificate. Automatically redirects all non-secure (HTTP) traffic to secure (HTTPS) port.
See links/instructions below on how to configure each of the suggested options.
HTTP Load Balancer
Execute the following command to deploy plain http load balancer:
1
kubectl apply -f receipts/http-load-balancer.yml
The process of load balancer provisioning may take some time. You may periodically check the status of the load balancer using the following command:
1
kubectl get ingress
Once provisioned, you should see similar output:
1
2
NAME CLASS HOSTS ADDRESS PORTS AGE
tb-http-loadbalancer <none> * 34.111.24.134 80 7m25s
Now, you may use the address (the one you see instead of 34.111.24.134 in the command output) to access HTTP web UI (port 80) and connect your devices via HTTP API Use the following default credentials:
- System Administrator: [email protected] / sysadmin
- Tenant Administrator: [email protected] / tenant
- Customer User: [email protected] / customer
HTTPS Load Balancer
For using ssl certificates we can add our certificate directly in Azure ApplicationGateWay using command:
1
2
3
4
5
6
az network application-gateway ssl-cert create \
--resource-group $(az aks show --name $TB_CLUSTER_NAME --resource-group $AKS_RESOURCE_GROUP --query nodeResourceGroup | tr -d '"') \
--gateway-name $AKS_GATEWAY\
--name ThingsBoardHTTPCert \
--cert-file YOUR_CERT \
--cert-password YOUR_CERT_PASS
After we added certificate to Application Load balancer we can execute:
1
kubectl apply -f receipts/https-load-balancer.yml
The process of load balancer provisioning may take some time. You may periodically check the status of the load balancer using the following command:
1
kubectl get ingress
Once provisioned, you should see similar output:
1
2
NAME CLASS HOSTS ADDRESS PORTS AGE
tb-https-loadbalancer <none> * 34.111.24.134 80 7m25s
9.2. Configure MQTT Load Balancer (Optional)
Configure MQTT load balancer if you plan to use MQTT protocol to connect devices.
Create TCP load balancer using following command:
1
kubectl apply -f receipts/mqtt-load-balancer.yml
The load balancer will forward all TCP traffic for ports 1883 and 8883.
MQTT over SSL
This type of the load balancer requires you to provision and maintain valid SSL certificate on your own. Follow the generic MQTT over SSL guide to configure required environment variables in the transport/tb-mqtt-transport.yml file.
9.3. Configure CoAP Load Balancer (Optional)
Configure CoAP load balancer if you plan to use CoAP protocol to connect devices.
Create CoAP load balancer using following command:
1
kubectl apply -f receipts/coap-load-balancer.yml
The load balancer will forward all UDP traffic for the following ports:
- 5683 - CoAP server non-secure port
- 5684 - CoAP server secure DTLS port.
CoAP over DTLS
This type of the load balancer requires you to provision and maintain valid SSL certificate on your own. Follow the generic CoAP over DTLS guide to configure required environment variables in the transport/tb-coap-transport.yml file.
9.4. Configure LwM2M Load Balancer (Optional)
Configure LwM2M load balancer if you plan to use LwM2M protocol to connect devices.
Create LwM2M UDP load balancer using following command:
1
kubectl apply -f receipts/lwm2m-load-balancer.yml
The load balancer will forward all UDP traffic for the following ports:
- 5685 - LwM2M server non-secure port.
- 5686 - LwM2M server secure DTLS port.
- 5687 - LwM2M bootstrap server DTLS port.
- 5688 - LwM2M bootstrap server secure DTLS port.
LwM2M over DTLS
This type of the load balancer requires you to provision and maintain valid SSL certificate on your own. Follow the generic LwM2M over DTLS guide to configure required environment variables in the transport/tb-lwm2m-transport.yml file.
9.5. Configure Edge Load Balancer (Optional)
Configure the Edge load balancer if you plan to connect Edge instances to your ThingsBoard server.
To create a TCP Edge load balancer, apply the provided YAML file using the following command:
1
kubectl apply -f receipts/edge-load-balancer.yml
The load balancer will forward all TCP traffic on port 7070.
After the Edge load balancer is provisioned, you can connect Edge instances to your ThingsBoard server.
Before connecting Edge instances, you need to obtain the external IP address of the Edge load balancer. To retrieve this IP address, execute the following command:
1
kubectl get services | grep "EXTERNAL-IP\|tb-edge-loadbalancer"
You should see output similar to the following:
1
2
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
tb-edge-loadbalancer LoadBalancer 10.44.5.255 104.154.29.225 7070:30783/TCP 85m
Make a note of the external IP address and use it later in the Edge connection parameters as CLOUD_RPC_HOST.
Step 10. Using
Now you can open ThingsBoard web interface in your browser using IP address of the load balancer.
You can see DNS name (the ADDRESS
column) of the HTTP load-balancer using command:
1
kubectl get ingress
You should see the similar picture:
To connect to the cluster via MQTT or COAP you’ll need to get corresponding service, you can do it with command:
1
kubectl get service
You should see the similar picture:
There are two load-balancers:
- tb-mqtt-loadbalancer - for TCP (MQTT) protocol
- tb-udp-loadbalancer - for UDP (COAP/LwM2M) protocol
Use EXTERNAL-IP
field of the load-balancers to connect to the cluster.
Use the following default credentials:
- System Administrator: [email protected] / sysadmin
If you installed database with demo data (using --loadDemo
flag) you can also use the following credentials:
- Tenant Administrator: [email protected] / tenant
- Customer User: [email protected] / customer
In case of any issues you can examine service logs for errors. For example to see ThingsBoard node logs execute the following command:
1
kubectl logs -f tb-node-0
Or use kubectl get pods
to see the state of the pod.
Or use kubectl get services
to see the state of all the services.
Or use kubectl get deployments
to see the state of all the deployments.
See kubectl Cheat Sheet command reference for details.
Execute the following command to delete tb-node and load-balancers:
1
./k8s-delete-resources.sh
Execute the following command to delete all data (including database):
1
./k8s-delete-all.sh
Upgrading
In case when database upgrade is needed, execute the following commands:
1
2
3
./k8s-delete-resources.sh
./k8s-upgrade-tb.sh --fromVersion=[FROM_VERSION]
./k8s-deploy-resources.sh
Where:
FROM_VERSION
- from which version upgrade should be started. See Upgrade Instructions for validfromVersion
values.
Next steps
-
Getting started guides - These guides provide quick overview of main ThingsBoard features. Designed to be completed in 15-30 minutes.
-
Connect your device - Learn how to connect devices based on your connectivity technology or solution.
-
Data visualization - These guides contain instructions on how to configure complex ThingsBoard dashboards.
-
Data processing & actions - Learn how to use ThingsBoard Rule Engine.
-
IoT Data analytics - Learn how to use rule engine to perform basic analytics tasks.
-
Hardware samples - Learn how to connect various hardware platforms to ThingsBoard.
-
Advanced features - Learn about advanced ThingsBoard features.
-
Contribution and Development - Learn about contribution and development in ThingsBoard.