Stand with Ukraine flag
Try it now Pricing
MQTT Broker
Community Edition Professional Edition Cloud Edge PE Edge IoT Gateway License Server Trendz Analytics Mobile Application PE Mobile Application MQTT Broker
Installation > Cluster setup with Kubernetes on AWS
Getting Started Documentation
Architecture API FAQ
On this page

Cluster setup using AWS infrastructure

This guide will help you to set up TBMQ in AWS EKS.

Prerequisites

Install and configure tools

To deploy TBMQ on EKS cluster you’ll need to install kubectl, eksctl and awscli tools.

Afterward you need to configure Access Key, Secret Key and default region. To get Access and Secret keys please follow this guide. The default region should be the ID of the region where you’d like to deploy the cluster.

1

aws configure

Step 1. Open TBMQ K8S scripts repository

1
2

git clone -b release-1.3.0 https://github.com/thingsboard/tbmq.git
cd tbmq/k8s/aws

Step 2. Configure and create EKS cluster

In the cluster.yml file you can find suggested cluster configuration. Here are the fields you can change depending on your needs:

  • region - should be the AWS region where you want your cluster to be located (the default value is us-east-1)
  • availabilityZones - should specify the exact IDs of the region’s availability zones (the default value is [us-east-1a,us-east-1b,us-east-1c])
  • instanceType - the type of the instance with TBMQ node (the default value is m7a.large)

Note: If you don’t make any changes to instanceType and desiredCapacity fields, the EKS will deploy 2 nodes of type m7a.large.

Doc info icon

In case you want to secure access to the PostgreSQL and MSK, you’ll need to configure the existing VPC or create a new one, set it as the VPC for TBMQ cluster, create security groups for PostgreSQL and MSK, set them for managed node-group in TBMQ cluster and configure the access from TBMQ cluster nodes to PostgreSQL/MSK using another security group.

You can find more information about configuring VPC for eksctl here.

Command to create AWS cluster:

1

eksctl create cluster -f cluster.yml

Step 3. Create AWS load-balancer controller

Once the cluster is ready you’ll need to create AWS load-balancer controller. You can do it by following this guide. The cluster provisioning scripts will create several load balancers:

  • tb-broker-http-loadbalancer - AWS ALB that is responsible for the web UI and REST API;
  • tb-broker-mqtt-loadbalancer - AWS NLB that is responsible for the MQTT communication.

Provisioning of the AWS load-balancer controller is a very important step that is required for those load balancers to work properly.

Step 4. Amazon PostgreSQL DB Configuration

You’ll need to set up PostgreSQL on Amazon RDS. One of the ways to do it is by following this guide.

Note: Some recommendations:

  • Make sure your PostgreSQL version is 15.x;
  • Use ‘Production’ template for high availability. It enables a lot of useful settings by default;
  • Consider creation of custom parameters group for your RDS instance. It will make change of DB parameters easier;
  • Consider deployment of the RDS instance into private subnets. This way it will be nearly impossible to accidentally expose it to the internet.
  • You may also change username field and set or auto-generate password field (keep your postgresql password in a safe place).

Note: Make sure your database is accessible from the cluster, one of the way to achieve this is to create the database in the same VPC and subnets as TBMQ cluster and use eksctl-tbmq-cluster-ClusterSharedNodeSecurityGroup-* security group. See screenshots below.

AWS RDS Connectivity - choose VPC with the name of your cluster
AWS RDS Connectivity - choose VPC with the name of your cluster
AWS RDS SG - choose "eksctl-tbmq-cluster-ClusterSharedNodeSecurityGroup-" security group
AWS RDS SG - choose "eksctl-tbmq-cluster-ClusterSharedNodeSecurityGroup-*" security group
AWS RDS Additional config - type "thingsboard_mqtt_broker" for the initial database name
AWS RDS Additional config - type "thingsboard_mqtt_broker" for the initial database name

Step 5. Amazon MSK Configuration

You’ll need to set up Amazon MSK. To do so you need to open AWS console, MSK submenu, press Create cluster button and choose Custom create mode. You should see the similar image:

AWS MSK - create cluster
AWS MSK - create cluster

Note: Some recommendations:

  • Apache Kafka version can be safely set to the 3.5.1 version as TBMQ is fully tested on it;
  • Use m5.large or similar instance types;
  • Consider creation of custom cluster configuration for your MSK. It will make change of Kafka parameters easier;
  • Use default ‘Monitoring’ settings or enable ‘Enhanced topic-level monitoring’.

Note: Make sure your MSK instance is accessible from TBMQ cluster. The easiest way to achieve this is to deploy the MSK instance in the same VPC. We also recommend to use private subnets. This way it will be nearly impossible to accidentally expose it to the internet;

AWS MSK - choose TBMQ cluster’s VPC
AWS MSK - choose TBMQ cluster’s VPC
AWS MSK - choose "eksctl-tbmq-cluster-ClusterSharedNodeSecurityGroup-" security group
AWS MSK - choose "eksctl-tbmq-cluster-ClusterSharedNodeSecurityGroup-*" security group
AWS MSK - enable Plaintext communication between clients and brokers
AWS MSK - enable Plaintext communication between clients and brokers

At the end, carefully review the whole configuration of the MSK and then finish the cluster creation.

Step 6. Amazon ElastiCache (Redis) Configuration (Optional)

Optionally, you can set up ElastiCache for Redis. TBMQ uses cache to improve performance and avoid frequent DB reads.

We recommend enabling this in case you have several thousand MQTT clients (devices) connected to TBMQ. It is useful when clients connect to TBMQ with the authentication enabled. For every connection, the request is made to find MQTT client credentials that can authenticate the client. Thus, there could be an excessive amount of requests to be processed for a large number of connecting clients at once.

Please open AWS console and navigate to ElastiCache->Redis clusters->Create Redis cluster.

Note: Some recommendations:

  • Specify Redis Engine version 7.x and node type with at least 1 GB of RAM;
  • Make sure your Redis cluster is accessible from the TBMQ cluster. The easiest way to achieve this is to deploy the Redis cluster in the same VPC. We also recommend to use private subnets. Use eksctl-tbmq-cluster-ClusterSharedNodeSecurityGroup-* security group;
  • Disable automatic backups.
AWS ElastiCache - create Redis cluster
AWS ElastiCache - create Redis cluster
AWS ElastiCache - choose 7.x engine version and appropriate Node type
AWS ElastiCache - choose 7.x engine version and appropriate Node type
AWS ElastiCache - choose TBMQ VPC and private subnets
AWS ElastiCache - choose TBMQ VPC and private subnets
AWS ElastiCache - choose "eksctl-tbmq-cluster-ClusterSharedNodeSecurityGroup-" security group
AWS ElastiCache - choose "eksctl-tbmq-cluster-ClusterSharedNodeSecurityGroup-*" security group

Amazon RDS PostgreSQL

Once the database switch to the ‘Available’ state, on AWS Console get the Endpoint of the RDS PostgreSQL and paste it to SPRING_DATASOURCE_URL in the tb-broker-db-configmap.yml instead of RDS_URL_HERE part.

AWS RDS Details
AWS RDS Details

Also, you’ll need to set SPRING_DATASOURCE_USERNAME and SPRING_DATASOURCE_PASSWORD with PostgreSQL username and password corresponding.

Amazon MSK

Once the MSK cluster switch to the ‘Active’ state, to get the list of brokers execute the next command:

1

aws kafka get-bootstrap-brokers --region us-east-1 --cluster-arn $CLUSTER_ARN

Where $CLUSTER_ARN is the Amazon Resource Name (ARN) of the MSK cluster:

AWS MSK Details
AWS MSK Details

You’ll need to paste data from the BootstrapBrokerString to the TB_KAFKA_SERVERS environment variable in the tb-broker.yml file.

Otherwise, click View client information seen on the screenshot above. Copy bootstrap server information in plaintext.

Amazon ElastiCache

Once the Redis cluster switch to the ‘Available’ state, open the ‘Cluster details’ and copy Primary endpoint without “:6379” port suffix, it`s YOUR_REDIS_ENDPOINT_URL_WITHOUT_PORT.

AWS ElastiCache Details
AWS ElastiCache Details

Edit tb-broker-cache-configmap.yml and replace YOUR_REDIS_ENDPOINT_URL_WITHOUT_PORT.

Step 8. Installation

Execute the following command to run installation:

1

./k8s-install-tbmq.sh

After this command finish you should see the next line in the console:

1

INFO  o.t.m.b.i.ThingsboardMqttBrokerInstallService - Installation finished successfully!
Doc info icon

Otherwise, please check if you set the PostgreSQL URL and PostgreSQL password in the tb-broker-db-configmap.yml correctly.

Step 9. Starting

Execute the following command to deploy the broker:

1

./k8s-deploy-tbmq.sh

After a few minutes, you may execute the next command to check the state of all pods.

1

kubectl get pods

If everything went fine, you should be able to see tb-broker-0 and tb-broker-1 pods. Every pod should be in the READY state.

Step 10. Configure Load Balancers

10.1 Configure HTTP(S) Load Balancer

Configure HTTP(S) Load Balancer to access web interface of your TBMQ instance. Basically you have 2 possible options of configuration:

  • http - Load Balancer without HTTPS support. Recommended for development. The only advantage is simple configuration and minimum costs. May be good option for development server but definitely not suitable for production.
  • https - Load Balancer with HTTPS support. Recommended for production. Acts as an SSL termination point. You may easily configure it to issue and maintain a valid SSL certificate. Automatically redirects all non-secure (HTTP) traffic to secure (HTTPS) port.

See links/instructions below on how to configure each of the suggested options.

HTTP Load Balancer

Execute the following command to deploy plain http load balancer:

1

kubectl apply -f receipts/http-load-balancer.yml

The process of load balancer provisioning may take some time. You may periodically check the status of the load balancer using the following command:

1

kubectl get ingress

Once provisioned, you should see similar output:

1
2

NAME                          CLASS    HOSTS   ADDRESS                                                                  PORTS   AGE
tb-broker-http-loadbalancer   <none>   *       k8s-thingsbo-tbbroker-000aba1305-222186756.eu-west-1.elb.amazonaws.com   80      3d1h
HTTPS Load Balancer

Use AWS Certificate Manager to create or import SSL certificate. Note your certificate ARN.

Edit the load balancer configuration and replace YOUR_HTTPS_CERTIFICATE_ARN with your certificate ARN:

1

nano receipts/https-load-balancer.yml

Execute the following command to deploy plain https load balancer:

1

kubectl apply -f receipts/https-load-balancer.yml

10.2 Configure MQTT Load Balancer

Configure MQTT load balancer to be able to use MQTT protocol to connect devices.

Create TCP load balancer using following command:

1

kubectl apply -f receipts/mqtt-load-balancer.yml

The load balancer will forward all TCP traffic for ports 1883 and 8883.

One-way TLS

The simplest way to configure MQTTS is to make your MQTT load balancer (AWS NLB) to act as a TLS termination point. This way we set up the one-way TLS connection, where the traffic between your devices and load balancers is encrypted, and the traffic between your load balancer and TBMQ is not encrypted. There should be no security issues, since the ALB/NLB is running in your VPC. The only major disadvantage of this option is that you can’t use “X.509 certificate” MQTT client credentials, since information about client certificate is not transferred from the load balancer to the TBMQ.

To enable the one-way TLS:

Use AWS Certificate Manager to create or import SSL certificate. Note your certificate ARN.

Edit the load balancer configuration and replace YOUR_MQTTS_CERTIFICATE_ARN with your certificate ARN:

1

nano receipts/mqtts-load-balancer.yml

Execute the following command to deploy plain MQTTS load balancer:

1

kubectl apply -f receipts/mqtts-load-balancer.yml
Two-way TLS

The more complex way to enable MQTTS is to obtain valid (signed) TLS certificate and configure it in the TBMQ. The main advantage of this option is that you may use it in combination with “X.509 certificate” MQTT client credentials.

To enable the two-way TLS:

Follow this guide to create a .pem file with the SSL certificate. Store the file as server.pem in the working directory.

You’ll need to create a config-map with your PEM file, you can do it by calling command:

1
2
3
4

kubectl create configmap tbmq-mqtts-config \
 --from-file=server.pem=YOUR_PEM_FILENAME \
 --from-file=mqttserver_key.pem=YOUR_PEM_KEY_FILENAME \
 -o yaml --dry-run=client | kubectl apply -f -
  • where YOUR_PEM_FILENAME is the name of your server certificate file.
  • where YOUR_PEM_KEY_FILENAME is the name of your server certificate private key file.

Then, uncomment all sections in the ‘tb-broker.yml’ file that are marked with “Uncomment the following lines to enable two-way MQTTS”.

Execute command to apply changes:

1

kubectl apply -f tb-broker.yml

Finally, deploy the “transparent” load balancer:

1

kubectl apply -f receipts/mqtt-load-balancer.yml

Step 11. Validate the setup

Now you can open TBMQ web interface in your browser using DNS name of the load balancer.

You can get DNS name of the load-balancers using the next command:

1

kubectl get ingress

You should see the similar picture:

1
2

NAME                          CLASS    HOSTS   ADDRESS                                                                  PORTS   AGE
tb-broker-http-loadbalancer   <none>   *       k8s-thingsbo-tbbroker-000aba1305-222186756.eu-west-1.elb.amazonaws.com   80      3d1h

Use ADDRESS field of the tb-broker-http-loadbalancer to connect to the cluster.

You should see TBMQ login page. Use the following default credentials for System Administrator:

Username:

1

[email protected]

Password:

1

sysadmin

On the first user log-in you will be asked to change the default password to the preferred one and then re-login using the new credentials.

Validate MQTT access

To connect to the cluster via MQTT you will need to get corresponding service IP. You can do this with the command:

1

kubectl get services

You should see the similar picture:

1
2

NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)                         AGE
tb-broker-mqtt-loadbalancer   LoadBalancer   10.100.119.170   k8s-thingsbo-tbbroker-b9f99d1ab6-1049a98ba4e28403.elb.eu-west-1.amazonaws.com   1883:30308/TCP,8883:31609/TCP   6m58s

Use EXTERNAL-IP field of the load-balancer to connect to the cluster via MQTT protocol.

Troubleshooting

In case of any issues you can examine service logs for errors. For example to see TBMQ logs execute the following command:

1

kubectl logs -f tb-broker-0

Use the next command to see the state of all statefulsets.

1

kubectl get statefulsets

See kubectl Cheat Sheet command reference for more details.

Upgrading

Doc info icon

Please note:
For the TBMQ 1.3.0 version, the installation scripts were updated to contain a new 8084 port for MQTT over WebSockets. This is needed for the correct work with the WebSocket client page.

Please pull the latest configuration files or modify your existing ones to include a new port entry. To find more details please visit the following link.

Once the required changes are made, you should be able to connect the MQTT client on the WebSocket client page. Otherwise, please contact us, so we can answer any questions and provide our help if needed.

In case you would like to upgrade, please pull the recent changes from the latest release branch:

1

git pull origin release-1.3.0

Note: Make sure custom changes of yours if available are not lost during the merge process.

If you encounter conflicts during the merge process that are not related to your changes, we recommend accepting all the new changes from the remote branch.

You could revert the merge process by executing the following:

1

git merge --abort

And repeat the merge by accepting theirs changes.

1

git pull origin release-1.3.0 -X theirs

There are several useful options for the default merge strategy:

  • -X ours - this option forces conflicting hunks to be auto-resolved cleanly by favoring our version.
  • -X theirs - this is the opposite of ours. See more details here.

After that execute the following command:

1

./k8s-upgrade-tbmq.sh --fromVersion=FROM_VERSION

Where FROM_VERSION - from which version upgrade should be started. See Upgrade Instructions for valid fromVersion values.

Note: You may optionally stop the TBMQ pods while you run the upgrade of the database with the below command.

1

./k8s-delete-tbmq.sh

This will cause downtime, but will make sure that the DB state will be consistent after the update. Most of the updates do not require the TBMQ to be stopped.

Once completed, execute deployment of the resources again. This will cause rollout restart of the TBMQ with the newest version.

1

./k8s-deploy-tbmq.sh

Cluster deletion

Execute the following command to delete TBMQ nodes:

1

./k8s-delete-tbmq.sh

Execute the following command to delete all TBMQ nodes and configmaps:

1

./k8s-delete-all.sh

Execute the following command to delete the EKS cluster (you should change the name of the cluster and the region if those differ):

1

eksctl delete cluster -r us-east-1 -n tbmq -w

Next steps