AWS EKS Microservices Setup

This guide walks you through deploying ThingsBoard PE in microservices mode on AWS EKS. We use Amazon RDS for managed PostgreSQL, Amazon MSK for managed Kafka, and Amazon ElastiCache for managed Redis.

Install kubectl, eksctl, and AWS CLI.

Configure your AWS credentials. To get Access and Secret keys, follow this guide. The default region should be the ID of the region where you want to deploy the cluster.

aws configure

Verify that you can pull the images from Docker Hub:

docker pull thingsboard/tb-pe-node:4.3.1.1PE
docker pull thingsboard/tb-pe-web-report:4.3.1.1PE
docker pull thingsboard/tb-pe-web-ui:4.3.1.1PE
docker pull thingsboard/tb-pe-js-executor:4.3.1.1PE
docker pull thingsboard/tb-pe-http-transport:4.3.1.1PE
docker pull thingsboard/tb-pe-mqtt-transport:4.3.1.1PE
docker pull thingsboard/tb-pe-coap-transport:4.3.1.1PE
docker pull thingsboard/tb-pe-lwm2m-transport:4.3.1.1PE
docker pull thingsboard/tb-pe-snmp-transport:4.3.1.1PE

git clone -b release-4.3 https://github.com/thingsboard/thingsboard-pe-k8s.git --depth 1
cd thingsboard-pe-k8s/aws/microservices

In the cluster.yml file you can find the suggested cluster configuration. Key fields you can change:

Create the cluster:

eksctl create cluster -f cluster.yml

Once the cluster is ready, create the AWS load-balancer controller by following this guide.

The cluster provisioning scripts create several load balancers:

Set up PostgreSQL on Amazon RDS. ThingsBoard uses it as the main database for devices, dashboards, rule chains, and device telemetry. Follow this guide, but take into account the following requirements:

• Keep your PostgreSQL password in a safe place. We will refer to it later as YOUR_RDS_PASSWORD.
• Make sure your PostgreSQL version is latest 16.x.
• Make sure your PostgreSQL RDS instance is accessible from the ThingsBoard cluster. The easiest way is to deploy the RDS instance in the same VPC and use the eksctl-thingsboard-cluster-ClusterSharedNodeSecurityGroup-* security group.
• Make sure you use “thingsboard” as the initial database name. If you do not specify a database name, Amazon RDS does not create one.

Recommendations:

• Use Production template for high availability.
• Use Provisioned IOPS for better performance.
• Consider creating a custom parameters group for your RDS instance.
• Consider deploying the RDS instance into private subnets.

Make sure your PostgreSQL version is latest 16.x. Keep your PostgreSQL master password in a safe place. Use "Provisioned IOPS" for better performance. Deploy the RDS instance in the same VPC and use the cluster shared security group. Use "thingsboard" as initial database name. Disable "auto minor version update".

Once the database switches to the Available state, navigate to Connectivity and Security and copy the endpoint value. We will refer to it as YOUR_RDS_ENDPOINT_URL.

Using Cassandra is optional. We recommend it if you plan to insert more than 5K data points per second or want to optimize storage space.

Provision additional node groups to host Cassandra instances. At least 4 vCPUs and 16 GB of RAM is recommended.

Create 3 separate node pools with 1 node per zone:

eksctl create nodegroup --config-file=<path> --include='cassandra-*'

kubectl apply -f tb-namespace.yml
kubectl config set-context $(kubectl config current-context) --namespace=thingsboard

Deploy Cassandra:

kubectl apply -f receipts/cassandra.yml

Monitor the process:

kubectl get pods

Don’t forget to replace YOUR_AWS_REGION with the name of your AWS region.

echo "  DATABASE_TS_TYPE: cassandra" >> tb-node-db-configmap.yml
echo "  CASSANDRA_URL: cassandra:9042" >> tb-node-db-configmap.yml
echo "  CASSANDRA_LOCAL_DATACENTER: YOUR_AWS_REGION"  >> tb-node-db-configmap.yml

Verify:

cat tb-node-db-configmap.yml | grep DATABASE_TS_TYPE

kubectl exec -it cassandra-0 -- bash -c "cqlsh -e \
    \"CREATE KEYSPACE IF NOT EXISTS thingsboard \
    WITH replication = { \
        'class' : 'NetworkTopologyStrategy', \
        'us-east' : '3' \
    };\""

ThingsBoard uses Kafka as an external queue for exchanging data between microservices, storing unprocessed messages, and more. By default, the deployment uses local Kafka, but ThingsBoard is also compatible with managed services such as Amazon MSK.

Steps to create a basic Kafka MSK cluster:

• Open the AWS console, go to MSK and click Create Cluster.
• Select Custom creation method.
• Specify a name for your cluster and select Cluster type → Provisioned.
• Select Apache Kafka version 3.8.x to use Express brokers or version 4.0.x for Standard brokers.
• Choose kafka.m7.large or similar instance types.
• Select the storage size for the broker (with default ThingsBoard partition settings, Kafka can use up to 100 GB).
• Deploy the MSK instance in the same VPC as the ThingsBoard cluster. Use private subnets.
• Use the default security settings. Make sure Plaintext mode is enabled.
• Use either Basic monitoring or Enhanced topic-level monitoring settings.

Select Custom creation, Provisioned cluster type, and Apache Kafka version. Choose kafka.m7.large or similar instance types. Select the storage size for the broker. Deploy the MSK instance in the same VPC as the ThingsBoard cluster. Use default security settings. Make sure Plaintext mode is enabled. Use Basic monitoring or Enhanced topic-level monitoring.

Once the MSK cluster switches to the Active state, navigate to Details and click View client information. Copy the bootstrap server information in plaintext — this is your Kafka endpoint.

Click "View client information" from the Details tab. Copy the bootstrap server information in plaintext.

Edit the tb-kafka.yml file, find the StatefulSet section named tb-kafka, and set spec.replicas to 0 to disable the default local Kafka deployment.

Edit tb-kafka-configmap.yml and replace TB_KAFKA_SERVERS value with your MSK endpoint.

ThingsBoard uses cache to improve performance and reduce frequent database reads. By default, the deployment uses a local Valkey cache, but ThingsBoard is also compatible with managed services such as Amazon ElastiCache.

Steps to create a basic ElastiCache Valkey cluster:

• Open the AWS console and navigate to ElastiCache Valkey caches and click Create.
• Choose the Deployment option Serverless or Design your own cache.
• Specify Valkey Engine version 8.x and a node type with at least 1 GB of RAM.
• Deploy the Valkey cluster in the same VPC as the ThingsBoard cluster. Use private subnets and your group ID.
• Disable the Enable automatic backups option.

Specify Valkey Engine version 8.x and node type with at least 1 GB of RAM. Deploy the Valkey cluster in the same VPC. Use private subnets. Disable the "Enable automatic backups" option.

Once the Valkey cluster switches to the Available state, navigate to the Details section and copy the Endpoint field without the “:6379” port suffix.

Edit the tb-valkey.yml file, locate the StatefulSet section named tb-valkey, and set spec.replicas to 0 to disable the default local Valkey deployment.

Then, edit tb-cache-configmap.yml and replace the REDIS_HOST value with your Valkey endpoint.

Edit tb-node-db-configmap.yml and replace YOUR_RDS_ENDPOINT_URL and YOUR_RDS_PASSWORD with the values obtained during Step 4.

Edit tb-kafka-configmap.yml and replace YOUR_MSK_BOOTSTRAP_SERVERS_PLAINTEXT with the value obtained during Step 5.

Edit tb-redis-configmap.yml and replace YOUR_REDIS_ENDPOINT_URL_WITHOUT_PORT with the value obtained during Step 6.

We assume you have already chosen your subscription plan or decided to purchase a perpetual license. If not, navigate to the pricing page. See How to get pay-as-you-go subscription or How to get perpetual license for details.

Create a docker secret with your license key:

export TB_LICENSE_KEY=PUT_YOUR_LICENSE_KEY_HERE
kubectl create -n thingsboard secret generic tb-license --from-literal=license-key=$TB_LICENSE_KEY

The scripts have preconfigured values of resources for each service. You can change them in .yml files under the resources section.

Recommended CPU/memory resources allocation:

Execute the following command to run the initial setup of the database:

./k8s-install-tb.sh --loadDemo

Where --loadDemo is an optional argument to load additional demo data.

After this command finishes you should see:

Installation finished successfully!

Deploy ThingsBoard services:

./k8s-deploy-resources.sh

After a few minutes, call kubectl get pods. If everything went fine, you should see:

• 5x tb-js-executor
• 1x tb-node (scale to more nodes if you have additional license instances)
• 2x tb-web-ui
• 3x zookeeper

Every pod should be in the READY state.

Deploy the transport microservices you need. Omit protocols you don’t use to save resources:

# HTTP Transport (optional)
kubectl apply -f transports/tb-http-transport.yml

# MQTT Transport (optional)
kubectl apply -f transports/tb-mqtt-transport.yml

# CoAP Transport (optional)
kubectl apply -f transports/tb-coap-transport.yml

# LwM2M Transport (optional)
kubectl apply -f transports/tb-lwm2m-transport.yml

# SNMP Transport (optional)
kubectl apply -f transports/tb-snmp-transport.yml

You have 2 options:

• HTTP — recommended for development. Simple configuration and minimum costs.
• HTTPS — recommended for production. Acts as an SSL termination point with automatic redirect from HTTP to HTTPS.

kubectl apply -f receipts/http-load-balancer.yml

Check the status:

kubectl get ingress

Once provisioned, you should see:

NAME                   CLASS    HOSTS   ADDRESS         PORTS   AGE
tb-http-loadbalancer   <none>   *       34.111.24.134   80      7m25s

Use the address to access the HTTP web UI (port 80) and connect devices via HTTP API.

Default credentials:

• System Administrator: [email protected] / sysadmin
• Tenant Administrator: [email protected] / tenant (if demo data loaded)
• Customer User: [email protected] / customer (if demo data loaded)

Use AWS Certificate Manager to create or import an SSL certificate. Note your certificate ARN.

Edit the load balancer configuration and replace YOUR_HTTPS_CERTIFICATE_ARN:

nano receipts/https-load-balancer.yml

Deploy:

kubectl apply -f receipts/https-load-balancer.yml

kubectl apply -f receipts/mqtt-load-balancer.yml

The load balancer forwards all TCP traffic for ports 1883 and 8883.

Make the AWS NLB act as a TLS termination point. Traffic between devices and the load balancer is encrypted.

Use AWS Certificate Manager to create or import an SSL certificate. Edit the load balancer configuration:

nano receipts/mqtts-load-balancer.yml

Replace YOUR_MQTTS_CERTIFICATE_ARN, then deploy:

kubectl apply -f receipts/mqtts-load-balancer.yml

Follow the MQTT over SSL guide to create a .pem file.

Create a config-map:

kubectl create configmap tb-mqtts-config \
 --from-file=server.pem=YOUR_PEM_FILENAME \
 --from-file=mqttserver_key.pem=YOUR_PEM_KEY_FILENAME \
 -o yaml --dry-run=client | kubectl apply -f -

Uncomment all sections in tb-services.yml marked with “Uncomment the following lines to enable two-way MQTTS”, then apply:

kubectl apply -f tb-services.yml

Deploy the “transparent” load balancer:

kubectl apply -f receipts/mqtt-load-balancer.yml

kubectl apply -f receipts/udp-load-balancer.yml

The load balancer forwards all UDP traffic for ports:

For CoAP over DTLS, follow the CoAP over DTLS guide. For LwM2M over DTLS, follow the LwM2M over DTLS guide.

kubectl apply -f receipts/edge-load-balancer.yml

The load balancer forwards all TCP traffic on port 7070.

To get the external IP address:

kubectl get services | grep "EXTERNAL-IP\|tb-edge-loadbalancer"

Use the external IP address as CLOUD_RPC_HOST in Edge connection parameters.

docker pull thingsboard/trendz:1.15.1
docker pull thingsboard/trendz-python-executor:1.15.1

Edit trendz/trendz-secret.yml and replace YOUR_RDS_ENDPOINT_URL and YOUR_RDS_PASSWORD, then apply:

kubectl apply -f ./trendz/trendz-secret.yml
kubectl apply -f ./trendz/trendz-create-db.yml

Check logs:

kubectl logs job/trendz-create-db -n thingsboard

./k8s-deploy-trendz.sh

After this command finishes you should see:

Trendz installed successfully!

Get the DNS name of the HTTP load balancer:

kubectl get ingress

Use the address to open the ThingsBoard web interface in your browser.

Default credentials:

• System Administrator: [email protected] / sysadmin
• Tenant Administrator: [email protected] / tenant (if demo data loaded)
• Customer User: [email protected] / customer (if demo data loaded)

kubectl get service

Two load balancers are available:

• tb-mqtt-loadbalancer-external — for MQTT protocol
• tb-coap-loadbalancer-external — for CoAP protocol

Use the EXTERNAL-IP field of the load balancers to connect to the cluster.

To examine ThingsBoard node logs:

kubectl logs -f tb-node-0

Other useful commands:

• kubectl get pods — see the state of all pods
• kubectl get services — see the state of all services
• kubectl get deployments — see the state of all deployments

See the kubectl Cheat Sheet for more details.

Merge your local changes with the latest release branch from the repo you cloned in Step 1.

If a database upgrade is needed:

./k8s-upgrade-tb.sh --fromVersion=[FROM_VERSION]

Where FROM_VERSION is the starting version. See Upgrade Instructions for valid fromVersion values. You must upgrade versions one by one (e.g., 3.6.1 → 3.6.2 → 3.6.3).

Once completed, re-deploy resources:

./k8s-deploy-resources.sh

Pull the latest changes:

git pull origin master

Then execute:

./k8s-upgrade-trendz.sh

Delete all ThingsBoard pods:

./k8s-delete-resources.sh

Delete all ThingsBoard pods and configmaps:

./k8s-delete-all.sh

Delete the EKS cluster (change cluster name and region as needed):

eksctl delete cluster -r us-east-1 -n thingsboard -w