GKE Microservices Setup

This guide walks you through deploying ThingsBoard PE in microservices mode on Google Kubernetes Engine. We use Google Cloud SQL for managed PostgreSQL.

Prerequisites

Install and configure tools

Install kubectl and gcloud. See before you begin for more info.

Create a new GCP project (recommended) or choose an existing one:

gcloud init

Enable GCP services

gcloud services enable container.googleapis.com sql-component.googleapis.com sqladmin.googleapis.com

Pull ThingsBoard PE images

Verify that you can pull the images from Docker Hub:

docker pull thingsboard/tb-pe-node:4.3.1.1PE
docker pull thingsboard/tb-pe-web-report:4.3.1.1PE
docker pull thingsboard/tb-pe-web-ui:4.3.1.1PE
docker pull thingsboard/tb-pe-js-executor:4.3.1.1PE
docker pull thingsboard/tb-pe-http-transport:4.3.1.1PE
docker pull thingsboard/tb-pe-mqtt-transport:4.3.1.1PE
docker pull thingsboard/tb-pe-coap-transport:4.3.1.1PE
docker pull thingsboard/tb-pe-lwm2m-transport:4.3.1.1PE
docker pull thingsboard/tb-pe-snmp-transport:4.3.1.1PE

Step 1. Clone ThingsBoard PE K8S scripts repository

git clone -b release-4.3 https://github.com/thingsboard/thingsboard-pe-k8s.git --depth 1
cd thingsboard-pe-k8s/gcp/microservices

Step 2. Define environment variables

export GCP_PROJECT=$(gcloud config get-value project)
export GCP_REGION=us-central1
export GCP_ZONE=us-central1
export GCP_ZONE1=us-central1-a
export GCP_ZONE2=us-central1-b
export GCP_ZONE3=us-central1-c
export GCP_NETWORK=default
export TB_CLUSTER_NAME=tb-pe-msa
export TB_DATABASE_NAME=tb-db
echo "Project: $GCP_PROJECT, region: $GCP_REGION, zones: $GCP_ZONE1,$GCP_ZONE2,$GCP_ZONE3, network: $GCP_NETWORK, cluster: $TB_CLUSTER_NAME, database: $TB_DATABASE_NAME"

Variable	Default	Description
`GCP_PROJECT`	(auto-detected)	Your GCP project ID
`GCP_REGION`	`us-central1`	Compute region
`GCP_ZONE1/2/3`	`us-central1-a/b/c`	Availability zones for the regional cluster
`GCP_NETWORK`	`default`	GCP network name
`TB_CLUSTER_NAME`	`tb-pe-msa`	GKE cluster name
`TB_DATABASE_NAME`	`tb-db`	Cloud SQL instance name

Step 3. Configure and create GKE cluster

Create a regional cluster distributed across 3 zones. The example provisions one e2-standard-4 node per zone (3 nodes total). You can modify the machine type and node count to suit your workload. See GCP machine types for options.

gcloud container clusters create $TB_CLUSTER_NAME \
  --release-channel stable \
  --region $GCP_REGION \
  --network=$GCP_NETWORK \
  --node-locations $GCP_ZONE1,$GCP_ZONE2,$GCP_ZONE3 \
  --enable-ip-alias \
  --num-nodes=1 \
  --node-labels=role=main \
  --machine-type=e2-standard-4

Alternatively, see the regional cluster setup guide.

Step 4. Update the context of kubectl

gcloud container clusters get-credentials $TB_CLUSTER_NAME --region $GCP_REGION

Step 5. Provision databases

5.1 Google Cloud SQL (PostgreSQL) instance

Prerequisites

Enable service networking:

gcloud services enable servicenetworking.googleapis.com --project=$GCP_PROJECT

gcloud compute addresses create google-managed-services-$GCP_NETWORK \
  --global \
  --purpose=VPC_PEERING \
  --prefix-length=16 \
  --network=projects/$GCP_PROJECT/global/networks/$GCP_NETWORK

gcloud services vpc-peerings connect \
  --service=servicenetworking.googleapis.com \
  --ranges=google-managed-services-$GCP_NETWORK \
  --network=$GCP_NETWORK \
  --project=$GCP_PROJECT

Create database server instance

gcloud beta sql instances create $TB_DATABASE_NAME \
  --database-version=POSTGRES_16 \
  --region=$GCP_REGION --availability-type=regional \
  --no-assign-ip --network=projects/$GCP_PROJECT/global/networks/$GCP_NETWORK \
  --cpu=2 --memory=7680MB

Note the IP address (YOUR_DB_IP_ADDRESS) from the command output.

Set database password

gcloud sql users set-password postgres \
  --instance=$TB_DATABASE_NAME \
  --password=secret

Create database

gcloud sql databases create thingsboard --instance=$TB_DATABASE_NAME

5.2 Cassandra (optional)

Using Cassandra is optional. We recommend it if you plan to insert more than 5K data points per second or want to optimize storage space.

Provision additional node groups

Create 3 separate node pools with 1 node per zone. At least 4 vCPUs and 16 GB of RAM is recommended.

gcloud container node-pools create cassandra1 --cluster=$TB_CLUSTER_NAME --zone=$GCP_ZONE --node-locations=$GCP_ZONE1 \
  --node-labels=role=cassandra --num-nodes=1 --min-nodes=1 --max-nodes=1 --machine-type=e2-standard-4
gcloud container node-pools create cassandra2 --cluster=$TB_CLUSTER_NAME --zone=$GCP_ZONE --node-locations=$GCP_ZONE2 \
  --node-labels=role=cassandra --num-nodes=1 --min-nodes=1 --max-nodes=1 --machine-type=e2-standard-4
gcloud container node-pools create cassandra3 --cluster=$TB_CLUSTER_NAME --zone=$GCP_ZONE --node-locations=$GCP_ZONE3 \
  --node-labels=role=cassandra --num-nodes=1 --min-nodes=1 --max-nodes=1 --machine-type=e2-standard-4

Deploy Cassandra stateful set

kubectl apply -f tb-namespace.yml
kubectl config set-context $(kubectl config current-context) --namespace=thingsboard
kubectl apply -f receipts/cassandra.yml

Update DB settings

echo "  DATABASE_TS_TYPE: cassandra" >> tb-node-db-configmap.yml
echo "  CASSANDRA_URL: cassandra:9042" >> tb-node-db-configmap.yml
echo "  CASSANDRA_LOCAL_DATACENTER: $GCP_REGION" >> tb-node-db-configmap.yml

Create keyspace

kubectl exec -it cassandra-0 -- bash -c "cqlsh -e \
    \"CREATE KEYSPACE IF NOT EXISTS thingsboard \
    WITH replication = { \
        'class' : 'NetworkTopologyStrategy', \
        'us-central1' : '3' \
    };\""

Step 6. Obtain and configure license key

We assume you have already chosen your subscription plan or decided to purchase a perpetual license. If not, navigate to the pricing page. See How to get pay-as-you-go subscription or How to get perpetual license for details.

Create a docker secret with your license key:

export TB_LICENSE_KEY=PUT_YOUR_LICENSE_KEY_HERE
kubectl create -n thingsboard secret generic tb-license --from-literal=license-key=$TB_LICENSE_KEY

Step 7. Installation

Edit tb-node-db-configmap.yml and replace YOUR_DB_IP_ADDRESS and YOUR_DB_PASSWORD:

nano tb-node-db-configmap.yml

Run the installation:

./k8s-install-tb.sh --loadDemo

Where --loadDemo is an optional argument to load additional demo data.

After this command finishes you should see:

Installation finished successfully!

Step 8. Starting

Deploy thirdparty components (Zookeeper, Kafka, Redis) and main ThingsBoard microservices:

./k8s-deploy-resources.sh

After a few minutes, call kubectl get pods. If everything went fine, you should see tb-node-0 pod in the READY state.

Deploy transport microservices

Deploy the transport microservices you need. Omit protocols you don’t use to save resources:

# HTTP Transport (optional)
kubectl apply -f transports/tb-http-transport.yml

# MQTT Transport (optional)
kubectl apply -f transports/tb-mqtt-transport.yml

# CoAP Transport (optional)
kubectl apply -f transports/tb-coap-transport.yml

# LwM2M Transport (optional)
kubectl apply -f transports/tb-lwm2m-transport.yml

# SNMP Transport (optional)
kubectl apply -f transports/tb-snmp-transport.yml

Step 9. Configure load balancers

9.1 Configure HTTP(S) load balancer

You have 3 options:

HTTP — recommended for development.
HTTPS — recommended for production. Uses Google-managed SSL certificate.
Transparent — forwards traffic to ThingsBoard HTTP/HTTPS ports. Requires your own SSL certificate.

HTTP load balancer

kubectl apply -f receipts/http-load-balancer.yml

Check the status:

kubectl get ingress

Use the address to access the web UI and connect devices via HTTP API.

Default credentials:

System Administrator: [email protected] / sysadmin
Tenant Administrator: [email protected] / tenant (if demo data loaded)
Customer User: [email protected] / customer (if demo data loaded)

HTTPS load balancer

See the official documentation for Google-managed SSL certificates. Reserve a static IP:

gcloud compute addresses create thingsboard-http-lb-address --global

Edit receipts/https-load-balancer.yml and replace PUT_YOUR_DOMAIN_HERE, then deploy:

kubectl apply -f receipts/https-load-balancer.yml

Assign the domain name to the load balancer IP and wait for the certificate to provision (up to 60 minutes):

kubectl describe managedcertificate managed-cert

Transparent load balancer

Follow the HTTPS (TLS) configuration guide to configure SSL in tb-node.yml. Then deploy:

kubectl apply -f receipts/transparent-http-load-balancer.yml

9.2 Configure MQTT load balancer (optional)

kubectl apply -f receipts/mqtt-load-balancer.yml

The load balancer forwards all TCP traffic for ports 1883 and 8883.

For MQTT over SSL, follow the MQTT over SSL guide to configure transport/tb-mqtt-transport.yml.

9.3 Configure CoAP load balancer (optional)

kubectl apply -f receipts/coap-load-balancer.yml

The load balancer forwards UDP traffic for ports 5683 (CoAP non-secure) and 5684 (CoAP secure DTLS).

For CoAP over DTLS, follow the CoAP over DTLS guide to configure transport/tb-coap-transport.yml.

9.4 Configure LwM2M load balancer (optional)

kubectl apply -f receipts/lwm2m-load-balancer.yml

The load balancer forwards UDP traffic for ports 5685–5688.

For LwM2M over DTLS, follow the LwM2M over DTLS guide to configure transport/tb-lwm2m-transport.yml.

9.5 Configure Edge load balancer (optional)

kubectl apply -f receipts/edge-load-balancer.yml

The load balancer forwards all TCP traffic on port 7070.

Step 10. Configure Trendz (optional)

10.1 Pull Trendz images

docker pull thingsboard/trendz:1.15.1
docker pull thingsboard/trendz-python-executor:1.15.1

10.2 Create a Trendz database in the existing Cloud SQL instance

Edit trendz/trendz-secret.yml and replace YOUR_DB_IP_ADDRESS and YOUR_DB_PASSWORD, then apply:

kubectl apply -f ./trendz/trendz-secret.yml
kubectl apply -f ./trendz/trendz-create-db.yml

Check logs:

kubectl logs job/trendz-create-db -n thingsboard

10.3 Trendz starting

./k8s-deploy-trendz.sh

After this command finishes you should see:

Trendz installed successfully!

Step 11. Validate the setup

Validate Web UI access

kubectl get ingress

Validate MQTT/CoAP access

kubectl get service

Troubleshooting

kubectl logs -f tb-node-0

See the kubectl Cheat Sheet for more details.

Upgrading

Upgrading to new ThingsBoard version

./k8s-delete-resources.sh
./k8s-upgrade-tb.sh --fromVersion=[FROM_VERSION]
./k8s-deploy-resources.sh

Where FROM_VERSION is the starting version. See Upgrade Instructions for valid values. Upgrade versions one by one.

Upgrading to new Trendz version (optional)

git pull origin master
./k8s-upgrade-trendz.sh

Cluster deletion

Delete ThingsBoard pods and load balancers:

./k8s-delete-resources.sh

Delete all data including database:

./k8s-delete-all.sh