17 March 2024

Guide: Preview environments in Kubernetes Engine Google Cloud

This is a guide to show you how we migrated our review apps from Heroku to Google Cloud.

Thanks to this migration we have full control of our review apps and we can control every step of the installation, plus Kubernetes lets us control each pod's resources, and is easy to scale.Preparing the DNSWe are going to use Cloudflare as our DNS provider so we need to create two types of DNS: A and CNAME.Create a DNS type A and use review for the NAME but let it keep empty for content, you will fill it later.After that create a DNS type CNAME and use *.review for the name with the following content: review.your_company.com. Notice the asterisk, this will be needed so you can create URLs like reviewapp1.review.your_company.comSetting up the clusterResearch your preferred cloud provider, then go ahead and create a cluster with an installed gateway API.Prerequisites

Install Helm in your machine

A Kubernetes cluster

Cert Manager and Gateway InstallationOnce you have your cluster, we are going to install cert-manager which is required to issue the certificates for your preview environments and access through HTTPS.I'm going to show you how to do it with Cloudflare, but if you choose use another provider, the cert manager site has plenty of helpful docs.The following command will install cert-manager in its own namespace with the gateway API support enabled:helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set "extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}"After that, we are going to install the Custom Resource Definitions. This helped me to avoid running errors with cert-manager pods.helm upgrade cert-manager jetstack/cert-manager --namespace cert-manager --set installCRDs=trueIt's time to install the next resources and we are going to need a Cloudflare token generated with special settings. Go to Cloudflare > Profile > API Tokens > API Tokens and create the token with these settings:

Permissions:

Zone - DNS - Edit
Zone - Zone - Read

Zone Resources

Include - All Zones

We have to push this secret to our cluster inside the cert-manager namespace

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: <CLOUDFLARE_TOKEN>

cloudflare-api-secret.yaml

Save it and apply it kubectl apply -f cloudflare-api-token.yamlNext step is to create the issuer that will generate the certificate for our review apps, and we need to install a cluster issuer so we can install review apps in different namespaces. It will use letsencrypt.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: acme-issuer
  namespace: cert-manager
spec:
  acme:
    email: <YOUR_CLOUDFLARE_EMAIL>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: example-issuer-account-key
    solvers:
    - dns01:
        cloudflare:
          email: <YOUR_CLOUDFLARE_EMAIL>
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token

clusterissuer.yaml

Save it and apply it kubectl apply -f cluster-issuer.yamlFinally the gateway, that will create a load balancer with a public IP that will be needed for setting the DNS:

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  namespace: default
  name: external-http
  annotations:
    cert-manager.io/cluster-issuer: acme-issuer
spec:
  gatewayClassName: <GATEWAY_CLASS_NAME>
  listeners:
    - name: https
      hostname: <WILDCARD_HOSTNAME>
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - name: review-cert
            kind: Secret
            group: ""

gateway.yaml

You have to replace <GATEWAY_CLASS_NAME> with one provided by your cluster. In Google Cloud I used gke-l7-global-external-managed so, as I said, it will depend on your cluster.<WILDCARD_HOSTNAME> is the DNS type CNAME we set in Cloudflare, e.g. it could be something like *.review.your_company.com.Save it and apply it kubectl apply -f gateway.yamlAfter some time you should be able to get a public IP when inspecting the gateway: kubectl get gateway and that IP should be used as the content for the DNS type A you created before."Helmifiying" your applicationIf you have a dockerised application you can create a custom helm chart to make it easy to run it in the Kubernetes cluster you have created.Run the following command to create your chart and then you can start adding the bits you need for your app:helm create YOUR_APPYou can add dependencies, that are other charts, to your application using the Charts.yaml file e.g. add databases as dependencies

dependencies:
  - name: redis
    version: 18.6.1
    repository: https://charts.bitnami.com/bitnami
  - name: memcached
    version: 6.7.1
    repository: https://charts.bitnami.com/bitnami
  - name: postgresql
    version: 13.2.25
    repository: https://charts.bitnami.com/bitnami
  - name: elasticsearch
    version: 19.13.14
    repository: https://charts.bitnami.com/bitnami

Charts.yaml

The advantage is that these charts are highly customised so in the values.yaml file you can configure them as you wish:

redis:
  image:
    tag: 6.2
  architecture: standalone
  auth:
    enabled: false
  master:
    persistence:
      enabled: false

values.yaml

Build the dependencies helm dependency build and it's time to configure your templates.Inside the templates folder you have to configure all the Kubernetes yaml files that your app will have, my review app has deployments, jobs and config maps.There are two special yaml files that we need for connecting with the gateway, first your app needs to get accessed by people so we need to create a Service to expose the app, something like the following would work but depends on your needs:

apiVersion: v1
kind: Service
metadata:
  name: hubbado-web
spec:
  selector:
    app: hubbado-web
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 3000
  - name: http
    port: 9393
    protocol: TCP
    targetPort: 9393

service-web.yaml

And now we need the HTTPRoute

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: hubbado-web-http-route
spec:
  parentRefs:
  - kind: Gateway
    name: external-http
    namespace: default
  hostnames:
  - "<YOUR_BRANCH_NAME>.<WILDCARD_HOSTNAME>"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: hubbado-web
      port: 443

http-route.yaml

This will connect the Gateway with the Service . Additionally you could need a HealthCheckPolicy

apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: hubbado-web-healthcheck
  namespace: {{ .Release.Namespace }}
spec:
  default:
    checkIntervalSec: 30
    logConfig:
      enabled: true
    config:
      type: HTTP
      httpHealthCheck:
        port: 9393
        requestPath: /_liveness
  targetRef:
    group: ""
    kind: Service
    name: hubbado-web

health-check-policy-web.yaml

Why do we need this health check? To properly configure all the routing and tell the gateway that everything was working.You will need more configurations depending on your application.Now it is time to store your custom chart in a registry that allows installation later on. In our case, we used Github packages to push the chart.

Create the package: helm package hubbado-review-app --version "$VERSION_NUMBER"

Logging to the registry: echo "GITHUB_TOKEN" | helm registry login https://ghcr.io/v2 --username GITHUB_USERNAME --password-stdin

The GITHUB_TOKEN needs access to packages scope so be sure to configure it with that permission.

Push the package: helm push hubbado-review-app-$VERSION_NUMBER.tgz oci://ghcr.io/hubbado

Now it is time to use Github actions (or something similar) to deploy the app.Github actionsWe use Github actions as CI and also to deploy our review apps depending on the git branch and I would like to show you a workflow example to share how we do it:

name: Review app - Install

on: workflow_dispatch

jobs:
  build:
    # Configure this step to build your app using docker

  deploy:
    runs-on: self-hosted

    needs: build

    steps:
      - id: 'auth'
        uses: 'google-github-actions/auth@v2'
        with:
          credentials_json: '${{ secrets.GOOGLE_CREDENTIALS }}'

      - id: 'get-credentials'
        uses: 'google-github-actions/get-gke-credentials@v2'
        with:
          cluster_name: ${{ secrets.CLUSTER_NAME }}
          location: ${{ secrets.CLUSTER_LOCATION }}
          project_id: ${{ secrets.PROJECT_ID }}

      - name: Log in Helm into github package registry
        run: |
          echo ${{ secrets.GITHUB_TOKEN }} | \
          helm registry login https://ghcr.io/v2 \
            --username ${{ github.repository_owner }} \
            --password-stdin

      - name: Pull review app chart
        run: |
          helm pull \
            --untar oci://ghcr.io/hubbado/hubbado-review-app

      - name: Install review app
        run: |
          helm install \
            --namespace ${{ github.ref_name }} \
            --create-namespace review-app hubbado-review-app \
            --set image.branch=${{ github.ref_name }}

review_app-install.yaml

Now wait until your app is deployed and you will have a full review app similar to production where you can test all the changes you want.