Blogging on the Blog that Habitat Built

After working with Kubernetes and setting up Cross Cloud Kubernetes I wanted to get this blog site up and running on the cluster. In addition I wanted to enable continuous deployment so I do not have to worry about remembering to perform a release.

The site is already packaged up and deployed as a Habitat package so it made sense to take advantage of the Habitat Updater that Chef developed for Kubernetes.

There are a few moving parts to this and this post will describe how I got it all working.

Motivation

This site has always been built as a Habitat package. I used it as a way to understand how Habitat works. However I have not been using all of the capabilities that it provides, namely Continuous Delivery (CD).

With my shiny new Kubernetes clusters I wanted to get this working properly. To do this I have used the following Chef tools:

Environment

I have two environments configured, they are both Kubernetes clusters with the components in this article installed.

One of the environments is my development cluetsr and runs across 4 machines on my local LAN at home.

The second is my production cluster which is run across 4 machines in different geographical regions across Europe and cloud providers. (Please refer to Cross Cloud Kubernetes for more information regarding this setup.)

Build Process

I currently use Hexo to generate a static site from Markdown files. Hexo is a NodeJS based application that has support for many different plugins and themes.

Unfortunately Habitat does not have a core plan for Hexo so my Habitat plan file has to install Hexo so that the site can be generated. I also need to set Nginx as a dependency so that the site gets served. The downside of this is that Hexo is needlessly bundled into the final package as I cannot declare it as a build dependency.

My plan.sh file is as follows:

pkg_origin=russellseymour
pkg_name=turtleblog
pkg_maintainer="Russell Seymour <russell.seymour@turtlesystems.co.uk>"
pkg_upstream_url=
pkg_deps=(core/nginx)
pkg_build_deps=(core/node core/make)
pkg_exports=(
    [port]=http.listen.port
)
pkg_exposes=(port)
pkg_svc_run="nginx"
pkg_svc_user="root"

trim(){
    [[ "$1" =~ ^[[:space:]]*(.*[^[:space:]])[[:space:]]*$ ]]
    printf "%s" "${BASH_REMATCH[1]}"
}

pkg_version() {

    # Set a local package version
    version="0.0.0"

    # Check to see if version file exists, if it does
    # read into the version variable
    if [ -f $PLAN_CONTEXT/../buildnumber ]
    then
        version=`cat $PLAN_CONTEXT/../buildnumber`
        version=`trim $version`
    fi

    echo $version
}

do_before() {
    do_default_before
    update_pkg_version
}

do_build() {

    # Copy the contents of the src directory onto the CACHE_PATH as this is where
    # habitat expects things to be stored
    mkdir $CACHE_PATH/scaffolds $CACHE_PATH/source $CACHE_PATH/themes
    cp -vr $PLAN_CONTEXT/../src/scaffolds/* $CACHE_PATH/scaffolds
    cp -vr $PLAN_CONTEXT/../src/source/* $CACHE_PATH/source
    cp -vr $PLAN_CONTEXT/../src/themes/* $CACHE_PATH/themes
    cp -vr $PLAN_CONTEXT/../src/_config.yml $CACHE_PATH
    cp -vr $PLAN_CONTEXT/../src/db.json $CACHE_PATH
    cp -vr $PLAN_CONTEXT/../src/package.json $CACHE_PATH

    # Ensure that all the site dependencies are installed
    cd $CACHE_PATH
    npm install

    # Generate the Hexo site
    node node_modules/hexo/bin/hexo generate

}

do_install() {

    # Copy the contents of the build directory to the pkg_prefix so that
    # it can be served by nginx
    echo "Copy from $CACHE_PATH/generated/* to $pkg_prefix"
    cp -r $CACHE_PATH/generated/* ${pkg_prefix}
}

The execution of this plan is performed by Azure DevOps using some of the Habitat Extension tasks:

  • Install Habitat
    • Ensure that the Habitat application is installed on the Azure DevOps agent
  • Signing Origin Key
    • Ensure that the keys associated with my origin are configured on the agent
  • Build Habitat Plan
    • Build the package using the above plan.sh file
  • Expose Build Variables
    • After each successful build Habitat creates an environment file with details about the package, this is read and the values made available to other tasks in the build pipeline

The build output is a Habitat hart file which is used in the release pipeline.

The Azure DevOps YAML file for this build definition is shown below:

pool:
  name: home-agents
  demands: sh
steps:
- task: chef-software.vsts-habitat-tasks-preview.vsts-habitat-tasks-install.vsts-habitat-install-preview@3
  displayName: 'Install Habitat'
- task: chef-software.vsts-habitat-tasks-preview.vsts-habitat-tasks-signing-key.vsts-habitat-signing-key-preview@3
  displayName: 'Signing Origin Key'
  inputs:
    habitatOrigin: 'Russell Seymour Origin 2'
- task: ShellScript@2
  displayName: 'Set Version number'
  inputs:
    scriptPath: setBuildNumber.sh
    args: '$(Build.BuildNumber)'
- task: chef-software.vsts-habitat-tasks-preview.vsts-habitat-tasks-build.vsts-habitat-build-preview@3
  displayName: 'Build Habitat plan'
  inputs:
    habitatOrigin: 'Russell Seymour Origin 2'
- task: chef-software.vsts-habitat-tasks-preview.vsts-expose-habitat-build-vars.vsts-habitat-expose-habitat-build-vars-preview@3
  displayName: 'Expose Habitat Build Variables'
  inputs:
    habitatSetBuildNumber: true
- task: CopyFiles@2
  displayName: 'Copy Files to: $(build.artifactstagingdirectory)'
  inputs:
    SourceFolder: '$(Build.SourcesDirectory)/results'
    Contents: |
     *-$(Build.BuildNumber)-*.hart
     last_build.env
    TargetFolder: '$(build.artifactstagingdirectory)'
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: drop'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)'

Release Process

The Release pipeline is configured with two environments.

  1. Habitat Depot
  2. Docker Hub

These are not the Kubernetes clusters I run on which my applications are deployed. In this case they are targets for where the built Habitat archive should be sent.

Each of the environments uses some tasks from the Azure DevOps Habitat extension

  • Install Habitat
    • Ensure that the Habitat application is installed on the Azure DevOps agent
  • Signing Origin Key
    • Ensure that the keys associated with my origin are configured on the agent
  • Package Upload
    • Upload the HART file to the Habitat Depot using the keys and Habitat Auth Token supplied by the ‘Signing Origin Key’ task
  • Expose Build Variables
    • After each successful build Habitat creates an environment file with details about the package, this is read and the values made available to other tasks in the release pipeline

In the ‘Docker Hub’ environment Docker tasks are used to tag the exported image and upload to Docker Hub. The ‘Expose Build Variables’ task is used to identify which image to upload.

The hart file is published to my origin in the Habitat Depot on the dev channel. This is triggered on every successful build.

Habitat Depot Packages

The image above shows two packages in the Depot for this blog. The version 1.2.66 is in the stable channel, which means it is live and version 1.2.71 has only just been built so it is in the dev channel.

The Docker Hub environment exports the built package as docker image. This is then uploaded to the Docker Hub using my credentials. This image is required for the initial deployment of the application into Kubernetes.

Azure DevOps Release

Habitat Kubernetes Components

As mentioned I use two Chef developed Kubernetes components.

Habitat Operator

The Habitat Operator creates a new type of deployment which signifies to Kubernetes that the image being deployed is a Habitat application. This is why the release process above requires creates a docker image for deployment.

The habitat-operator requires some role based access control (RBAC) to be configured as well as deploying the updater itself. The following two commands show how this can be done from the GitHub repo.

kubectl apply -f https://raw.githubusercontent.com/habitat-sh/habitat-operator/master/examples/rbac/rbac.yml
kubectl apply -f https://raw.githubusercontent.com/habitat-sh/habitat-operator/master/examples/rbac/habitat-operator.yml

The above commands will deploy the habitat-operator into the “default” namespace.

For the deployment of the operator into my Kubernetes clusters I chose to deploy into the kube-system namespace. In hindsight this is probably not the best location for it, but when I was starting with Kubernetes I wanted it in a central location and this name space seemed the most logical choice.

Habitat Operator Pod

Blog Deployment

Now that the operator is installed applications can be installed that are based on Habitat. The following manifest file is an example of the one used to deploy this blog site into the cluster.

apiVersion: habitat.sh/v1beta1 
kind: Habitat
metadata:
  name: blog
customVersion: v1beta2
spec:
  v1beta2:
    image: russellseymour/turtleblog:latest
    count: 1
    env:
      - name: HAB_BLDR_CHANNEL
        value: dev
      - name: HAB_LICENSE
        value: accept
    service:
      name: blog
      topology: standalone
---
apiVersion: v1
kind: Service
metadata:
  name: blog
spec:
  selector:
    habitat-name: blog
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: blog
  labels:
    app: blog
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: blog.home.turtlesystems.co.uk
    http:
      paths:
      - backend: 
          serviceName: blog
          servicePort: 80

This manifest deploys three Kubernetes items:

  1. Docker image of the blog as a Habitat application
    • This image is not publicly available. To access it I have patched the service account in the default namespace with a reference to a Kubernetes secret containing the credentials to access my provate repos in Docker Hub.
    • In order to use Habitat, the license needs to be accepted. To do this the manifest sets the HAB_LICENSE environment to accept so that the application will startup without error.
  2. Service for the blog to answer requests on HTTP
  3. Ingress to allow external access into the application

This is the manifest file for my development Kubernetes cluster, and as such the pod is deployed with the HAB_BLDR_CHANNEL environment variable set to dev. This is the key to allowing me to see all builds of the blog on my development cluster without any intervention, they just appear.

For the production cluster I do not set the HAB_BLDR_CHANNEL environment variable, thus it will only look at the stable channel. This means that operator is only concerned with updates to this channel.

Habitat Updater

Kubernetes does not allow applications to update themselves, so this lightweight application was developed. When running it periodically analyses each Habitat based application in the cluster and determines the channel for that package. It then looks at the Habitat Depot for that package origin and determines if there is an update available. If there is it then informs the supervisor of the running application to perform an update.

Deployment

The Kubernetes manifest for deploying the updater into the cluster is in the Habitat GitHub repo. The following commands show how to deploy it:

# If the cluster is RBAC enabled then the updater needs to have permissions enabled first
kubectl -f https://github.com/habitat-sh/habitat-updater/blob/master/kubernetes/rbac/rbac.yml

# Deploy the updater into the cluster
kubectl -f https://github.com/habitat-sh/habitat-updater/blob/master/kubernetes/rbac/updater.yml

By default the updater is only able to pull packages from the Depot that are public, however this is not always practical as some packages, such as this blog, are not publicly accessible. To enable the updater to pull a private package the manifest needs to be updated to include a HAB_AUTH_TOKEN environment variable which is set to the Habitat account authentication token. For example:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: habitat-updater
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: habitat-updater
    spec:
      containers:
      - name: habitat-updater
        image: habitat/habitat-updater
        env:
        - name: HAB_AUTH_TOKEN
          valueFrom:
            secretKeyRef:
              name: tokens
              key: HAB_AUTH_TOKEN
      serviceAccountName: habitat-updater

In this case the HAB_AUTH_TOKEN environment variable is set from a Kubernetes secret which contains my authentication token.

When the updater has been deployed, it will appear as a pod in the Default namespace, for example:

Habitat Updater Pod

Status

As mentioned previously the updater is constantly checking any Habitat applications in the cluster to see if they are running the correct version. To see that the updater is checking versions of packages run the following command:

kubectl logs habitat-updater-b9996b4f-xv9wh

The output will be similar to the following.

Habitat Updater Polling

Now when an update occurs on either the stable or dev channels the updater will see that there is a newer version and inform the operator that there is a newer version of the application to deploy.

Final word

Overall I am really happy with this setup. I can easily create a new post, check it in and within a few mins see the results on my development cluster. When I happy with it I can promote the package and it deploys to my live site.

However there are a few improvements that I would like to make which would make this process a bit more streamlined.

  • Create a core plan for Hexo so that it is not being bundled in the final package or consider a different static site generator such as Hugo for which there is a core plan
  • Generate the Hexo site outside of habitat and then pull the results into Habitat. This would mean that Hexo does not get packaged into the resulting archive
  • Deploy more than one pod in Production cluster and perform the update sequentially so that I do not have any downtime during updates
Share Comments