Speed Up Local Test Kitchen runs

I had an initial thought and that was whether it would be possible to create a Squid proxy Docker container and configure Test Kitchen and related cookbooks to use it. This should be simple right? Wrong!

At first it was, with simple HTTP requests, but then came the issue if making Squid intercept and cache SSL objects. This meant:

  • Understanding how Squid can intercept SSL traffic
  • Squid package in Ubuntu is not compiled with SSL or the option to generate dynamic certificates
  • Compile own Squid with these two options
  • Create new Docker image using the new Squid
  • Update the CA certificates in Ubuntu
  • Rebuild image
  • Push image to dockerhub
  • Create self signed CA for Squid to create certificates with

This last one is where things are not as clean as they could be and that although I have created something that does work, it is not as perfect as I would like it to be and there are some caveats. Needless to say I have been doing a lot of yak shaving.

Reasoning

You may well be asking why I bothered doing any of this. When I first had the idea, it was to combat one scenario and that was to be able to continue cookbook development when I was not connected to the Internet, primarily for travelling. However when I was working on it I started to realise that it would also speed up local development and also reduce bandwidth.

So if I have a container running tht runs Squid and its proxy port is exposed then all my virtual machines and containers that I run can be configured to use it. So now when I have several things running and they are downloading things, Squid will cache it for me so I do not need to download it again thus reducing bandwidth. And as the machines are pulling from a vert near source the download times will be faster as well.

The above diagram shows my initial sketch for the idea.

Requirements

  • Docker

Squid Container

I have pushed the image that I created for this project to the Docker Hub so anyone is welcome to use it - https://hub.docker.com/r/russellseymour/squid3/.

In order to be able to intercept and cache SSL objects, specifically chef-client, Squid needs a Certificate Authority (CA) to work with. As this is for local development it does not have to be valid, although as I mention later on having a self signed CA does have issues.

When the container is created a number of volumes need to be mapped, this is so that items that have been downloaded are persisted between containers. It is a good idea to have these in a central location. For this example I have created the following folders in my home directory.

~\scratch
└───squid_proxy
    ├───data
    └───etc
        └───ssl

NB the ~ means home directory in Unix shells, e.g. bash, zsh etc, and PowerShell. Windows Command prompt does not understand this notation.

The data dir is where all the data that Squid downloads is stored and the etc is the configuration file and SSL certificates.

Create the CA

To create the CA you need to use the openssl command. As intended I have not mentioned anything regarding the platform that needs to run all this, however depending on your platform you may not have openssl available.

NB. If you are using Windows then please ensure ChefDK is installed.

cd ~/scratch/squid_proxy/etc/ssl
openssl req -new -newkey rsa:2048 -sha256 -days 365 -nodes -x509 -extensions v3_ca -keyout squid_ca.key -out squid_ca.pem

Configuration File

As each installation and use of the Squid proxy will be different for each user the container only has the default Squid configuration file installed. This can (and should) be overridden so that it can use the SSL configuration created above.

The following is an example of the configuration I used and should be saved at ~/scratch/squid_proxy/etc/squid.conf.

http_port 3128 ssl-bump cert=/etc/squid/ssl/squid_ca.pem key=/etc/squid/ssl/squid_ca.key generate-host-certificates=on dynamic_cert_mem_cache_size=4MB

sslcrtd_program /usr/bin/ssl_crtd -s /var/spool/squid_ssldb -M 4MB

acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1

acl localnet src 10.0.0.0/8     # RFC 1918 possible internal network

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT

acl step1 at_step SslBump1

ssl_bump peek step1
ssl_bump bump all

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow localnet
http_access deny all

maximum_object_size 1024 MB
cache_replacement_policy heap LFUDA
cache_dir aufs /var/spool/squid 5000 24 256
coredump_dir /var/spool/squid

refresh_pattern ^ftp:          1440    20%     10080
refresh_pattern ^gopher:       1440    0%      1440
refresh_pattern Packages\.bz2$ 0       20%     4320 refresh-ims
refresh_pattern Sources\.bz2$  0       20%     4320 refresh-ims
refresh_pattern Release\.gpg$  0       20%     4320 refresh-ims
refresh_pattern Release$       0       20%     4320 refresh-ims
refresh_pattern -i .(deb|rpm|exe|zip|tar|tgz|bz2|ram|rar|bin)$  129600 100% 129600 override-expire ignore-no-cache ignore-no-store
refresh_pattern .              0       20%     4320

Take note that the path to the CA certificate that was created is from the viewpoint of within the container and not the host machine running it. This will be configured when the container is started.

This configuration should ensure that all files and packages are downloaded and saved locally.

Start the container

Now that all the configuration is in place, the container can be started. As the paths are slightly different for Windows or Mac/Linux the following shows examples for both.

NB. In both cases the container is set to run interactively and remove it when it has stopped. If you wish to put it in the background and let it run remove the -it --rm arguments and replace with -d.

Windows

Run the following command in PowerShell

PS > docker run -it `
                --rm `
                --name squid `
                -v //c/users/russells/scratch/squid_proxy/data:/var/spool/squid `
                -v //c/users/russells/scratch/squid_proxy/etc:/etc/squid `
                -p 3128:3128 `
                russellseymour/squid3

Docker needs to have the path in a Unix like format, so the home path has to be expended. My username is russells so my home directory for Docker is //c/users/russells.

Windows normally runs a local firewall and unless it has been turned off a rule needs to be added to allow access to the exposed 3128 port. Run the following PowerShell command in an elevated prompt.

PS > New-NetFirewallRule -DisplayName "Squid Proxy" -Profile Private -LocalPort 3128 -Protocol tcp

MacOS / Linux

The following command should be run to get the container running on MacOS or Linux.

$> docker run -it \
              --rm \
              --name squid \
              -v ~/scratch/squid_proxy/data:/var/spool/squid \
              -v ~/scratch/squid_proxy/etc:/etc/squid \
              -p 3128:3128 \
              russellseymour/squid3

Test Kitchen

Now that the proxy is running it is possible to configure Test Kitchen and configure the necessary software on the target machine to use the proxy.

Parent image

Astute readers will have realised that as we are using a brand new self signed Certificate Authority there are going to be issues when wget or curl attempt to download the chef-client.

Currently, because it is not possible to tell either of these tools not to perform certificate verification, the parent image that the machine will be created from needs to be modified.

As my testing for this has been with Ubuntu I will show this as an example. On Ubuntu chef-client is downloaded by wget. In order to ignore the certificate from the proxy server --no-check-certificate needs to be appended to the command. As we cannot change the command that is run we have to change the settings of wget.

On the parent image run the following command:

$> echo check_certificate=off >> /etc/wgetrc

Recipe

The whole idea of this is to try and get all components using the proxy server. This means that the local repository for Ubuntu for example needs to be configured. To do this a recipe should be created with the relevant settings in it, recipes/proxy.rb

if node["proxy"]["http"] != "" &&
   node["proxy"]["https"] != ""

   case node["platform"]
   when "ubuntu"

        template "/etc/apt/apt.conf.d/99proxy-apt.conf" do
            source "proxy-apt.conf.erb"
            variables(
                "http_proxy" => node["proxy"]["http"],
                "https_proxy" => node["proxy"]["https"],
            )
        end
    end
end

In this example both the http_proxy and https_proxy have to be set for the settings to be applied to the machine. As this recipe is really only intended to be used as part of the Test Kitchen testing cycle the attributes do not need to be defaulted in the cookbook. However they must be configured in the .kitchen.yml file (see next section).

Although this has only be configured for Ubuntu it is using the node["platform"] attribute to determine how the settings should applied based on what is required. This allows for yum or chocolatey to be configured.

Test Kitchen Configuration

So the last thing to do is add the settings for the download of Chef Client through the proxy and to set the attributes for the recipe.

---
driver:
  name: hyperv
  parent_vhd_folder: C:\VirtualMachines\Hyper-V
  parent_vhd_name: ubuntu-16.04.vhdx
  vm_switch: InternalSwitch
  memory_startup_bytes: 2GB
  http_proxy: http://192.168.137.1:3128
  https_proxy: http://192.168.137.1:3128

provisioner:
  name: chef_zero
  chef_omnibus_url: http://www.getchef.com/chef/install.sh
  require_chef_omnibus: 13.0.118

transport:
  name: ssh
  password: P@ssw0rd!

verifier:
  name: inspec
  format: doc

platforms:
  - name: ubuntu-16.04

suites:
  - name: default
    run_list:
      - recipe[offline_proxy::default]
    attributes:
      proxy:
        http: http://192.168.137.1:3128
        https: http://192.168.137.1:3128

The example shows how I have configured the cookbook to work with Hyper-V, however the settings for the driver and the attributes should be the same for any provider.

It is imperative that the URL used for the Squid server can be addressed by the guest machine or container. In this case I am using http://192.168.137.1:3128 as my Windows 10 machine provides the network on this range and it is the first IP in the range.

Executing Test Kitchen

Now just run the Test Kitchen commands as you normally would and you should find that things start to get cached in the proxy server.

Troubleshooting

During the investigation and writing of this post there are a few things that I have learnt that help with troubleshooting the setup.

Check cache dir is being used

If you are not sure if things are being cached have a look at the size of directory on the host machine that is configured to hold the cache data. In the case of this post this would be ~/scratch/squid_proxy/data.

For Windows this can be done in Explorer by right clicking on the folder and selecting properties:

If on MacOS or Linux then run the command du -hs ~/scratch/squid_proxy/data

Watch the startup of Squid

If you run the container in interactive mode then Squid will show how many items it has loaded from the cache directory.

Tail the access log for Squid

Whenever the proxy is used it will log the access within the /var/log/squid/access.log. This can be tailed by access the container, e.g.

docker exec -it squid /bin/bash
root@125dfggf4f:/ $ tail -f /var/log/squid/access.log

Caveats

I stated at the beginning that there are a few issues with this setup, and whilst I have mentioned them in the main post here they are for easy reference.

  • When running on Windows the local firewall needs to be updated to allow port 3218
  • The CA is a self signed one so images need to be pre-configured with the wget or curl settings to allow them to download without verifying the certificate. This will limit to own machines for the moment
    • Turning off the certificate verification globally may cause the download of undesired packages
  • I have not tested this extensively at all and only with one platform, Ubuntu 16.04. However with the correct configuration in the .kitchen.yml file and setting up proxy on the host, for chocolately for example there is no reason it could not be used with any platform.
  • As it is running locally with an exposed port there is nothing to stop you from connecting your browser(s) to the proxy, however bear in mind that it is explicitly configured to cache SSL objects so you may end up caching your bank details for example.
  • A TK run has to be performed at least once to cache the items, so make sure it is done before you need it. Over time the cache should be become much more useful if you keep it running.

Conclusion

I am pleased I went through this and worked it all out and I am sure that it will be useful. I have not tried it enough on different platforms for me to be super confident about it but that should change over time.

I like the speed increases I am getting now and should continue to do the longer the proxy container works.

This ended up taking much longer and being much more involved than I thought it would be. I have tried to check commands and test everything but if you find any issues please let me know and I will update them.

After writing this I realise that it is not the end of the story and I need to write a follow up blog post, so watch my blog :-).

Share Comments