Setup local Airflow running on Minikube

Setup local Airflow running on Minikube

This note assumes OSX as development environment.

Requirements

• brew
• GCloud SDK
• Docker for Mac

Configuring Minikube

Install Minikube

brew cask install minikube

Install Kubectl

brew install kubernetes-cli

Start Minikube cluster with hyperkit (default VM driver).

minikube start

If you are running on a virtual machine, which does not support hyperkit natively, please install virtualbox

brew cask install virtualbox

then start Minikube cluster again with virtualbox as driver.

minikube start --vm-driver=hyperkit

The ~/.kube/config contains configurations of clusters so that kubectl can interact with. Let use Minkube instead of any other clustering contexts.

kubectl config use-context minikube

Review status of Minikube cluster

minikube status

Cross validate that kubectl is configured to communicate with youre expected Kubernetes on Minikube cluster

kubectl cluster-info

Open Kubernetes dashboard

minikube dashboard

Preparing Airflow image

Our customized Airflow image is on Google Container Registry,so that you need an authorized Google account. There are two options

• (Not recommended !!!!) Setting up Docker environment on a VM running within Minikube cluster is an option but you need to re-setup everytime after the VM is purged or corrupted.

    minikube ssh
    # Setting up Docker environment to build / pull images
    . . .
    

• Let set the Docker environment from the host machine to that Minikube VM

    eval $(minikube docker-env)
    

Then 

    gcloud docker --pull eu.gcr.io/bb-analytics-1/airflow:878afc8
    

If your docker client is above 18.03, the command gcloud dockeris not supported. Here it is [deprecation-notices](https://cloud.google.com/container-registry/docs/support/deprecation-notices). In that case:

    gcloud auth configure-docker
    docker pull eu.gcr.io/bb-analytics-1/airflow:878afc8
    

Initializing Helm

brew install kubernetes-helm

Initialize the local CLI and also install Tiller into your Kubernetes cluster.

helm init

Let create a service account named Tiller and give its permission to create/delete resources on a cluster, namely cluster-admin

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'      
helm init --service-account tiller --upgrade

Note that, Helm is CLI and Tiller is server side of Helm Chart.

Deploying Airflow on Minikube

• Mounting DAGs folders Mounting a DAGs folder into the Minikube

    minikube mount ~/Workspace/src/github.com/tv4/data-airflow-dags/dev:/dags
    

This folder with be forwarded (mounted) into the running pod.

• Deploying Airflow with Helm

 
    helm dep update ./charts/airflow
    helm dep build ./charts/airflow
    helm upgrade --install airflow ./charts/airflow --values ./charts/airflow/values-local.yaml
    

Cleaning

Optionally, force removal of the Docker images created:

docker rmi eu.gcr.io/bb-analytics-1/airflow:878afc8 -f

Stop the Minikube VM:

minikube stop
eval $(minikube docker-env -u)

Optionally, delete the Minikube VM to save disk space

minikube delete