As folks adopt DevOps principals they are using common applications to help them get there. One of those is Docker and usually in the same sentence Kubernetes is mentioned next. To review, Docker is essentially a wrapper for Linux containers (LXC), which similar to FreeBSD jails or Solaris Zones, provides a method for applications (and their dependencies) to be isolated in separate namespaces all while sharing the host system’s kernel. Docker containers are extremely portable as they just need the host server to have a LXC-compatible kernel and the Docker application installed. Kubernetes takes this concept to the next level by automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. For a more detailed intro on what Kubernetes is check out the “Sources” section below. Now to the meat of the post, what is NetApp Trident and where does it fit in to the Docker/Kubernetes equation? Well according to NetApp Trident’s GitHub page, “Trident provides storage orchestration for Kubernetes, integrating with its Persistent Volume framework to act as an external provisioner for NetApp ONTAP, SolidFire, and E-Series systems. Additionally, through its REST interface, Trident can also provide storage orchestration for non-Kubernetes deployments.” In other words, Trident allows one to attach persistent storage from NetApp FAS, E-Series, or SolidFire system(s) to containers allowing applications such as databases to easily operate in a containerized environment. Below are the steps I compiled needed to not only stand up a small 3-node Kubernetes cluster but also to deploy the NetApp Trident plugin:

• Kubernetes Host Preparation

  • Provision 3+ CentOS 7.x VMs with the latest updates (minimal package bundle recommended)

    • Server #1 - Master
    • Server #2-3 - Slave (aka minions)

  • Disable the firewalld service

    # systemctl stop firewalld && systemctl disable firewalld
    

  • Disable SELinux

    # setenforce permissive
    # vi /etc/sysconfig/selinux
    NOTE: Change "enforcing" to "disabled" and reboot
    

  • Set default version of NFS equal to version 3

    # sed -i 's/# Defaultvers=4/Defaultvers=3/g' /etc/nfsmount.conf
    

  • Define the following yum repos

    • docker

      # cat << EOF > /etc/yum.repos.d/docker.repo
      [dockerrepo]
      name=Docker Repository
      baseurl=https://yum.dockerproject.org/repo/main/centos/7/
      enabled=1
      gpgcheck=1
      gpgkey=https://yum.dockerproject.org/gpg"
      EOF
      NOTE: Due to compatability concerns it is recommended to use the CentOS-mainline version of Docker (currently 1.12 as of 2017-08-12)
      

    • kubernetes

      # cat << EOF > /etc/yum.repos.d/kubernetes.repo
      [kubernetes]
      name=Kubernetes
      baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
      enabled=1
      gpgcheck=0
      repo_gpgcheck=0
      EOF
      

  • Install the following RPM packages

    # yum install -y docker kubelet kubeadm iptables-services
    

  • Enable the following services to start at boot

    # systemctl enable docker && systemctl enable kubelet
    

  • Start the docker service

    # systemctl start docker
    

  • Login to the target NetApp ONTAP cluster and create an export policy that will be used by all the Kubernetes cluster nodes

    # export-policy create -vserver -policy prod_kubernetes_nodes
    # export-policy rule create -vserver -policy prod_kubernetes_nodes -rorule any -rwrule any -superuser any -protocol nfs -clientmatch <Kubernetes_Node#1_Storage_IP>
    # export-policy rule create -vserver -policy prod_kubernetes_nodes -rorule any -rwrule any -superuser any -protocol nfs -clientmatch <Kubernetes_Node#2_Storage_IP>
    # export-policy rule create -vserver -policy prod_kubernetes_nodes -rorule any -rwrule any -superuser any -protocol nfs -clientmatch <Kubernetes_Node#3_Storage_IP>
    

• Kubernetes Cluster Setup / Initialization
  • Setup the master node

    • Login to the Kubernetes Master server via SSH

    • Initialize the cluster using kubeadm

      # kubeadm init --pod-network-cidr=10.244.0.0/16
      # mkdir -p $HOME/.kube
      # cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      # chown $(id -u):$(id -g) $HOME/.kube/config
      NOTE: Save the "kubeadm join" command as the token will be used to join the other nodes to the new cluster.
      # kubectl get pods --all-namespaces
      NOTE: All should be in a "Running" state, if they are not (i.e. ContainerCreating) run the following commands:
          # kubectl describe pods | grep -i ContainerCreating
          # kubectl apply -f https://git.io/weave-kube
          # kubectl get pods --all-namespaces
      

    • Install a pod network (flannel)

      # kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
      

  • Setup the minion nodes

    • Login to the Kubernetes Slave servers (minions) via SSH

    • Join each of these servers to the cluster

      # kubeadm join --token=blahblahblah
      

    • Navigate to the SSH session for the master server and verify the state of the cluster

      # kubectl get nodes
      # kubectl get pods --all-namespaces
      NOTE: All should be in a "Running" state
      # kubectl describe svc
      

  • Dashboard (GUI) Setup

    • Create a service account and role binding for the user that will be used to access the dashboard

      # vi dashboard_service_account.yaml
      apiVersion: v1
      kind: ServiceAccount
      metadata:
      name: admin-user
      namespace: kube-system
      # vi dashboard_service_account_role_binding.yaml
      apiVersion: rbac.authorization.k8s.io/v1beta1
      kind: ClusterRoleBinding
      metadata:
      name: admin-user
      roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
      subjects:
      

• kind: ServiceAccount name: admin-user namespace: kube-system

kubectl create -f dashboard_service_account.yaml

kubectl create -f dashboard_service_account_role_binding.yaml

    - Record the value after "token:" for the user we just created which we will use when logging into the dashboard

      ```sh
      # kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

      ```

    - Deploy the "dashboard" pod which provides a GUI to manage the cluster

      ```sh
      # kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
      ```

    - From your workstation, you can access the GUI by running the following steps
      - NOTE: These steps assume the "kubectl" binary along with the "/etc/kubernetes/admin.conf" file from the Master server has been download to the workstation and stored under $HOME/.kube/ and has the filename of "config"
      - Setup a proxy session which will allow the API server to be accessed for authentication

        ```sh
        # kubectl proxy
        ```

      - Open up a web browser from your workstation and browse to the following URL
        - Setup a proxy session which will allow the API server to be accessed for authentication

          ```sh
          # http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
          NOTE: You will need to select "Token" and enter in the token value that we captured a few steps back.
          ```
- NetApp Trident Storage Orchestration
  - From the master node download and extract the latest version of the NetApp Trident tarball from [here](https://github.com/NetApp/trident/releases)
  - Change the current path to the NetApp Trident directory

    ```sh
    # cd trident-installer
    ```

  - Copy "sample-input/backend-ontap-nas.json" to "setup" and modify it as needed

    ```sh
    # cp sample-input/backend-ontap-nas.json setup/backend.json
    # vi setup/backend.json
    {
     "version": 1,
     "storageDriverName": "ontap-nas",
     "managementLIF": "10.0.10.16",
     "dataLIF": "10.0.13.16",
     "svm": "mn1_prod_svm_files",
     "username": "svc-netappdvp",
     "password": "PASSWORD_GOES_HERE"
    }
    NOTE: See the "https://github.com/NetApp/trident" link below for details on how to create an SVM account for Trident.
    ```

  - While still logged in to the aforementioned NetApp system via SSH, temporarily set the default export-policy rule to allow RW/Root to all clients (Required by Trident install)

    ```sh
    # export-policy rule modify -vserver -policy default -rorule any -rwrule any -superuser any -protocol any -ruleindex 1 -clientmatch 0.0.0.0/0
    ```

  - Start trident the installer

    ```sh
    # ./tridentctl install --dry-run -n trident
    # ./tridentctl install -n trident

    ```

  - Verify trident deployed successfully

    ```sh
    # kubectl logs -n trident trident-launcher -f
    # kubectl get pods --all-namespaces --watch=true
    <CTRL+C>
    ```

  - Copy the "tridentctl" binary to a path in the $PATH directory

    ```sh
    # cp tridentctl /usr/local/bin/
    # chmod +x /usr/local/bin/tridentctl
    ```

  - Register the backend with Trident

    ```sh
    # tridentctl create backend -n trident -f setup/backend.json
    ```

  - Login to the aforementioned NetApp system via SSH and revert the default export-policy rule to allow ONLY RO to all clients

    ```sh
    # export-policy rule modify -vserver -policy default -rorule any -rwrule none -superuser none -protocol any -ruleindex 1 -clientmatch 0.0.0.0/0 # vol modify -vserver -volume trident_trident
    ```

  - Copy "sample-input/storage-class-basic.yaml.templ" to "setup/storage\_class-default\_dburkland.yaml" and modify it as needed

    ```sh
    # cp sample-input/storage-class-basic.yaml.templ setup/storage_class-default_dburkland.yaml
    # vi setup/storage_class-default_dburkland.yaml
    apiVersion: storage.k8s.io/v1beta1
    kind: StorageClass
    metadata:
     name: default-dburkland
    provisioner: netapp.io/trident
    parameters:
     backendType: "ontap-nas"
    ```

  - Create the new storage class using the file created in the previous step

    ```sh
    # kubectl create -f setup/storage_class-default_dburkland.yaml
    ```

  - Define an example persistent volume claim (PVC)

    ```sh
    # vi setup/pvc-dburkland_harvest_carbon.yaml
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
     name: var-lib-carbon
     annotations:
     volume.beta.kubernetes.io/storage-class: default-dburkland
     trident.netapp.io/exportPolicy: "prod_kubernetes_nodes"
     trident.netapp.io/snapshotPolicy: "default-dburkland"
    spec:
     accessModes:
 - ReadWriteMany
 resources:
 requests:
 storage: 5Gi

• Create an example PVC

  # kubectl create namespace harvest
  # kubectl create -f setup/pvc-dburkland_harvest_carbon.yaml
  # kubectl get pvc --all-namespaces
  

  • NOTE: If you are interested in deploying an example application that utilizes NetApp Trident click here

• Troubleshooting Commands

  # kubectl get namespace
  # kubectl get pods --all-namespaces
  # kubectl get pvc --all-namespaces
  # kubectl describe
  # kubectl logs
  

• Sources

  • https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#kubeadm-maturity
  • https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/
  • https://github.com/kubernetes/dashboard
  • https://github.com/NetApp/trident
  • https://github.com/kubernetes/kubernetes/issues/35378
  • https://github.com/kelseyhightower/kubernetes-the-hard-way
  • https://kubernetes.io/docs/user-guide/docker-cli-to-kubectl