Kubernetes cluster setup using virtual machines

Posted 7 years ago by Pavel Balcárek

Preface

Recently I needed to setup k8s cluster on my local machine in order to see if I’m able to do it on bare-metal servers lately. So I decided to first give it a shot using Virtual Box machines. There were a few places where I was a bit stuck and spent some times googling. This article should summarize all of the problems and give you solution.

Most of knowledge used here can be found in official kubernetes documentation. I will also show you how to configure your virtual machines and their network in order to work well with k8s cluster.

If you are new to VirtualBox I suggest to visit official documentation as well as CentOS 7 specific tutorial:

 

Installing and configuring Virtual Box VMs

Our goal is to setup following infrastructure:

Machine name Hostname IP
kubemaster kubemaster 192.168.1.20
kubeslave1 kubeslave1 192.168.1.21
kubeslave2 kubeslave2 192.168.1.22

VirtualBox network configuration

First we need to install VirtualBox. On Ubuntu it’s a simple sudo apt install virtualbox command.

Now we are going to configure our host-only network, which will be used inside of VirtualBox environment. Open VirtualBox, go to File -> Preferences -> Network -> Host-only Network. If there is nothing configured yet, add new Host Only network(if you already have something configured, you need to either change IP as in this tutorial or make note what IPs to use later). You can now disable DHCP inside of the settings and also set IPv4 address to 192.168.99.1.

Once you have this configured, let’s create some VMs.

Create base virtual machine

Given all our machines will need the same base pieces of sw installed before setting up k8s, we are going to create just one VM now and clone the others later. We will use CentOS 7 as a OS for all VMs(K8s official tutorial uses them aswell, so you should be fine with this choice). Get ISO from https://www.centos.org/download/ . Minimal ISO is perfectly fine for our purpose. Create new virutal machine, called kubemaster, with at least following resources:

  • 1 CPU
  • 2 GB RAM

Assign host only network to second adapter and add downloaded CentOS iso file to storage.

Boot up the machine and install CentOS. In this tutorial, I’m using root accounts only on all 3 VMs. This is not recommended, but we will do it just to make it simpler now.

Once you have your system running, do following:

  • Update packages
    yum update
  • Install wget
    yum install wget
  • Set IP address for host only network with
    nmtui

    (usually it’s sitting on enp8s0 adapter)

  • Set hostname
    hostnamectl set-hostname kubemaster
  • Disable firewall
    systemctl disable firewalld && systemctl stop firewalld
  • Disable Selinux – http://idroot.net/tutorials/how-to-disable-selinux-on-centos-7/
  • Add following records to /etc/hosts:
    • 192.168.99.20 kubemaster.test.com kubemaster
    • 192.168.99.21 kubeslave1.test.com kubeslave1
    • 192.168.99.22 kubeslave2.test.com kubeslave2

Now you are ready to install base kubernetes pieces. This is from k8s tutorial mentioned above:

cat < /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF setenforce 0 yum install -y docker kubelet kubeadm kubectl kubernetes-cni systemctl enable docker && systemctl start docker systemctl enable kubelet && systemctl start kubelet

This should install all dependencies we need for k8s cluster.

Clone machines

Now create a linked clone machines from kubemaster machines created before. Once you’re done, boot into machine and change following things to match infrastructure:

  • Set IP address 192.168.99.21 (or 22 for second slave) for host only network.
  • Set hostname hostnamectl set-hostname kubeslave1 (or kubeslave2 for second slave) Everything else is already configured.

Setup Kubernetes cluster

Now we are going to setup whole cluster in just few steps.

Boot all 3 machines up.

Now we need to ensure that hostname matches our host only network ip. You can do it with hostname -i and it should return IP of machine. If that’s not the case, just try to restart VMs or check your network configuration.

Now on kubemaster, run following set of commands:

  • Get jq utility
    wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 && mv jq-linux64 /usr/bin/jq && chmod +x /usr/bin/jq
  • Initialize cluster and copy connect command
    kubeadm init --api-advertise-address=192.168.99.20
  • Workaround for kube-dns problem with multiple network interfaces(run on master node):
    kubectl -n kube-system get ds -l "component=kube-proxy" -o json | jq ".items[0].spec.template.spec.containers[0].command |= .+ [\"--proxy-mode=userspace\"]" | kubectl apply -f - && kubectl -n kube-system delete pods -l "component=kube-proxy"
  • Install network layer for k8s
    kubectl apply -f https://git.io/weave-kube
  • Connect every node with command saved from kubeadm init

And that is it. To ensure that your cluster is OK, try following:

[root@kubemaster ~] # kubectl get nodes
NAME         STATUS         AGE
kubemaster   Ready,master   1h
kubeslave1   Ready          1h
kubeslave2   Ready          1h
[root@kubemaster ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                 READY     STATUS    RESTARTS   AGE
kube-system   dummy-2088944543-mfrw9               1/1       Running   0          1h
kube-system   etcd-kubemaster                      1/1       Running   0          1h
kube-system   kube-apiserver-kubemaster            1/1       Running   5          1h
kube-system   kube-controller-manager-kubemaster   1/1       Running   0          1h
kube-system   kube-discovery-1769846148-pjdlm      1/1       Running   0          1h
kube-system   kube-dns-2924299975-b9nqt            4/4       Running   0          1h
kube-system   kube-proxy-4lltt                     1/1       Running   0          1h
kube-system   kube-proxy-8qhm6                     1/1       Running   0          1h
kube-system   kube-proxy-dgxgj                     1/1       Running   0          1h
kube-system   kube-scheduler-kubemaster            1/1       Running   0          1h
kube-system   weave-net-8tnbk                      2/2       Running   1          1h
kube-system   weave-net-b6fdl                      2/2       Running   0          1h
kube-system   weave-net-f59vj                      2/2       Running   0          1h

Pavel Balcárek

24 Responses to “Kubernetes cluster setup using virtual machines”

  1. Navid Al Hossain says:

    Could you suggest a guide to configure network on a CentOS VM? – Thanks!

  2. […] My issue is that I think I am lacking understanding of how to connect my VMs to one another beforehand. This is the resource I am using for the Kubernetes installation: https://www.profiq.com/kubernetes-cluster-setup-using-virtual-machines/ […]

  3. “–api-advertise-addresses” should be “–api-advertise-address”

    • Pavel Balcárek says:

      Looking to https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#24-initializing-your-master you are indeed right. I don’t know if it’s a typo or that flag changed since time of writing this post. Will change it.

      Cheers.

      • Venkat Padmanabhan says:

        Thanks for the excellent guide, I just finished installing 2 node cluster using k8s 1.9. I had to change some things to get it working.
        1) –apiserver-advertise-address (not api)
        2) Switched to use flannel. Weave net did not work for me.

        Cheers!

        • Pavel Balcárek says:

          Hello Venkat. Thanks for you comment and for sharing things you needed to change – I will try to update guide. I was using kube 1.5 or 1.6 in time of writing this post.

          Have a nice day!

  4. Soumya says:

    Hi,
    Running the command “Workaround for kube-dns problem” gives me an error :
    Error validating STDIN : ApiVersion, Kind not set

    Even if I turn validate off with –validate=false, I get the foll error:
    Unable to get type info from object “unstructured.Unstructured” : Object ‘Kind’ is missing in object has no kind field

    Any idea what this could be about?

    • Pavel Balcárek says:

      Hello Soumya, To be honest, i don’t really know what that can be about. What version of Kubernetes are you trying to deploy?

  5. Bill Timmins says:

    I added a sed s/kind/Kind/ to the pipe chain and the error went away, but not sure if everything is working yet

    • Yizhuan says:

      Bill, which part did you add the sed to ?

      • Bill Timmns says:

        Sorry Yizhuan, it’s been a few months ago now, but I think it was in the pipe chain in the Workaround for kubedns as listed just above, when the guy is complaining about kind and Kind

      • Bill Timmins says:

        Oh and that workaround of the kind/Kind issue in the Workaround for kube-dns did work. My slaves are working nicely now

  6. Rob says:

    Thanks for the guide, much appreciated. But couldn’t this be done with a script? Or better yet, can this be coded up into something like Terraform so there is a clear pathway to running this for staging and prod?

  7. ami says:

    hi ,

    got the following error : “bash: /etc/yum.repos.d/kubernetes.repo: No such file or directory” once trying to install the kubernetes pieces . can you please help ?

    Thanks

    • Pavel Balcárek says:

      You might need to run this as superuser ->

      cat < /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF setenforce 0 yum install -y docker kubelet kubeadm kubectl kubernetes-cni systemctl enable docker && systemctl start docker systemctl enable kubelet && systemctl start kubelet

      Either add sudo before each call or simply do sudo su and continue with whole command.

    • André says:

      I had the same problem.
      you need to write the repository as shown in these images from this link
      https://www.howtoforge.com/tutorial/centos-kubernetes-docker-cluster/
      I solved it: D

  8. Jayanth Acharya says:

    Great tutorial Andre. Considering following it to setup a K8s cluster with 3-4 VMs on my host (i7 8th gen, 12 core, 32GB RAM — so plenty of resources, I hope), and your tutorial should come in handy. However, do you suppose that due to 2years having passed between the original article and now, K8s have evolved so much that things may not work as expected ? If you’ve been keeping track of K8s evolutions, any specific areas where you suspect that one could anticipate issues ?

  9. Assaduzzaman Assad says:

    Hi Andre, I tried the steps on Ubuntu and creat vm and use host only network. Since it is host only network my vm not able to connect with internet. Cloud you please told me how you get access thing on internet.

  10. Mirketa says:

    Thanks for sharing

  11. Alagar says:

    Hi Pavel,

    How to deploy a docker with kubernetes orchestration on ubuntu 18.4 LTS, i was installing on Vmware workstation 14 but network doesn’t work.

    root@kube-master:~# kubeadm init –apiserver-advertise-address $(hostname -i) –pod-network-cidr=192.168.0.0/16
    unable to select an IP from lo network interface
    To see the stack trace of this error execute with –v=5 or higher
    root@kube-master:~# kubeadm init –apiserver-advertise-address $(hostname -i) –pod-network-cidr=192.168.0.0/16 –v=5
    I1228 18:19:08.563781 19314 initconfiguration.go:103] detected and using CRI socket: /var/run/dockershim.sock
    I1228 18:19:08.564121 19314 interface.go:208] Interface lo is up
    I1228 18:19:08.564212 19314 interface.go:256] Interface “lo” has 2 addresses :[127.0.0.1/8 ::1/128].
    I1228 18:19:08.564240 19314 interface.go:223] Checking addr 127.0.0.1/8.
    I1228 18:19:08.564249 19314 interface.go:233] Non-global unicast address found 127.0.0.1
    I1228 18:19:08.564260 19314 interface.go:223] Checking addr ::1/128.
    I1228 18:19:08.564267 19314 interface.go:236] ::1 is not an IPv4 address
    I1228 18:19:08.564666 19314 interface.go:208] Interface lo is up
    I1228 18:19:08.564732 19314 interface.go:256] Interface “lo” has 2 addresses :[127.0.0.1/8 ::1/128].
    I1228 18:19:08.564769 19314 interface.go:223] Checking addr 127.0.0.1/8.
    I1228 18:19:08.564780 19314 interface.go:236] 127.0.0.1 is not an IPv6 address
    I1228 18:19:08.564788 19314 interface.go:223] Checking addr ::1/128.
    I1228 18:19:08.564796 19314 interface.go:233] Non-global unicast address found ::1
    unable to select an IP from lo network interface

  12. AYUSH KUMAR says:

    I have configured kubenetes cluster and single worker node on oracle virtual box centos 7 vm’s. master node is fine but while connecting worker node it is always throwing below error. Please help on this. I have setup my host only adpator ip address as: 192.168.1.1 , master: 192.168.1.7, worker: 192.168.1.5 . master and worker can ping each other but any one can’t ping to 192.168.1.1

    Please help me on this.

    kubeadm join 192.168.1.7:6443 –token ur5tee.f40lpx85l11u7gxb –discovery-token-ca-cert-hash sha256:d6cc5a3945136b2f3931cabee4d0bdc2d9e18daa66c744067ffc217494bb6ede
    W0130 00:37:19.340456 8194 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
    [preflight] Running pre-flight checks
    [preflight] Reading configuration from the cluster…
    [preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
    [kubelet-start] Downloading configuration for the kubelet from the “kubelet-config-1.17” ConfigMap in the kube-system namespace
    [kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
    [kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
    [kubelet-start] Starting the kubelet
    [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…
    [kubelet-check] Initial timeout of 40s passed.
    [kubelet-check] It seems like the kubelet isn’t running or healthy.
    [kubelet-check] The HTTP call equal to ‘curl -sSL http://localhost:10248/healthz‘ failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.

  13. louis says:

    Thanks for Pavel, may i know is it possible with the bare metal machine that have a single interface only and use vlan to create multiple virtual interfaces?

  14. Great Blog! I have learned many things from this article. Thanks for sharing this excellent blog with us and posting on

    Iqra Technology

Leave a Reply

Related articles