Kubernetes cluster setup using virtual machines
Pavel Balcárek
8 years ago 2.3.2017
Preface
Recently I needed to setup k8s cluster on my local machine in order to see if I’m able to do it on bare-metal servers lately. So I decided to first give it a shot using Virtual Box machines. There were a few places where I was a bit stuck and spent some times googling. This article should summarize all of the problems and give you solution.
Most of knowledge used here can be found in official kubernetes documentation. I will also show you how to configure your virtual machines and their network in order to work well with k8s cluster.
If you are new to VirtualBox I suggest to visit official documentation as well as CentOS 7 specific tutorial:
- https://www.virtualbox.org/wiki/Documentation
- https://wiki.centos.org/HowTos/Virtualization/VirtualBox/CentOSguest
Installing and configuring Virtual Box VMs
Our goal is to setup following infrastructure:
| Machine name | Hostname | IP |
|---|---|---|
| kubemaster | kubemaster | 192.168.1.20 |
| kubeslave1 | kubeslave1 | 192.168.1.21 |
| kubeslave2 | kubeslave2 | 192.168.1.22 |
VirtualBox network configuration
First we need to install VirtualBox. On Ubuntu it’s a simple sudo apt install virtualbox command.
Now we are going to configure our host-only network, which will be used inside of VirtualBox environment. Open VirtualBox, go to File -> Preferences -> Network -> Host-only Network. If there is nothing configured yet, add new Host Only network(if you already have something configured, you need to either change IP as in this tutorial or make note what IPs to use later). You can now disable DHCP inside of the settings and also set IPv4 address to 192.168.99.1.
Once you have this configured, let’s create some VMs.
Create base virtual machine
Given all our machines will need the same base pieces of sw installed before setting up k8s, we are going to create just one VM now and clone the others later. We will use CentOS 7 as a OS for all VMs(K8s official tutorial uses them aswell, so you should be fine with this choice). Get ISO from https://www.centos.org/download/ . Minimal ISO is perfectly fine for our purpose. Create new virutal machine, called kubemaster, with at least following resources:
- 1 CPU
- 2 GB RAM
Assign host only network to second adapter and add downloaded CentOS iso file to storage.
Boot up the machine and install CentOS. In this tutorial, I’m using root accounts only on all 3 VMs. This is not recommended, but we will do it just to make it simpler now.
Once you have your system running, do following:
- Update packages
yum update - Install wget
yum install wget - Set IP address for host only network with
nmtui(usually it’s sitting on enp8s0 adapter)
- Set hostname
hostnamectl set-hostname kubemaster - Disable firewall
systemctl disable firewalld && systemctl stop firewalld - Disable Selinux – http://idroot.net/tutorials/how-to-disable-selinux-on-centos-7/
- Add following records to /etc/hosts:
- 192.168.99.20 kubemaster.test.com kubemaster
- 192.168.99.21 kubeslave1.test.com kubeslave1
- 192.168.99.22 kubeslave2.test.com kubeslave2
Now you are ready to install base kubernetes pieces. This is from k8s tutorial mentioned above:
cat < /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF setenforce 0 yum install -y docker kubelet kubeadm kubectl kubernetes-cni systemctl enable docker && systemctl start docker systemctl enable kubelet && systemctl start kubeletThis should install all dependencies we need for k8s cluster.
Clone machines
Now create a linked clone machines from kubemaster machines created before. Once you’re done, boot into machine and change following things to match infrastructure:
- Set IP address 192.168.99.21 (or 22 for second slave) for host only network.
- Set hostname
hostnamectl set-hostname kubeslave1 (or kubeslave2 for second slave)Everything else is already configured.
Setup Kubernetes cluster
Now we are going to setup whole cluster in just few steps.
Boot all 3 machines up.
Now we need to ensure that hostname matches our host only network ip. You can do it with hostname -i and it should return IP of machine. If that’s not the case, just try to restart VMs or check your network configuration.
Now on kubemaster, run following set of commands:
- Get jq utility
wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 && mv jq-linux64 /usr/bin/jq && chmod +x /usr/bin/jq - Initialize cluster and copy connect command
kubeadm init --api-advertise-address=192.168.99.20 - Workaround for kube-dns problem with multiple network interfaces(run on master node):
kubectl -n kube-system get ds -l "component=kube-proxy" -o json | jq ".items[0].spec.template.spec.containers[0].command |= .+ [\"--proxy-mode=userspace\"]" | kubectl apply -f - && kubectl -n kube-system delete pods -l "component=kube-proxy"
- Install network layer for k8s
kubectl apply -f https://git.io/weave-kube - Connect every node with command saved from kubeadm init
And that is it. To ensure that your cluster is OK, try following:
[root@kubemaster ~] # kubectl get nodes NAME STATUS AGE kubemaster Ready,master 1h kubeslave1 Ready 1h kubeslave2 Ready 1h
[root@kubemaster ~]# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system dummy-2088944543-mfrw9 1/1 Running 0 1h kube-system etcd-kubemaster 1/1 Running 0 1h kube-system kube-apiserver-kubemaster 1/1 Running 5 1h kube-system kube-controller-manager-kubemaster 1/1 Running 0 1h kube-system kube-discovery-1769846148-pjdlm 1/1 Running 0 1h kube-system kube-dns-2924299975-b9nqt 4/4 Running 0 1h kube-system kube-proxy-4lltt 1/1 Running 0 1h kube-system kube-proxy-8qhm6 1/1 Running 0 1h kube-system kube-proxy-dgxgj 1/1 Running 0 1h kube-system kube-scheduler-kubemaster 1/1 Running 0 1h kube-system weave-net-8tnbk 2/2 Running 1 1h kube-system weave-net-b6fdl 2/2 Running 0 1h kube-system weave-net-f59vj 2/2 Running 0 1h
Could you suggest a guide to configure network on a CentOS VM? – Thanks!
Hello Navid. Answers you are looking for might be found here: http://ask.xmodulo.com/configure-static-ip-address-centos7.html
[…] My issue is that I think I am lacking understanding of how to connect my VMs to one another beforehand. This is the resource I am using for the Kubernetes installation: https://www.profiq.com/kubernetes-cluster-setup-using-virtual-machines/ […]
“–api-advertise-addresses” should be “–api-advertise-address”
Looking to https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#24-initializing-your-master you are indeed right. I don’t know if it’s a typo or that flag changed since time of writing this post. Will change it.
Cheers.
Thanks for the excellent guide, I just finished installing 2 node cluster using k8s 1.9. I had to change some things to get it working.
1) –apiserver-advertise-address (not api)
2) Switched to use flannel. Weave net did not work for me.
Cheers!
Hello Venkat. Thanks for you comment and for sharing things you needed to change – I will try to update guide. I was using kube 1.5 or 1.6 in time of writing this post.
Have a nice day!
Hi,
Running the command “Workaround for kube-dns problem” gives me an error :
Error validating STDIN : ApiVersion, Kind not set
Even if I turn validate off with –validate=false, I get the foll error:
Unable to get type info from object “unstructured.Unstructured” : Object ‘Kind’ is missing in object has no kind field
Any idea what this could be about?
Hello Soumya, To be honest, i don’t really know what that can be about. What version of Kubernetes are you trying to deploy?
I added a sed s/kind/Kind/ to the pipe chain and the error went away, but not sure if everything is working yet
Bill, which part did you add the sed to ?
Sorry Yizhuan, it’s been a few months ago now, but I think it was in the pipe chain in the Workaround for kubedns as listed just above, when the guy is complaining about kind and Kind
Oh and that workaround of the kind/Kind issue in the Workaround for kube-dns did work. My slaves are working nicely now
Thanks for the guide, much appreciated. But couldn’t this be done with a script? Or better yet, can this be coded up into something like Terraform so there is a clear pathway to running this for staging and prod?
hi ,
got the following error : “bash: /etc/yum.repos.d/kubernetes.repo: No such file or directory” once trying to install the kubernetes pieces . can you please help ?
Thanks
You might need to run this as superuser ->
cat < /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF setenforce 0 yum install -y docker kubelet kubeadm kubectl kubernetes-cni systemctl enable docker && systemctl start docker systemctl enable kubelet && systemctl start kubeletEither add sudo before each call or simply do sudo su and continue with whole command.
I had the same problem.
you need to write the repository as shown in these images from this link
https://www.howtoforge.com/tutorial/centos-kubernetes-docker-cluster/
I solved it: D
Great tutorial Andre. Considering following it to setup a K8s cluster with 3-4 VMs on my host (i7 8th gen, 12 core, 32GB RAM — so plenty of resources, I hope), and your tutorial should come in handy. However, do you suppose that due to 2years having passed between the original article and now, K8s have evolved so much that things may not work as expected ? If you’ve been keeping track of K8s evolutions, any specific areas where you suspect that one could anticipate issues ?
Hi Andre, I tried the steps on Ubuntu and creat vm and use host only network. Since it is host only network my vm not able to connect with internet. Cloud you please told me how you get access thing on internet.
Thanks for sharing
Hi Pavel,
How to deploy a docker with kubernetes orchestration on ubuntu 18.4 LTS, i was installing on Vmware workstation 14 but network doesn’t work.
root@kube-master:~# kubeadm init –apiserver-advertise-address $(hostname -i) –pod-network-cidr=192.168.0.0/16
unable to select an IP from lo network interface
To see the stack trace of this error execute with –v=5 or higher
root@kube-master:~# kubeadm init –apiserver-advertise-address $(hostname -i) –pod-network-cidr=192.168.0.0/16 –v=5
I1228 18:19:08.563781 19314 initconfiguration.go:103] detected and using CRI socket: /var/run/dockershim.sock
I1228 18:19:08.564121 19314 interface.go:208] Interface lo is up
I1228 18:19:08.564212 19314 interface.go:256] Interface “lo” has 2 addresses :[127.0.0.1/8 ::1/128].
I1228 18:19:08.564240 19314 interface.go:223] Checking addr 127.0.0.1/8.
I1228 18:19:08.564249 19314 interface.go:233] Non-global unicast address found 127.0.0.1
I1228 18:19:08.564260 19314 interface.go:223] Checking addr ::1/128.
I1228 18:19:08.564267 19314 interface.go:236] ::1 is not an IPv4 address
I1228 18:19:08.564666 19314 interface.go:208] Interface lo is up
I1228 18:19:08.564732 19314 interface.go:256] Interface “lo” has 2 addresses :[127.0.0.1/8 ::1/128].
I1228 18:19:08.564769 19314 interface.go:223] Checking addr 127.0.0.1/8.
I1228 18:19:08.564780 19314 interface.go:236] 127.0.0.1 is not an IPv6 address
I1228 18:19:08.564788 19314 interface.go:223] Checking addr ::1/128.
I1228 18:19:08.564796 19314 interface.go:233] Non-global unicast address found ::1
unable to select an IP from lo network interface
I have configured kubenetes cluster and single worker node on oracle virtual box centos 7 vm’s. master node is fine but while connecting worker node it is always throwing below error. Please help on this. I have setup my host only adpator ip address as: 192.168.1.1 , master: 192.168.1.7, worker: 192.168.1.5 . master and worker can ping each other but any one can’t ping to 192.168.1.1
Please help me on this.
kubeadm join 192.168.1.7:6443 –token ur5tee.f40lpx85l11u7gxb –discovery-token-ca-cert-hash sha256:d6cc5a3945136b2f3931cabee4d0bdc2d9e18daa66c744067ffc217494bb6ede
W0130 00:37:19.340456 8194 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
[kubelet-start] Downloading configuration for the kubelet from the “kubelet-config-1.17” ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn’t running or healthy.
[kubelet-check] The HTTP call equal to ‘curl -sSL http://localhost:10248/healthz‘ failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
Thanks for Pavel, may i know is it possible with the bare metal machine that have a single interface only and use vlan to create multiple virtual interfaces?
Great Blog! I have learned many things from this article. Thanks for sharing this excellent blog with us and posting on
Iqra Technology