- kube-scheduler: The Kubernetes scheduler - Fri, Sep 29 2023
- Kubernetes logs: Pod logs, container logs, Docker logs, kubelet logs, and master node logs - Mon, Sep 25 2023
- Kubernetes DaemonSets - Wed, Sep 6 2023
Basic information
In a previous post, we mentioned that a Proxmox cluster is multimaster, meaning there are no master–slave nodes, and the cluster can be effectively managed from any node. Proxmox uses a Corosync cluster engine, which uses a quorum-based voting technique to ensure consistent configuration among all the nodes of a cluster. The cluster configuration is distributed to all nodes with the help of a Proxmox cluster file system (pmxcfs).
Prerequisites
- The date and time need to be synchronized.
- SSH connectivity on TCP port 22 is required.
- UDP port range 5405–5412 is required for Corosync to work.
- The root password of the cluster node is needed to add nodes.
- Proxmox VE needs to be installed on all nodes.
- Hostnames and static IP addresses need to be configured on all nodes.
- A dedicated network adapter is highly recommended for cluster traffic, although it is not required.
Create a cluster
Once you have all the nodes ready, access them in a web browser, and log in to the Proxmox VE web GUI. I connected to my first Proxmox server (pmox-host1). Click the Datacenter option at the top, choose Cluster, and then click the Create Cluster button.
Create Cluster option in Proxmox VE web interface
Give a unique name to your Proxmox cluster, and select a link for the cluster network. I will keep the default link (0) selected. It is a good idea to choose a network link that isn't used for other high-traffic needs, such as network storage.
Create a cluster in Proxmox VE
You can click the Add button to add a failover link if multiple network links are available on your node. Click the Create button, and you will see a task viewer window showing the status of cluster creation.
Viewing cluster creation progress in Proxmox VE
Your Proxmox cluster has now been created.
Add nodes to the cluster
When adding nodes to your Proxmox cluster, make sure no VM or container is running on the node. When you add a node to the cluster, it will inherit the cluster configuration, which will overwrite all current local configurations. If you have an important VM or container running, you can create a backup, remove it, and import it later after joining the cluster.
To join a node to the cluster, you need to copy some information from the first node where you created the cluster. To do that, navigate to Datacenter > Cluster and then click the Join Information button, as shown in the screenshot.
Viewing the join information for a cluster in Proxmox VE
Click the Copy Information button to copy the cluster join information to your clipboard.
Copying the join information for a cluster in Proxmox VE
Now, connect the other Proxmox node that you want to join to the cluster. Navigate to Datacenter > Cluster, and click the Join Cluster button, as shown below:
Join Cluster option in Proxmox VE
In the Information box, paste the join information, type the root password of the first node, and click the Join button.
Add a node to the Proxmox cluster
The second node will now be added to the Proxmox cluster. As soon as the node is added to the cluster, the server certificate is changed, so you need to reload the page and log in again to the Proxmox VE web interface. The screenshot below shows that our proxmox-lab cluster now has two nodes: pmox-host1 and pmox-host2.
Viewing the cluster information in Proxmox VE
The information about cluster nodes is listed under the Cluster option. In addition, any shared storage that is attached to the first node is automatically attached to the new node.
Once you have a Proxmox cluster up and running, you can start migrating your VMs or containers when needed, as shown in the screenshot below:
Performing live VM migration in a Proxmox cluster
The migration is pretty fast, particularly when you're using shared storage (e.g., NAS or Ceph), so your VM will experience little to no downtime at all. The cool feature you get with a Proxmox cluster is that you can still migrate your VM (or container), even if you're using local storage. Of course, it takes longer to migrate, but it works.
Problem with a 2-node cluster
It is entirely possible to set up a Proxmox cluster with just two nodes, as we did above. However, as mentioned earlier, the Proxmox cluster depends on quorum votes. For a quorum to exist in a cluster, the majority of nodes need to be online. With just two nodes in a cluster, both get an equal number of votes, so when either node goes down, the quorum will be lost, and the Proxmox cluster switches to read-only mode. As a result, the cluster will stop functioning as expected. This makes server maintenance hard in a two-node cluster setup.
To deal with such a situation, there are two options:
- Manually set the expected votes to 1 by running the pvecm expected 1 command. This command changes the expected votes temporarily, which revert after restart. But it will help maintain the quorum so that your Proxmox cluster can continue to work with just a single node while you perform maintenance on the other node.
- Alternatively, you could set up a corosync-qdevice in Raspberry Pi or Docker and use it as a tiebreaker for quorum votes when one of two Proxmox nodes is down. The purpose of qdevice is to vote for the active node to maintain a quorum when there are just two nodes (or an even number) in the Proxmox cluster. The use of qdevice is not recommended when there are an odd number of nodes, as it could hurt cluster stability.
Considering the above situation, it is clear that it is a good idea to have at least three nodes in a Proxmox cluster, particularly in a production environment. Furthermore, you will be able to use the additional features, such as high availability, when there are three or more nodes in the cluster.
Subscribe to 4sysops newsletter!
Conclusion
Creating a Proxmox cluster can greatly enhance the performance and scalability of your virtualization environment. By utilizing multiple nodes, you can distribute the workload across the cluster and ensure high availability for your virtual machines. The process of setting up a Proxmox cluster may seem daunting at first, but by following the steps outlined in this article, you can easily create a robust and reliable cluster that meets your needs.
