Setting up a Cluster With Proxmox

By
anismaj

Setting up a Cluster With Proxmox

In this article we try to make a cluster using Proxmox. These tests were performed with two nodes, although it is more interesting to make a cluster with at least three nodes.

For the system, we took the ISO Proxmox VE 3.1 from here.

The installation is the same for base, so no problems to remark (we just added Puppet for automated management configurations).

You can check here our article Proxmox Install Step by Step

The setup

Update:

For starters, you need an updated system of nodes. But since version 3 of Proxmox, publishers have changed the repository, so you have a version “without support”, we invite you to edit your sources.list

nano /etc/apt/sources.list

And edit :

deb https://enterprise.proxmox.com/debian wheezy pve-enterprise

By :

deb http://download.proxmox.com/debian wheezy pve-no-subscription

Then do a: (repo update, update the system, and then reboot at the end):

apt-get update && apt-get upgrade -y && reboot

Clustering:

Very simple to do, choose one of the nodes to be the master,

pvecm create name-of-cluster
Restarting pve cluster filesystem: pve-cluster[dcdb] notice: wrote new cluster config '/etc/cluster/cluster.conf'
Starting cluster:
    Checking if cluster has been disabled at boot... [  OK  ]
    Checking Network Manager... [  OK  ]
    Global setup... [  OK  ]
    Loading kernel modules... [  OK  ] 
    Mounting configfs... [  OK  ]
   Starting cman... [  OK  ]
   Waiting for quorum... [  OK  ]
   Starting fenced... [  OK  ]
   Starting dlm_controld... [  OK  ]
   Tuning DLM kernel config... [  OK  ]
   Unfencing self... [  OK  ]

Then we check Cluster creation:

pvecm status
Version: 6.2.0
Config Version: 2
Cluster Name: unixmen-Cluster
Cluster Id: 52776
Cluster Member: Yes
Cluster Generation: 256
Membership state: Cluster-Member
Nodes: 1
Expected votes: 1
Total votes: 1
Node votes: 1
Quorum: 1 
Active subsystems: 5
Flags:
Ports Bound: 0 
Node name: srv-1
Node ID: 1
Multicast addresses: 239.192.206.246
Node addresses: 10.10.32.100

And we add the second server:

pvecm add 10.10.32.100

And valid exchange ssh keys:

The authenticity of host '10.10.32.100 (10.10.32.100)' can't be established.
ECDSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xxx:xx:xx:xx:xx:xx:xx.
Are you sure you want to continue connecting (yes/no)? yes
root@10.10.32.100's password:
copy corosync auth key
stopping pve-cluster service
Stopping pve cluster filesystem: pve-cluster.
backup old database
Starting pve cluster filesystem : pve-clustercan't create shared ssh key database '/etc/pve/priv/authorized_keys'
Starting cluster:
   Checking if cluster has been disabled at boot... [  OK  ]
   Checking Network Manager... [  OK  ]
   Global setup... [  OK  ]
   Loading kernel modules... [  OK  ]
   Mounting configfs... [  OK  ]
   Starting cman... [  OK  ]
   Waiting for quorum... [  OK  ]
   Starting fenced... [  OK  ]
   Starting dlm_controld... [  OK  ]
   Tuning DLM kernel config... [  OK  ]
   Unfencing self... [  OK  ]
generating node certificates
merge known_hosts file
restart services
Restarting PVE Daemon: pvedaemon.
Restarting PVE API Proxy Server: pveproxy.
successfully added node 'yvain' to cluster.

We will check now the status of nodes:

pvecm nodes
Node  Sts   Inc   Joined               Name
   1   M    244   2014-02-13 16:36:02  srv-1
   2   M    256   2014-02-13 16:36:34  srv-2

You can also verify the presence of two nodes, by connecting to the web interface of any of your servers: https: // ipduserveur: 8006

1

Your cluster is now in place, before starting to play with, you have to add shared storage.

2

Storage:

To change the storage, go to “data center” on the “Storage” tab and click “Add”:

3

For simplicity we chose NFS mounts (although we intend to experiment with Ceph later):

The addition is quite simple (there just follow the instructions of the wizard), and you have the ability to assign storage resources to very specific uses: Storage VMs, ISOs, backups

4

Once the VM is created, you can access its options, preferences … in the left menu, depending on the host on which it was started:

5

You can test live migration:

6

Improvements

For basic use, the default installation is fine, but for a professional environment, it is good to make some changes:

Network Settings

To use our VLANs and virtual IP for the cluster, I had to adapt the /etc/network/interfaces,

Here is what happens at home (that is adjusted according to your needs)

#loopback
auto lo
iface lo inet loopback

#déclaration ip virtuelle du cluster
auto lo:1
iface lo:1 inet static
        address 10.10.32.103
        netmask 255.255.255.255

#bonding with double attachement
auto bond0
iface bond0 inet manual
        slaves eth4 eth5
        bond_mode balance-xor
        bond_miimon 100
        bond_downdelay 200
        bond_updelay 200
        pre-up /bin/ip link set eth4 mtu 1500
        pre-up /bin/ip link set eth5 mtu 1500

#Default bridge interface
auto vmbr0
iface vmbr0 inet static
        bridge_ports bond0
        bridge_fd 0
        bridge_maxwait 0
        bridge_stp off
        address 10.10.32.103
        netmask 255.255.255.0
        network 10.10.32.0
        broadcast 10.10.32.255
        gateway 10.10.32.1
        dns-nameservers 10.10.32.30 10.10.32.130
        dns-search mondomaine

#declaration VLAN 7
auto vlan7
iface vlan7 inet manual
        vlan_raw_device bond0
        up /bin/ip link set vlan7 mtu 1500

#déclaration VLAN 72
auto vlan72
iface vlan72 inet manual
        vlan_raw_device bond0
        up /bin/ip link set vlan72 mtu 1500

#declaration VLAN 73
auto vlan73
iface vlan73 inet manual
        vlan_raw_device bond0
        up /bin/ip link set vlan73 mtu 1500


#VLAN interface bridge 72
auto vmbr72
iface vmbr72 inet static
        bridge_ports vlan72
        bridge_fd 0
        bridge_maxwait 0
        bridge_stp off
        address 10.14.31.54
        netmask 255.255.255.0
        network 10.14.31.0
        broadcast 10.14.31.255

#VLAN interface bridge 73
auto vmbr73
iface vmbr73 inet static
        bridge_ports vlan73
        bridge_fd 0
        bridge_maxwait 0
        bridge_stp off
        address 10.14.30.57
        netmask 255.255.255.0
        network 10.14.30.0
        broadcast 10.14.30.255

Failover with Keepalive

In order to failover, we obviously have a working installation of keepalive servers, and add a new instance

vrrp_instance VI_1 {
state MASTER
interface bond0
virtual_router_id 1
priority 100
advert_int 1
lvs_sync_daemon_interface bond0
authentication {
 auth_type PASS
 auth_pass password
}
virtual_ipaddress {
10.10.32.103/24 brd 10.10.32.255 dev bond0 # Proxmox

Then we will use the following commands to make the changes:

##############################################
## Proxmox
##############################################

virtual_server 10.10.32.103 443 {
    delay_loop 6
    lb_algo wlc
    lb_kind DR
    protocol TCP
    persistence_timeout 120

    sorry_server 10.10.32.103 443

    real_server 10.10.32.103 443 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
        }
    }
}
virtual_server 10.10.32.103 5900 {
    delay_loop 6
    lb_algo wlc
    lb_kind DR
    protocol TCP
    persistence_timeout 120

    sorry_server 10.10.32.100 5900

    real_server 10.10.32.103 5900 {
        weight 1
        MISC_CHECK {
            misc_path "/usr/lib/nagios/plugins/check_tcp -H 10.10.32.103 -p 443"
        }
    }
}

The virtual cluster IP is 10.10.32.103.

As we said the master is 10.10.32.103 and keepalive via nagios did a test on port 443 of the master, if it does not respond in less than 3s, it switches to “HS” and it is the slave (10.14. 32.100)

this configuration is applied to port 443 and port 5900, used to the virtual console in VNC.

Adding a reverse proxy and SSL Certificate

Since version 3.0, Proxmox no longer uses Apache as the web server, but an internal process “pvcecluster” There are no changes possible..

We will insert a Nginx (lightweight, easily changeable and highly efficient)

apt-get install nginx-light

Then we will create the virtual host:

On port 443, you take into account the ssl certificate, created for the common domain servers (domain that points to the virtual cluster IP), then redirects locally on the 8006 port.

server {
        listen          80;
        server_name     proxmox.unixmen.com;
        rewrite         ^ https://$hostname.mondomaine.fr$request_uri? permanent;
}

server {
        listen                  443 ssl;
        server_name             proxmox.unixmen.com;
        ssl on;
        ssl_certificate         /etc/nginx/conf.d/cert-20434-proxmox.unixmen.com.pem;
        ssl_certificate_key     /etc/nginx/conf.d/proxmox.unixmen.com.key;
        location / { proxy_pass https://127.0.0.1:8006; }
}

We activate the configuration now:

rm -f /etc/nginx/sites-enabled/default
ln -sf /etc/nginx/sites-available/pveproxy /etc/nginx/sites-enabled/

we modify pveproxy, for that you need to create the following file:

nano /etc/default/pveproxy
ALLOW_FROM="127.0.0.1,10.10.32.103,10.10.32.100"
DENY_FROM="all"
POLICY="allow"

It is important to add the ips of the cluster nodes. Otherwise the web interface will no longer work properly (the other nodes can no longer question them).

It remains to restart what is necessary:

/etc/init.d/pveproxy restart
/etc/init.d/nginx restart

That’s all. Thank you.