Setup clustered LVM as shared VM storage for KVM
This How-To explains how to set up a shared storage space for VM virtual disks using clustered LVM (CLVMD). CLVMD (Clustered LVM Daemon) is a service which runs on top of regular LVM. It is using the "CMAN" HA-framework for the cluster communications. This How-To explains the configuration of 2 (or more) additional bare-metal systems used as KVM virtualization hosts which are configured with a shared clustered LVM volume group. For this How-To we are using the CentOS Linux distribution for the additional systems.
Requirements
- Three physical Server. Alternatively openQRM itself can be installed within a Virtual Machine
- at least 1 GB of Memory
- at least 100 GB of Diskspace
- VT (Virtualization Technology) enabled in the Systems BIOS for the Xen Host system so it can run HVM Virtual Machines later
- Minimal Debian installation on a physical server.
- openQRM 5.1 or later, that has been initialised.
NOTE: For this How-To, we will be using the same openQRM server used in the Virtualisation with KVM and openQRM on Debian How-To, so it is recommended to complete that How-To before following this one. If you do not have openQRM installed yet, follow this How-To: Install openQRM on Debian.
Install CentOS on the KVM Hosts
Install the latest CentOS on two (or more) physical systems dedicated for the KVM virtualization hosts. Partition the disk as you like and use a minimal package installation. openQRM will later automatically fetch all further needed package dependencies. Please configure both systems with a static IP address from the openQRM management network. After the installation reboot and SSH into the 2 systems.
For this How-To we are going to use "clvmkvm1" as hostname with the IP address 192.168.178.135 for the first system and "clvmkvm2" as hostname with the IP address 192.168.178.136 for the second system.
- Please notice!
After the CentOS installation, make sure to enable the additional "rpmforge" and "epel" package repositories. Please check http://wiki.centos.org/AdditionalResources/Repositories/RPMForge and http://www.rackspace.com/knowledge_center/article/installing-rhel-epel-repo-on-centos-5x-or-6x how to enable those 2 package repositories for CentOS.
Post configuration on the CentOS KVM Hosts
As the first step after the OS installation please configure one (or more) bridges. On both systems please run:
yum install bridge-utils
to install the bridge-utils.
In /etc/sysconfig/network-scripts/ please adapt ifcfg-eth0 as following:
# Interface for bridge br0
DEVICE=eth0
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no
USERCTL=no
BRIDGE=br0
Then create an ifcfg-br0 as following (here for example on the first system):
DEVICE=br0
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.88.135
NETMASK=255.255.255.0
GATEWAY=192.168.88.1
DELAY=0
HELLO=2
MAXAGE=12
STP=no
TYPE=Bridge
NOTE:
For the second system please use IPADDR=192.168.88.136
After that, reboot both systems to activate the new network configuration. It may be sufficient to restart the network service instead of rebooting.
If you plan to attach further external networks to your VMs please setup bridges to the external network(s) in the same way as for the openQRM management network described above.
Now please configure /etc/hosts on both systems and add the following lines:
192.168.88.135 clvmkvm1
192.168.88.136 clvmkvm2
This is to make sure the clustered systems know each other by hostname.
NOTE:
Please disable SELinux by adapting the /etc/selinux/config to:
SELINUX=disabled
Further make sure to also disable the iptables and ip6tables firewall by running:
chkconfig --del iptables
chkconfig --del ip6tables
Now please reboot the 2 systems to deactivate selinux and the firewall.
To recheck your SELinux configuration is really disabled please run:
selinuxenabled
echo $?
It should return "1" now (disabled).
To recheck your iptables firewall is really disabled please run:
iptables -L
Connect the SAN/iSCSI-target to both systems
The configuration details for connecting a SAN and/or iSCSI device can be different in your setup! For this How-To we are using a simple iSCSI-target created with "ietd" without any CHAP authentication on the openQRM server system. The easiest way is to use the "iscsi-storage" plugin in openQRM to create such an iSCSI target.
On both systems please install the "iscsi-initiator-utils".
yum install iscsi-initiator-utils
Now discover the iSCSI-target.
[root@clvmkvm1 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.178.5:3260
192.168.178.5:3260,1 clvmsan 192.168.178.5
The iSCSI discovery action provides the exact target name and record to use for the iSCSI target login. In our example the target name is "clvmsan":
iscsiadm -m node -T clvmsan -p 192.168.178.5:3260 --login
Now the iSCSI target is connected and appears in /proc/partitions as new device "sdb"
[root@clvmkvm1 ~]# cat /proc/partitions
major minor #blocks name
8 0 312571224 sda
8 1 512000 sda1
8 2 307200000 sda2
8 3 204800 sda3
8 16 51200000 sdb
253 0 2048000 dm-0
[root@clvmkvm1 ~]#
Please create a partition on the iSCSI target device using fdisk. Set the type of the partition to "8e" (LVM). It should look similar to the following fdisk output:
[root@clvmkvm1 ~]# fdisk /dev/sdb
Command (m for help): p
Disk /dev/sdb: 52.4 GB, 52428800000 bytes
64 heads, 32 sectors/track, 50000 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xa60f79fa
Device Boot Start End Blocks Id System
/dev/sdb1 1 50000 51199984 8e Linux LVM
Command (m for help):
Now please setup the new partition as physical device for LVM and create a LVM volume group. For this How-To we have used "clvmsan" for the LVM volume group name:
pvcreate /dev/sdb1
vgcreate clvmsan /dev/sdb1
The new created LVM volume group will now appear on both systems but it is not yet clustered.
Adapt the LVM configuration for clustering
On both systems please edit /etc/lvm/lvm.conf and set the following parameters:
locking_type = 3
fallback_to_clustered_locking = 0
fallback_to_local_locking = 0
After that all LVM commands will fail because the underlaying LVM cluster is not yet started.
Setup the LVM cluster
On both systems please install the "lvm2-cluster" and "cman" packages.
yum install lvm2-cluster cman
To configure the cluster topology please create a /etc/cluster/cluster.conf configuration file equal on both nodes as the following:
<cluster name="clvmkvm" config_version="1">
<cman two_node="1" expected_votes="1"/>
<clusternodes>
<clusternode name="clvmkvm1" nodeid="1">
</clusternode>
<clusternode name="clvmkvm2" nodeid="2">
</clusternode>
</clusternodes>
</cluster>
NOTE:
Since a 2 node cluster cannot form a quorum on behalf of 2 votes please make sure to have second line configured as in the above example. Cluster with 3 or more nodes do not need this special configuration option.
Now please restart the "cman" service on both systems to activate the cluster:
/etc/init.d/cman restart
To check the cluster status please use the "cman_tool":
[root@clvmkvm1 ~]# cman_tool nodes
Node Sts Inc Joined Name
1 M 312 2014-02-20 11:14:16 clvmkvm1
2 M 324 2014-02-20 11:15:45 clvmkvm2
[root@clvmkvm1 ~]#
Now please restart the "clvmd" service on both systems to activate the clustered volume group:
/etc/init.d/clvmd restart
Running the regular "vgs" command shows the clustered volume group. Please notice the 'c' attributes which stands for "clusterd"
[root@clvmkvm1 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
clvmsan 1 1 0 wz--nc 48.82g 46.87g
[root@clvmkvm1 ~]#
For managing the now clustered LVM volume the regular LVM commands can be used. The CLVMD service will make sure to use a clustered locking mechanism and to distribute the LVM meta data across the cluster nodes.
Integrate the VM Hosts into openQRM
Copy (scp) the openqrm-local-server integration tool from the openQRM server to the 2 systems dedicated for the VM Hosts:
scp /usr/share/openqrm/plugins/local-server/bin/openqrm-local-server 192.168.178.135:
scp /usr/share/openqrm/plugins/local-server/bin/openqrm-local-server 192.168.178.136:
Then login (ssh) to each system and run the openqrm-local-server tool with the 'integrate' parameter. On the first system:
./openqrm-local-server integrate -u openqrm -p openqrm -q 192.168.178.5 -n clvmkvm1 -i br0 -s http
and on the second system:
./openqrm-local-server integrate -u openqrm -p openqrm -q 192.168.178.5 -n clvmkvm2 -i br0 -s http
Here how it looks like on the first system terminal console:
root@clvmkvm1:~# chmod +x openqrm-local-server
root@clvmkvm1:~# ./openqrm-local-server
Usage : ./openqrm-local-server integrate -u -p -q [ -n ] [-i ] [-s ]
./openqrm-local-server remove -u -p -q [ -n ] [-s ]
root@clvmkvm11:~# ./openqrm-local-server integrate -u openqrm -p openqrm -q 192.168.178.5 -n clvmkvm1 -i br0 -s http
Integrating system to openQRM-server at 192.168.178.5
-> could not find dropbear. Trying to automatically install it ...
Reading package lists... Done
Building dependency tree
........... (more output and automatic package installation)
root@clvmkvm1:~#
The "local-server" integration automatically adds the 2 systems to openQRM and creates for a each system a "server" object. Please edit the "server" object of both systems and set the "Virtualization" type to "KVM Host". This will automatically create 2 new "storage" objects in openQRM which can be used to manage the clustered LVM VM disk space.
NOTE:
Please notice that LVM snapshots are NOT supported for clustered LVM! In case you are using openQRM IaaS cloud please adapt the following cloud-deployment hook for KVM. Edit:
/usr/share/openqrm/plugins/kvm/web/openqrm-kvm-lvm-deployment-cloud-hook.php
and change the line:
$image_clone_cmd="$OPENQRM_SERVER_BASE_DIR/openqrm/plugins/kvm/bin/openqrm-kvm snap -n ".$image_location_name." -v ".$volume_group." -s ".$image_clone_name." -m ".$disk_size." -t ".$deployment->type." -u ".$openqrm_admin_user->name." -p ".$openqrm_admin_user->password;
to
$image_clone_cmd="$OPENQRM_SERVER_BASE_DIR/openqrm/plugins/kvm/bin/openqrm-kvm clone -n ".$image_location_name." -v ".$volume_group." -s ".$image_clone_name." -m ".$disk_size." -t ".$deployment->type." -u ".$openqrm_admin_user->name." -p ".$openqrm_admin_user->password;
This configures the openQRM IaaS Cloud to use the "clone" command instead of "snap".
NOTE:
In the upcoming openQRM Enterprise release the "clone" action for KVM got enhanced to allow automatic resizing from small master images to bigger volumes for e.g. Cloud deployment. A new "Image cache" feature in 5.1.2 provides the capability to locally cache master images and efficiently clone them from the local cache.
Congratulations!!
You have successfully completed this How-To!
