View source for Create a Backup Cloud Product with Puppet and BackupPC for openQRM Cloud

[[Category:Tutorial]]
[[Category:Backup]]
[[Category:Puppet]]
[[Category:BackupPC]]
[[Category:Cloud]]
This How-To is about how to create a Cloud Product for an Automatic-Backup solution on the Cloud systems (VMs and Bare-Metal) with Puppet and BackupPC on an openQRM Cloud environment.

=== Requirements ===

* One (or more) physical Server
* at least 1 GB of Memory
* at least 100 GB of Diskspace
* VT (Virtualization Technology) enabled in the Systems BIOS so that the openQRM Server can run Virtual Machines later
* Debian Installation on a physical Server
'''Note:'''

For this How-To, we assume you have successfully [https://wiki.openqrm-enterprise.com/index.php%3Ftitle=Install_openQRM_on_Debian installed openQRM] and have at least referenced our [https://wiki.openqrm-enterprise.com/index.php%3Ftitle=Cloud_Computing_with_openQRM Cloud Computing with openQRM on Debian How-To] before. If not, it is highly suggested to complete both before continuing 

=== Goal ===
The Goal for this How-to is to create a "Backup" Cloud Product which the following properties:

* It needs to be fully automated without regular manual tasks for the system administrator(s)
* It needs to "selectable" as a regular Cloud Product in the openQRM Cloud Portal and in openQRM Enterprise Cloud Zones.
* It needs to provide an Cloud user UI to schedule the backup and to restore data (probably also in another location/system)

=== Characteristic of BackupPC ===
BackupPC is a open-source backup solution licensed under the GPL which supports backup and restore for Unix and Windows based systems. It provides several different transport mechanisms (rsync over ssh, rsyncd, tar, SMB etc.) and support deduplication to reduce the overall storage consumption. BackupPC is written in Perl and includes user-friendly configuration files plus a web based UI.

Please check <nowiki>http://backuppc.sourceforge.net/info.html</nowiki> for more information about BackupPC.

=== Setup ===
Since the "Backup" Cloud Product should be "purchasable" by the openQRM Cloud customer as any other Cloud Application Product it must be defined in the openQRM Cloud Product configuration. The best options to integrate the automated Nagios configuration is with either Puppet or Ansible. In this How-To we are going to use Puppet.

The first step of this How-To describes the installation of a separated BackupPC server. The further steps of this How-To assume the BackupPC server is running on a separated dedicated system. Only the UI of the BackupPC server is displayed on the openQRM server.

'''NOTE''' In the following configuration commands and Puppet classes please replace:<blockquote>BACKUPPC_HOSTNAME with the real hostname of your BackupPC server.

OPENQRM_SERVER_HOSTNAME with your openQRM Servers hostname

OPENQRM_DOMAIN_NAME with your actual configured Domain name in the DNS-Plugin configuration

SSH_SYSTEM_USER with your actual Unix system account name

SECRET_DB_PASSWORD with your password for the openQRM Database.</blockquote>

=== Installation of the dedicated BackupPC server ===
- Please make sure the following Perl dependencies are installed: Compress::Zlib, Archive::Zip, XML::RSS and File::RsyncP. Those dependencies can be checked by e.g.
 perldoc Compress::Zlib
To install those dependencies please run e.g.
 cpan install Compress::Zlib
Create a system user and group "backuppc"

Download BackupPC and unpack it in a temporary directory and run the installer by:
 perl configure.pl
The following configuration section is self-explaining.

Edit the
 /etc/BackupPC/config.pl
as following:
 $Conf{EMailFromUserName} = 'ADMINISTRATOR_EMAIL@OPENQRM_DOMAIN_NAME';
 $Conf{EMailAdminUserName} = 'ADMINISTRATOR_EMAIL@OPENQRM_DOMAIN_NAME';
 $Conf{Language} = 'de';
 $Conf{BackupFilesOnly} = \['/app', '/var/app'\];
 $Conf{XferMethod} = 'rsync';
 $Conf{RsyncShareName} = \['/app', '/var/app'\];
 
 # One task is the Hostkey which needs to be accepted by the new system.
 # Another task is that ip are going to be recycled. There we added the following 2 configuration lines
 
 $Conf{RsyncClientCmd} = '$sshPath -q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -x -l root $host $rsyncPath $argList+';
 $Conf{RsyncClientRestoreCmd} = '$sshPath -q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -x -l root $host $rsyncPath $argList+';
Create and enable the BackupPC init script located in the temporary installation directory

Create a rsa SSH Key. This is for the automatic authentication of the clients for the "rsync over ssh" transport mechanism.
 ssh-keygen -t rsa
The
 /home/SSH_SYSTEM_USER/.ssh/id_rsa.pub
file is later distributed to all Cloud servers via Puppet.

=== Creating the Puppet recipe for the Backup product ===
Currently the standard resource for BackupPC are not included in the Puppet version on the mainstream Linux distribution for now. Therefore we are using a mechanism which just reports that a Cloud user bought the backup product. A cron job on the central openQRM server then automatically configures BackupPC.

'''NOTE:''' The requirement for that mechanism is to have "stored configuration" enabled and enabled in Puppet. How to archive this is explained in detail in the previous How-To about "How-To create a Monitoring Cloud Product with Puppet and Nagios for openQRM Cloud 5.1"

Here's the Puppet class "g_backup":
 class c_backup {
   ssh_authorized_key {'backuppc@BACKUPPC_HOSTNAME':
   ensure => 'present',
   user => 'root',
   key => '[content-of-the-previously-created-.ssh/id_rsa.pub-key-file]'
   type => 'ssh-rsa',
 }
 @@file {"/app/openqrm/backup/clients/$hostname":
   content => "$fqdn $ipaddress",
   tag => 'backup',
   }
 }
To include this puppet class into openQRM create another recipe in the "groups" directory of the openQRM Puppet configuration with the following content:
 class g_backup {
   include c_backup
 }
 
After adding this class to Puppet the "g_backup" application can be added as Cloud product in openQRM.

Here's the Puppet class for the openQRM server itself. It cleans-up a local directory for the client data nd runs the
 gen_backuppc_hosts_config
shell script which then generates the BackupPC configuration from the Puppet DB informations of the Cloud system.
 class backup {
   exec { 'housekeeping':
     command => 'rm -rf /app/openqrm/backup/clients/*',
     cwd => '/app/openqrm/backup/clients',
     path => '/sbin:/usr/sbin:/usr/bin:/bin',
   }
   File <<| tag == "backup" |>> {
   require => Exec ['housekeeping'],
   }
   # get the contact and email for the cloud system from the openQRM DB
   exec { 'gen_backuppc_hosts_config':
     command => '/app/openqrm/tools/gen_backuppc_hosts_config',
     cwd => '/app/openqrm/tools',
     path => '/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/global/bin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
   }
 }
 node 'OPENQRM_SERVER_HOSTNAME.OPENQRM_DOMAIN_NAME' {
   include backup
 }
 
The following script,
 gen_backuppc_hosts_config
gets the owner of the Cloud system from the openQRM database and generates the BackupPC hosts.pl configuration. The resulting generated configuration is then copied locally (for the UI) and also remotely to the BackupPC server.
 #!/bin/bash
 #
 # gen_backuppc_hosts_config
 #
 # generates the BackupPC hosts config file
 #
 # host  dhcp  user  moreUsers  #
 # SSH_SYSTEM_USER.de 980 5957
 #
 BACKUP_CLIENTS="/app/openqrm/backup/clients"
 MYSQL="/usr/bin/mysql -uroot -pSECRET_DB_PASSWORD openqrm -s -N"
 TMP_DIR="/tmp/$$"
 [ -d ${TMP_DIR} ] && rm -rf ${TMP_DIR}
 mkdir ${TMP_DIR}
 TMP_FILE="/${TMP_DIR}/gen_backuppc_hosts_config.$$"
 # Header ausgeben
 echo "#
 # The first non-comment non-empty line gives the field names and should
 # not be edited!!
 #
 host  dhcp  user  moreUsers  # <--- do not edit this line
 #farside  0  craig  jill,jeff  # <--- example static IP host entry
 #larson  1  bill  # <--- example DHCP host entry
 OPENQRM_SERVER_HOSTNAME.OPENQRM_DOMAIN_NAME 0  SSH_SYSTEM_USER" >${TMP_FILE}
 
 for DATEI in ${BACKUP_CLIENTS}/*
 do
   HOSTNAME="`basename ${DATEI}`"
   USERNAME="`echo "select u.cu_name from cloud_requests r INNER JOIN cloud_users u ON r.cr_cu_id = u.cu_id where cr_status='3' AND cr_appliance_hostname = '${HOSTNAME}';"|${MYSQL}`"
   EMAIL="`echo "select u.cu_email from cloud_requests r INNER JOIN cloud_users u ON r.cr_cu_id = u.cu_id where cr_status='3' AND cr_appliance_hostname = '${HOSTNAME}';"|${MYSQL}`"
   IP="`cat ${DATEI}|head -1 | awk '{ print $2 }'`"
   FQDN="`cat ${DATEI}|head -1 | awk '{ print $1 }'`"
   if [ "X${USERNAME}" == "X" ]
   then
     rm ${DATEI}
     continue
   fi
   if [ "X${EMAIL}" == "X" ]
   then
     rm ${DATEI}
     continue
   fi
   if [ "X${IP}" == "X" ]
   then
     rm ${DATEI}
     continue
   fi
   echo "${FQDN} 0 ${USERNAME}" >> ${TMP_FILE}
 done
 scp ${TMP_FILE} BACKUPPC_HOSTNAME:/etc/BackupPC/hosts
 ssh BACKUPPC_HOSTNAME "/etc/init.d/backuppc reload"
 rm ${TMP_FILE}
 [ -d ${TMP_DIR} ] && rm -rf ${TMP_DIR}
 
The only thing left to do is to define the "g_backup" application as Backup Cloud product in openQRM.

'''Congratulations!!'''

You have successfully completed this How-To!