Difference between revisions of "Automated Amazon EC2 Cloud deployments with openQRM on Debian"

From openQRM
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
This HowTo is about how to manage Public and Hybrid Cloud deployments with openQRM. As the deployment manager for Amazon EC2 and its API compatible derivatives (e.g Eucalyptus) openQRM is capable to fully automate Instance provisioning and to add additional value by attaching automated application deployment via Puppet, automated monitoring via Nagios and also high availability on Infrastructure-Level to the providers cloud features. The whole workflow of Instance-deployment in openQRM is exactly the same as for local resources in the internal IT-environment.
+
This How-To is about how to manage Public and Hybrid Cloud deployments with openQRM. As the deployment manager for Amazon EC2 and its API compatible derivatives (e.g Eucalyptus) openQRM is capable to fully automate Instance provisioning and to add additional value by attaching automated application deployment via Puppet, automated monitoring via Nagios and also high availability on Infrastructure-Level to the providers cloud features. The whole workflow of Instance-deployment in openQRM is exactly the same as for local resources in the internal IT-environment.
  
 
==Requirements==
 
==Requirements==
  
 
#One physical Server. Alternatively, the installation can be also done within a Virtual Machine
 
#One physical Server. Alternatively, the installation can be also done within a Virtual Machine
#Atleast 1 GB of Memory
+
#At least 1 GB of Memory
#Atleast 100 GB of Diskspace
+
#At least 100 GB of Diskspace
 
#Optional VT (Virtualization Technology) enabled in the Systems BIOS so that the openQRM Server can run KVM Virtual Machines later
 
#Optional VT (Virtualization Technology) enabled in the Systems BIOS so that the openQRM Server can run KVM Virtual Machines later
  
Line 11: Line 11:
 
#Install a minimal Debian on a physical Server
 
#Install a minimal Debian on a physical Server
 
#Install and initialize openQRM
 
#Install and initialize openQRM
A detailed Howto about the above initial starting point is available at [[Install openQRM on Debian]]
+
A detailed How-To about the above initial starting point is available at [[Install openQRM on Debian]]
  
For this howto, we have used the same openQRM server as for the howto [[Virtualization with KVM and openQRM on Wheezy]].
+
For this How-To, we have used the same openQRM server as for the How-To [[Virtualisation with KVM and openQRM on Debian]].
That means with this howto we are going to add functionality to an existing openQRM setup. This is to show that openQRM manages all different virtualization and deployment types seamlessly.
+
That means with this How-To we are going to add functionality to an existing openQRM setup. This is to show that openQRM manages all different virtualization and deployment types seamlessly.
  
This means you can use either the [[Install openQRM on Debian]] or [[Virtualization with KVM and openQRM on Debian]] howto as a starting point.
+
This means you can use either the [[Install openQRM on Debian]] or [[Virtualisation with KVM and openQRM on Debian]] How-To as a starting point.
  
==1. Set a custom Domain name==
+
== Step-by-Step ==
 +
=== Set a custom Domain name ===
  
 
As the first step after the openQRM installation and initialisation, it is recommended to configure a custom domain name for the openQRM management network.
 
As the first step after the openQRM installation and initialisation, it is recommended to configure a custom domain name for the openQRM management network.
  
In this Use-Case the openQRM Server has the private Class C IP address 192.168.178.5/255.255.255.0 based on the previous Howto [[Install openQRM on Debian]]. Since the openQRM management network is a private one any syntactically correct domain name can be used e.g. 'my123cloud.net'.
+
In this Use-Case the openQRM Server has the private Class C IP address 192.168.178.5/255.255.255.0 based on the previous How-To [[Install openQRM on Debian]]. Since the openQRM management network is a private one any syntactically correct domain name can be used e.g. 'my123cloud.net'.
 
The default domain name pre-configured in the DNS plugin is "oqnet.org".
 
The default domain name pre-configured in the DNS plugin is "oqnet.org".
  
Line 40: Line 41:
  
 
== 2. Enabling Plugins ==
 
== 2. Enabling Plugins ==
For this HowTo please enable and start the following plugins in the sequence below:
+
For this How-To please enable and start the following plugins in the sequence below:
[[File:Csm 01-openqrm-ec2 01cc83dc1c.png|thumb|350px|Plugin Manager]]
 
 
#dns plugin - ''type Networking''
 
#dns plugin - ''type Networking''
 
#dhcpd plugin - ''type Networking''
 
#dhcpd plugin - ''type Networking''
Line 52: Line 52:
 
#sshterm plugin - ''type Management''
 
#sshterm plugin - ''type Management''
 
#hybrid-cloud - ''type Deployment''
 
#hybrid-cloud - ''type Deployment''
Hint: You can use the filter in the plugin list to find plugins by their type easily!
+
'''Hint''': You can use the filter in the plugin list to find plugins by their type easily!
 
+
[[File:Csm 01-openqrm-ec2 01cc83dc1c.png|thumb|390x390px|Plugin Manager|alt=|none]]
  
 
== 3. Install the latest Amazon EC2 Tools ==
 
== 3. Install the latest Amazon EC2 Tools ==
Go to Plugins -> Deployment -> Hybrid-Cloud -> About
+
i. Go to Plugins -> Deployment -> Hybrid-Cloud -> About
[[File:Csm 02-openqrm-ec2 032d98a502.png|thumb|350px|Plugin Manager]]
+
[[File:Csm 02-openqrm-ec2 032d98a502.png|thumb|390x390px|Plugin Manager|alt=|none]]
 
There you can find the URLs and information about the latest Amazon EC2 API- and AMI-Tools.
 
There you can find the URLs and information about the latest Amazon EC2 API- and AMI-Tools.
  
Here the steps to install the Amazon EC2 Tools. Please SSH-login to the openQRM server as 'root' and run the following commands:
+
ii. Follow these steps to install the Amazon EC2 Tools.  
 +
 
 +
Please SSH-login to the openQRM server as 'root' and run the following commands:
  
 
<blockquote>
 
<blockquote>
Line 69: Line 71:
 
''mkdir /usr/local/ec2''<br>
 
''mkdir /usr/local/ec2''<br>
 
''cp -r ec2-ami-tools-1.4.0.9/* /usr/local/ec2/''<br>
 
''cp -r ec2-ami-tools-1.4.0.9/* /usr/local/ec2/''<br>
''cp -r ec2-api-tools-1.6.8.1/* /usr/local/ec2/''<br>
+
''cp -r ec2-api-tools-1.6.8.1/* /usr/local/ec2/''
 +
 
 
''apt-get update && apt-get install default-jdk''</blockquote>
 
''apt-get update && apt-get install default-jdk''</blockquote>
  
Line 76: Line 79:
 
Then please add the following to the system-wide profile /etc/profile:
 
Then please add the following to the system-wide profile /etc/profile:
  
<blockquote>''# EC2 Tools''<br>
+
<blockquote>''# EC2 Tools''
 
''export EC2_HOME=/usr/local/ec2''<br>
 
''export EC2_HOME=/usr/local/ec2''<br>
''export PATH=$PATH:$EC2_HOME/bin''<br>
+
''export PATH=$PATH:$EC2_HOME/bin''
 +
 
 
''export JAVA_HOME=/usr''<br></blockquote>
 
''export JAVA_HOME=/usr''<br></blockquote>
  
Line 114: Line 118:
  
 
== 5. Create a Hybrid-Cloud Account ==
 
== 5. Create a Hybrid-Cloud Account ==
Go to Plugins -> Deployment -> Hybrid-Cloud -> Actions and click on 'Add new Account'
+
i. Go to Plugins -> Deployment -> Hybrid-Cloud -> Actions and click on 'Add new Account'
[[File:csm_03-openqrm-ec2_abdea0a4c9.png|thumb|350px|Account Selection Screen]]
+
[[File:csm_03-openqrm-ec2_abdea0a4c9.png|thumb|390x390px|Account Selection Screen|alt=|none]]
  
Provide an account name and the AWS Access and Secret Key plus a description for the account. Then click on submit.
+
ii. Provide an account name and the AWS Access and Secret Key plus a description for the account. Then click on submit.
[[File:csm_04-openqrm-ec2_9625f7f297.png|thumb|350px|Add new Account screen]]
+
[[File:csm_04-openqrm-ec2_9625f7f297.png|thumb|390x390px|Add new Account screen|alt=|none]]
  
Adding the account is then checking to get access via the provided credentials. If configured correctly the account is added as seen below.
+
iii. Adding the account is then checking to get access via the provided credentials. If configured correctly the account is added as seen below.
  
 
You can now easily access all kind of Amazon EC2 functionalities through the different action buttons.
 
You can now easily access all kind of Amazon EC2 functionalities through the different action buttons.
[[File:csm_05-openqrm-ec2_74b883d91e.png|thumb|350px|Account Selection Screen]]
+
[[File:csm_05-openqrm-ec2_74b883d91e.png|thumb|390x390px|Account Selection Screen|alt=|none]]
  
 
== 6. Choose AMIs for deployment==
 
== 6. Choose AMIs for deployment==
Go to Plugins -> Deployment -> Hybrid-Cloud -> About
+
i. Go to Plugins -> Deployment -> Hybrid-Cloud -> About
[[File:csm_02-openqrm-ec2_032d98a502.png|thumb|350px|About Hybrid-Cloud page]]
+
[[File:csm_02-openqrm-ec2_032d98a502.png|thumb|390x390px|About Hybrid-Cloud page|alt=|none]]
 +
 
 +
ii. In the section 'Manage and automate public and private clouds' -> AMIs you can find a URL to some current Ubuntu AMIs. Please open the url and find an AMI of your choice in the region of your choice. For this How-To will will use a 'Ubuntu 13.04 64bit' AMI named       
  
In the section 'Manage and automate public and private clouds' -> AMIs you can find a URL to some current Ubuntu AMIs. Please open the url and find an AMI of your choice in the region of your choice. For this howto will will use a 'Ubuntu 13.04 64bit' AMI named 'ami-23a9b057'.
+
'ami-23a9b057'.
  
 
'''Note:''' Those AMIs are updated frequently so the AMI name may change!
 
'''Note:''' Those AMIs are updated frequently so the AMI name may change!
[[File:csm_06-openqrm-ec2_1cfbb96e0d.png|thumb|350px|Ubuntu AMIs Selection Screen]]
+
[[File:csm_06-openqrm-ec2_1cfbb96e0d.png|thumb|390x390px|Ubuntu AMIs Selection Screen|alt=|none]]
 
 
Now go to Plugins -> Deployment -> Hybrid-Cloud -> Actions -> AMI. This will list available AMIs in the selected region.
 
[[File:csm_07-openqrm-ec2_7c578d7f38.png|thumb|350px|Add/Remove Image for AMIs page]]
 
  
 +
iii. Now go to Plugins -> Deployment -> Hybrid-Cloud -> Actions -> AMI. This will list available AMIs in the selected region.
 +
[[File:csm_07-openqrm-ec2_7c578d7f38.png|thumb|390x390px|Add/Remove Image for AMIs page|alt=|none]]
  
Click on the AMI filter 'U' button and find the AMI you have selected on the Ubuntu AMI page. Click on 'Add Image' for that AMI.
+
iv. Click on the AMI filter 'U' button and find the AMI you have selected on the Ubuntu AMI page. Click on 'Add Image' for that AMI.
[[File:csm_08-openqrm-ec2_0bec5042bc.png|thumb|350px|Add Image to AMI]]
+
[[File:csm_08-openqrm-ec2_0bec5042bc.png|thumb|390x390px|Add Image to AMI|alt=|none]]
  
 
+
v. This creates a new available Image object in the openQRM server.
This creates a new available Image object in the openQRM server.
+
[[File:csm_09-openqrm-ec2_ffb87a7a55.png|thumb|390x390px|New Image object is created|alt=|none]]
[[File:csm_09-openqrm-ec2_ffb87a7a55.png|thumb|350px|New Image object is created]]
 
  
 
== 7. Create a custom auto-configuration script to the EC2 Instance on S3 ==
 
== 7. Create a custom auto-configuration script to the EC2 Instance on S3 ==
Line 154: Line 158:
  
 
Now go to Plugins -> Deployment -> Hybrid-Cloud -> Actions -> S3 and create a new S3 bucket.
 
Now go to Plugins -> Deployment -> Hybrid-Cloud -> Actions -> S3 and create a new S3 bucket.
 
+
[[File:Csm 10-openqrm-ec2 194d0af9a0.png|none|thumb|390x390px|Create S3 bucket page]]
 
 
 
Click on 'Files in bucket' to list the files in the bucket.
 
Click on 'Files in bucket' to list the files in the bucket.
 
+
[[File:Csm 11-openqrm-ec2 d9809cefc0.png|none|thumb|390x390px|Bucket list page]]
 
 
 
Click on 'Upload file' button to upload the custom 'my-custom-auto-configure.sh' script from your desktop.
 
Click on 'Upload file' button to upload the custom 'my-custom-auto-configure.sh' script from your desktop.
 
+
[[File:Csm 12-openqrm-ec2 7a76a2eba9.png|none|thumb|390x390px|Upload 'my-custom-auto-configure.sh' script]]
 
+
Select the 'my-custom-auto-configure.sh' script from your desktop, set the File Permission to 'public-read' and submit.
Select the 'my-custom-auto-configure.sh' srcript from your desktop, set the File Permission to 'public-read' and submit.
+
[[File:Csm 13-openqrm-ec2 8fa6ddd09d.png|none|thumb|390x390px|Configure uploaded script settings]]
 
+
The 'my-custom-auto-configure.sh' script got uploaded to the S3 bucket and is available via http. Please copy the URL of the uploaded script, we are going to paste the URL in the following 'Instance Add' dialog.
 
+
[[File:Csm 14-openqrm-ec2 dc080e6f68.png|none|thumb|390x390px|Fully configured custom script]]
The 'my-custom-auto-configure.sh' srcript got uploaded to the S3 bucket and is available via http. Please copy the URL of the uploaded script, we are going to paste the URL in the following 'Instance Add' dialog.
 
 
 
  
 
== 8. Pre-configure Nagios service checks ==
 
== 8. Pre-configure Nagios service checks ==
 
Now go to Plugins -> Monitoring -> Nagios3 -> Services. and login to the embedded Nagios server with the openqrm credentials.
 
Now go to Plugins -> Monitoring -> Nagios3 -> Services. and login to the embedded Nagios server with the openqrm credentials.
 
+
[[File:Csm 32-openqrm-ec2 a7a2765715 (1).png|none|thumb|390x390px|Nagios Server login]]
 
 
 
Here the standard Nagios configuration after the Debian package is installed.
 
Here the standard Nagios configuration after the Debian package is installed.
 
+
[[File:Csm 33-openqrm-ec2 2902ec1579.png|none|thumb|390x390px|Standard Nagios configuration installation]]
 
 
 
Now go to Plugins -> Monitoring -> Nagios3 -> Config -> Services. and click on 'Add new Service'
 
Now go to Plugins -> Monitoring -> Nagios3 -> Config -> Services. and click on 'Add new Service'
 
+
[[File:Csm 37-openqrm-ec2 9227b98e67.png|none|thumb|390x390px|Nagios3 Add New Service]]
 
 
 
Select the http service (Port 80) and click on 'Submit'
 
Select the http service (Port 80) and click on 'Submit'
 
+
[[File:Csm 36-openqrm-ec2 ac02e01129.png|none|thumb|390x390px|New service configuration]]
 
+
Your Nagios Service check list should look like this after adding the http service
Here the available Nagios Service check list after adding the http service
+
[[File:Csm 37-openqrm-ec2 9227b98e67.png|none|thumb|390x390px|Nagios Service check list]]
 
 
 
 
Here after adding some more service checks
 
 
 
  
 
== 9. Create a new Instance on Amazon EC2 ==
 
== 9. Create a new Instance on Amazon EC2 ==
Go to Datacenter -> Server. and click on 'Add a new Server'
+
Go to Datacentre -> Server. and click on 'Add a new Server'
 
+
[[File:Csm 15-openqrm-ec2 14cf155c65.png|none|thumb|390x390px|Add new server]]
 
 
 
Provide a name for the new server. Easiest is to use the 'Generate name' button.
 
Provide a name for the new server. Easiest is to use the 'Generate name' button.
 
+
[[File:Csm 16-openqrm-ec2 d220c08913.png|none|thumb|390x390px|Give the server a name]]
 
 
 
In the Resource-Select please click on 'New Resource'
 
In the Resource-Select please click on 'New Resource'
 
+
[[File:Csm 17-openqrm-ec2 cf734acce8.png|none|thumb|390x390px|Allocate resource for new server]]
 
 
 
Here the list of available Resource types. Please select 'Cloud (localboot) Virtual Machine'
 
Here the list of available Resource types. Please select 'Cloud (localboot) Virtual Machine'
 
+
[[File:Csm 18-openqrm-ec2 658b0f67ed.png|none|thumb|390x390px|Select Cloud (localboot) Virtual Machine]]
 
 
 
This forwards to the Hybrid-Cloud plugin actions. Please click on 'Instances'
 
This forwards to the Hybrid-Cloud plugin actions. Please click on 'Instances'
 
+
[[File:Csm 19-openqrm-ec2 dacbf66243.png|none|thumb|390x390px|Select Account Screen]]
 
 
 
The following screenshot shows the empty Instance list. Please click on 'Add Instance'
 
The following screenshot shows the empty Instance list. Please click on 'Add Instance'
 +
[[File:Csm 20-openqrm-ec2 2206ecc055.png|none|thumb|390x390px|Instances for new server]]
 +
In the Instance - Select the AMI, the availability zone, type, keypair and security group.
  
 
+
'''Hint:''' Keypairs can be managed via the 'Keypair' action, security groups can be managed with the 'Groups' action!
In the Instance-Add for please select the AMI, the availability zone, type, keypair and security group.
 
 
 
Hint: Keypairs can be managed via the 'Keypair' action, security groups can be managed with the 'Groups' action!
 
  
 
Click on submit when finished.
 
Click on submit when finished.
 
+
[[File:Csm 21-openqrm-ec2 6c2635adc6.png|none|thumb|390x390px|Click on submit]]
 
 
 
Creating a new Instance automatically creates a new resource in openQRM and forward back into the server wizard. Please select the just created new resource and click on 'Submit'
 
Creating a new Instance automatically creates a new resource in openQRM and forward back into the server wizard. Please select the just created new resource and click on 'Submit'
 
+
[[File:Csm 22-openqrm-ec2 9ad9e37f36.png|none|thumb|390x390px|Resource allocation]]
 
 
 
Next select the Image object created from the AMI before.
 
Next select the Image object created from the AMI before.
 
+
[[File:Csm 23-openqrm-ec2 0d73331be7.png|none|thumb|390x390px|Select the image from the AMI]]
 
 
 
On this screen please click on 'Submit' to edit the Image parameter.
 
On this screen please click on 'Submit' to edit the Image parameter.
 
+
[[File:Csm 24-openqrm-ec2 fbcf81a51f.png|none|thumb|390x390px|Click Submit]]
 
+
The Edit Image form allows to set a custom password for the server.
The Image-Edit form allows to set a custom password for the server.
+
[[File:Csm 25-openqrm-ec2 814a254e2f.png|none|thumb|390x390px|Set a password]]
 
+
'''Please notice:''' Normally SSH-Access to Amazon EC2 instances works only on behalf of a private- and public keypair. Amazon EC2 keypairs can be easily managed through the 'keypair' action. Anyway openQRM also allows to simply set a password in the Image-Edit section. Setting a password there automatically sets the password in the Instance AMI and enables to allow SSH-Login with password.
 
 
Please notice: Normally SSH-Access to Amazon EC2 instances works only on behalf of a private- and public keypair. Amazon EC2 keypairs can be easily managed through the 'keypair' action. Anyway openQRM also allows to simply set a password in the Image-Edit section. Setting a password there automatically sets the password in the Instance AMI and enables to allow SSH-Login with password.
 
  
 
Here the final Server-Edit screen. Click on 'Submit' to save the server configuration.
 
Here the final Server-Edit screen. Click on 'Submit' to save the server configuration.
 
+
[[File:Csm 26-openqrm-ec2 e518821b49.png|none|thumb|390x390px|Click on submit]]
  
 
== 10. Configure Puppet recipes for the EC2 Instance ==
 
== 10. Configure Puppet recipes for the EC2 Instance ==
Go again to Datacenter -> Server. and edit the just created server object.
+
Go again to Datacentre -> Server. and edit the just created server object.
 
+
[[File:Csm 27-openqrm-ec2 17087b845c.png|none|thumb|390x390px|Edit the newly created server object]]
 
 
 
Click on Deployment -> Puppet to add a custom Puppet recipe to the server.
 
Click on Deployment -> Puppet to add a custom Puppet recipe to the server.
 
+
[[File:Csm 28-openqrm-ec2 f34b3bc6ba.png|none|thumb|390x390px|Add a custom Puppet recipe]]
 
 
 
Here we choose the included 'webserver' puppet recipe which automatically installs and starts apache.
 
Here we choose the included 'webserver' puppet recipe which automatically installs and starts apache.
 
+
[[File:Csm 39-openqrm-ec2 97c6c68d47.png|none|thumb|390x390px|Select 'webserver']]
 
 
 
The overview of the Puppet deployment configuration looks now like this:
 
The overview of the Puppet deployment configuration looks now like this:
 
+
[[File:Csm 30-openqrm-ec2 eed9152a2a.png|none|thumb|390x390px|Puppet deployment configuration]]
  
 
== 11. Set up monitoring for the EC2 Instance ==
 
== 11. Set up monitoring for the EC2 Instance ==
Go again to Datacenter -> Server and edit the just created server object. Click on Deployment -> Nagios3
+
Go again to Datacentre -> Server and edit the just created server object. Click on Deployment -> Nagios3
 
+
[[File:Csm 31-openqrm-ec2 4eed9fc746.png|none|thumb|390x390px|Server deployment screen]]
 
 
 
In the Service-Edit form please select the 'http' service and click on 'Submit'
 
In the Service-Edit form please select the 'http' service and click on 'Submit'
 
+
[[File:Csm 39-openqrm-ec2 97c6c68d47.png|none|thumb|390x390px|Click Submit]]
 
+
The overview of the Nagios check configuration looks now like this:
The overview about the Nagios check configuration looks now like this:
+
[[File:Csm 40-openqrm-ec2 1c0a0daf90.png|none|thumb|390x390px|Nagios Check overview]]
 
 
  
 
== 12. Starting the EC2 Instance ==
 
== 12. Starting the EC2 Instance ==
To start the configured Amazon EC2 simply start its server object in Datacenter -> Server. This will create and start the Instance on the Amazon Public Cloud, apply the Image password configuration, apply the puppet recipes, configure WebSSH and execute the custom auto-configuration script we have attached to the Instance from S3.
+
To start the configured Amazon EC2 simply start its server object in Datacentre -> Server. This will create and start the Instance on the Amazon Public Cloud, apply the Image password configuration, apply the puppet recipes, configure WebSSH and execute the custom auto-configuration script we have attached to the Instance from S3.
 
 
Go to Datacenter -> Server. Select the new created server object and start it
 
 
 
 
 
The server object is now activated and the deployment of the Amazon EC2 instances has started.
 
 
 
 
 
Here a screenshot of the Amazon EC2 console after we have started the deployment.
 
 
 
 
 
You can now use the 'ssh' button in the server list at Datacenter -> Server to easily login to the Instance.
 
 
 
Please notice: Your browser will warn because of a self-signed ssl certificate for the WebSSH login! Please accept to login.
 
 
 
 
 
A quick check that the webserver is up+running
 
 
 
 
 
Here a screenshot of the embedded Nagios console with the http service check activated.
 
  
 +
i. Go to Datacentre -> Server. Select the new created server object and start it
 +
[[File:Csm 41-openqrm-ec2 cc31dfa6b6.png|thumb|390x390px|Datacentre Server Selection page|alt=|none]]
  
Also please re-check /tmp/my-custom-auto-configure.log on the Instance to see you custom script got executed.
+
ii. The server object is now activated and the deployment of the Amazon EC2 instances has started.[[File:Csm 42-openqrm-ec2 90a06ab354.png|thumb|390x390px|Amazon EC2 console after we have started the deployment.|alt=|none]]
  
And here the Datacenter Dashboard after we have created the Amazon EC2 Instance
+
[[File:Csm 43-openqrm-ec2 6d8e970430.png|thumb|390x390px|alt=|none]]
  
 +
iv. You can now use the 'ssh' button in the server list at Datacenter -> Server to easily login to the Instance.
  
You can now fully automate your Amazon EC2 deployment with openQRM
+
'''Note:''' Your browser will warn because of a self-signed ssl certificate for the WebSSH login! Please accept to login.
 +
[[File:Csm 48-openqrm-ec2 e720c7fc36.png|thumb|390x390px|A quick check that the webserver is up+running|alt=|none]]
  
Hope you enjoyed this Howto!
+
v. Here a screenshot of the embedded Nagios console with the http service check activated.[[File:Csm 46-openqrm-ec2 2d5b45ef14.png|thumb|390x390px|alt=|none]]
  
Add more functionalities to your openQRM Setup
+
vi. Please re-check /tmp/my-custom-auto-configure.log on the Instance to see you custom script got executed.
To continue and further enhance your openQRM KVM Virtualization Setup there are several things to do:
 
  
Enable the highavailability plugin to automatically gain HA for your server
+
vii. And here's the Datacentre Dashboard after we have created the Amazon EC2 Instance
Enable the cloud plugin for a complete Self-Service deployment of your Server and Software stack to end-users
+
[[File:Csm 47-openqrm-ec2 9d58093544.png|thumb|390x390px|You can now fully automate your Amazon EC2 deployment with openQRM|alt=|none]]
Enable further Virtualization plugins and integrate remote Virtulization hosts for a fully distributed Cloud environment
+
[[Category:Tutorial]]
Enable further Storage and Deployment plugins to automatically provision your Virtualization Hosts and other physical systems
+
[[Category:Automate]]
... and more.
+
[[Category:Amazon EC2]]
Links
+
[[Category:Deployment]]
openQRM Community: http://www.openqrm.com/
+
[[Category:Debian]]
openQRM Project at sourceforge: http://sourceforge.net/projects/openqrm/
+
[[Category:Cloud]]
openQRM Enterprise: http://www.openqrm-enterprise.com/
 
openQRM at Twitter: https://twitter.com/openQRM
 
openQRM at Facebook: https://www.facebook.com/pages/openQRM-Enterprise/324904179687
 
Amazon EC2: http://aws.amazon.com/ec2/http://linuxcoe.sourceforge.net/
 

Latest revision as of 10:12, 5 November 2020

This How-To is about how to manage Public and Hybrid Cloud deployments with openQRM. As the deployment manager for Amazon EC2 and its API compatible derivatives (e.g Eucalyptus) openQRM is capable to fully automate Instance provisioning and to add additional value by attaching automated application deployment via Puppet, automated monitoring via Nagios and also high availability on Infrastructure-Level to the providers cloud features. The whole workflow of Instance-deployment in openQRM is exactly the same as for local resources in the internal IT-environment.

Requirements

  1. One physical Server. Alternatively, the installation can be also done within a Virtual Machine
  2. At least 1 GB of Memory
  3. At least 100 GB of Diskspace
  4. Optional VT (Virtualization Technology) enabled in the Systems BIOS so that the openQRM Server can run KVM Virtual Machines later

Install openQRM on Debian

  1. Install a minimal Debian on a physical Server
  2. Install and initialize openQRM

A detailed How-To about the above initial starting point is available at Install openQRM on Debian

For this How-To, we have used the same openQRM server as for the How-To Virtualisation with KVM and openQRM on Debian. That means with this How-To we are going to add functionality to an existing openQRM setup. This is to show that openQRM manages all different virtualization and deployment types seamlessly.

This means you can use either the Install openQRM on Debian or Virtualisation with KVM and openQRM on Debian How-To as a starting point.

Step-by-Step

Set a custom Domain name

As the first step after the openQRM installation and initialisation, it is recommended to configure a custom domain name for the openQRM management network.

In this Use-Case the openQRM Server has the private Class C IP address 192.168.178.5/255.255.255.0 based on the previous How-To Install openQRM on Debian. Since the openQRM management network is a private one any syntactically correct domain name can be used e.g. 'my123cloud.net'. The default domain name pre-configured in the DNS plugin is "oqnet.org".

The best practice is to use the 'openqrm' commandline util to setup the domain name for the DNS plugin. Please login to the openQRM Server system and run the following command as 'root' in a terminal:

/usr/share/openqrm/bin/openqrm boot-service configure -n dns -a default -k OPENQRM_SERVER_DOMAIN -v my123cloud.net

The output of the above command will look like:

root@debian:~# /usr/share/openqrm/bin/openqrm boot-service configure -n dns -a default -k OPENQRM_SERVER_DOMAIN -v my123cloud.net

Setting up default Boot-Service Konfiguration of plugin dns

root@debian:~#

To (re)view the current configuration of the DNS plugin please run:

/usr/share/openqrm/bin/openqrm boot-service view -n dns -a default

2. Enabling Plugins

For this How-To please enable and start the following plugins in the sequence below:

  1. dns plugin - type Networking
  2. dhcpd plugin - type Networking
  3. tftpd plugin - type Networking
  4. device-manager plugin - type Management
  5. nfs-storage - type Storage
  6. lvm-storage - type Storage
  7. nagios3 - type Monitoring
  8. puppet - type Deployment
  9. sshterm plugin - type Management
  10. hybrid-cloud - type Deployment

Hint: You can use the filter in the plugin list to find plugins by their type easily!

Plugin Manager

3. Install the latest Amazon EC2 Tools

i. Go to Plugins -> Deployment -> Hybrid-Cloud -> About

Plugin Manager

There you can find the URLs and information about the latest Amazon EC2 API- and AMI-Tools.

ii. Follow these steps to install the Amazon EC2 Tools.

Please SSH-login to the openQRM server as 'root' and run the following commands:

wget s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
wget s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
unzip ec2-api-tools.zip
unzip ec2-ami-tools.zip
mkdir /usr/local/ec2
cp -r ec2-ami-tools-1.4.0.9/* /usr/local/ec2/
cp -r ec2-api-tools-1.6.8.1/* /usr/local/ec2/

apt-get update && apt-get install default-jdk

Note: The version numbers may be different when newer EC2 Tools become available!

Then please add the following to the system-wide profile /etc/profile:

# EC2 Tools

export EC2_HOME=/usr/local/ec2
export PATH=$PATH:$EC2_HOME/bin

export JAVA_HOME=/usr

The EC2 API- and AMI Tools are now installed and available in the system path.

Now please log out of the openQRM server and re-login. This is to activate the new profile settings in the environment. After re-login please restart the openQRM server to also activate the profile in its environment by running:

/etc/init.d/openqrm restart

To re-check the configuration please run:

ec2-describe-regions -O [your-aws-access-key] -W [your-aws-secret-key]

The output of the above command looks like:

root@debian:~# ec2-describe-regions -O XXXXXXXXXXXXXXXXXXXXX -W YYYYYYYYYYYYYYYYYYYYYYYY

REGION eu-west-1 ec2.eu-west-1.amazonaws.com
REGION sa-east-1 ec2.sa-east-1.amazonaws.com
REGION us-east-1 ec2.us-east-1.amazonaws.com
REGION ap-northeast-1 ec2.ap-northeast-1.amazonaws.com
REGION us-west-2 ec2.us-west-2.amazonaws.com
REGION us-west-1 ec2.us-west-1.amazonaws.com
REGION ap-southeast-1 ec2.ap-southeast-1.amazonaws.com
REGION ap-southeast-2 ec2.ap-southeast-2.amazonaws.com

root@debian:~#

4. Configure which Amazon EC2 regions to use

Best practice is to use the 'openqrm' commandline util to setup which Amazon regions to use for the hybrid-cloud plugin. Please login to the openQRM Server system and run the following command as 'root' in a terminal:

/usr/share/openqrm/bin/openqrm boot-service configure -n hybrid-cloud -a default -k OPENQRM_PLUGIN_HYBRID_CLOUD_REGIONS -v "eu-west-1, us-west-1"

To (re)view the current configuration of the Hybrid-Cloud plugin please run:

/usr/share/openqrm/bin/openqrm boot-service view -n hybrid-cloud -a default

5. Create a Hybrid-Cloud Account

i. Go to Plugins -> Deployment -> Hybrid-Cloud -> Actions and click on 'Add new Account'

Account Selection Screen

ii. Provide an account name and the AWS Access and Secret Key plus a description for the account. Then click on submit.

Add new Account screen

iii. Adding the account is then checking to get access via the provided credentials. If configured correctly the account is added as seen below.

You can now easily access all kind of Amazon EC2 functionalities through the different action buttons.

Account Selection Screen

6. Choose AMIs for deployment

i. Go to Plugins -> Deployment -> Hybrid-Cloud -> About

About Hybrid-Cloud page

ii. In the section 'Manage and automate public and private clouds' -> AMIs you can find a URL to some current Ubuntu AMIs. Please open the url and find an AMI of your choice in the region of your choice. For this How-To will will use a 'Ubuntu 13.04 64bit' AMI named

'ami-23a9b057'.

Note: Those AMIs are updated frequently so the AMI name may change!

Ubuntu AMIs Selection Screen

iii. Now go to Plugins -> Deployment -> Hybrid-Cloud -> Actions -> AMI. This will list available AMIs in the selected region.

Add/Remove Image for AMIs page

iv. Click on the AMI filter 'U' button and find the AMI you have selected on the Ubuntu AMI page. Click on 'Add Image' for that AMI.

Add Image to AMI

v. This creates a new available Image object in the openQRM server.

New Image object is created

7. Create a custom auto-configuration script to the EC2 Instance on S3

The integration with Amazon EC2 in openQRM allows to attach a custom script to a starting Instance. The Instance is then running this script on system startup. This can be used in combination with the Puppet integration to fully pre-configure an Instance in EC2. The easiest way to create such a custom auto-configuration script is to use the S3 action in the account overview. This provides you with a File-Manager for S3 and allows to easily upload files to S3. Those files, if set to 'public-read' permission is directly available via http. As an example we create a small bash-script which actually just outputs some text to a file.

On your Desktop create a new file named 'my-custom-auto-configure.sh' with the following content:

  1. !/bin/bash

echo "Here custom commands are running on instance startup" > /tmp/my-custom-auto-configure.log

Now go to Plugins -> Deployment -> Hybrid-Cloud -> Actions -> S3 and create a new S3 bucket.

Create S3 bucket page

Click on 'Files in bucket' to list the files in the bucket.

Bucket list page

Click on 'Upload file' button to upload the custom 'my-custom-auto-configure.sh' script from your desktop.

Upload 'my-custom-auto-configure.sh' script

Select the 'my-custom-auto-configure.sh' script from your desktop, set the File Permission to 'public-read' and submit.

Configure uploaded script settings

The 'my-custom-auto-configure.sh' script got uploaded to the S3 bucket and is available via http. Please copy the URL of the uploaded script, we are going to paste the URL in the following 'Instance Add' dialog.

Fully configured custom script

8. Pre-configure Nagios service checks

Now go to Plugins -> Monitoring -> Nagios3 -> Services. and login to the embedded Nagios server with the openqrm credentials.

Nagios Server login

Here the standard Nagios configuration after the Debian package is installed.

Standard Nagios configuration installation

Now go to Plugins -> Monitoring -> Nagios3 -> Config -> Services. and click on 'Add new Service'

Nagios3 Add New Service

Select the http service (Port 80) and click on 'Submit'

New service configuration

Your Nagios Service check list should look like this after adding the http service

Nagios Service check list

9. Create a new Instance on Amazon EC2

Go to Datacentre -> Server. and click on 'Add a new Server'

Add new server

Provide a name for the new server. Easiest is to use the 'Generate name' button.

Give the server a name

In the Resource-Select please click on 'New Resource'

Allocate resource for new server

Here the list of available Resource types. Please select 'Cloud (localboot) Virtual Machine'

Select Cloud (localboot) Virtual Machine

This forwards to the Hybrid-Cloud plugin actions. Please click on 'Instances'

Select Account Screen

The following screenshot shows the empty Instance list. Please click on 'Add Instance'

Instances for new server

In the Instance - Select the AMI, the availability zone, type, keypair and security group.

Hint: Keypairs can be managed via the 'Keypair' action, security groups can be managed with the 'Groups' action!

Click on submit when finished.

Click on submit

Creating a new Instance automatically creates a new resource in openQRM and forward back into the server wizard. Please select the just created new resource and click on 'Submit'

Resource allocation

Next select the Image object created from the AMI before.

Select the image from the AMI

On this screen please click on 'Submit' to edit the Image parameter.

Click Submit

The Edit Image form allows to set a custom password for the server.

Set a password

Please notice: Normally SSH-Access to Amazon EC2 instances works only on behalf of a private- and public keypair. Amazon EC2 keypairs can be easily managed through the 'keypair' action. Anyway openQRM also allows to simply set a password in the Image-Edit section. Setting a password there automatically sets the password in the Instance AMI and enables to allow SSH-Login with password.

Here the final Server-Edit screen. Click on 'Submit' to save the server configuration.

Click on submit

10. Configure Puppet recipes for the EC2 Instance

Go again to Datacentre -> Server. and edit the just created server object.

Edit the newly created server object

Click on Deployment -> Puppet to add a custom Puppet recipe to the server.

Add a custom Puppet recipe

Here we choose the included 'webserver' puppet recipe which automatically installs and starts apache.

Select 'webserver'

The overview of the Puppet deployment configuration looks now like this:

Puppet deployment configuration

11. Set up monitoring for the EC2 Instance

Go again to Datacentre -> Server and edit the just created server object. Click on Deployment -> Nagios3

Server deployment screen

In the Service-Edit form please select the 'http' service and click on 'Submit'

Click Submit

The overview of the Nagios check configuration looks now like this:

Nagios Check overview

12. Starting the EC2 Instance

To start the configured Amazon EC2 simply start its server object in Datacentre -> Server. This will create and start the Instance on the Amazon Public Cloud, apply the Image password configuration, apply the puppet recipes, configure WebSSH and execute the custom auto-configuration script we have attached to the Instance from S3.

i. Go to Datacentre -> Server. Select the new created server object and start it

Datacentre Server Selection page

ii. The server object is now activated and the deployment of the Amazon EC2 instances has started.

Amazon EC2 console after we have started the deployment.

iv. You can now use the 'ssh' button in the server list at Datacenter -> Server to easily login to the Instance.

Note: Your browser will warn because of a self-signed ssl certificate for the WebSSH login! Please accept to login.

A quick check that the webserver is up+running

v. Here a screenshot of the embedded Nagios console with the http service check activated.

vi. Please re-check /tmp/my-custom-auto-configure.log on the Instance to see you custom script got executed.

vii. And here's the Datacentre Dashboard after we have created the Amazon EC2 Instance

You can now fully automate your Amazon EC2 deployment with openQRM