openQRM - User contributions [en]

How to build Proxmox tmpfs image

2025-02-23T09:53:14Z

Stvsyf: change ownership of /var/log/pveproxy/

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''Ensure apparmor is removed;'''

apt remove --assume-yes --purge apparmor

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## mount --bind /tmp /exports/proxmox_image/tmp
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common python3 snmpd
## apt-get install python-is-python3
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
<code>[Service]

TTYVTDisallocate=no
</code>
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
#Remember to change ownership of /var/log/pveproxy/
##chown -R www-data:www-data /var/log/pveproxy
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
## umount /exports/proxmox_image/tmp
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### #/bin/systemctl disable ksm.service ksmtuned.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service open-iscsi.service --root /exports/proxmox_image/
### #/bin/systemctl disable iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pve-daily-update.timer --root /exports/proxmox_image/
### #/bin/systemctl disable pvesr.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service --root /exports/proxmox_image/
### /bin/systemctl disable dm-event.socket rbdmap.service --root /exports/proxmox_image/
### #/bin/systemctl disable smartd.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable rsync.service --root /exports/proxmox_image/
### #/bin/systemctl disable netdiag.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable dropbear nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### #/bin/systemctl disable nfs-ganesha-lock --root /exports/proxmox_image/
### /bin/systemctl disable nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall pvestatd pveproxy pvedaemon spiceproxy qmeventd rrdcached lxc pve-ha-crm pve-ha-lrm pve-lxc-syscalld lxcfs lxc-net lxc-monitord --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pveproxy pve-cluster corosync pvestatd rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-11-25T21:35:27Z

Stvsyf: added tmp mounts

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''Ensure apparmor is removed;'''

apt remove --assume-yes --purge apparmor

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## mount --bind /tmp /exports/proxmox_image/tmp
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common python3 snmpd
## apt-get install python-is-python3
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
<code>[Service]

TTYVTDisallocate=no
</code>
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
## umount /exports/proxmox_image/tmp
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### #/bin/systemctl disable ksm.service ksmtuned.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service open-iscsi.service --root /exports/proxmox_image/
### #/bin/systemctl disable iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pve-daily-update.timer --root /exports/proxmox_image/
### #/bin/systemctl disable pvesr.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service --root /exports/proxmox_image/
### /bin/systemctl disable dm-event.socket rbdmap.service --root /exports/proxmox_image/
### #/bin/systemctl disable smartd.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable rsync.service --root /exports/proxmox_image/
### #/bin/systemctl disable netdiag.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable dropbear nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### #/bin/systemctl disable nfs-ganesha-lock --root /exports/proxmox_image/
### /bin/systemctl disable nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall pvestatd pveproxy pvedaemon spiceproxy qmeventd rrdcached lxc pve-ha-crm pve-ha-lrm pve-lxc-syscalld lxcfs lxc-net lxc-monitord --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pveproxy pve-cluster corosync pvestatd rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-09-17T04:50:10Z

Stvsyf: updating systemctl

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''Ensure apparmor is removed;'''

apt remove --assume-yes --purge apparmor

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common python3 snmpd
## apt-get install python-is-python3
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
<code>[Service]

TTYVTDisallocate=no
</code>
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### #/bin/systemctl disable ksm.service ksmtuned.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service open-iscsi.service --root /exports/proxmox_image/
### #/bin/systemctl disable iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pve-daily-update.timer --root /exports/proxmox_image/
### #/bin/systemctl disable pvesr.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service --root /exports/proxmox_image/
### /bin/systemctl disable dm-event.socket rbdmap.service --root /exports/proxmox_image/
### #/bin/systemctl disable smartd.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable rsync.service --root /exports/proxmox_image/
### #/bin/systemctl disable netdiag.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable dropbear nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### #/bin/systemctl disable nfs-ganesha-lock --root /exports/proxmox_image/
### /bin/systemctl disable nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall pvestatd pveproxy pvedaemon spiceproxy qmeventd rrdcached lxc pve-ha-crm pve-ha-lrm pve-lxc-syscalld lxcfs lxc-net lxc-monitord --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pveproxy pve-cluster corosync pvestatd rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-09-17T04:23:33Z

Stvsyf: remove deprecated & non-existant services

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''Ensure apparmor is removed;'''

apt remove --assume-yes --purge apparmor

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common python3 snmpd
## apt-get install python-is-python3
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
<code>[Service]

TTYVTDisallocate=no
</code>
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### #/bin/systemctl disable ksm.service ksmtuned.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service open-iscsi.service --root /exports/proxmox_image/
### #/bin/systemctl disable iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pve-daily-update.timer --root /exports/proxmox_image/
### #/bin/systemctl disable pvesr.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service --root /exports/proxmox_image/
### /bin/systemctl disable dm-event.socket rbdmap.service --root /exports/proxmox_image/
### #/bin/systemctl disable smartd.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable rsync.service --root /exports/proxmox_image/
### #/bin/systemctl disable netdiag.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable dropbear nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### #/bin/systemctl disable nfs-ganesha-lock --root /exports/proxmox_image/
### /bin/systemctl disable nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pveproxy pve-cluster corosync pvestatd rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-08-20T04:55:41Z

Stvsyf: adding snmpd

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''Ensure apparmor is removed;'''

apt remove --assume-yes --purge apparmor

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common python3 snmpd
## apt-get install python-is-python3
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
<code>[Service]

TTYVTDisallocate=no
</code>
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-08-19T05:41:13Z

Stvsyf:

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''Ensure apparmor is removed;'''

apt remove --assume-yes --purge apparmor

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common python3
## apt-get install python-is-python3
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
<code>[Service]

TTYVTDisallocate=no
</code>
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-08-19T05:40:31Z

Stvsyf:

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''Ensure apparmor is removed;'''

apt remove --assume-yes --purge apparmor

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common python3
## apt-get install python-is-python3
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
<code>[Service]
TTYVTDisallocate=no
</code>
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-08-19T05:39:29Z

Stvsyf:

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''Ensure apparmor is removed;'''

apt remove --assume-yes --purge apparmor

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common python3
## apt-get install python-is-python3
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
<code>
[Service]
TTYVTDisallocate=no
</code>
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-08-19T05:14:10Z

Stvsyf: adding python3

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''Ensure apparmor is removed;'''

apt remove --assume-yes --purge apparmor

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common python3
## apt-get install python-is-python3
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-08-19T03:30:16Z

Stvsyf: remove apparmor

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''Ensure apparmor is removed;'''

apt remove --assume-yes --purge apparmor

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-08-19T00:47:32Z

Stvsyf: comment out bind bind and make-rprivate

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## #mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## #mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-08-19T00:39:17Z

Stvsyf: add extra bind and make-rprivate for debian 12

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## mount --bind /exports/proxmox_image/ /exports/proxmox_image/
## mount --make-rprivate /exports/proxmox_image/
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-08-18T23:44:07Z

Stvsyf: update docs

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## Boostrap with either Debian 11 or 12
### Debian 11:
#### debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
### Debian 12:
#### debootstrap --arch amd64 bookworm /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get update; apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-07-03T02:09:38Z

Stvsyf: adding Debian 12 support

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) -
## [http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb Proxmox 8 - http://download.proxmox.com/debian/dists/bookworm/pve-no-subscription/binary-amd64/proxmox-kernel-6.5.13-5-pve_6.5.13-5_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb Proxmox 7 (Stable) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb]
## [http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve 6.2.9-1 amd64.deb Proxmox 7 (Testing) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-6.2.9-1-pve_6.2.9-1_amd64.deb]
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get -y install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @
### Proxmox 7 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### Proxmox 8 - https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
#### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get -y install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get -y install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-07-03T02:02:39Z

Stvsyf: adding yes option to apt-get

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install -y debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2024-01-14T19:30:41Z

Stvsyf: adding about openqrm

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''About openQRM:'''

openQRM Enterprise is a Turn Key Deployment and Management Platform, with over 55 plugins allowing variety of deployment options. This article describes the deployment methods to convert Proxmox into a tmpfs image allowing servers to PXE boot and Run Proxmox as a memory resident operating system requiring now attached storage. This is perfect for compute nodes and allow KVM and LXC to operate as normal. Proxmox can connect to a variety of storage options including; NFS, Ceph, Gluster, iSCSI and more.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-12-20T23:45:20Z

Stvsyf: disable rc.local

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable rc-local --root /exports/proxmox_image/
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

Ceph OSD Creation Failure

2023-12-19T19:38:22Z

Stvsyf: update osds

[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Ceph]]

Sometimes the bootstrap-osd keyring can not be found.

Run this command;

<code>ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring</code>

To rebuild /var/lib/ceph/osd, try this command;

<code>ceph-volume lvm activate --all</code>

Ceph OSD Creation Failure

2023-12-19T06:03:37Z

Stvsyf: adding ceph volume lvm activate

[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Ceph]]

Sometimes the bootstrap-osd keyring can not be found.

Run this command;

<code>ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring</code>

Sometimes when running a tmpfs node you may lose /var/lib/ceph/osd, try this command;

<code>ceph-volume lvm activate --all</code>

How to build Proxmox tmpfs image

2023-12-18T00:56:39Z

Stvsyf: disable frr pvenetcommit

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable frr.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-12-17T11:31:47Z

Stvsyf: remove disable ssh.service

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

Ceph OSD Creation Failure

2023-12-17T05:49:47Z

Stvsyf:

[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Ceph]]

Sometimes the bootstrap-osd keyring can not be found.

Run this command;

ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

Ceph OSD Creation Failure

2023-12-17T05:49:05Z

Stvsyf:

[[Category:Tutorials]]
Sometimes the bootstrap-osd keyring can not be found.

Run this command;

ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

Ceph OSD Creation Failure

2023-12-17T05:48:39Z

Stvsyf: Ceph OSD Creation Failure

Sometimes the bootstrap-osd keyring can not be found.

Run this command;

ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

How to build Proxmox tmpfs image

2023-12-17T04:34:16Z

Stvsyf: disable extra services managed through ATU plugin

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/| grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service collectd.service ksm.service ksmtuned.service proxmox-boot-cleanup.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-12-15T00:25:53Z

Stvsyf: adding zsh-common dependency

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool procmail zsh-common
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-12-14T23:34:33Z

Stvsyf: symlink ssh to sshd

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool procmail
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
#symlink ssh.service to sshd.service required for pve-cluster;
##ln -s /usr/lib/systemd/system/ssh.service /etc/systemd/system/sshd.service
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-12-11T00:31:16Z

Stvsyf: add procmail dependency

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool procmail
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to unpack and pack an initrd

2023-11-27T22:31:08Z

Stvsyf:

[[Category:tutorial]]

Brief instructions to unpack and repack an image as an initrd;

'''Unpack an initrd'''

<code>unmkinitramfs initrd .</code>

'''Pack an initrd'''

<code>cd initrd

find . | cpio -o -H newc | gzip -9 > ../KERNEL_NAME.img
<code>

Configuring IPMI under Linux using ipmitool

2023-11-27T22:29:48Z

Stvsyf: adding category tutorial

[[Category:tutorial]]

Under Linux, the <code>ipmitool</code> command (<nowiki>http://ipmitool.sourceforge.net/</nowiki>) can be used for configuring IPMI for a server.

== Contents ==

* 1Hardware and Software Requirements
* 2LAN Configuration
** 2.1ipmitool lan print 1
* 3User Configuration
** 3.1Users at the USER Privilege Level

== Hardware and Software Requirements ==
The following example will show how to configure IPMI on a Linux server. The /dev/ipmi0 device file must exist so that configuration can be carried out. If it does not exist, you can create it as follows:

* under SuSE, Red Hat or CentOS: <code>/etc/init.d/ipmi start</code> (requires the OpenIPMI package. The OpenIPMI-tools package will be required later, as well.)
* under Debian 4: <code>/usr/share/ipmitool/ipmi.init.basic</code> (If the error message, ''ipmi_kcs_drv not found'', appears, you will have to comment the corresponding if-condition out, see also [1].)
* under Debian 5: <code>modprobe ipmi_devintf; modprobe ipmi_si</code>

The approach described below has been tested on an Intel SR2500 under CentOS 4 using ipmitool version 1.8.7. In principle, the configuration should be configured similarly on other systems with IPMI support.

== ipmitool lan print 1 ==
You can check the configuration using <code>ipmitool lan print 1</code>.
[root@sr2500 ~]# ipmitool lan print 1
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback :
: User :
: Operator :
: Admin : MD5
: OEM :
IP Address Source : Static Address
IP Address : 192.168.1.211
Subnet Mask : 255.255.255.0
MAC Address : 00:0e:0c:ea:92:a2
SNMP Community String :
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 192.168.1.254
Default Gateway MAC : 00:0e:0c:aa:8e:13
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
RMCP+ Cipher Suites : None
Cipher Suite Priv Max : XXXXXXXXXXXXXXX
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
[root@sr2500 ~]#

== LAN Configuration ==
The first IPMI LAN channel will now be configured. Thereby, the configured IP address can be accessed at the first LAN port for the server. For the default gateway, both its IP address and MAC address must be configured.
[root@sr2500 ~]# ipmitool lan set 1 ipsrc static
[root@sr2500 ~]# ipmitool lan set 1 ipaddr 192.168.1.211
Setting LAN IP Address to 192.168.1.211
[root@sr2500 ~]# ipmitool lan set 1 netmask 255.255.255.0
Setting LAN Subnet Mask to 255.255.255.0
[root@sr2500 ~]# ipmitool lan set 1 defgw ipaddr 192.168.1.254
Setting LAN Default Gateway IP to 192.168.1.254
[root@sr2500 ~]# ipmitool lan set 1 defgw macaddr 00:0e:0c:aa:8e:13
Setting LAN Default Gateway MAC to 00:0e:0c:aa:8e:13
[root@sr2500 ~]# ipmitool lan set 1 arp respond on
Enabling BMC-generated ARP responses
[root@sr2500 ~]# ipmitool lan set 1 auth ADMIN MD5
[root@sr2500 ~]# ipmitool lan set 1 access on
When configuring LANs, older versions of ipmitool would not automatically reset ''Set in Progress'' to ''Set Complete''. This can be done manually using a raw command (regarding this, see <nowiki>http://www.mail-archive.com/ipmitool-devel@lists.sourceforge.net/msg00095.html</nowiki>)
[root@sr2500 ~]# ipmitool lan print 1
Set in Progress : Set In Progress
[...]
[root@sr2500 ~]# ipmitool raw 0x0c 1 1 0 0

=== User Configuration ===
A user will now be setup with admin rights.
[root@sr2500 ~]# ipmitool user set name 2 admin
[root@sr2500 ~]# ipmitool user set password 2
Password for user 2:
Password for user 2:
[root@sr2500 ~]# ipmitool channel setaccess 1 2 link=on ipmi=on callin=on privilege=4
[root@sr2500 ~]# ipmitool user enable 2
[root@sr2500 ~]#
The server can now be controlled by this user as described in Using ipmitool for Remote Control of Servers.

=== Users at the USER Privilege Level ===
If a user should only be used for querying sensor data, a custom privilege level can be setup for that. This user then has no rights for activating or deactivating the server, for example. A user named monitor will be created for this in the following example:
[root@sr2500 ~]# ipmitool user set name 3 monitor
[root@sr2500 ~]# ipmitool user set password 3
Password for user 3:
Password for user 3:
[root@sr2500 ~]# ipmitool channel setaccess 1 3 link=on ipmi=on callin=on privilege=2
[root@sr2500 ~]# ipmitool user enable 3
[root@sr2500 ~]# ipmitool channel getaccess 1 3
Maximum User IDs : 15
Enabled User IDs : 2

User ID : 3
User Name : monitor
Fixed Name : No
Access Available : call-in / callback
Link Authentication : enabled
IPMI Messaging : enabled
Privilege Level : USER
[root@sr2500 ~]#
The importance of the various privilege numbers will be displayed when <code>ipmitool channel</code> is called without any additional parameters:
[root@sr2500 ~]# ipmitool channel
Channel Commands: authcap <channel number> <max privilege>
getaccess <channel number> [user id]
setaccess <channel number> <user id> [callin=on|off] [ipmi=on|off] [link=on|off] [privilege=level]
info [channel number]
getciphers <ipmi | sol> [channel]

Possible privilege levels are:
1 Callback level
2 User level
3 Operator level
4 Administrator level
5 OEM Proprietary level
15 No access
[root@sr2500 ~]#
The user just created (named 'monitor') has been assigned the USER privilege level. So that LAN access is allowed for this user, you must activate MD5 authentication for LAN access for this user group (USER privilege level):
[root@sr2500 ~]# ipmitool lan set 1 auth USER MD5
[root@sr2500 ~]#
MD5 will now also be listed as User Auth Type Enable for LAN Channel 1:
[root@sr2500 ~]# ipmitool lan print 1
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback :
: User : MD5
: Operator :
: Admin : MD5
: OEM :
IP Address Source : Static Address
IP Address : 192.168.1.211
Subnet Mask : 255.255.255.0
MAC Address : 00:0e:0c:ea:92:a2
SNMP Community String :
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 192.168.1.254
Default Gateway MAC : 00:0e:0c:aa:8e:13
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14
Cipher Suite Priv Max : XXXXXXXXXXXXXXX
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
[root@sr2500 ~]#
'''Please specify the option "-L USER" for ipmitool when using a user with USER privilege.''' Otherwise you will get an error message stating:
Activate Session error: Requested privilege level exceeds limit

Attribution: https://www.thomas-krenn.com/en/wiki/Configuring_IPMI_under_Linux_using_ipmitool

Configuring IPMI under Linux using ipmitool

2023-11-27T22:29:21Z

Stvsyf: Configuring IPMI under Linux using ipmitool

Under Linux, the <code>ipmitool</code> command (<nowiki>http://ipmitool.sourceforge.net/</nowiki>) can be used for configuring IPMI for a server.

== Contents ==

* 1Hardware and Software Requirements
* 2LAN Configuration
** 2.1ipmitool lan print 1
* 3User Configuration
** 3.1Users at the USER Privilege Level

== Hardware and Software Requirements ==
The following example will show how to configure IPMI on a Linux server. The /dev/ipmi0 device file must exist so that configuration can be carried out. If it does not exist, you can create it as follows:

* under SuSE, Red Hat or CentOS: <code>/etc/init.d/ipmi start</code> (requires the OpenIPMI package. The OpenIPMI-tools package will be required later, as well.)
* under Debian 4: <code>/usr/share/ipmitool/ipmi.init.basic</code> (If the error message, ''ipmi_kcs_drv not found'', appears, you will have to comment the corresponding if-condition out, see also [1].)
* under Debian 5: <code>modprobe ipmi_devintf; modprobe ipmi_si</code>

The approach described below has been tested on an Intel SR2500 under CentOS 4 using ipmitool version 1.8.7. In principle, the configuration should be configured similarly on other systems with IPMI support.

== ipmitool lan print 1 ==
You can check the configuration using <code>ipmitool lan print 1</code>.
[root@sr2500 ~]# ipmitool lan print 1
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback :
: User :
: Operator :
: Admin : MD5
: OEM :
IP Address Source : Static Address
IP Address : 192.168.1.211
Subnet Mask : 255.255.255.0
MAC Address : 00:0e:0c:ea:92:a2
SNMP Community String :
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 192.168.1.254
Default Gateway MAC : 00:0e:0c:aa:8e:13
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
RMCP+ Cipher Suites : None
Cipher Suite Priv Max : XXXXXXXXXXXXXXX
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
[root@sr2500 ~]#

== LAN Configuration ==
The first IPMI LAN channel will now be configured. Thereby, the configured IP address can be accessed at the first LAN port for the server. For the default gateway, both its IP address and MAC address must be configured.
[root@sr2500 ~]# ipmitool lan set 1 ipsrc static
[root@sr2500 ~]# ipmitool lan set 1 ipaddr 192.168.1.211
Setting LAN IP Address to 192.168.1.211
[root@sr2500 ~]# ipmitool lan set 1 netmask 255.255.255.0
Setting LAN Subnet Mask to 255.255.255.0
[root@sr2500 ~]# ipmitool lan set 1 defgw ipaddr 192.168.1.254
Setting LAN Default Gateway IP to 192.168.1.254
[root@sr2500 ~]# ipmitool lan set 1 defgw macaddr 00:0e:0c:aa:8e:13
Setting LAN Default Gateway MAC to 00:0e:0c:aa:8e:13
[root@sr2500 ~]# ipmitool lan set 1 arp respond on
Enabling BMC-generated ARP responses
[root@sr2500 ~]# ipmitool lan set 1 auth ADMIN MD5
[root@sr2500 ~]# ipmitool lan set 1 access on
When configuring LANs, older versions of ipmitool would not automatically reset ''Set in Progress'' to ''Set Complete''. This can be done manually using a raw command (regarding this, see <nowiki>http://www.mail-archive.com/ipmitool-devel@lists.sourceforge.net/msg00095.html</nowiki>)
[root@sr2500 ~]# ipmitool lan print 1
Set in Progress : Set In Progress
[...]
[root@sr2500 ~]# ipmitool raw 0x0c 1 1 0 0

=== User Configuration ===
A user will now be setup with admin rights.
[root@sr2500 ~]# ipmitool user set name 2 admin
[root@sr2500 ~]# ipmitool user set password 2
Password for user 2:
Password for user 2:
[root@sr2500 ~]# ipmitool channel setaccess 1 2 link=on ipmi=on callin=on privilege=4
[root@sr2500 ~]# ipmitool user enable 2
[root@sr2500 ~]#
The server can now be controlled by this user as described in Using ipmitool for Remote Control of Servers.

=== Users at the USER Privilege Level ===
If a user should only be used for querying sensor data, a custom privilege level can be setup for that. This user then has no rights for activating or deactivating the server, for example. A user named monitor will be created for this in the following example:
[root@sr2500 ~]# ipmitool user set name 3 monitor
[root@sr2500 ~]# ipmitool user set password 3
Password for user 3:
Password for user 3:
[root@sr2500 ~]# ipmitool channel setaccess 1 3 link=on ipmi=on callin=on privilege=2
[root@sr2500 ~]# ipmitool user enable 3
[root@sr2500 ~]# ipmitool channel getaccess 1 3
Maximum User IDs : 15
Enabled User IDs : 2

User ID : 3
User Name : monitor
Fixed Name : No
Access Available : call-in / callback
Link Authentication : enabled
IPMI Messaging : enabled
Privilege Level : USER
[root@sr2500 ~]#
The importance of the various privilege numbers will be displayed when <code>ipmitool channel</code> is called without any additional parameters:
[root@sr2500 ~]# ipmitool channel
Channel Commands: authcap <channel number> <max privilege>
getaccess <channel number> [user id]
setaccess <channel number> <user id> [callin=on|off] [ipmi=on|off] [link=on|off] [privilege=level]
info [channel number]
getciphers <ipmi | sol> [channel]

Possible privilege levels are:
1 Callback level
2 User level
3 Operator level
4 Administrator level
5 OEM Proprietary level
15 No access
[root@sr2500 ~]#
The user just created (named 'monitor') has been assigned the USER privilege level. So that LAN access is allowed for this user, you must activate MD5 authentication for LAN access for this user group (USER privilege level):
[root@sr2500 ~]# ipmitool lan set 1 auth USER MD5
[root@sr2500 ~]#
MD5 will now also be listed as User Auth Type Enable for LAN Channel 1:
[root@sr2500 ~]# ipmitool lan print 1
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback :
: User : MD5
: Operator :
: Admin : MD5
: OEM :
IP Address Source : Static Address
IP Address : 192.168.1.211
Subnet Mask : 255.255.255.0
MAC Address : 00:0e:0c:ea:92:a2
SNMP Community String :
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 192.168.1.254
Default Gateway MAC : 00:0e:0c:aa:8e:13
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14
Cipher Suite Priv Max : XXXXXXXXXXXXXXX
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
[root@sr2500 ~]#
'''Please specify the option "-L USER" for ipmitool when using a user with USER privilege.''' Otherwise you will get an error message stating:
Activate Session error: Requested privilege level exceeds limit

Attribution: https://www.thomas-krenn.com/en/wiki/Configuring_IPMI_under_Linux_using_ipmitool

How to build Proxmox tmpfs image

2023-11-27T02:38:42Z

Stvsyf: add ipmitool

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping ipmitool
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to unpack and pack an initrd

2023-11-26T22:47:16Z

Stvsyf:

[[Category:tutorial]]

Breif instructions to unpack and repack an image as an initrd;

'''Unpack an initrd'''

<code>unmkinitramfs initrd .</code>

'''Pack an initrd'''

cd initrd

find . | cpio -o -H newc | gzip -9 > ../KERNEL_NAME.img

How to unpack and pack an initrd

2023-11-26T22:46:49Z

Stvsyf: adding initrd

Breif instructions to unpack and repack an image as an initrd;

'''Unpack an initrd'''

<code>unmkinitramfs initrd .</code>

'''Pack an initrd'''

cd initrd

find . | cpio -o -H newc | gzip -9 > ../KERNEL_NAME.img

How to unpack and pack a debian archive

2023-11-26T22:33:02Z

Stvsyf:

[[Category:tutorial]]

The primary command to manipulate deb packages is <code>dpkg-deb</code>.

To unpack the package, create an empty directory and switch to it, then run <code>dpkg-deb</code> to extract its control information and the package files. Use <code>dpkg-deb -b</code> to rebuild the package.
<code>mkdir tmp
dpkg-deb -R original.deb tmp
# edit DEBIAN/postinst
dpkg-deb -b tmp fixed.deb</code>
Beware that unless your script is running as root, the files' permissions and ownership will be corrupted at the extraction stage. One way to avoid this is to run your script under <code>fakeroot</code>. Note that you need to run the whole sequence under <code>fakeroot</code>, not each <code>dpkg-deb</code> individually, since it's the <code>fakeroot</code> process that keeps the memory of the permissions of the files that can't be created as they are.
<code>fakeroot sh -c '
mkdir tmp
dpkg-deb -R original.deb tmp
# edit DEBIAN/postinst
dpkg-deb -b tmp fixed.deb
'</code>
Rather than mess with permissions, you can keep the data archive intact and modify only the control archive. <code>dpkg-deb</code> doesn't provide a way to do that. Fortunately, deb packges are in a standard format: they're <code>ar</code> archives. So you can use <code>ar</code> to extract the control archive, modify its files, and use <code>ar</code> again to replace the control archive by a new version.
<code>mkdir tmp
cd tmp
ar p ../original.deb control.tar.gz | tar -xz
# edit postinst
cp ../original.deb ../fixed.deb
tar czf control.tar.gz *[!z]
ar r ../fixed.deb control.tar.gz</code>
You should '''add a changelog entry and change the version number''' if you modify anything in the package. The infrastructure to manipulate Debian packages assumes that if two packages have the same name and version, they're the same package. Add a suffix to the ''debian_revision'' part at the end of the version number; for sorting reasons the suffix should start with <code>~</code>, e.g. <code>1.2.3-4.1</code> becomes <code>1.2.3-4.1~johnjumper1</code>.

Instead of using shell tools, you can use Emacs. The <code>dpkg-dev-el</code> package (which is its own upstream as this is a native Debian package) contains modes to edit <code>.deb</code> files and to edit Debian changelogs. Emacs can be used interactively or scripted.

Attribution: https://unix.stackexchange.com/questions/138188/easily-unpack-deb-edit-postinst-and-repack-deb

How to unpack and pack a debian archive

2023-11-26T22:31:57Z

Stvsyf: adding how to unpack and pack a debian archive

The primary command to manipulate deb packages is <code>dpkg-deb</code>.

To unpack the package, create an empty directory and switch to it, then run <code>dpkg-deb</code> to extract its control information and the package files. Use <code>dpkg-deb -b</code> to rebuild the package.
<code>mkdir tmp
dpkg-deb -R original.deb tmp
# edit DEBIAN/postinst
dpkg-deb -b tmp fixed.deb</code>
Beware that unless your script is running as root, the files' permissions and ownership will be corrupted at the extraction stage. One way to avoid this is to run your script under <code>fakeroot</code>. Note that you need to run the whole sequence under <code>fakeroot</code>, not each <code>dpkg-deb</code> individually, since it's the <code>fakeroot</code> process that keeps the memory of the permissions of the files that can't be created as they are.
<code>fakeroot sh -c '
mkdir tmp
dpkg-deb -R original.deb tmp
# edit DEBIAN/postinst
dpkg-deb -b tmp fixed.deb
'</code>
Rather than mess with permissions, you can keep the data archive intact and modify only the control archive. <code>dpkg-deb</code> doesn't provide a way to do that. Fortunately, deb packges are in a standard format: they're <code>ar</code> archives. So you can use <code>ar</code> to extract the control archive, modify its files, and use <code>ar</code> again to replace the control archive by a new version.
<code>mkdir tmp
cd tmp
ar p ../original.deb control.tar.gz | tar -xz
# edit postinst
cp ../original.deb ../fixed.deb
tar czf control.tar.gz *[!z]
ar r ../fixed.deb control.tar.gz</code>
You should '''add a changelog entry and change the version number''' if you modify anything in the package. The infrastructure to manipulate Debian packages assumes that if two packages have the same name and version, they're the same package. Add a suffix to the ''debian_revision'' part at the end of the version number; for sorting reasons the suffix should start with <code>~</code>, e.g. <code>1.2.3-4.1</code> becomes <code>1.2.3-4.1~johnjumper1</code>.

Instead of using shell tools, you can use Emacs. The <code>dpkg-dev-el</code> package (which is its own upstream as this is a native Debian package) contains modes to edit <code>.deb</code> files and to edit Debian changelogs. Emacs can be used interactively or scripted.

Attribution: https://unix.stackexchange.com/questions/138188/easily-unpack-deb-edit-postinst-and-repack-deb

How to build Proxmox tmpfs image

2023-11-25T22:06:18Z

Stvsyf: disable postfix

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket postfix --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-11-25T09:35:45Z

Stvsyf: disable pvescheduler

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed pvescheduler.service --root /exports/proxmox_image/
### /bin/systemctl disable nfs-blkmap iscsid.socket --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-11-25T09:33:53Z

Stvsyf: disabling some extra services

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### /bin/systemctl disable iscsi dropbear nfs-ganesha-lock nvmefc-boot-connections nvmf-autoconnect zfs-zed --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

Install openQRM on Debian

2023-11-24T22:42:59Z

Stvsyf: adding note to add extra kernel

This How-To explains installing the openQRM Datacentre Management and Cloud Computing platform on Debian. It is the starting point for a set of openQRM How-Tos explaining different Use-cases with the focus on virtualisation, automation and cloud computing.

'''Requirements'''

* One physical Server. Alternatively, the installation can be also done within a Virtual Machine
* At least 1 GB of Memory
* at least 40 GB of Diskspace
* Optional VT for Intel CPUs or AMD-V for AMD CPUs (Virtualization Technology) enabled in the Systems BIOS so that the openQRM Server can run Virtual Machines later

=== Install Debian ===

# Install a minimal Debian on a physical Server. During the installation select 'manual network' configuration and provide a static IP address. In this tutorial we will use 192.168.178.5/255.255.255.0 as the IP configuration for the openQRM Server system.
# Remember to use/set a Fully Qualified Domain Name (FQDN) as the sytem's hostname and domain. It does not need to resolve, but its important to be set.
# In the partitioning setup, select 'manual' and create one partition for the root-filesystem, one as swap space plus a dedicated partition to be used as storage space for the Virtual Machines later. In the configuration of the dedicated storage partition select 'do not use'.
# In the software selection dialog select just 'SSH-Server'
# After the installation finished please login to the system and update its packaging system as 'root':

<blockquote>''apt-get update && apt-get upgrade'' </blockquote>

=== Install openQRM - the short version ===
1. download and un-compress openQRM Community or Enterprise to /usr/src

2. un tar and install ;

Community;

''tar -zxf openQRM-5.3.50-Community-Edition.tgz ; cd openQRM-5.3.50-Community-Edition/ ; ./install-openqrm.sh''

Enterprise;

''tar -zxf openQRM-5.3.50-Enterprise-Edition.tgz ; cd openQRM-5.3.50-Enterprise-Edition/ ; ./install-openqrm.sh''

This process can take a short while, whilst it installs the supporting openQRM packages. Minimum 10 minutes if you have fast internet and a decent kvm.

4. suggestion: add PVE kernel to KVM first, if used in conjunction with a Proxmox VE installation,

wget <nowiki>http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-7_amd64.deb</nowiki>

5. create mysql openqrm user and password, flush privileges

mysql -e "grant all on openqrm.* to 'openqrm'@'localhost' identified by 'openqrm'; flush privileges"

6. reboot, this is useful to the current KVM kernel (now the pve kernel) will be used as the default linux kernel.

7. then configure openqrm via web (remember the steps above the username and password are both openqrm), the last screen will take 5-10 minutes whilst openQRM rebuilds the current kernel's initrd into an openQRM compabille boot

=== Install openQRM - the longer version ===
Purchase and download openQRM

openQRM is available from openQRM Enterprise at

<nowiki>http://www.openqrm-enterprise.com/</nowiki>

You can also use the instructions below to install openQRM from the source repository or by packages.

The installation procedure for openQRM is straight forward.

# Unpack the openqrm-enterprise.tar.gz file:

<blockquote>''tar -xvzf openqrm-enterprise.tar.gz''</blockquote>2. Then run as 'root'<blockquote>''cd openqrm-enterprise''

''./install-openqrm.sh''</blockquote>Make sure to set a password for the mysql-server and nagios4 package.
[[File:Csm 02-openqrm-install e007a58550.png|none|thumb|390x390px|Setting a password]]
The installation also asks for the mail-configuration. If unsure please select "local only" and go on with the suggested system name.
[[File:Csm 04-openqrm-install b8db1a2840.png|none|thumb|390x390px|Mail-Configuration]]
The last step of the installation provides you with the URL, username and password to login to the openQRM Server
[[File:Csm 06-openqrm-install fa5cc24876.png|none|thumb|390x390px|openQRM login credentials]]

=== Installation by packages ===
To install openQRM by distribution packages please request the package installation from openQRM Enterprise

=== Configure and initialize openQRM ===
After a successful installation the openQRM Server web interface is available at<blockquote>''<nowiki>http://static-ip-configured-during-the-Debian-installation/openqrm</nowiki>''</blockquote>If you have set the suggested IP address for this howto the openQRM URL will be<blockquote>''<nowiki>http://192.168.178.5/openqrm</nowiki>''</blockquote>Open this URL in your Web browser. Login with the username 'openqrm' and the password 'openqrm'. Then select the network interface to use for the openQRM management network
[[File:Csm 07-openqrm-install 602e7d0524.png|none|thumb|390x390px|Network Interface selection screen]]Then select 'mysql' as the database type
[[File:Csm 08-openqrm-install 5a821c5713.png|none|thumb|390x390px|Database type selection]]
At the next step, provide the database credentials. Ensure the database credentials are valid before proceeding. Please note this stage can take up to 8 minutes. The screen will be blank, please be patient. You can always;

tail -n 100 -f /var/log/syslog

for activity.
[[File:Csm 09-openqrm-install 10b3d582f9.png|none|thumb|390x390px|Database configuration]]
For the openQRM Enterprise Edition, the following page provides an simple option to upload the licence keys
[[File:Csm 10-openqrm-install 3a06f002ca.png|none|thumb|390x390px|Upload licence keys]]

Once the license keys are provided openQRM will rebuild the current kernel into an openQRM initramdisk, this may take 5-10 minutes.

'''Congratulations!!'''

openQRM is now installed and successfully initialized ready to manage all aspects of your datacentre

[[File:Csm 11-openqrm-install 8cdf50bf05.png|frameless|390x390px]]

Note Well:

It is always wise to install a duplicate kernel (this process can take 5-10mins or more on slower computers);

To add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables

# /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
# /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

*

[[Category:Tutorial]]
[[Category:Debian]]
[[Category:Howto]]

Install openQRM on Debian

2023-11-24T08:13:34Z

Stvsyf: db pre-existing and time takes

This How-To explains installing the openQRM Datacentre Management and Cloud Computing platform on Debian. It is the starting point for a set of openQRM How-Tos explaining different Use-cases with the focus on virtualisation, automation and cloud computing.

'''Requirements'''

* One physical Server. Alternatively, the installation can be also done within a Virtual Machine
* At least 1 GB of Memory
* at least 40 GB of Diskspace
* Optional VT for Intel CPUs or AMD-V for AMD CPUs (Virtualization Technology) enabled in the Systems BIOS so that the openQRM Server can run Virtual Machines later

=== Install Debian ===

# Install a minimal Debian on a physical Server. During the installation select 'manual network' configuration and provide a static IP address. In this tutorial we will use 192.168.178.5/255.255.255.0 as the IP configuration for the openQRM Server system.
# Remember to use/set a Fully Qualified Domain Name (FQDN) as the sytem's hostname and domain. It does not need to resolve, but its important to be set.
# In the partitioning setup, select 'manual' and create one partition for the root-filesystem, one as swap space plus a dedicated partition to be used as storage space for the Virtual Machines later. In the configuration of the dedicated storage partition select 'do not use'.
# In the software selection dialog select just 'SSH-Server'
# After the installation finished please login to the system and update its packaging system as 'root':

<blockquote>''apt-get update && apt-get upgrade'' </blockquote>

=== Install openQRM - the short version ===
1. download and un-compress openQRM Community or Enterprise to /usr/src

2. un tar and install ;

Community;

''tar -zxf openQRM-5.3.50-Community-Edition.tgz ; cd openQRM-5.3.50-Community-Edition/ ; ./install-openqrm.sh''

Enterprise;

''tar -zxf openQRM-5.3.50-Enterprise-Edition.tgz ; cd openQRM-5.3.50-Enterprise-Edition/ ; ./install-openqrm.sh''

This process can take a short while, whilst it installs the supporting openQRM packages. Minimum 10 minutes if you have fast internet and a decent kvm.

4. suggestion: add PVE kernel to KVM first, if used in conjunction with a Proxmox VE installation,

wget <nowiki>http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-7_amd64.deb</nowiki>

5. create mysql openqrm user and password, flush privileges

mysql -e "grant all on openqrm.* to 'openqrm'@'localhost' identified by 'openqrm'; flush privileges"

6. reboot, this is useful to the current KVM kernel (now the pve kernel) will be used as the default linux kernel.

7. then configure openqrm via web (remember the steps above the username and password are both openqrm), the last screen will take 5-10 minutes whilst openQRM rebuilds the current kernel's initrd into an openQRM compabille boot

=== Install openQRM - the longer version ===
Purchase and download openQRM

openQRM is available from openQRM Enterprise at

<nowiki>http://www.openqrm-enterprise.com/</nowiki>

You can also use the instructions below to install openQRM from the source repository or by packages.

The installation procedure for openQRM is straight forward.

# Unpack the openqrm-enterprise.tar.gz file:

<blockquote>''tar -xvzf openqrm-enterprise.tar.gz''</blockquote>2. Then run as 'root'<blockquote>''cd openqrm-enterprise''

''./install-openqrm.sh''</blockquote>Make sure to set a password for the mysql-server and nagios4 package.
[[File:Csm 02-openqrm-install e007a58550.png|none|thumb|390x390px|Setting a password]]
The installation also asks for the mail-configuration. If unsure please select "local only" and go on with the suggested system name.
[[File:Csm 04-openqrm-install b8db1a2840.png|none|thumb|390x390px|Mail-Configuration]]
The last step of the installation provides you with the URL, username and password to login to the openQRM Server
[[File:Csm 06-openqrm-install fa5cc24876.png|none|thumb|390x390px|openQRM login credentials]]

=== Installation by packages ===
To install openQRM by distribution packages please request the package installation from openQRM Enterprise

=== Configure and initialize openQRM ===
After a successful installation the openQRM Server web interface is available at<blockquote>''<nowiki>http://static-ip-configured-during-the-Debian-installation/openqrm</nowiki>''</blockquote>If you have set the suggested IP address for this howto the openQRM URL will be<blockquote>''<nowiki>http://192.168.178.5/openqrm</nowiki>''</blockquote>Open this URL in your Web browser. Login with the username 'openqrm' and the password 'openqrm'. Then select the network interface to use for the openQRM management network
[[File:Csm 07-openqrm-install 602e7d0524.png|none|thumb|390x390px|Network Interface selection screen]]Then select 'mysql' as the database type
[[File:Csm 08-openqrm-install 5a821c5713.png|none|thumb|390x390px|Database type selection]]
At the next step, provide the database credentials. Ensure the database credentials are valid before proceeding. Please note this stage can take up to 8 minutes. The screen will be blank, please be patient. You can always;

tail -n 100 -f /var/log/syslog

for activity.
[[File:Csm 09-openqrm-install 10b3d582f9.png|none|thumb|390x390px|Database configuration]]
For the openQRM Enterprise Edition, the following page provides an simple option to upload the licence keys
[[File:Csm 10-openqrm-install 3a06f002ca.png|none|thumb|390x390px|Upload licence keys]]

Once the license keys are provided openQRM will rebuild the current kernel into an openQRM initramdisk, this may take 5-10 minutes.

'''Congratulations!!'''

openQRM is now installed and successfully initialized ready to manage all aspects of your datacentre

[[File:Csm 11-openqrm-install 8cdf50bf05.png|frameless|390x390px]]

*

[[Category:Tutorial]]
[[Category:Debian]]
[[Category:Howto]]

How to build Proxmox tmpfs image

2023-11-20T04:42:20Z

Stvsyf: adding path to openqrm

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## /usr/share/openqrm/bin/openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## /usr/share/openqrm/bin/openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-11-15T23:57:51Z

Stvsyf: adding packages dropbear iputils-ping

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl dropbear iputils-ping
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-11-14T02:32:49Z

Stvsyf: adding package libpve-network-perl

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl libpve-network-perl
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-11-14T02:15:20Z

Stvsyf: adding extra packages

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd ifupdown2 dnsutils ethtool curl unzip screen iftop lshw smartmontools nvme-cli lsscsi sysstat htop mc rpl
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-11-14T02:09:28Z

Stvsyf: adding xinetd

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo xinetd
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

Debian 12 how to bootstrap an image

2023-11-13T22:07:45Z

Stvsyf: added create image tarball

[[Category:Debian]] [[Category:Tutorial]]

'''Starting from a fresh Debian 12 installation do the following;'''

apt-get install debootstrap

export MY_CHROOT=/exports/debian12

mkdir -p $MY_CHROOT/dev/pts $MY_CHROOT/proc $MY_CHROOT/var/run

cd $MY_CHROOT/..

mount --bind /dev/ $MY_CHROOT/dev/

mount --bind /dev/pts $MY_CHROOT/dev/pts

mount --bind /proc $MY_CHROOT/proc

debootstrap --arch amd64 bookworm $MY_CHROOT/ <nowiki>https://deb.debian.org/debian/</nowiki>

cp /etc/passwd $MY_CHROOT/etc/

cp /etc/shadow $MY_CHROOT/etc/

cp /etc/group $MY_CHROOT/etc/

cp /etc/apt/sources.list $MY_CHROOT/etc/apt/

cp /usr/share/keyrings/*gpg $MY_CHROOT/etc/apt/trusted.gpg.d/

chroot $MY_CHROOT

apt-get update

apt-get install wget net-tools screen locales tzdata collectd telnet whois traceroute nfs-kernel-server jq bash dialog iptables

dpkg-reconfigure locales

dpkg-reconfigure tzdata

exit

umount $MY_CHROOT/dev/pts

umount $MY_CHROOT/dev

umount $MY_CHROOT/proc

'''Create a tarball for this image;'''

export TEMPLATE_NAME="debian12"

cd $MY_CHROOT

tar --exclude=etc/hostname --exclude=var/openqrm/openqrm.conf --exclude=root/.*history --exclude=root/.joe_state --exclude=root/.ssh/* --exclude=etc/ssh/ssh_host_* --exclude=etc/dropbear/*key --exclude=etc/dropbear-initramfs/*key --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/share/pve-edk2-firmware --exclude=usr/share/man --numeric-owner -czf ../$TEMPLATE_NAME.new.tgz .

cd ../

ls -la

How to build Proxmox tmpfs image

2023-11-13T22:03:11Z

Stvsyf: added ceph install pacakges and additional supporting packages

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash rsyslog portmap open-iscsi rsync sudo
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
## To install ceph support, add the relavent repository and add packages;
### apt-get install ceph ceph-common ceph-fuse ceph-mds ceph-volume gdisk nvme-cli
## To add FRRouting add the relavent repository and add packages;
### apt-get install frr frr-pythontools
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-11-13T06:19:27Z

Stvsyf: comment out make-rprivate

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## #mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]

How to build Proxmox tmpfs image

2023-11-13T06:18:20Z

Stvsyf: change order of mounting dev/pts

Follow the steps below to convert Proxmox VE to a pxe booted tmpfs memory resident operating system.

Once you have a running openQRM Server you can follow these steps.

This process is supported in both the community and enterprise versions of openQRM.

You will need the following plugins enabled; dhcpd, tftp, nfs-storage, tmpfs-storage, atu (optional, available in the enterprise package)

Pre-built Proxmox VE templates are available for download in the customer portal.

'''Why is this solution so exciting ?'''

When data centre operators deploy compute nodes, they no longer need network or attached storage to run that node. This solution allows a compute node to pxe network boot an operating system into a ram disk. This ram disk is essentially the local storage for the server. Being memory resident the system ram is exceptionally fast, several times faster in order of magnitude than NVMe. So if the node lost network connectivity it would still be able to function as the node would have already been booted and running just like it had local attached storage.

'''Hold on this is too good to be true, what are the down sides ?'''

Well its memory resident, so if power is lost the local configuration would be lost. However if the node is part of a cluster then the cluster would hold the PVE configuration and if using the ATU plugin is used the configuration would be synchronised and retained on the openQRM server.

'''Requirements:'''
* openQRM Community or Enterprise (a KVM is the suggested option)
* optional: openQRM ATU Plugin for advanced server and cluster configuration and boot management
* CPU 64bit Intel EMT64 or AMD64
* PCI(e) passthrough requires VT-d/AMD-d CPU flag support
* Debian 11 Bullseye
'''Suggest minimum specification for:'''
* openQRM Server: 1GB & 1 CPU
* Virtual or Hardware Node (booted via tmpfs) 6-8GB. 4GB for tmpfs and 2-4GB for OS and Services.
* The clustering requires co-ordinated initialisation and configuration backup. The ATU Plugin orchestrates these steps for cluster management and configuration backup.
'''What is the ATU plugin ?'''

The ATU plugin is available in openQRM Enterprise. It allows the configuration synchronisation of the server to be maintain during reboots and power loss events. The ATU plugin is open source and written in bash and allows the start up sequence to be controlled and important configuration and service start sequences especially important for Proxmox VE.

'''<big>Let's Start:</big>'''

'''1. Adding a Proxmox Kernel to openQRM:'''
# Download PVE Kernel (check to see if there is a newer kernel) - http://download.proxmox.com/debian/dists/bullseye/pve-no-subscription/binary-amd64/pve-kernel-5.11.22-3-pve_5.11.22-6_amd64.deb
# Install Kernel locally
# then add the Kernel to openQRM. Replace KERNEL_NAME, KERNEL_VER, OPENQRM_UI_USER, OPENQRM_UI_PASS, SERVER_NAME with the appropriate variables)
## openqrm kernel add -n KERNEL_NAME -v KERNEL_VER -u OPENQRM_UI_USER -p OPENQRM_UI_PASS -l / -i initramfs -m csiostor
## openqrm kernel add -n pve-5.11.22-6 -v 5.11.22-3-pve -u openqrm -p openqrm -l / -i initramfs -m csiostor

'''2. Creating Image suitable to TMPFS Boot:'''
# Create Image - To create an image for Proxmox VE (image will be named "proxmox_image") which can be used as a tmpfs image, follow these steps;
## apt-get install debootstrap
## mkdir -p /exports/proxmox_image/dev/pts /exports/proxmox_image/proc /exports/proxmox_image/var/run/dbus
## debootstrap --arch amd64 buster /exports/proxmox_image/ <nowiki>https://deb.debian.org/debian/</nowiki>
## mount --bind /dev/ /exports/proxmox_image/dev/
## mount --bind /dev/pts /exports/proxmox_image/dev/pts
## mount --bind /proc /exports/proxmox_image/proc
## mount --make-rprivate /exports/proxmox_image/
## mount --bind /var/run/dbus /exports/proxmox_image/var/run/dbus
## chroot /exports/proxmox_image
## apt-get install wget net-tools screen locales collectd telnet whois traceroute nfs-kernel-server jq bash
## dpkg-reconfigure locales
## dpkg-reconfigure tzdata
## Follow steps (Start at "Install Proxmox VE") @ https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye
### We do not need to install grub or any other boot loaders
##'''set root password; passwd'''
## (optional) implement noclear for getty/inittab;
### mkdir -p /etc/systemd/system/getty@tty1.service.d/
### edit file; /etc/systemd/system/getty@tty1.service.d/noclear.conf add contents;
[Service]

TTYVTDisallocate=no
#'''Remember: /etc/hosts needs a valid hostname with your ip address'''
## This is managed with the ATU plugin
# exit chroot, type exit
# umount binds;
## umount /exports/proxmox_image/dev/pts
## umount /exports/proxmox_image/dev
## umount /exports/proxmox_image/proc
## umount /exports/proxmox_image/var/run/dbus
# (optional) If using the ATU Plugin follow these steps;
## (if using the ATU plugin) For reference only; since Proxmox/Debian uses systemd management of services needs to be done externally of the chroot. To find enabled services;
### systemctl list-unit-files --root /exports/proxmox_image/ | grep -v disabled | grep enabled
## (if using the ATU plugin) These services are managed by the ATU plugin. Since the ATU plugins manages cluster initialisation these services need to be started in an orderly fashion by the plugin. So we then remove services from startup, systemd is not friendly, so we need to point systemctl to the root directory as follows;
### /bin/systemctl disable pve-cluster.service corosync.service pve-guests.service --root /exports/proxmox_image/
### /bin/systemctl disable lvm2-lvmpolld.socket lvm2-monitor.service --root /exports/proxmox_image/
### /bin/systemctl disable lxc.service lxc-net.service lxcfs.service lxc-monitord.service --root /exports/proxmox_image/
### /bin/systemctl disable portmap.service rpcbind.service nfs-client.target --root /exports/proxmox_image/
### /bin/systemctl disable iscsid.service iscsi.service open-iscsi.service --root /exports/proxmox_image/
### /bin/systemctl disable pve-firewall.service pvefw-logger.service pvesr.timer pve-daily-update.timer --root /exports/proxmox_image/
### /bin/systemctl disable pve-ha-crm.service pve-ha-lrm.service pve-lxc-syscalld.service --root /exports/proxmox_image/
### /bin/systemctl disable pvebanner.service pvedaemon.service pvenetcommit.service --root /exports/proxmox_image/
### /bin/systemctl disable pveproxy.service pvestatd.service --root /exports/proxmox_image/
### /bin/systemctl disable qmeventd.service spiceproxy.service ssh.service --root /exports/proxmox_image/
### /bin/systemctl disable rsyslog.service syslog.service --root /exports/proxmox_image/
### /bin/systemctl disable smartd.service dm-event.socket rbdmap.service --root /exports/proxmox_image/
### /bin/systemctl disable ceph.target ceph-fuse.target frr.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable zfs-import.target zfs-import-cache.service zfs-volumes.target zfs-volume-wait.service zfs-share.service --root /exports/proxmox_image/
### /bin/systemctl disable netdiag.service rsync.service console-setup.service --root /exports/proxmox_image/
### If you have ceph installed disable;
#### /bin/systemctl disable ceph-crash.service ceph-mds.target ceph-mgr.target ceph-mon.target ceph-osd.target remote-fs.target --root /exports/proxmox_image/
### If you have Ganesha installed for nfs;
#### /bin/systemctl disable nfs-ganesha.service nfs-ganesha-lock.service nfs-common.service --root /exports/proxmox_image/
### /bin/systemctl disable puppet --root /exports/proxmox_image/
### /bin/systemctl disable zfs.target zfs-mount.service nfs-kernel-server --root /exports/proxmox_image/
## (if using the ATU plugin) disable services (some services may not exist):
### /bin/systemctl disable pvedaemon pve-proxy pve-manager pve-cluster cman corosync ceph pvestatd qemu-server rrdcached spiceproxy --root /exports/proxmox_image/
# Tar the Image;
## mkdir -p /usr/share/openqrm/web/boot-service/tmpfs/
## cd /exports/proxmox_image
## tar --exclude=usr/src --exclude=var/lib/apt/lists --exclude=usr/lib/jvm --exclude=var/lib/apt/lists --exclude=usr/share/man --exclude=usr/share/doc --exclude=usr/share/icons --numeric-owner -czf /usr/share/openqrm/web/boot-service/tmpfs/proxmox_image.tgz .
# When taring the image above, there are other directories that are not required that can be excluded. We suggest the uncompressed image size to be 55-60% of the available tmpfs volume size allocated (4GB as below).

'''3. Configuring openQRM to support above template:'''
# Activate dhcpd plugin then the tftp plugin
# Activate NFS Storage (if not already done so)
## Under Plugins -> Storage -> NFS-Storage
## Add NFS Storage;
## name "openqrm-nfs"
## Deployment Type: "nfs-deployment"
# Add NFS Volume (this triggers tmpfs storage)
## Under Plugins -> Storage -> NFS-Storage -> Volume Admin -> Edit -> proxmox_image "ADD IMAGE"
# <s>restart openQRM server/vm in case of duplicate services started from chroot image initialisation</s>
# Now create a TmpFs-Storage: Plugins -> Storage -> Tmpfs-storage -> Volume Admin -> New Storage
## Name: openqrm-tmpfs
## Deployment Type: tmpfs-storage
# Now Create an Image: Components -> Image -> Add new Image -> Tmpfs-root deployment -> click edit on the "openqrm-tmpfs" -> Click "ADD NEW VOLUME"
## Name: pve7
## Size: 4 GB
## Description: proxmox ve 7
# Now network boot a new node either a KVM or Physical machine, you will need to link this resource to a server. A resource is a blank system/server and a Server is a configuration applied to a resource/system/server. So when a system has booted via dhcp/pxe then system will enter an "idle" state and will be selectable as "idle" for this next step.
## Click "ADD A NEW SERVER"
## Select the resource
## then select the image for server, select the pve7 = tmpfs-deployment as previously setup (leave the tick on edit image details after selection.)
## then click "Install from NAS/NFS" select the "proxmox_image" as above then click submit
## then select the kernel pve-5.11.22-6 then click submit
## Done
# You will then need to "start" the server, click "start", the idle resource will then reboot and boot the image as created above
# Once booted you may need to restart sshd and pve-cluster
## systemctl restart ssh pve-cluster

'''Notes/Customisations:'''
# Postfix may error a warning on boot, edit /etc/mailname
#'''Nodes booted with out the ATU plugin will lose configuration upon reboot!'''
# when changing kernel versions, a stop and start of the server is required

This technology preview displays the tmpfs memory resident capabilities to support Proxmox VE as a memory resident operating system.

'''About the ATU Plugin:'''

The ATU plugin is a server service management configuration tool. It supports generic systems as well as Proxmox VE. It is responsible for boot management and the configuration and cluster configuration synchronisation with the openQRM server. Orchestrating the system service start/stop with configuration synchronisation with the openQRM server. This is a vital plugin for tmpfs based operating systems.

'''About openQRM:'''

openQRM is available in both community and enterprise versions. Both versions are open source with the enterpise package available for commercial support and numerous additional plugins being available. With over 60 plugins available openQRM manages storage, network, monitoring, cloud, management and virtualisation. It is the toolkit of choice for data centres.
[[Category:Howto]]
[[Category:Tutorial]]
[[Category:Debian]]