Adding a disk to an instance

Disk	<role>	Volume type	Mounted on	Use
Data	data	high-speed-gen2	/mnt/data	MariaDB and /home, on a fast disk
Backup	backup	classic	/mnt/backup	daily copy of each site, quota per account

Both disks use XFS:

• quotas built into the file system: per user and per group on /mnt/data (uquota, gquota), per directory on /mnt/backup (project quotas, prjquota), with no quota file to maintain;
• online growth: once the volume has been enlarged at OVH, xfs_growfs extends the file system without unmounting it;
• performance on large files and parallel writes, those of a database as well as those of a backup.

The trade-off: an XFS file system cannot be shrunk. Start with a modest size, then grow it.

1. Terraform

After the instance, in resource-openstack-compute-xxx.tf:

resource "openstack_blockstorage_volume_v3" "xxx_<role>" {
  provider             = openstack.ovh
  region               = openstack_compute_instance_v2.xxx.region
  name                 = "xxx_<role>"
  size                 = <size in GB>
  volume_type          = "<volume type>"
  enable_online_resize = true
  lifecycle { prevent_destroy = true }
}

resource "openstack_compute_volume_attach_v2" "xxx_<role>_attach" {
  provider    = openstack.ovh
  region      = openstack_compute_instance_v2.xxx.region
  instance_id = openstack_compute_instance_v2.xxx.id
  volume_id   = openstack_blockstorage_volume_v3.xxx_<role>.id
}

The plan must announce 2 creations.

2. Formatting (erases the disk)

Terraform gives the volume identifier, from the project directory:

bash

terraform state show openstack_blockstorage_volume_v3.xxx_<role> | grep -E '^ +id '

It is also shown in the OVH customer area, Public Cloud > Block Storage.

On the instance, the disk appears under /dev/disk/by-id/ with the name scsi-0QEMU_QEMU_HARDDISK_<identifier>.

bash

ID=<volume identifier>
DISK=$(ls /dev/disk/by-id/*"$ID")   # the disk alone, without its partitions
echo "$DISK"
wipefs -n "$DISK"          # must be empty
parted -s "$DISK" mklabel gpt mkpart <role> xfs 0% 100%
udevadm settle
mkfs.xfs -L <role> "${DISK}-part1"

3. Data disk

Puppet

In data/nodes/xxx.yaml:

yaml

min::data_disk: true
min::mariadb::on_data_disk: true
min::home::on_data_disk: true

On the next run, Puppet mounts /mnt/data, declares the binds of /var/lib/mysql and /home without mounting them, and installs the migration scripts. MariaDB keeps running on the internal disk.

Until the migration is done, MariaDB no longer restarts: after a reboot, the database stays stopped.

Moving MariaDB

Sites are down during the copy.

bash

mariadb-move-to-data-disk

The script suspends Puppet, copies the database, mounts the bind, restarts MariaDB, then re-enables Puppet. If it fails, Puppet stays disabled: check the state, then puppet agent --enable.

Moving /home

Services using /home (nginx, PHP, FTP, mail, cron) are down during the copy.

bash

min-move-home-to-data-disk

Works the same way as for MariaDB. If /home is still in use, the script moves nothing, restarts the services and re-enables Puppet.

On a Virtualmin instance, then set the quotas on /home in Virtualmin, and run:

bash

virtualmin modify-domain --all-domains --apply-quotas

Checking

bash

findmnt /var/lib/mysql
findmnt /home
mysql -e 'SHOW DATABASES'

Then delete /var/lib/mysql.old-<date> and /home.old-<date>.

4. Backup disk

Without this disk, no site backup is enabled.

Puppet

Only once the disk is formatted, in data/nodes/xxx.yaml:

yaml

min::backup_disk: true

On the next run, Puppet:

• mounts the disk on /mnt/backup with project quotas (prjquota);
• creates /mnt/backup/wcf/<account> for each site and mounts it on /home/wcf/<account>/backup;
• limits the space of each account;
• schedules the backup of each site around 3 a.m.: an archive /home/wcf/<account>/backup/<YYYY-MM-DD>/<account>-<YYYY-MM-DD>.tar.zst with data/, app/, public_html/ and the database in db.sql.

The prjquota option is only taken into account at mount time. If the disk was mounted without it, unmount it and run Puppet again.

Settings

Setting	Default	Where
Days kept	1 (the previous day is deleted once the day's backup has succeeded)	backup_keep of the site
Space per account	10G	min::wcf::backup_quota, or backup_quota of the plan in min::wcf::dolibarrs::offers (Biz: 80G)
Backup of a site	enabled	backup: false on the site to turn it off

Checking

bash

findmnt /mnt/backup
xfs_quota -x -c 'report -p -h' /mnt/backup
ls /home/wcf/*/backup/

A backup failing for 36 hours triggers the MinBackupStale alert: missing disk, account quota reached or script error.