Overview

Vertical Backup is provided as a single executable file. It is recommended to download it to a datastore directory:

mkdir /vmfs/volumes/datastore1/verticalbackup
cd /vmfs/volumes/datastore1/verticalbackup
wget http://verticalbackup.com/esxi/vertical
chmod u+x vertical
sha256sum vertical # e6d55b2999f6994cb366870618cf66914683f908b2f175c991ef38f0a54e88a1

Once we have the executable ready, we first need to choose a host id, and the storage to back up to:

vertical init esxi55 sftp://esxi@192.168.1.100/storage

Here we use esxi55 as the host id. Each ESXi host should have a unique host id. Backups will be uploaded to 192.168.1.100 via SFTP with the username being esxi and the storage directory will be storage (relative to the home directory). Multiple ESXi hosts can back up to the same storage, and in fact this is the recommended way as this enables cross-host deduplication.

Now we are ready to run the backup command. This is to back up the virtual machine named vm-ubuntu:

vertical backup vm-ubuntu

Vertical Backup assigns increasing revision numbers to backups, starting at 1 for the first backup. We can list previous backups using the list command:

vertical list

The restore command restores a virtual machine to a previous revision:

vertical restore vm-ubuntu -r 1

This command sets the virtual machine vm-ubuntu to be backed up 10 minutes after every hour:

vertical cron 10 vm-ubuntu

We can also provide the time in the hh:mm format to schedule a daily backup job.

vertical cron 11:30 vm-ubuntu

To set up email notifications, first run the email command:

vertical email smtp.gmail.com 587 username password from@gmail.com to@gmail.com

Then when we can run the backup command with the --email option. An email will be send to to@gmail.com after the backup is finished.

vertical backup vm-ubuntu --email

The --email option can also be provided to the cron command to enable email notifications for a scheduled backup job

vertical cron 0:30 vm-ubuntu --email


Commands


Below are all the commands and global options supported by Vertical Backup:

usage: vertical [-h] [-v] [--log] [--disable-script] [--print-stack]
                {restore,prune,license,list,email,cron,init,backup,check,help}
                ...

commands:
  {restore,prune,license,list,email,cron,init,backup,check,help}
    init                initialize the backup storage
    backup              back up virtual machines to the storage
    restore             restore virtual machine from previous backup
    list                list backups
    check               check the integrity of backups
    prune               remove specified backups
    cron                set up a cron job
    email               set up email notifications
    license             manage the license
    benchmark           run performance benchmarks
    help                display help information

optional arguments:
  -h, --help            show this help message and exit
  -v                    increase output verbosity
  --log                 enable log-style output
  --disable-script      don't run scripts before and after the command
  --print-stack         print the stack trace on error

Init


usage: vertical init [-h] [-c SIZE] [-e] [--temp-dir DIR] host_id storage_url

positional arguments:
  host_id                       a unique id to identify the host machine
  storage_url                   the url of the storage to back up to

optional arguments:
  -h, --help                    show this help message and exit
  -c SIZE, --chunk-size SIZE    the size of chunks (default is 1M)
  -e, --encrypt                 encrypt to the storage with a master password
  --temp-dir DIR                directory for storing temporary files

The init command assigns the given host id that will be used by the backup command later. It also tries to connect to the storage backend specified by the storage url. If the storage backend has not been initialized before, it will initialize the storage backend by uploading a configuration file containing the chunk size as specified and other parameters. If the configuration file already exists it will download the file and use the existing configuration parameters instead, ignoring the chunk size provided.

The chunk size parameter indicates the size of chunks that Vertical Backup will split the disk files into. It is recommended to be a multiple of 1024, and the default size is 1 megabytes.

The -e option encrypts the storage backend. If the storage backend has not been initialized, then a storage password must be entered. If the storage has been initialized with this option before, then this option becomes required.

Currently there are 5 types of storage urls:

In the SFTP url, the storage path is relative to the user's home directory. An extra slash is needed between the server and the storage path for an absolute path.

Not that if an SFTP storage is used, you must configure the ESXi host to enable outgoing SSH connections.

Depending on the storage backend you specify, you may be prompted to enter passwords/keys required to access the specified storage backend. To avoid entering them every time the storage is accessed, you can save them in environment variables or a special file. Please refer to the Managing Credential sections for details.

Under the hood, this command creates a subdirectory named .verticalbackup under the directory where the Vertical Backup executable is. It then saves the host id and the storage url to a file named preference under .verticalbackup. Later on other commands always read from this file to get the host id and the storage url, among other things.

The .verticalbackup directory is also the place for storing other auxiliary files. For instance, Vertical Backups caches chunks referenced by backup files and these chunks are temporarily stored in .verticalbackup/cache . You can use the --temp-dir option to change location of this cache directory.

This command also creates a custom firewall rule that enables outgoing HTTPS connections via port 443. This is needed since Vertical Backup needs to contact our license server at https://vertical-portal.appspot.com in order to activate licenses.


Backup


usage: vertical backup [-h] [-p] [--no-quiesce] [-t TAG] [--threads THREADS]
                              [-l LIMIT_RATE] [-e EXCLUDES] [--email]
                              [--subject EMAIL_SUBJECT]
                              [--failure-subject FAILURE_SUBJECT] [--parallel]
                              [--exclude-disk EXCLUDED_DISKS]
                              [vmname [vmname ...]]

positional arguments:
  vmname                      the name of the virtual machine to back up

optional arguments:
  -h, --help                  show this help message and exit
  -p, --preserve-snapshots    don't remove or create snapshots
  --no-quiesce                do not quiesce the virtual machine before the backup
  -t TAG, --tag TAG           specify a backup tag
  --threads THREADS           number of uploading threads
  -l RATE, --limit-rate RATE  limit the upload speed (KiloBytes/sec)
  -e VMNAME, --exclude VMNAME do not backup the virtual machine with this name
  --email                     email log after backup is done
  --subject EMAIL_SUBJECT     the subject of the email to be sent
  --failure-subject SUBJECT   the email subject to use when backup fails
  --parallel                  allow multiple backups at the same time
  --exclude-disk DISKNAME     do not backup the disk with this name
  --no-chunk-listing          do not list chunks for initial backup
  --snapshot-timeout TIMEOUT  snasphot operation timeout (in seconds)  
  --power-off                 power off the virtual machine before backup
  --power-on                  power on the virtual machine after backup

The backup command creates backups for one or more that one virtual machine, and uploads them to the storage backend specified by the init command.

By default, for each virtual machine to be backed up, all existing snapshots are removed at first and then a new one is created. This is to ensure that disk files are in a most recent state. After the backup is finished, the newly created snapshot will be removed. The -p option disables all these snapshot operations.

If --no-quiesce option is specified, the latest snapshot will be created with the Quiesce flag.

You can assign a tag to the backup to be created by using the -t option. In other commands you can select backups by their tags.

The backup operation supports multithreading if a number greater than 1 is given to the --threads option, or rate limiting if a non-zero number is given to the -l option.

The -e option excludes virtual machines from the list to be backed up. Wildcard characters such as * and ? can be used in both the -e optin and the vmname argument.

To send out an email after the backup is done, specify the --email option and optionally the --subject option to use a customized subject. The subject can be changed by the --failure-subject option if any error occurs during the backup. The email command must be invoked first to set up email configuration.

By default, Vertical Backup will not start if there is another process that has been running. The --parallell option will ignore the previous instance

The --exclude-disk option is used to exclude disks with certain names. Note that the virtual machine restored from a backup created with this option may not work properly due to the missing disks.

The --no-chunk-listing option can be used to avoid listing all chunks on the storage when running the first backup for a virtual machine. By default, when a virtual machine is backed up for the first time, Vertical Backup will construct a chunk cache in memory from the complete chunk list on the storage. The advantage of doing so is that if a chunk already in the storage, a lookup API call can be avoided. However, if there are too many chunks in the storage this chunk cache may consume a lot of memory. This option effectively disable the chunk cache at the cost of one lookup API call for each chunk to be uploaded or skipped.

The --snapshot-timeout option set the maximum time for the snapshot creation operation. The default is 120 seconds.


Restore


usage: vertical restore [-h] -r REVISION [-f VM] [--threads THREADS] [-l RATE] vmname [file [file ...]]

positional arguments:
  vmname                        the name of the virtual machine to restore
  file                          only restore specified files

required arguments:
  -r REVISION                   the revision of the backup to restore from

optional arguments:
  -h, --help                    show this help message and exit
  -f VM, --restore-from VM      restore from a backup of another virtual machine
  --power-off                   power off the virtual machine before restore
  --power-on                    power on the virtual machine after restore
  --threads THREADS             number of downloading threads
  -l RATE, --limit-rate RATE    limit the download speed (kilobytes/sec)

The restore command restores a given virtual machine to a previous revision. By default the virtual machine must be stopped before running this command, unless the --power-off option is specified

If the name of the virtual machine is the single argument, then it will restore all files contained in the backup. You can pass the name of files you wish to restore after the first argument, then these files will be the only files to be restored.

The first argument can also be a name of a directory. In this case the files to be restored will be written to this directory. The --restore-from option becomes required, which specifies the name of the virtual machine to be restored from. To restore from a virtual machine running on a different host but backing up to the same storage url, specify the virtual machine name in the format of vmname@hostid.

The restore operation supports multithreading if a number greater than 1 is given to the --threads option, or rate limiting if a non-zero number is given to the -l option.

The --power-off option can be used to power off the virtual machine before the restore operation, and the --power-on option to power on the virtual machine after the restore operation.

List


usage: vertical list [-h] [-a] [-f] [vmname [vmname ...]]

positional arguments:
  vmname                        list previous backups of the specified virtual machines

optional arguments:
  -h, --help                    show this help message and exit
  -a, --all                     list all backups in the storage
  -f, --files                   show file information for each backup

The list command lists existing backups. By default it only lists backups created locally, but the -a option will force it to list all backups present on the same storage, regardless of their host ids.

Virtual machines can also be specified in the format of vmname@hostid, and wildcard patterns are allowed. Therefore, *@host1 will check all virtual machines on the host with host id host1, and *@* is equivalent to the -a option.

The -f option will also list detailed information about each file.


Check


usage: vertical check [-h] [-a] [-r REVISIONS] [-t TAG] [-v] [vmname [vmname ...]]

positional arguments:
  vmname                        verify backups of the specified virtual machines

optional arguments:
  -h, --help                    show this help message and exit
  -a, --all                     check all virtual machines
  -r REVISIONS                  the revisions to verify
  -t TAG, --tag TAG             only verify backups with the specified tag
  -v, --verify                  verify integrity of all chunks

The check command verifies that all chunks referenced by backups of the specified virtual machines exist.

If the -a option is specified, it will check backups of every virtual machine in the storage (including those in different hosts). Otherwise, it will only check specified virtual machines.

Virtual machines can also be specified in the format of vmname@hostid, and wildcard patterns are allowed. Therefore, *@host1 will check all virtual machines on the host with host id host1, and *@* is equivalent to the -a option.

The -r options, which can occur multiple times, limit the check to backups with the specified revisions numbers. Ranges of revision numbers can also be specified in the format of a-b, where a is the starting revision number and b is the ending revision number.

If the -t option is specified, the check will be further limited to backups with the specified tag.

The -v option will not only check the existence of chunks, but also make sure that each chunk is not corrupted. Backups that pass the -v check are guaranteed to be complete. Note that this option makes the check operation much slower (due to the need to download every chunk).


Prune


usage: vertical prune [-h] [-k POLICY] [-r REVISIONS] [-t TAGS] [-d]
                                [--exhaustive] [--exclusive] [--collect-only] [--delete-only]
                                [vmname [vmname ...]]

positional arguments:
  vmname                        delete backups of the specified virtual machines

optional arguments:
  -h, --help                    show this help message and exit
  -k POLICY, --keep POLICY      specify the retention policy
  -r REVISIONS                  delete backups with the specified revision numbers
  -t TAGS, --tag TAGS           delete backups with the specified tags
  -d, --dry-run                 don't actually remove
  --exhaustive                  find all unreferenced chunks
  --exclusive                   assume no backup is in progress
  --collect-only                skip the fossil deletion step
  --delete-only                 skip the fossil collection step

The prune command deletes backups of specified virtual machines from the storage. Unless the --exclusive option is specified, when a backup is removed from the storage by this command, all chunks it references will not immediately be removed and instead remain in the storage. However they cannot be accessible by new backups created after the prune command is finished. Next time the prune command is run, it will check if certain conditions are met. If they are met, those unreferenced chunks will be permanently removed. This is the central idea of Two-Step Fossil Collection algorithm that enables Lock-Free Deduplication.

Virtual machines can also be specified in the format of vmname@hostid, and wildcard patterns are allowed. Therefore, *@host1 will check all virtual machines on the host with host id host1.

If no virtual machines are specified, then the 3 backup selection options, -k, -r, and -t, apply to all virtual machines in the storage.

The retention policies are specified by the -k option, which accepts an argument in the format of two numbers n:m, where n indicates the number of days between two consecutive backups to keep, and m indicates that the policy only applies to backups at least m day old. If n is zero, any backups older than m days will be removed.

Here are a few sample retention policies:

vertical prune -k 1:7       # Keep 1 backup per day for backups older than 7 days
vertical prune -k 7:30      # Keep 1 backup every 7 days for backups older than 30 days
vertical prune -k 30:180    # Keep 1 backup every 30 days for backups older than 180 days
vertical prune -k 0:360     # Keep no backup older than 360 days

Multiple -k options must be sorted by their m values in decreasing order. For instance, to combine the above policies into one line, it would become:

vertical prune -k 0:360 -k 30:180 -k 7:30 -k 1:7

The -r options, which can occur multiple times, limit the pruning to backups with the specified revisions numbers. Ranges of revision numbers can also be specified in the format of a-b, where a is the starting revision number and b is the ending revision number.

If the -t option is specified, the pruning will be further limited to backups with the specified tag.

The --exhaustive option will scan the list of all chunks in the storage, therefore it will find not only unreferenced chunks from deleted backups, but also chunks that become unreferenced for other reasons (for instance those caused by incomplete backups). It will also find any file that does not look like a chunk fileĀ at all. In contrast, a default prune command will only identify chunks referenced by backups to be deleted, but not by others.

The --exclusive option will assume that no other ESXi hosts are accessing the same storage, effectively disabling the Two-Step Fossil Collection algorithm. With this option, the prune command will immediately remove unreferenced chunks.

The -d option is used to test what changes the prune command would have done. It is guaranteed not to make any changes on the storage. The following command checks if the chunk directory is clean (i.e., if there are any unreferenced chunks, temporary files, or anything else):

vertical prune -d --exclusive --exhaustive    #  prints unreferenced chunks 

The --delete-only option will skip the fossil collection step, while the --collect-only option will skip the fossil deletion step.

If there are multiple ESXi hosts connected to the same storage (which is in fact the suggested use case), it is recommended that only one host will perform periodic prune operations. Although the prune operation is not known to be resource consuming, it is even better if the prune operation is performed on a non-production ESXi host, or even by Duplicacy on a non-ESXi computer.


Cron


usage: vertical cron [-h] time ...

positional arguments:
  time                          the scheduled time to run the job
  ...                           virtual machines to back up followed by options

optional arguments:
  -h, --help                    show this help message and exit

The cron command schedules a backup job to be run at the specified time.

If the time is a one digit or two digit number, then it is scheduling an hourly backup job and the time is the number of minutes after the hour.

If the time is in the format of hour:minute, then it is scheduling a daily backup job to be run at the specified time.

For more flexibility, you can also specify the time in the standard 5 field crontab format.

If the time is the string disable, then it will disable the current job that has been scheduled.

You can specify any valid options of the backup command after the names of virtual machines to be backed up. For instance, this command schedules an hourly job to backup a virtual machine named vm1 with email notifications enabled:

 vertical cron 0 vm1 --email --subject '"Backup for vm1 has been completed successfully"'

Email


usage: vertical email [-h] [--tls] [--test]
                      server port username password from_address to_address

positional arguments:
  server        SMTP server hostname or ip
  port          SMTP port (normally 25, 465, or 587)
  username      username on the SMTP server
  password      password to the SMTP server
  from_address  sender address
  to_address    receiver address

optional arguments:
  -h, --help    show this help message and exit
  --tls         enable SSL/TLS when sending emails
  --test        send a test email

The email command sets up email configuration for the backup command. Note that this command doesn't send out any email, unless the --test option is given.

The --tls option indicates that the TLS protocol will be used when talking to the email server.

The --test option will instruct the command to send out a test email after saving the configuration.

This command also creates a custom firewall rule that enables outgoing SMTP or SMTPS connections via the specified port.


Benchmark


usage: vertical benchmark [-h] [-l] [datastore]

positional arguments:
  datastore   specify the directory for testing disk access speeds

optional arguments:
  -h, --help  show this help message and exit
  -l, --long  run longer tests

The benchmark command runs a few tests to measure the write and read speeds of the datastore, as well as the download and upload speeds of the storage.

By default this command selects the first datastore as the test directory, which can be changed by providing the optional datastore argument.

The -l option doubles the test time which may produce more accurate results.


License


usage: vertical license [-h] {activate,show} ...

positional arguments:
  {activate,show}
    show           display license information
    activate       activate a new license

optional arguments:
  -h, --help       show this help message and exit

The license command supports 2 sub-commands.

The license show sub-command lists license information.

The license activate takes a license activation code, downloads the assigned license from our license server (at https://vertical-port.appspot.com), and then save the license which will be checked by the backup command.


Managing Credentials


Storage backends supported by Vertical Backup require various kinds of credentials, such as passwords, account ids, or keys. If you choose to encrypt the storage backend, then there is also the storage password.

Before prompting for user input, Vertical Backup will first check if the corresponding environment variables exist. If they do, Vertical Backup will read the credentials from these environment variables instead of keyboard input. If they don't, Vertical Backup will also check a file named passwords under the directory .verticalbackup (which is under the same directory as the Vertical Backup executable), where passwords can be stored as key-value pairs in json format.

The following table summarizes the name of the environment variable or the key in the passwords file for each type of credential:

Credential Environment variable Key in .verticalbackup/passwords
storage password VB_PASS pass
SFTP password VB_SSH_PASS ssh_pass
SFTP key file VB_SSH_KEY ssh_key
SFTP key passphrase VB_SSH_KEY_PASS ssh_key_pass
Amazon S3 access id VB_S3_ID s3_id
Amazon S3 secret key VB_S3_KEY s3_key
Microsoft Azure account VB_AZURE_ACCOUNT azure_account
Microsoft Azure key VB_AZURE_KEY azure_key
Backblaze B2 account id VB_B2_ACCOUNT b2_account
Backblaze B2 application key VB_B2_KEY b2_key

For instance, the following is a sample passwords file:

[root@esxi55:/vmfs/volumes/datastore1/vertical] cat .verticalbackup/passwords
{
"pass": "storage password",
"ssh_pass": "ssh password"
}

Pre-Command and Post-Command Scripts

You can instruct Vertical Backup to run a script before or after executing a command. For example, if you create a script with the name pre-prune under the .verticalbackup/scripts directory, this script will be executed before the prune operation starts. A script named post-prune will be executed at the end of the prune operation. This rule applies to backup, restore, list, check, and prune commands.