Tech

How to set up a standby OnApp Control Panel for recovery, using the cron job scheduler to back it up

Here’s a quick guide to preparing a standby OnApp Control Panel (CP) server, so that normal service can be recovered quickly if your primary CP server has an outage, and cannot be started.

 

Part 1 – configure the standby CP server

1. On the standby CP, perform the normal CP server installation steps, here: https://docs.onapp.com/ig/latest/install-control-panel-server. Ensure that the installed package versions are the same on both your standby and primary control panel servers.

 

2. On the standby CP, create a folder to store copies of CP configuration files and db dumps.

[root@onapp-cp2 ~]# mkdir /root/BACKUP_DIR/

3. From the primary CP, share the root and OnApp users’ public SSH keys with the standby CP.

[root@onapp-cp1 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@[Standby CP IP address]
[root@onapp-cp1 ~]# ssh-copy-id -i /home/onapp/.ssh/id_rsa.pub root@[Standby CP IP address]

4. Add the standby CP’s management IP address to the following line in the file /onapp/onapp-cp.conf (on the active CP).

DB_DUMP_SERVER=""

Also change the line:

DB_DUMP_SERVER_ROOT="/onapp/backups"

So it reads:

DB_DUMP_SERVER_ROOT="/root/BACKUP_DIR/"

After these changes, hourly database dumps will be transferred to the standby CP automatically.

 

5. From the Primary CP, copy these configuration files:

[root@onapp-cp1 ~]# mkdir /BACKUP_DIR/
[root@onapp-cp1 ~]# cd /BACKUP_DIR/ 
[root@onapp-cp1 ~]# tar czf etc.tar.gz /etc/ 
[root@onapp-cp1 ~]# tar cjf onapp_interface.tar.bz2 --exclude=/onapp/interface/log --exclude=/onapp/interface/db/dump /onapp/interface
[root@onapp-cp1 ~]# cp -a /root/.ssh ./ssh_root 
[root@onapp-cp1 ~]# cp -a /home/onapp/.ssh ./ssh_onapp
[root@onapp-cp1 ~]# rsync --progress -avz -e ssh /BACKUP_DIR/ root@[Standby CP IP address]:/root/BACKUP_DIR/


6. From the Primary CP, add the following lines to /etc/crontab to enable automatic (hourly) backup and replication of the redis database:

10 * * * * root echo "SAVE" | redis-cli -s /var/run/redis/redis.sock -x
20 * * * * root rsync -az -e ssh /var/lib/redis/onapp.rdb root@[Standby CP IP address]:/root/BACKUP_DIR/


7. If cloudboot is used, also add the following lines to /etc/crontab to enable automatic (daily) replication of cloudboot configuration files.

25 0 * * * root rsync -az -e ssh /tftpboot root@[Standby CP IP address]:/root/BACKUP_DIR/
30 0 * * * root rsync -az -e ssh /onapp/configuration/dhcp/dhcpd.conf root@[Standby CP IP address]:/root/BACKUP_DIR/


8. Final step – wait for 24 hours!

Part 2 – activating the standby CP

If there is an outage to the Primary CP, you can use Steps 9 and onwards in the guide at https://docs.onapp.com/misc/control-panel-migration-guide  to activate the standby CP and restore services.

Please note: for step 24, the rabbitMQ credentials can be found using the following command:

[root@onapp-cp2 ~]# cat /root/BACKUP_DIR/onapp/interface/config/on_app.yml|grep rabbit

 

I hope you found that helpful. As always, if you have any questions, email architecture@onapp.com. Thanks!