Index: openacs-4/packages/acs-core-docs/www/xml/install-guide/recovery.xml =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-core-docs/www/xml/install-guide/recovery.xml,v diff -u -r1.6 -r1.7 --- openacs-4/packages/acs-core-docs/www/xml/install-guide/recovery.xml 11 Nov 2003 16:57:20 -0000 1.6 +++ openacs-4/packages/acs-core-docs/www/xml/install-guide/recovery.xml 11 Dec 2003 21:39:49 -0000 1.7 @@ -1,16 +1,15 @@ %myvars; ]> Backup and Recovery - by Don Baccus - - with additions by Joel Aufrecht - + By Don Baccus with additions + by Joel Aufrecht + Backup Strategy @@ -20,63 +19,63 @@ - Store everything on a fault-tolerant disk array (RAID 1 or 5 - or better). - + Store everything on a fault-tolerant disk array (RAID 1 or 5 + or better). + - Use battery backup. - + Use battery backup. + - Use more reliable hardware, such as SCSI instead of IDE. - - + Use more reliable hardware, such as SCSI instead of IDE. + + - These steps improve the chances of successful recovery: - - - Store backups on a third disk on another controller - - - Store backups on a different computer on a different network - in a different physical location. (Compared to off-line - backup such as tapes and CDRs, on-line backup is faster and - more likely to succeed, but requires maintenance of another machine.) - - - Plan and configure for recovery from the beginning. - - - Test your recovery strategy from time to time. - - - Make it easy to maintain and test your recovery strategy, so - that you are more likely to do it. - - + These steps improve the chances of successful recovery: + + + Store backups on a third disk on another controller + + + Store backups on a different computer on a different network + in a different physical location. (Compared to off-line + backup such as tapes and CDRs, on-line backup is faster and + more likely to succeed, but requires maintenance of another machine.) + + + Plan and configure for recovery from the beginning. + + + Test your recovery strategy from time to time. + + + Make it easy to maintain and test your recovery strategy, so + that you are more likely to do it. + + - - OpenACS installations comprise files and database contents. - If you follow the reference install and put all files, - including configuration files, in - /var/lib/aolserver/service0/, - and back up the database nightly to a file in - /var/lib/aolserver/service0/database-backup, - then you can apply standard file-based backup strategies to - /var/lib/aolserver/service0 - + + OpenACS installations comprise files and database contents. + If you follow the reference install and put all files, + including configuration files, in + /var/lib/aolserver/service0/, + and back up the database nightly to a file in + /var/lib/aolserver/service0/database-backup, + then you can apply standard file-based backup strategies to + /var/lib/aolserver/service0 + - Snapshot backup and recovery + Manual backup and recovery - This section describes how to make a one-time backup and - restore of the files and database. This is useful for rolling back to - known-good versions of a service, such as at initial - installation and just before an upgrade. First, you back up the - database to a file within the file tree. Then, you back up the - file tree. All of the information needed to rebuild the site, - including the AOLserver config files, is then in tree for - regular file system backup. + This section describes how to make a one-time backup and + restore of the files and database. This is useful for rolling back to + known-good versions of a service, such as at initial + installation and just before an upgrade. First, you back up the + database to a file within the file tree. Then, you back up the + file tree. All of the information needed to rebuild the site, + including the AOLserver config files, is then in tree for + regular file system backup. @@ -91,196 +90,196 @@ - Download the backup script. Save the file export-oracle.txt as - /tmp/export-oracle.txt - + Download the backup script. Save the file export-oracle.txt as + /tmp/export-oracle.txt + - Login as root. The following commands will install the export script: - - -joeuser:~$ su - -Password: *********** -root:~# cp /tmp/export-oracle.txt /usr/sbin/export-oracle -root:~# chmod 700 /usr/sbin/export-oracle - + Login as root. The following commands will install the export script: + + + joeuser:~$ su - + Password: *********** + root:~# cp /tmp/export-oracle.txt /usr/sbin/export-oracle + root:~# chmod 700 /usr/sbin/export-oracle + - - Setup the export directory; this is the directory where backups will - be stored. We recommend the directory - /ora8/m02/oracle-exports. + + Setup the export directory; this is the directory where backups will + be stored. We recommend the directory + /ora8/m02/oracle-exports. - -root:~# mkdir /ora8/m02/oracle-exports -root:~# chown oracle.dba /ora8/m02/oracle-exports -root:~# chmod 770 /ora8/m02/oracle-exports - + + root:~# mkdir /ora8/m02/oracle-exports + root:~# chown oracle.dba /ora8/m02/oracle-exports + root:~# chmod 770 /ora8/m02/oracle-exports + - - Now edit - /usr/sbin/export-oracle and - change the SERVICE_NAME and - DATABASE_PASSWORD fields to - their correct values. If you want to use a directory other than - /ora8/m02/oracle-exports, you - also need to change the - exportdir setting. - - - Test the export procedure by running the command: - + + Now edit + /usr/sbin/export-oracle and + change the SERVICE_NAME and + DATABASE_PASSWORD fields to + their correct values. If you want to use a directory other than + /ora8/m02/oracle-exports, you + also need to change the + exportdir setting. + + + Test the export procedure by running the command: + - -root:~# /usr/sbin/export-oracle -mv: /ora8/m02/oracle-exports/oraexport-service_name.dmp.gz: No such file or directory + + root:~# /usr/sbin/export-oracle + mv: /ora8/m02/oracle-exports/oraexport-service_name.dmp.gz: No such file or directory -Export: Release 8.1.6.1.0 - Production on Sun Jun 11 18:07:45 2000 + Export: Release 8.1.6.1.0 - Production on Sun Jun 11 18:07:45 2000 -(c) Copyright 1999 Oracle Corporation. All rights reserved. + (c) Copyright 1999 Oracle Corporation. All rights reserved. -Connected to: Oracle8i Enterprise Edition Release 8.1.6.1.0 - Production -With the Partitioning option -JServer Release 8.1.6.0.0 - Production -Export done in US7ASCII character set and US7ASCII NCHAR character set -. exporting pre-schema procedural objects and actions -. exporting foreign function library names for user SERVICE_NAME -. exporting object type definitions for user SERVICE_NAME -About to export SERVICE_NAME's objects ... -. exporting database links -. exporting sequence numbers -. exporting cluster definitions -. about to export SERVICE_NAME's tables via Conventional Path ... -. exporting synonyms -. exporting views -. exporting stored procedures -. exporting operators -. exporting referential integrity constraints -. exporting triggers -. exporting indextypes -. exporting bitmap, functional and extensible indexes -. exporting posttables actions -. exporting snapshots -. exporting snapshot logs -. exporting job queues -. exporting refresh groups and children -. exporting dimensions -. exporting post-schema procedural objects and actions -. exporting statistics -Export terminated successfully without warnings. + Connected to: Oracle8i Enterprise Edition Release 8.1.6.1.0 - Production + With the Partitioning option + JServer Release 8.1.6.0.0 - Production + Export done in US7ASCII character set and US7ASCII NCHAR character set + . exporting pre-schema procedural objects and actions + . exporting foreign function library names for user SERVICE_NAME + . exporting object type definitions for user SERVICE_NAME + About to export SERVICE_NAME's objects ... + . exporting database links + . exporting sequence numbers + . exporting cluster definitions + . about to export SERVICE_NAME's tables via Conventional Path ... + . exporting synonyms + . exporting views + . exporting stored procedures + . exporting operators + . exporting referential integrity constraints + . exporting triggers + . exporting indextypes + . exporting bitmap, functional and extensible indexes + . exporting posttables actions + . exporting snapshots + . exporting snapshot logs + . exporting job queues + . exporting refresh groups and children + . exporting dimensions + . exporting post-schema procedural objects and actions + . exporting statistics + Export terminated successfully without warnings. - PostGreSQL + PostgreSQL Create a backup file and verify that it was created and has a reasonable size (several megabytes). [root@localhost root]# su - service0 -[service0@localhost service0]$ pg_dump -f /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp service0 -[service0@localhost service0]$ ls -al /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp --rw-rw-r-x 1 service0 service0 4005995 Feb 21 18:28 /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp -[service0@localhost service0]$ exit -[root@localhost root]# -su - service0 -pg_dump -f /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp openacs-dev -ls -al /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp -exit + [service0@localhost service0]$ pg_dump -f /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp service0 + [service0@localhost service0]$ ls -al /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp + -rw-rw-r-x 1 service0 service0 4005995 Feb 21 18:28 /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp + [service0@localhost service0]$ exit + [root@localhost root]# + su - service0 + pg_dump -f /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp openacs-dev + ls -al /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp + exit - Back up the file system - Back up all of the files in the service, including the - database backup file but excluding the auto-generated - supervise directory, which is - unneccesary and has complicated permissions. + Back up the file system + Back up all of the files in the service, including the + database backup file but excluding the auto-generated + supervise directory, which is + unneccesary and has complicated permissions. - In the tar command, - - - c create a - new tar archive - - - p preserves permissions. - - - s preserves file sort order - - - j compresses the output with bz2. - - - The --exclude clauses skips some daemontools files that - are owned by root and thus cannot be backed up by the - service owner. These files are autogenerated and we don't - break anything by omitting them. - - - The --file clause - specifies the name of the output file to be generated; we - manually add the correct extensions. - - - The last clause, - /var/lib/aolserver/service0/, - specifies the starting point for backup. Tar defaults to - recursive backup. - - - [root@yourserver root]# su - service0 -[service0@yourserver service0]$ tar -cpsj --exclude /var/lib/aolserver/service0/etc/daemontools/supervise --file /tmp/service0-backup.tar.bz2 /var/lib/aolserver/service0/ -tar: Removing leading `/' from member names -[service0@yourserver service0]$ - + In the tar command, + + + c create a + new tar archive + + + p preserves permissions. + + + s preserves file sort order + + + j compresses the output with bz2. + + + The --exclude clauses skips some daemontools files that + are owned by root and thus cannot be backed up by the + service owner. These files are autogenerated and we don't + break anything by omitting them. + + + The --file clause + specifies the name of the output file to be generated; we + manually add the correct extensions. + + + The last clause, + /var/lib/aolserver/service0/, + specifies the starting point for backup. Tar defaults to + recursive backup. + + + [root@yourserver root]# su - service0 + [service0@yourserver service0]$ tar -cpsj --exclude /var/lib/aolserver/service0/etc/daemontools/supervise --file /tmp/service0-backup.tar.bz2 /var/lib/aolserver/service0/ + tar: Removing leading `/' from member names + [service0@yourserver service0]$ + - Suffer a catastrophic failure on your production system + Suffer a catastrophic failure on your production system (We'll simulate this step) [root@yourserver root]# svc -d /service/service0 -[root@yourserver root]# mv /var/lib/aolserver/service0/ /var/lib/aolserver/service0.lost -[root@yourserver root]# rm /service/service0 -rm: remove symbolic link `/service/service0'? y -[root@yourserver root]# ps -auxw | grep service0 -root 1496 0.0 0.0 1312 252 ? S 16:58 0:00 supervise service0 -[root@yourserver root]# kill 1496 -[root@yourserver root]# ps -auxw | grep service0 -[root@yourserver root]# su - postgres -[postgres@yourserver pgsql]$ dropdb service0 -DROP DATABASE -[postgres@yourserver pgsql]$ dropuser service0 -DROP USER -[postgres@yourserver pgsql]$ exit -logout -[root@yourserver root]# + [root@yourserver root]# mv /var/lib/aolserver/service0/ /var/lib/aolserver/service0.lost + [root@yourserver root]# rm /service/service0 + rm: remove symbolic link `/service/service0'? y + [root@yourserver root]# ps -auxw | grep service0 + root 1496 0.0 0.0 1312 252 ? S 16:58 0:00 supervise service0 + [root@yourserver root]# kill 1496 + [root@yourserver root]# ps -auxw | grep service0 + [root@yourserver root]# su - postgres + [postgres@yourserver pgsql]$ dropdb service0 + DROP DATABASE + [postgres@yourserver pgsql]$ dropuser service0 + DROP USER + [postgres@yourserver pgsql]$ exit + logout + [root@yourserver root]# - + Recovery - - - Restore the operating system and required software. - You can do this with standard backup processes or by - keeping copies of the install material (OS CDs, OpenACS - tarball and supporting software) and repeating the install - guide. Recreate the service user (service0). - + + Restore the operating system and required software. + You can do this with standard backup processes or by + keeping copies of the install material (OS CDs, OpenACS + tarball and supporting software) and repeating the install + guide. Recreate the service user (service0). + + Restore the OpenACS files and database backup file. -[root@yourserver root]# su - service0 -[service0@yourserver service0]$ cd /var/lib/aolserver -[service0@yourserver aolserver]$ tar xjf /tmp/service0-backup.tar.bz2 -[service0@yourserver aolserver]$ chmod -R 775 service0 -[service0@yourserver aolserver]$ chown -R service0.web service0 + [root@yourserver root]# su - service0 + [service0@yourserver service0]$ cd /var/lib/aolserver + [service0@yourserver aolserver]$ tar xjf /tmp/service0-backup.tar.bz2 + [service0@yourserver aolserver]$ chmod -R 775 service0 + [service0@yourserver aolserver]$ chown -R service0.web service0 - + Restore the database @@ -293,12 +292,12 @@ Set up a clean Oracle database user and - tablespace (more information). + tablespace with the same names as the ones exported from (more information). Invoke the import command - imp service0/service0 FILE=/var/lib/aolserver/service0/database-backup/nighty_backup.dmp + imp service0/service0 FILE=/var/lib/aolserver/service0/database-backup/nighty_backup.dmp FULL=Y @@ -308,30 +307,30 @@ Because of a bug in Postgres backup-recovery, database objects are not guaranteed to be created in the right order. To compensate, we pre-creating some critical items first, which leads to some harmless errors. - [root@yourserver root]# su - postgres -[postgres@yourserver pgsql]$ createuser service0 -Shall the new user be allowed to create databases? (y/n) y -Shall the new user be allowed to create more new users? (y/n) y -CREATE USER -[service0@yourserver web]$ createdb service0 -CREATE DATABASE -[service0@yourserver web]$ psql -f /var/lib/aolserver/service0/packages/acs-kernel/sql/postgresql/postgresql.sql service0 -(many lines omitted) -[service0@yourserver web]$ psql service0 < /var/lib/aolserver/service0/database-backup/database-backup.dmp -(many lines omitted) -[service0@yourserver web]$ exit -[postgres@yourserver pgsql]$ exit -logout - + [root@yourserver root]# su - postgres + [postgres@yourserver pgsql]$ createuser service0 + Shall the new user be allowed to create databases? (y/n) y + Shall the new user be allowed to create more new users? (y/n) y + CREATE USER + [service0@yourserver web]$ createdb service0 + CREATE DATABASE + [service0@yourserver web]$ psql -f /var/lib/aolserver/service0/packages/acs-kernel/sql/postgresql/postgresql.sql service0 + (many lines omitted) + [service0@yourserver web]$ psql service0 < /var/lib/aolserver/service0/database-backup/database-backup.dmp + (many lines omitted) + [service0@yourserver web]$ exit + [postgres@yourserver pgsql]$ exit + logout + Activate the service -[root@yourserver root]# ln -s /var/lib/aolserver/service0/etc/daemontools /service/service0 -[root@yourserver root]# sleep 10 -[root@yourserver root]# svgroup web /service/service0 -[root@yourserver root]# + [root@yourserver root]# ln -s /var/lib/aolserver/service0/etc/daemontools /service/service0 + [root@yourserver root]# sleep 10 + [root@yourserver root]# svgroup web /service/service0 + [root@yourserver root]# @@ -340,177 +339,63 @@ Automated Backup (OPTIONAL) - Backup can encompass all files in - /var/lib/aolserver/service0. Use one cron job to back up the database to a file in - /var/lib/aolserver/service0/database-backup, and a second cron job to back up the entire file tree. - - Postgres automatic backup - Backing up the database consists of creating a file - which is a picture of the database at a particular moment. - Postgres can be backed up while running. - Depending on your overall backup strategy, you can - create a series of database backup files (recommended for a development server where you are using ), or you can create a - single nightly backup file which is then collected into a - bigger backup file that includes the other parts of the - service. The latter technique is more generally recommended. To make a new file every - night, edit the crontab file for service0: - [service0@yourserver service0]$ export EDITOR=emacs;crontab -e - Add this line to the file. The numbers and stars at the beginning are cron columns that specify when the program should be run - in this case, whenever the minute is 0 and the hour is 1, i.e., 1:00 am every day. - 0 1 * * * /usr/local/pgsql/bin/pg_dump -f /var/lib/aolserver/service0/database-backup/service0_`date +\%Y-\%m-\%d`.dmp service0 - If you plan to back up the whole /var/lib/aolserver/service0 directory, then it would be redundant to keep a history of database backups. In that case, set up the cron job to overwrite the previous backup each time: - 0 1 * * * /usr/local/pgsql/bin/pg_dump -f /var/lib/aolserver/service0/database-backup/service0_nightly.dmp service0 - - - - A full Backup/Recovery cycle - On a test service, make sure that your backup-recovery process work. After backing up the database and file system, delete the service as detailed below and then recover it. - - - - Other Backup Strategies - Earlier strategies, included here because this section - hasn't been fully updated yet. - - Set Up Nightly Oracle Exports - Edit the backup script to save the backup - file in /var/lib/aolserver/service0/database-backup. - While you're working with Oracle, you should configure it to do - automatic exports. An export is a separate backup copy of the - database. This copy includes all of the database's state at the - time that the export was initiated. If your database is corrupted, - you can restore from one of these backups. You should do this step as - root. - - - Automating backups is accomplished using the UNIX - crontab facility. - - - While still root, run the - following command. You can replace the - EDITOR="emacs -nw" - portion with whatever editor your prefer, such as - EDITOR=vi. - - -root:~# export EDITOR="emacs -nw" -root:~# crontab -e - - Now add the following line on a line by itself - - -0 23 * * * /usr/sbin/export-oracle - + The recommended backup strategy for a production sit is to use an automated script which first backs up the database to a file in /var/lib/aolserver/service0/database-backup and then backs up all of /var/lib/aolserver/service0 to a single zip file, and then copies that zip file to another computer. + + + Make sure that the manual backup process described above works. + + + Customize the default backup script. Edit /var/lib/aolserver/service0/etc/backup.sh with your specific parameters. + + - Save the file, exit the editor. Verify that the addition - succeeded by checking the output of the following command. - - -root:~# crontab -l | grep export-oracle -0 23 * * * /usr/sbin/export-oracle -root:~# exit -; Logout - - If you see the line, go ahead and log out. - - - Set up nightly Postgres exports - - This is an alternate method to the crontabls - backup. - Dowload this script - to /tmp. At the top of the script - are several variables that you'll need to customize: - - - - - bak - location where you want - local backups to be saved - - - - servername - name of your server - (and database instance) - - - - ftp_user - username on your ftp - account - - - - ftp_password - password on your - ftp account - - - - ftp_dir - path on the remote - server where your backups will be uploaded - - - - ftp_server - your ftp server - - - - - - Next, we'll save this file to our server's - tcl directory so that it will be - loaded on startup. It will automatically be run every night at - midnight. Note that this script only backs up the database - not the - OpenACS scripts and file content. - - - -joeuser:~$ cp /tmp/acs-pgbackup-init.txt ~/var/lib/aolserver/birdnotes/tcl/acs-pgbackup-init.tcl -joeuser:~$ restart-aolserver birdnotes - + Make sure the file is executable: +chmod +x backup.sh + + - That's it! The script will email you with each successful backup (or - if it fails, it will send you an email with the reason) - - - + Set this file to run automatically by adding a line to root's crontab. (Typically, with export EDITOR=emacs; crontab -e.) This example runs the backup script at 1:30 am every day. + 30 1 * * * * sh /var/lib/aolserver/service0/etc/backup.sh + + + + Using CVS for backup-recovery - If you are already using CVS, you probably don't - need to do anything to back up your files. Just make - sure that your current work is checked into the system. - You can then roll back based on date - note the - current system time, down to the minute. For maximum - safety, you can apply a tag to your current - files. You will still need to back up your database. - Note that, if you did the CVS options in this document, the /var/lib/aolserver/service0/etc directory is not included in cvs and you may want to add it. - [root@localhost root]# su - service0 -[service0@localhost service0]$ cd /var/lib/aolserver/service0 -[service0@localhost service0]$ cvs commit -m "last-minute commits before upgrade to 4.6" -cvs commit: Examining . -cvs commit: Examining bin -(many lines omitted) -[service0@localhost service0]$ cvs tag before_upgrade_to_4_6 -cvs server: Tagging bin -T bin/acs-4-0-publish.sh -T bin/ad-context-server.pl -(many lines omitted) -[service0@localhost service0]$ exit -[root@localhost root]# -su - service0 -cd /var/lib/aolserver/service0 -cvs commit -m "last-minute commits before upgrade to 4.6" -cvs tag before_upgrade_to_4_6 -exit - To restore files from a cvs tag such as the one used above: - [root@localhost root]# su - service0 -[service0@localhost service0]$ cd /var/lib/aolserver/service0 -[service0@localhost service0]$ cvs up -r current -[service0@localhost service0]$ exit -su - service0 -cd /var/lib/aolserver/service0 -cvs up -r current + CVS-only backup is often appropriate for development sites. If you are already using CVS and your data is not important, you probably don't + need to do anything to back up your files. Just make + sure that your current work is checked into the system. + You can then roll back based on date - note the + current system time, down to the minute. For maximum + safety, you can apply a tag to your current + files. You will still need to back up your database. + Note that, if you did the CVS options in this document, the /var/lib/aolserver/service0/etc directory is not included in cvs and you may want to add it. + [root@localhost root]# su - service0 + [service0@localhost service0]$ cd /var/lib/aolserver/service0 + [service0@localhost service0]$ cvs commit -m "last-minute commits before upgrade to 4.6" + cvs commit: Examining . + cvs commit: Examining bin + (many lines omitted) + [service0@localhost service0]$ cvs tag before_upgrade_to_4_6 + cvs server: Tagging bin + T bin/acs-4-0-publish.sh + T bin/ad-context-server.pl + (many lines omitted) + [service0@localhost service0]$ exit + [root@localhost root]# + su - service0 + cd /var/lib/aolserver/service0 + cvs commit -m "last-minute commits before upgrade to 4.6" + cvs tag before_upgrade_to_4_6 + exit + To restore files from a cvs tag such as the one used above: + [root@localhost root]# su - service0 + [service0@localhost service0]$ cd /var/lib/aolserver/service0 + [service0@localhost service0]$ cvs up -r current + [service0@localhost service0]$ exit + su - service0 + cd /var/lib/aolserver/service0 + cvs up -r current - ($Id$) - - - \ No newline at end of file + \ No newline at end of file