Index: openacs-4/packages/acs-core-docs/www/xml/install-guide/recovery.xml =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-core-docs/www/xml/install-guide/recovery.xml,v diff -u -r1.5 -r1.6 --- openacs-4/packages/acs-core-docs/www/xml/install-guide/recovery.xml 21 Oct 2003 10:04:49 -0000 1.5 +++ openacs-4/packages/acs-core-docs/www/xml/install-guide/recovery.xml 11 Nov 2003 16:57:20 -0000 1.6 @@ -64,22 +64,121 @@ then you can apply standard file-based backup strategies to /var/lib/aolserver/service0 - + + + + Snapshot backup and recovery - - Snapshot backup and recovery - - This section describes how to make a one-time backup of - the files and database. This is useful for rolling back to + This section describes how to make a one-time backup and + restore of the files and database. This is useful for rolling back to known-good versions of a service, such as at initial - installation and just before an upgrade. - - - - PostGreSQL - Create a backup file and verify that it was created and has a reasonable size (several megabytes). - - [root@localhost root]# su - service0 + installation and just before an upgrade. First, you back up the + database to a file within the file tree. Then, you back up the + file tree. All of the information needed to rebuild the site, + including the AOLserver config files, is then in tree for + regular file system backup. + + + + Back up the database to a file + + + + + + Oracle + + + + + Download the backup script. Save the file export-oracle.txt as + /tmp/export-oracle.txt + + + + Login as root. The following commands will install the export script: + + +joeuser:~$ su - +Password: *********** +root:~# cp /tmp/export-oracle.txt /usr/sbin/export-oracle +root:~# chmod 700 /usr/sbin/export-oracle + + + + Setup the export directory; this is the directory where backups will + be stored. We recommend the directory + /ora8/m02/oracle-exports. + + +root:~# mkdir /ora8/m02/oracle-exports +root:~# chown oracle.dba /ora8/m02/oracle-exports +root:~# chmod 770 /ora8/m02/oracle-exports + + + + Now edit + /usr/sbin/export-oracle and + change the SERVICE_NAME and + DATABASE_PASSWORD fields to + their correct values. If you want to use a directory other than + /ora8/m02/oracle-exports, you + also need to change the + exportdir setting. + + + Test the export procedure by running the command: + + + +root:~# /usr/sbin/export-oracle +mv: /ora8/m02/oracle-exports/oraexport-service_name.dmp.gz: No such file or directory + +Export: Release 8.1.6.1.0 - Production on Sun Jun 11 18:07:45 2000 + +(c) Copyright 1999 Oracle Corporation. All rights reserved. + + +Connected to: Oracle8i Enterprise Edition Release 8.1.6.1.0 - Production +With the Partitioning option +JServer Release 8.1.6.0.0 - Production +Export done in US7ASCII character set and US7ASCII NCHAR character set +. exporting pre-schema procedural objects and actions +. exporting foreign function library names for user SERVICE_NAME +. exporting object type definitions for user SERVICE_NAME +About to export SERVICE_NAME's objects ... +. exporting database links +. exporting sequence numbers +. exporting cluster definitions +. about to export SERVICE_NAME's tables via Conventional Path ... +. exporting synonyms +. exporting views +. exporting stored procedures +. exporting operators +. exporting referential integrity constraints +. exporting triggers +. exporting indextypes +. exporting bitmap, functional and extensible indexes +. exporting posttables actions +. exporting snapshots +. exporting snapshot logs +. exporting job queues +. exporting refresh groups and children +. exporting dimensions +. exporting post-schema procedural objects and actions +. exporting statistics +Export terminated successfully without warnings. + + + + + + + PostGreSQL + Create a backup file and verify that it was created and has a reasonable size (several megabytes). + + [root@localhost root]# su - service0 [service0@localhost service0]$ pg_dump -f /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp service0 [service0@localhost service0]$ ls -al /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp -rw-rw-r-x 1 service0 service0 4005995 Feb 21 18:28 /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp @@ -89,110 +188,17 @@ pg_dump -f /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp openacs-dev ls -al /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp exit - - - - File tree with CVS - If you are already using CVS, you probably don't - need to do anything to back up your data. Just make - sure that your current work is checked into the system. - You can then roll back based on date - note the - current system time, down to the minute. For maximum - safety, you can apply a tag to your current - files. - Note that, if you did the CVS options in this document, the /var/lib/aolserver/service0/etc directory is not included in cvs and you may want to add it. - - [root@localhost root]# su - service0 -[service0@localhost service0]$ cd /var/lib/aolserver/service0 -[service0@localhost service0]$ cvs commit -m "last-minute commits before upgrade to 4.6" -cvs commit: Examining . -cvs commit: Examining bin -(many lines omitted) -[service0@localhost service0]$ cvs tag before_upgrade_to_4_6 -cvs server: Tagging bin -T bin/acs-4-0-publish.sh -T bin/ad-context-server.pl -(many lines omitted) -[service0@localhost service0]$ exit -[root@localhost root]# -su - service0 -cd /var/lib/aolserver/service0 -cvs commit -m "last-minute commits before upgrade to 4.6" -cvs tag before_upgrade_to_4_6 -exit - To restore files from a cvs tag such as the one used above: - [root@localhost root]# su - service0 -[service0@localhost service0]$ cd /var/lib/aolserver/service0 -[service0@localhost service0]$ cvs up -r current -[service0@localhost service0]$ exit -su - service0 -cd /var/lib/aolserver/service0 -cvs up -r current - - - File tree without CVS - If you don't use cvs, you may want to back up the working directory. The simplest way is just to copy it. - - [root@localhost root]# su - service0 -[service0@localhost service0]$ cp -r /var/lib/aolserver/service0 /var/lib/aolserver/service0-before-upgrade-to-4.6 -[service0@localhost service0]$ exit -[root@localhost root]# -su - service0 -cp -r /var/lib/aolserver/service0 /var/lib/aolserver/service0-before-upgrade-to-4.6 -exit - To restore the files, copy the directory back (do this while the service is stopped): - [root@localhost root]# su - service0 -[service0@localhost service0]$ mv /var/lib/aolserver/service0 /var/lib/aolserver/openacs-failed-upgrade -[service0@localhost service0]$ mv /var/lib/aolserver/service0-before-upgrade-to-4.6 /var/lib/aolserver/service0 -[service0@localhost service0]$ exit -[root@localhost root]# - - - - Database Rollback - Restore the database to its backed-up state. - - [root@localhost root]# su - service0 -[service0@localhost service0]$ svc -d /service/service0 -[service0@localhost service0]$ dropdb service0 -DROP DATABASE -[service0@localhost service0]$ createdb service0 -CREATE DATABASE - -PostGreSQL's dump command does not guarantee to back up all of the -procedures and things in the right order for them to be reassembled. -In practice, OpenACS users have found that rebuilding some of the -common procedures before running the restore usually addresses this. -You will see a number of "already exists" errors when you run the -database restore; these can be ignored. This forum thread has more information. -[service0@localhost service0]$ psql -f /var/lib/aolserver/service0/packages/acs-kernel/sql/postgresql/postgresql.sql service0 -[service0@localhost service0]$ psql service0 < /backup/openacs/openacs_dev_before_upgrade_to_4.6.dmp -[service0@localhost service0]$ svc -u /service/service0 -[service0@localhost service0]$ exit -All commands for a database rollback: -su - service0 -svc -d /service/service0 -dropdb service0 -createdb service0 -psql -f /var/lib/aolserver/service0/packages/acs-kernel/sql/postgresql/postgresql.sql service0 -psql service0 < /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp -svc -u /service/service0 -exit - - - - - - Back up the file system - Here's a quick manual way to back up a reference install - - it should be replaced by an automated script within - OpenACS. The command excludes the auto-generated + + + + + Back up the file system + Back up all of the files in the service, including the + database backup file but excluding the auto-generated supervise directory, which is - unneccesary and has complicated permissions. Make sure that you - are using the cron job to back up the database to a file in - /var/lib/aolserver/service0/database-backup - so that the tar command will include the database. + unneccesary and has complicated permissions. + In the tar command, @@ -230,41 +236,13 @@ [service0@yourserver service0]$ tar -cpsj --exclude /var/lib/aolserver/service0/etc/daemontools/supervise --file /tmp/service0-backup.tar.bz2 /var/lib/aolserver/service0/ tar: Removing leading `/' from member names [service0@yourserver service0]$ - - - - Automated Backup (OPTIONAL) - Backup can encompass all files in - /var/lib/aolserver/service0. For a development - server, putting the files in cvs, and backing up the database nightly, is sufficient. (It's important then to back up the cvs repository!) - - Postgres automatic backup - Backing up the database consists of creating a file - which is a picture of the database at a particular moment. - Postgres can be backed up while running. A quick way to automate database backup is a cron job. This - is not recommended for production and is not part of the Reference - Platform, because it is not cross-platform and can fail silently. - A more thorough solution using the cronjob OpenACS package is - planned. - Depending on your overall backup strategy, you can - create a series of database backup files, or you can create a - single nightly backup file which is then collected into a - bigger backup file that includes the other parts of the - service (web pages, content, code). To make a new file every - night, edit the crontab file for service0: - [service0@yourserver service0]$ export EDITOR=emacs;crontab -e - Add this line to the file. The numbers and stars at the beginning are cron columns that specify when the program should be run - in this case, whenever the minute is 0 and the hour is 1, i.e., 1:00 am every day. - 0 1 * * * /usr/local/pgsql/bin/pg_dump -f /var/lib/aolserver/service0/database-backup/service0_`date +\%Y-\%m-\%d`.dmp service0 - If you plan to back up the whole /var/lib/aolserver/service0 directory, then it would be redundant to keep a history of database backups. In that case, set up the cron job to overwrite the previous backup each time: - 0 1 * * * /usr/local/pgsql/bin/pg_dump -f /var/lib/aolserver/service0/database-backup/service0_nightly.dmp service0 - - - - A full Backup/Recovery cycle - On a test service, make sure that your backup-recovery process work. After backing up the database and file system, delete the service as detailed below and then recover it. - - Delete the Service - [root@yourserver root]# svc -d /service/service0 + + + + Suffer a catastrophic failure on your production system + (We'll simulate this step) + + [root@yourserver root]# svc -d /service/service0 [root@yourserver root]# mv /var/lib/aolserver/service0/ /var/lib/aolserver/service0.lost [root@yourserver root]# rm /service/service0 rm: remove symbolic link `/service/service0'? y @@ -280,156 +258,134 @@ [postgres@yourserver pgsql]$ exit logout [root@yourserver root]# - - - - Recovery + + + + Recovery + + Restore the operating system and required software. You can do this with standard backup processes or by keeping copies of the install material (OS CDs, OpenACS - tarball and supporting software) and repeating the install guide. + tarball and supporting software) and repeating the install + guide. Recreate the service user (service0). - - Restore the OpenACS service. Assuming the user already exists, restore the database and files from backup and restore the daemontools link. (Because of a bug in Postgres backup-recovery, not all database objects are created in the correct order. To compensate, pre-creating some objects usually work.) + + Restore the OpenACS files and database backup file. +[root@yourserver root]# su - service0 +[service0@yourserver service0]$ cd /var/lib/aolserver +[service0@yourserver aolserver]$ tar xjf /tmp/service0-backup.tar.bz2 +[service0@yourserver aolserver]$ chmod -R 775 service0 +[service0@yourserver aolserver]$ chown -R service0.web service0 + + + + + Restore the database + + + + Oracle + + + + + Set up a clean Oracle database user and + tablespace (more information). + + + Invoke the import command + imp service0/service0 FILE=/var/lib/aolserver/service0/database-backup/nighty_backup.dmp + + + + + + Postgres + + + Because of a bug in Postgres backup-recovery, database objects are not guaranteed to be created in the right order. To compensate, we pre-creating some critical items first, which leads to some harmless errors. [root@yourserver root]# su - postgres [postgres@yourserver pgsql]$ createuser service0 Shall the new user be allowed to create databases? (y/n) y Shall the new user be allowed to create more new users? (y/n) y CREATE USER -[postgres@yourserver pgsql]$ exit -logout - -[root@yourserver root]# su - service0 -[service0@yourserver service0]$ cd /var/lib/aolserver -[service0@yourserver web]$ tar xjf /tmp/service0-backup.tar.bz2 -[service0@yourserver web]$ chmod -R 700 service0 [service0@yourserver web]$ createdb service0 CREATE DATABASE [service0@yourserver web]$ psql -f /var/lib/aolserver/service0/packages/acs-kernel/sql/postgresql/postgresql.sql service0 (many lines omitted) [service0@yourserver web]$ psql service0 < /var/lib/aolserver/service0/database-backup/database-backup.dmp (many lines omitted) [service0@yourserver web]$ exit -[root@yourserver root]# ln -s /var/lib/aolserver/service0/etc/daemontools /service/service0 +[postgres@yourserver pgsql]$ exit +logout + + + + + + Activate the service +[root@yourserver root]# ln -s /var/lib/aolserver/service0/etc/daemontools /service/service0 [root@yourserver root]# sleep 10 [root@yourserver root]# svgroup web /service/service0 [root@yourserver root]# - - + + + + + + + + Automated Backup (OPTIONAL) + Backup can encompass all files in + /var/lib/aolserver/service0. Use one cron job to back up the database to a file in + /var/lib/aolserver/service0/database-backup, and a second cron job to back up the entire file tree. + + Postgres automatic backup + Backing up the database consists of creating a file + which is a picture of the database at a particular moment. + Postgres can be backed up while running. + Depending on your overall backup strategy, you can + create a series of database backup files (recommended for a development server where you are using ), or you can create a + single nightly backup file which is then collected into a + bigger backup file that includes the other parts of the + service. The latter technique is more generally recommended. To make a new file every + night, edit the crontab file for service0: + [service0@yourserver service0]$ export EDITOR=emacs;crontab -e + Add this line to the file. The numbers and stars at the beginning are cron columns that specify when the program should be run - in this case, whenever the minute is 0 and the hour is 1, i.e., 1:00 am every day. + 0 1 * * * /usr/local/pgsql/bin/pg_dump -f /var/lib/aolserver/service0/database-backup/service0_`date +\%Y-\%m-\%d`.dmp service0 + If you plan to back up the whole /var/lib/aolserver/service0 directory, then it would be redundant to keep a history of database backups. In that case, set up the cron job to overwrite the previous backup each time: + 0 1 * * * /usr/local/pgsql/bin/pg_dump -f /var/lib/aolserver/service0/database-backup/service0_nightly.dmp service0 + + A full Backup/Recovery cycle + On a test service, make sure that your backup-recovery process work. After backing up the database and file system, delete the service as detailed below and then recover it. + Other Backup Strategies Earlier strategies, included here because this section hasn't been fully updated yet. Set Up Nightly Oracle Exports - - - (This has not yet been updated to fit with the Reference - install. To do so, edit the backup script to save the backup - file in /var/lib/aolserver/service0/database-backup). + Edit the backup script to save the backup + file in /var/lib/aolserver/service0/database-backup. While you're working with Oracle, you should configure it to do automatic exports. An export is a separate backup copy of the database. This copy includes all of the database's state at the time that the export was initiated. If your database is corrupted, you can restore from one of these backups. You should do this step as root. - - - - Download the backup script. Save the file export-oracle.txt as - /tmp/export-oracle.txt - - - - Login as root. The following commands will install the export script: - - -joeuser:~$ su - -Password: *********** -root:~# cp /tmp/export-oracle.txt /usr/sbin/export-oracle -root:~# chmod 700 /usr/sbin/export-oracle - - - - Setup the export directory; this is the directory where backups will - be stored. We recommend the directory - /ora8/m02/oracle-exports. - - -root:~# mkdir /ora8/m02/oracle-exports -root:~# chown oracle.dba /ora8/m02/oracle-exports -root:~# chmod 770 /ora8/m02/oracle-exports - - - - Now edit - /usr/sbin/export-oracle and - change the SERVICE_NAME and - DATABASE_PASSWORD fields to - their correct values. If you want to use a directory other than - /ora8/m02/oracle-exports, you - also need to change the - exportdir setting. - - - Test the export procedure by running the command: - - - -root:~# /usr/sbin/export-oracle -mv: /ora8/m02/oracle-exports/oraexport-service_name.dmp.gz: No such file or directory - -Export: Release 8.1.6.1.0 - Production on Sun Jun 11 18:07:45 2000 - -(c) Copyright 1999 Oracle Corporation. All rights reserved. - - -Connected to: Oracle8i Enterprise Edition Release 8.1.6.1.0 - Production -With the Partitioning option -JServer Release 8.1.6.0.0 - Production -Export done in US7ASCII character set and US7ASCII NCHAR character set -. exporting pre-schema procedural objects and actions -. exporting foreign function library names for user SERVICE_NAME -. exporting object type definitions for user SERVICE_NAME -About to export SERVICE_NAME's objects ... -. exporting database links -. exporting sequence numbers -. exporting cluster definitions -. about to export SERVICE_NAME's tables via Conventional Path ... -. exporting synonyms -. exporting views -. exporting stored procedures -. exporting operators -. exporting referential integrity constraints -. exporting triggers -. exporting indextypes -. exporting bitmap, functional and extensible indexes -. exporting posttables actions -. exporting snapshots -. exporting snapshot logs -. exporting job queues -. exporting refresh groups and children -. exporting dimensions -. exporting post-schema procedural objects and actions -. exporting statistics -Export terminated successfully without warnings. - - If you don't have any warnings, proceed to automate the - backups. - - - + Automating backups is accomplished using the UNIX crontab facility. - - + + While still root, run the following command. You can replace the EDITOR="emacs -nw" @@ -455,12 +411,9 @@ root:~# exit ; Logout - If you see the line, go ahead and log out. - - - - - + If you see the line, go ahead and log out. + + Set up nightly Postgres exports This is an alternate method to the crontabls - backup. @@ -519,6 +472,43 @@ + + Using CVS for backup-recovery + If you are already using CVS, you probably don't + need to do anything to back up your files. Just make + sure that your current work is checked into the system. + You can then roll back based on date - note the + current system time, down to the minute. For maximum + safety, you can apply a tag to your current + files. You will still need to back up your database. + Note that, if you did the CVS options in this document, the /var/lib/aolserver/service0/etc directory is not included in cvs and you may want to add it. + [root@localhost root]# su - service0 +[service0@localhost service0]$ cd /var/lib/aolserver/service0 +[service0@localhost service0]$ cvs commit -m "last-minute commits before upgrade to 4.6" +cvs commit: Examining . +cvs commit: Examining bin +(many lines omitted) +[service0@localhost service0]$ cvs tag before_upgrade_to_4_6 +cvs server: Tagging bin +T bin/acs-4-0-publish.sh +T bin/ad-context-server.pl +(many lines omitted) +[service0@localhost service0]$ exit +[root@localhost root]# +su - service0 +cd /var/lib/aolserver/service0 +cvs commit -m "last-minute commits before upgrade to 4.6" +cvs tag before_upgrade_to_4_6 +exit + To restore files from a cvs tag such as the one used above: + [root@localhost root]# su - service0 +[service0@localhost service0]$ cd /var/lib/aolserver/service0 +[service0@localhost service0]$ cvs up -r current +[service0@localhost service0]$ exit +su - service0 +cd /var/lib/aolserver/service0 +cvs up -r current + ($Id$)