Index: openacs-4/packages/acs-core-docs/www/xml/install-guide/maintenance.xml =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-core-docs/www/xml/install-guide/maintenance.xml,v diff -u -r1.12.2.1 -r1.12.2.2 --- openacs-4/packages/acs-core-docs/www/xml/install-guide/maintenance.xml 17 Apr 2004 14:11:03 -0000 1.12.2.1 +++ openacs-4/packages/acs-core-docs/www/xml/install-guide/maintenance.xml 5 Jul 2004 19:47:36 -0000 1.12.2.2 @@ -16,7 +16,7 @@ Starting and Stopping an OpenACS instance. - The simplest way to start and stop and OpenACS site is to run the startup shell script provided, /var/lib/aolserver/service0/etc/daemontools/run. This runs as a regular task, and logs to the logfile. To stop the site, kill the script. + The simplest way to start and stop and OpenACS site is to run the startup shell script provided, /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/daemontools/run. This runs as a regular task, and logs to the logfile. To stop the site, kill the script. A more stable way to run OpenACS is with a "keepalive" mechanism of some sort, so that whenever the server halts or is stopped for a reset, it restarts automatically. This is recommended for development and production servers. The Reference Platform uses Daemontools to control AOLserver. A simpler method, using init, is here. @@ -49,62 +49,62 @@ Hence, the AOLserver instance for your development server is started by the file - /service/service0/run. + /service/$OPENACS_SERVICE_NAME/run. But we use a symlink to make it easier to add and remove stuff from the /service, so the actual location is - /var/lib/aolserver/service0etc/daemontools/run. + /var/lib/aolserver/$OPENACS_SERVICE_NAMEetc/daemontools/run. Daemontools creates additional files and directories to track status and log. A daemontools directory is included in the OpenACS tarball at - /var/lib/aolserver/service0/etc/daemontools. To use it, first ill any existing AOLserver instances. As root, link the daemontools directory into the /service directory. Daemontools' svscan process checks this directory every five seconds, and will quickly execute run. + /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/daemontools. To use it, first ill any existing AOLserver instances. As root, link the daemontools directory into the /service directory. Daemontools' svscan process checks this directory every five seconds, and will quickly execute run. - [service0 etc]$ killall nsd + [$OPENACS_SERVICE_NAME etc]$ killall nsd nsd: no process killed -[service0 etc]$ emacs /var/lib/aolserver/service0/etc/daemontools/run -[service0 etc]$ exit +[$OPENACS_SERVICE_NAME etc]$ emacs /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/daemontools/run +[$OPENACS_SERVICE_NAME etc]$ exit -[root root]# ln -s /var/lib/aolserver/service0/etc/daemontools/ /service/service0 +[root root]# ln -s /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/daemontools/ /service/$OPENACS_SERVICE_NAME Verify that AOLserver is running. [root root]# ps -auxw | grep nsd -service0 5562 14.2 6.2 22436 15952 ? S 11:55 0:04 /usr/local/aolserver/bin/nsd -it /var/lib/aolserver/service0/etc/config.tcl -u serve +$OPENACS_SERVICE_NAME 5562 14.2 6.2 22436 15952 ? S 11:55 0:04 /usr/local/aolserver/bin/nsd -it /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/config.tcl -u serve root 5582 0.0 0.2 3276 628 pts/0 S 11:55 0:00 grep nsd [root root]# - The user service0 can now control the service service0 with these commands: + The user $OPENACS_SERVICE_NAME can now control the service $OPENACS_SERVICE_NAME with these commands: - svc -d /service/service0 - + svc -d /service/$OPENACS_SERVICE_NAME - Bring the server down - svc -u /service/service0 - + svc -u /service/$OPENACS_SERVICE_NAME - Start the server up and leave it in keepalive mode. - svc -o /service/service0 - + svc -o /service/$OPENACS_SERVICE_NAME - Start the server up once. Do not restart it if it stops. - svc -t /service/service0 - + svc -t /service/$OPENACS_SERVICE_NAME - Stop and immediately restart the server. - svc -k /service/service0 - + svc -k /service/$OPENACS_SERVICE_NAME - Sends the server a KILL signal. This is like KILL -9. AOLserver exits immediately. If svc -t fails to fully kill AOLserver, use this option. This does not take the server out of keepalive mode, so it should still bounce back up immediately. @@ -115,20 +115,20 @@ Install a script to automate the stopping and starting - of AOLserver services via daemontools. You can then restart a service via restart-aolserver service0 - [root root]# cp /var/lib/aolserver/service0/packages/acs-core-docs/www/files/restart-aolserver-daemontools.txt /usr/local/bin/restart-aolserver + of AOLserver services via daemontools. You can then restart a service via restart-aolserver $OPENACS_SERVICE_NAME + [root root]# cp /var/lib/aolserver/$OPENACS_SERVICE_NAME/packages/acs-core-docs/www/files/restart-aolserver-daemontools.txt /usr/local/bin/restart-aolserver [root root]# chmod 755 /usr/local/bin/restart-aolserver [root root]# At this point, these commands will work only for the - root user. Grant permission for the web group to use svc commands on the service0 server. - [root root]# svgroup web /service/service0 + root user. Grant permission for the web group to use svc commands on the $OPENACS_SERVICE_NAME server. + [root root]# svgroup web /service/$OPENACS_SERVICE_NAME [root root]# - Verify that the controls work. You may want to tail -f /var/lib/aolserver/service0/log/service0-error.log in another window, so you can see what happens when you type these commands. + Verify that the controls work. You may want to tail -f /var/lib/aolserver/$OPENACS_SERVICE_NAME/log/$OPENACS_SERVICE_NAME-error.log in another window, so you can see what happens when you type these commands. @@ -140,7 +140,7 @@ How it Works - + Program @@ -165,10 +165,10 @@ aolserver supervise (a child of svscanboot) - /service/service0/run - /var/lib/aolserver/service0/log/error.log - /var/lib/aolserver/service0/log/service0.log - svc -k /service/service0 + /service/$OPENACS_SERVICE_NAME/run + /var/lib/aolserver/$OPENACS_SERVICE_NAME/log/error.log + /var/lib/aolserver/$OPENACS_SERVICE_NAME/log/$OPENACS_SERVICE_NAME.log + svc -k /service/$OPENACS_SERVICE_NAME postgresql @@ -335,15 +335,15 @@ Services on different ports To run a different service on another port but the same ip, simply repeat replacing - service0, and change the + $OPENACS_SERVICE_NAME, and change the set httpport 8000 set httpsport 8443 to different values. Services on different host names For example, suppose you want to support -http://foo.com and +http://service0.com and http://bar.com on the same machine. The easiest way is to assign each one a different ip address. Then you can install two services as above, but with @@ -369,7 +369,7 @@ Multiple-server configuration - + @@ -381,7 +381,7 @@ By Joel Aufrecht - This section describes minimal-risk methods for deploying changes on a production network. The important characteristics of a safe change deployment include: (THIS SECTION IN DEVELOPMENT) + This section describes two minimal-risk methods for deploying changes on a production network. The important characteristics of a safe change deployment include: (THIS SECTION IN DEVELOPMENT) Control: You know for sure that the change you are making is the change that you intend to make and is the change that you tested. @@ -392,29 +392,104 @@ - Deployment with CVS - With this method, we control the files on a site via CVS. In this example, there is one development site and one production site. The only way files should move between the two is via cvs. The production site could run "HEAD" from cvs (raw example from chat log:) + Method 1: Deployment with CVS + With this method, we control the files on a site via + CVS. This example uses one developmental server (service0-dev) and one + production server (service0). Depending on your needs, you can also + have a staging server for extensive testing before you go + live. The only way files should move between the server + instances is via cvs. + + To set up a developmental installation, first set up + either your developmental installation or your production + installation, and follow the instructions for committing your + files to CVS. We'll assume in this example that you set up the + production server (service0). To set up the developmental instance, + you then follow the intall guide again, this time creating a new + user (service0-dev) that you'll use for the new installation. To get + the files for service0-dev, you check them out from cvs (check out + service0). + -1) change the file on dev as desired +su - service0-dev +co -d /cvsroot service0 +mv service0 /var/lib/aolserver/service0-dev +ln -s /home/service0-dev/web /var/lib/aolserver/service0-dev +emacs web/etc/config.tcl +emacs web/etc/daemontools/run + + + In the config.tcl file, you'll probably want to pay attention + the rollout support section. That will ensure that email on your + developmental server will not be sent out to the general world. + + Also, instead of going through the OpenACS online + installer, you'll actually load live data into your production + server. + + You can even automate the process of getting live data + from your production server. Copy something like this to + /home/service0-dev/bin and put it in service0-dev's crontab to + run once a night. You'll need to make sure the database backups + are set up in service0's crontab, and that if the servers are on + different physical machines, that the database backup is copied + to the developmental machine once per night. + + +/usr/local/bin/svc -d /service/service0-dev +/bin/sleep 60 +# this deletes the dev database! +/usr/local/pgsql/bin/dropdb service0-dev +/usr/local/pgsql/bin/createdb -E UNICODE service0-dev +# this is not necessary from Postgres 7.4 on +/usr/local/pgsql/bin/psql -f /var/lib/aolserver/service0-dev/packages/acs-kernel/sql/postgresql/postgresql.sql service0 +mv /var/lib/aolserver/service0/database-backup/service0-nightly-backup.dmp.gz /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup-old.dmp.gz +/bin/gunzip /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup.dmp.gz +/usr/bin/perl -pi -e "s/^\\connect service0$/\\connect service0-dev/" /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup.dmp +/usr/local/pgsql/bin/psql service0-dev < /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup.dmp +/usr/local/bin/svc -u /service/service0-dev +/bin/gzip /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup-old.dmp + + + Your developmental server will always have data about a day + old. + + To make changes on service0-dev: + + +1) change the file on service0-dev as desired 2) test the new file 3) commit the file: -if the file is /web/foo-dev/www/index.adp, do: +if the file is /var/lib/aolserver/service0-dev/www/index.adp, do: -cd /web/foo-dev/www +cd /var/lib/aolserver/service0-dev/www cvs diff index.adp (this is optional; it's just a reality check) the lines starting > will be added and the lines starting < will be removed, when you commit if that looks okay, commit with: cvs -m "changing text on front page for February conference" index.adp -the stuff in -m "foo" is a comment visible only from within cvs commands +the stuff in -m "service0" is a comment visible only from within cvs commands + + + To make these changes take place on service0: + + 4) update the file on production: -cd /web/foo-prod/www -cvs up index.adp +cd /var/lib/aolserver/service0/www +cvs up -Pd index.adp + + If you make changes that require changes to the database, + test them out first on service0-dev, using either -create.sql or + upgrade scripts. Once you've tested them, you then update and + run the upgrade scripts from the package manager. + +The production site can run "HEAD" from cvs. + The drawback to using HEAD as the live code is that you cannot commit new work on the development server without erasing the definition of 'working production code.' So a better method is to use a tag. This guarantees that, at any time in the future, you can retrieve exactly the same set of code. This is useful for both of the characteristics of safe change deployment. For control, you can use tags to define a body of code, test that code, and then know that what you are deploying is exactly that code. For rollback, you can use return to the last working tag if the new tag (or new, untagged changes) cause problems. .... example of using tags to follow ... - A/B Deployment + Method 2: A/B Deployment The approach taken in this section is to always create a new service with the desired changes, running in parallel with the existing site. This guarantees control, at least at the final step of the process: you know what changes you are about to make because you can see them directly. It does not, by itself, guarantee the entire control chain. You need additional measures to make sure that the change you are making is exactly and completely the change you intended to make and tested previously, and nothing more. Those additional measures typically take the form of source control tags and system version numbers. The parallel-server approach also guarantees rollback because the original working service is not touched; it is merely set aside. This approach can has limitations. If the database or file system regularly receiving new data, you must interrupt this function or risk losing data in the shuffle. It also requires extra steps if the database will be affected. @@ -423,23 +498,23 @@ Simple A/B Deployment - Step 1 - +
Simple A/B Deployment - Step 2 - +
Simple A/B Deployment - Step 3 - +
@@ -450,23 +525,23 @@ Complex A/B Deployment - Step 1 - +
Complex A/B Deployment - Step 2 - +
Complex A/B Deployment - Step 3 - +
@@ -487,17 +562,17 @@ Prepare a certificate directory for the service. - [service0 etc]$ mkdir /var/lib/aolserver/service0/etc/certs -[service0 etc]$ chmod 700 /var/lib/aolserver/service0/etc/certs -[service0 etc]$ -mkdir /var/lib/aolserver/service0/etc/certs -chmod 700 /var/lib/aolserver/service0/etc/certs + [$OPENACS_SERVICE_NAME etc]$ mkdir /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/certs +[$OPENACS_SERVICE_NAME etc]$ chmod 700 /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/certs +[$OPENACS_SERVICE_NAME etc]$ +mkdir /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/certs +chmod 700 /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/certs It takes two files to support an SSL connection. The certificate is the public half of the key pair - the server sends the certificate to browser requesting ssl. The key is the private half of the key pair. In addition, the certificate must be signed by Certificate Authority or browsers will protest. Each web browser ships with a built-in list of acceptable Certificate Authorities (CAs) and their keys. Only a site certificate signed by a known and approved CA will work smoothly. Any other certificate will cause browsers to produce some messages or block the site. Unfortunately, getting a site certificate signed by a CA costs money. In this section, we'll generate an unsigned certificate which will work in most browsers, albeit with pop-up messages. Use an OpenSSL perl script to generate a certificate and key. - [service0 service0]$ cd /var/lib/aolserver/service0/etc/certs -[service0 certs]$ perl /usr/share/ssl/misc/CA -newcert + [$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ cd /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/certs +[$OPENACS_SERVICE_NAME certs]$ perl /usr/share/ssl/misc/CA -newcert Using configuration from /usr/share/ssl/openssl.cnf Generating a 1024 bit RSA private key ...++++++ @@ -506,15 +581,15 @@ Enter PEM pass phrase: Enter a pass phrase for the CA certificate. Then, answer the rest of the questions. At the end you should see this: Certificate (and private key) is in newreq.pem -[service0 certs]$ +[$OPENACS_SERVICE_NAME certs]$ newreq.pem contains our certificate and private key. The key is protected by a passphrase, which means that we'll have to enter the pass phrase each time the server starts. This is impractical and unnecessary, so we create an unprotected version of the key. Security implication: if anyone gets access to the file keyfile.pem, they effectively own the key as much as you do. Mitigation: don't use this key/cert combo for anything besides providing ssl for the web site. [root misc]# openssl rsa -in newreq.pem -out keyfile.pem read RSA key Enter PEM pass phrase: writing RSA key -[service0 certs]$ +[$OPENACS_SERVICE_NAME certs]$ To create the certificate file, we take the combined file, copy it, and strip out the key. - [service0 certs]$ cp newreq.pem certfile.pem + [$OPENACS_SERVICE_NAME certs]$ cp newreq.pem certfile.pem [root misc]# emacs certfile.pem Strip out the section that looks like -----BEGIN RSA PRIVATE KEY----- @@ -538,46 +613,46 @@ the OpenACS tarball. - [root src]# su - service0 -[service0 service0]$ cd /var/lib/aolserver/service0 -[service0 service0]$ cp /var/lib/aolserver/service0/packages/acs-core-docs/www/files/analog.cfg.txt etc/analog.cfg -[service0 service0]$ mkdir www/log -[service0 service0]$ cp -r /usr/share/analog-5.31/images www/log/ -[service0 service0]$ -su - service0 -cd /var/lib/aolserver/service0 -cp /var/lib/aolserver/service0/packages/acs-core-docs/www/files/analog.cfg.txt etc/analog.cfg + [root src]# su - $OPENACS_SERVICE_NAME +[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ cd /var/lib/aolserver/$OPENACS_SERVICE_NAME +[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ cp /var/lib/aolserver/$OPENACS_SERVICE_NAME/packages/acs-core-docs/www/files/analog.cfg.txt etc/analog.cfg +[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ mkdir www/log +[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ cp -r /usr/share/analog-5.31/images www/log/ +[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ +su - $OPENACS_SERVICE_NAME +cd /var/lib/aolserver/$OPENACS_SERVICE_NAME +cp /var/lib/aolserver/$OPENACS_SERVICE_NAME/packages/acs-core-docs/www/files/analog.cfg.txt etc/analog.cfg mkdir www/log cp -r /usr/share/analog-5.31/images www/log/ Edit -/var/lib/aolserver/service0/etc/analog.cfg and change the variable in HOSTNAME "[my +/var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/analog.cfg and change the variable in HOSTNAME "[my organisation]" to reflect your website title. If you don't want the traffic log to be publicly visible, change -OUTFILE /var/lib/aolserver/service0/www/log/traffic.html to use a private +OUTFILE /var/lib/aolserver/$OPENACS_SERVICE_NAME/www/log/traffic.html to use a private directory. Run it. - [service0 service0]$ /usr/share/analog-5.31/analog -G -g/var/lib/aolserver/service0/etc/analog.cfg + [$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ /usr/share/analog-5.31/analog -G -g/var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/analog.cfg /usr/share/analog-5.31/analog: analog version 5.31/Unix /usr/share/analog-5.31/analog: Warning F: Failed to open DNS input file - /home/service0/dnscache: ignoring it + /home/$OPENACS_SERVICE_NAME/dnscache: ignoring it (For help on all errors and warnings, see docs/errors.html) /usr/share/analog-5.31/analog: Warning R: Turning off empty Search Word Report -[service0 service0]$ +[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ Verify that it works by browing to http://yourserver.test:8000/log/traffic.html Automate this by creating a file in /etc/cron.daily. - [service0 service0]$ exit + [$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ exit logout [root root]# emacs /etc/cron.daily/analog Put this into the file: #!/bin/sh -/usr/share/analog-5.31/analog -G -g/var/lib/aolserver/service0/etc/analog.cfg +/usr/share/analog-5.31/analog -G -g/var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/analog.cfg [root root]# chmod 755 /etc/cron.daily/analog Test it by running the script. [root root]# sh /etc/cron.daily/analog @@ -647,7 +722,7 @@ Identify a runaway Oracle query: first, use ps aux or top to get the UNIX process ID of a runaway Oracle process. Log in to SQL*Plus as the admin: - [service0 ~]$ svrmgrl + [$OPENACS_SERVICE_NAME ~]$ svrmgrl Oracle Server Manager Release 3.1.7.0.0 - Production @@ -684,8 +759,51 @@ - - ($Id$) + + Creating an appropriate tuning and monitoring environment + + The first task is to create an appropriate environment for finding out + what is going on inside Oracle. Oracle provides Statspack, a package to + monitor and save the state of the v$ performance views. These reports + help finding severe problems by exposing summary data about the Oracle + wait interface, executed queries. You'll find the installation + instructions in $ORACLE_HOME/rdbms/admin/spdoc.txt. Follow the + instructions carefully and take periodic snapshots, this way you'll be + able to look at historical performance data. + + + + Also turn on the timed_statistics in your init.ora file, so that + Statspack reports (and all other Oracle reports) are timed, which makes + them a lot more meaningful. The overhead of timing data is about 1% per + Oracle Support information. + + + + + To be able to get a overview of how Oracle executes a particular query, + install "autotrace". I usually follow the instructions here http://asktom.oracle.com/~tkyte/article1/autotrace.html. + + + Make sure, that the Oracle CBO works with adequate statistics + + + The Oracle Cost Based optimizer is a piece of software that tries to find + the "optimal" execution plan for a given SQL statement. For that it + estimates the costs of running a SQL query in a particular way (by default + up to 80.000 permutations are being tested in a Oracle 8i). To get an + adequate cost estimate, the CBO needs to have adequate statistics. For + that Oracle supplies the dbms_stats + package. + + + + + + + ($Id$)