Running OpenACS in Production Environments

By Joel Aufrecht

OpenACS docs are written by the named authors, and may be edited by OpenACS documentation staff.

Maintenance tasks, optional software, and alternate configurations for AOLserver.

Starting and Stopping an OpenACS instance.

The simplest way to start and stop and OpenACS site is to run the startup shell script provided, /var/lib/aolserver/service0/etc/daemontools/run. This runs as a regular task, and logs to the logfile. To stop the site, kill the script.

A more stable way to run OpenACS is with a "keepalive" mechanism of some sort, so that whenever the server halts or is stopped for a reset, it restarts automatically. This is recommended for development and production servers.

The Reference Platform uses Daemontools to control AOLserver. A simpler method, using init, is here.

  1. Daemontools must already be installed. If not, install it.

  2. Each service controlled by daemontools must have a directory in /service. That directory must have a file called run. Daemontools then creates additional files and directories to track status and log. A daemontools directory is included in the OpenACS tarball at /var/lib/aolserver/service0/etc/daemontools. To use it, first ill any existing AOLserver instances. As root, link the daemontools directory into the /service directory. Daemontools' svscan process checks this directory every five seconds, and will quickly execute run.

    [service0 etc]$ killall nsd
    nsd: no process killed
    [service0 etc]$ exit
    
    [root root]# ln -s /var/lib/aolserver/service0/etc/daemontools/ /service/service0

    Verify that AOLserver is running.

    [root root]# ps -auxw | grep nsd
    service0   5562 14.2  6.2 22436 15952 ?       S    11:55   0:04 /usr/local/aolserver/bin/nsd -it /var/lib/aolserver/service0/etc/config.tcl -u serve
    root      5582  0.0  0.2  3276  628 pts/0    S    11:55   0:00 grep nsd
    [root root]#
  3. The user service0 can now control the service service0 with these commands:

    • svc -d /service/service0 - Bring the server down

    • svc -u /service/service0 - Start the server up and leave it in keepalive mode.

    • svc -o /service/service0 - Start the server up once. Do not restart it if it stops.

    • svc -t /service/service0 - Stop and immediately restart the server.

    • svc -k /service/service0 - Sends the server a KILL signal. This is like KILL -9. AOLserver exits immediately. If svc -t fails to fully kill AOLserver, use this option. This does not take the server out of keepalive mode, so it should still bounce back up immediately.

  4. Install a script to automate the stopping and starting of AOLserver services via daemontools. You can then restart a service via restart-aolserver service0

    [root root]# cp /var/lib/aolserver/service0/packages/acs-core-docs/www/files/restart-aolserver-daemontools.txt /usr/local/bin/restart-aolserver
    [root root]# chmod 755 /usr/local/bin/restart-aolserver
    [root root]#
  5. At this point, these commands will work only for the root user. Grant permission for the web group to use svc commands on the service0 server.

    [root root]# svgroup web /service/service0
    [root root]#
  6. Verify that the controls work. You may want to tail -f /var/lib/aolserver/service0/log/service0-error.log in another window, so you can see what happens when you type these commands.

    Most of this information comes from Tom Jackson's AOLserver+Daemontools Mini-HOWTO.

AOLserver keepalive with inittab

This is an alternative method for keeping the AOLserver process running. The recommended method is to run AOLserver supervised.

This step should be completed as root. This can break every service on your machine, so proceed with caution.

  • There are 2 general steps to getting this working.

    1. Install a script called restart-aolserver. This script doesn't actually restart AOLserver - it just kills it.

    2. Ask the OS to restart our service whenever it's not running. We do this by adding a line to /etc/inittab.

    Calling restart-aolserver kills our service. The OS notices that our service is not running, so it automatically restarts it. Thus, calling restart-aolserver effectively restarts our service.

  • Copy this file into /tmp/restart-aolserver.txt.

  • This script needs to be SUID-root, which means that the script will run as root. This is necessary to ensure that the AOLserver processes are killed regardless of who owns them. However the script should be executable by the web group to ensure that the users updating the web page can use the script, but that general system users cannot run the script. You also need to have Perl installed and also a symbolic link to it in /usr/local/bin.

    [joeuser ~]$ su - 
    Password: ***********
    [root ~]# cp /tmp/restart-aolserver.txt /usr/local/bin/restart-aolserver
    [root ~]# chown root.web /usr/local/bin/restart-aolserver
    [root ~]# chmod 4750 /usr/local/bin/restart-aolserver
    [root ~]# ln -s /usr/bin/perl /usr/local/bin/perl
    [root ~]# exit
  • Test the restart-aolserver script. We'll first kill all running servers to clean the slate. Then, we'll start one server and use restart-aolserver to kill it. If it works, then there should be no more servers running. You should see the following lines.

    [joeuser ~]$ killall nsd
    nsd: no process killed
    [joeuser ~]$ /usr/local/aolserver/bin/nsd-postgres -t ~/var/lib/aolserver/birdnotes/nsd.tcl
    [joeuser ~]$ restart-aolserver birdnotes
    Killing 23727 
    [joeuser ~]$ killall nsd
    nsd: no process killed

    The number 23727 indicates the process id(s) (PIDs) of the processes being killed. It is important that no processes are killed by the second call to killall. If there are processes being killed, it means that the script is not working.

  • Assuming that the restart-aolserver script worked, login as root and open /etc/inittab for editing.

    [joeuser ~]$ su -
    Password: ************
    [root ~]# emacs -nw /etc/inittab
  • Copy this line into the bottom of the file as a template, making sure that the first field nss1 is unique.

    nss1:345:respawn:/usr/local/aolserver/bin/nsd-postgres -i -u nobody -g web -t /home/joeuser/var/lib/aolserver/birdnotes/nsd.tcl
  • Important: Make sure there is a newline at the end of the file. If there is not a newline at the end of the file, the system may suffer catastrophic failures.

  • Still as root, enter the following command to re-initialize /etc/inittab.

    [root ~]# killall nsd    
    nsd: no process killed
    [root ~]# /sbin/init q
  • See if it worked by running the restart-aolserver script again.

    [root ~]# restart-aolserver birdnotes
    Killing 23750

If processes were killed, congratulations, your server is now automated for startup and shutdown.

Running AOLserver on Port 80

If you want your webserver to be http://yourserver.com, it must run on port 80, the default HTTP port. You set this in the config.tcl file. You will need to start the service as root. If you follow the instructions above for automating startup, this will be taken care of, but if you ever start the server from the command line, be sure to su - first.

Port 80 is a privileged port. Only certain users can claim it. When you start nsd as root, it obtains the port, and then changes to run as whatever user you specify in the server configuration file. This ensures a high level of security, as the server, once started, is not running as root. This mean that if someone was able to exploit your web server to execute a command on your server, they would not be able to gain root access.

Running multiple services on one machine

Services on different ports. To run a different service on another port but the same ip, simply repeat Install OpenACS replacing service0, and change the

set httpport              8000
set httpsport             8443 

to different values.

Services on different host names. For example, suppose you want to support http://foo.com and http://bar.com on the same machine. The easiest way is to assign each one a different ip address. Then you can install two services as above, but with different values for

set hostname               [ns_info hostname]
set address                127.0.0.1 

If you want to install two services with different host names sharing the same ip, you'll need nsvhr to redirect requests based on the contents of the tcp headers. See AOLserver Virtual Hosting with TCP by markd.

Installing SSL Support

nsopenssl is an open-sounce module for AOLserver which adds support for the ssl encryption layer. To use it, you must install the software, create or purchase certificates, and configure your OpenACS instance to use it.

  1. Uncomment this line from config.tcl.

    #ns_param   nsopenssl       ${bindir}/nsopenssl.so
    
  2. Prepare a certificate directory for the service.

    [service0 etc]$ mkdir /var/lib/aolserver/service0/etc/certs
    [service0 etc]$ chmod 700 /var/lib/aolserver/service0/etc/certs
    [service0 etc]$ 
    mkdir /var/lib/aolserver/service0/etc/certs
    chmod 700 /var/lib/aolserver/service0/etc/certs
  3. It takes two files to support an SSL connection. The certificate is the public half of the key pair - the server sends the certificate to browser requesting ssl. The key is the private half of the key pair. In addition, the certificate must be signed by Certificate Authority or browsers will protest. Each web browser ships with a built-in list of acceptable Certificate Authorities (CAs) and their keys. Only a site certificate signed by a known and approved CA will work smoothly. Any other certificate will cause browsers to produce some messages or block the site. Unfortunately, getting a site certificate signed by a CA costs money. In this section, we'll generate an unsigned certificate which will work in most browsers, albeit with pop-up messages.

    Use an OpenSSL perl script to generate a certificate and key.

    [service0 service0]$ cd /var/lib/aolserver/service0/etc/certs
    [service0 certs]$ perl /usr/share/ssl/misc/CA -newcert
    Using configuration from /usr/share/ssl/openssl.cnf
    Generating a 1024 bit RSA private key
    ...++++++
    .......++++++
    writing new private key to 'newreq.pem'
    Enter PEM pass phrase:

    Enter a pass phrase for the CA certificate. Then, answer the rest of the questions. At the end you should see this:

    Certificate (and private key) is in newreq.pem
    [service0 certs]$

    newreq.pem contains our certificate and private key. The key is protected by a passphrase, which means that we'll have to enter the pass phrase each time the server starts. This is impractical and unnecessary, so we create an unprotected version of the key. Security implication: if anyone gets access to the file keyfile.pem, they effectively own the key as much as you do. Mitigation: don't use this key/cert combo for anything besides providing ssl for the web site.

    [root misc]# openssl rsa -in newreq.pem -out keyfile.pem
    read RSA key
    Enter PEM pass phrase:
    writing RSA key
    [service0 certs]$ 

    To create the certificate file, we take the combined file, copy it, and strip out the key.

    [service0 certs]$ cp newreq.pem certfile.pem
    [root misc]# emacs certfile.pem

    Strip out the section that looks like

    -----BEGIN RSA PRIVATE KEY-----
    Proc-Type: 4,ENCRYPTED
    DEK-Info: DES-EDE3-CBC,F3EDE7CA1B404997
    S/Sd2MYA0JVmQuIt5bYowXR1KYKDka1d3DUgtoVTiFepIRUrMkZlCli08mWVjE6T
    (11 lines omitted)
    1MU24SHLgdTfDJprEdxZOnxajnbxL420xNVc5RRXlJA8Xxhx/HBKTw==
    -----END RSA PRIVATE KEY-----

Set up Log Analysis Reports - OPTIONAL

Analog is a program with processes webserver access logs, performs DNS lookup, and outputs HTML reports. Analog should already be installed. A modified configuration file is included in the OpenACS tarball.

  1. [root src]# su - service0
    [service0 service0]$ cd /var/lib/aolserver/service0
    [service0 service0]$ cp /var/lib/aolserver/service0/packages/acs-core-docs/www/files/analog.cfg.txt etc/analog.cfg
    [service0 service0]$ mkdir www/log
    [service0 service0]$ cp -r /usr/share/analog-5.31/images www/log/
    [service0 service0]$ 
    su - service0
    cd /var/lib/aolserver/service0
    cp /var/lib/aolserver/service0/packages/acs-core-docs/www/files/analog.cfg.txt etc/analog.cfg
    mkdir www/log
    cp -r /usr/share/analog-5.31/images www/log/

    Edit /var/lib/aolserver/service0/etc/analog.cfg and change the variable in HOSTNAME "[my organisation]" to reflect your website title. If you don't want the traffic log to be publicly visible, change OUTFILE /var/lib/aolserver/service0/www/log/traffic.html to use a private directory.

  2. Run it.

    [service0 service0]$ /usr/share/analog-5.31/analog -G -g/var/lib/aolserver/service0/etc/analog.cfg
    /usr/share/analog-5.31/analog: analog version 5.31/Unix
    /usr/share/analog-5.31/analog: Warning F: Failed to open DNS input file
      /home/service0/dnscache: ignoring it
      (For help on all errors and warnings, see docs/errors.html)
    /usr/share/analog-5.31/analog: Warning R: Turning off empty Search Word Report
    [service0 service0]$

    Verify that it works by browing to http://yourserver.test:8000/log/traffic.html

  3. Automate this by creating a file in /etc/cron.daily.

    [service0 service0]$ exit
    logout
    
    [root root]# emacs /etc/cron.daily/analog

    Put this into the file:

    #!/bin/sh
    
    /usr/share/analog-5.31/analog -G -g/var/lib/aolserver/service0/etc/analog.cfg
    [root root]# chmod 755 /etc/cron.daily/analog

    Test it by running the script.

    [root root]# sh /etc/cron.daily/analog

    Browse to http://yourserver.test/log/traffic.html

External uptime validation

The OpenACS uptime site can monitor your site and send you an email whenever your site fails to respond. If you test the url http://yourserver.test/SYSTEM/dbtest.tcl, you should get back the string success.

($Id: maintenance-web.html,v 1.17 2004/02/04 16:47:33 joela Exp $)
View comments on this page at openacs.org