<html>
<!--AD_DND-->
<HEAD><TITLE>ArsDigita Server Architecture Setup Instructions</TITLE></HEAD>
<BODY bgcolor=#ffffff text=#000000>
<h2>ArsDigita Server Architecture Setup Instructions</h2>
by <A href="http://teadams.com">Tracy Adams</a> and <A href="http://haverkamp.com">Greg Haverkamp</a>, a companion to
<a href="http://photo.net/wtr/arsdigita-server-architecture.html">ArsDigita Server Architecture</a>
<hr>
<ul>
<li>bug reports: <a href="mailto:acs-bugs@arsdigita.com">acs-bugs@arsdigita.com</a>
<li>philosophy:
<a href="http://photo.net/wtr/arsdigita-server-architecture.html">ArsDigita Server Architecture</a>
</ul>
This document describes the installation procedures for a server conforming to the ArsDigita Server Architecture.
<h3>Prerequisites</h3>
This document assumes you have a server preconfigured with the following:
<ul>
<li>Oracle 8 (or higher)
<li>AOLserver
<li><a href="http://www.arsdigita.com/free-tools/oracle-driver.html">ArsDigita Oracle driver for AOLserver</a>
<li>A working MTA (Mail Transfer Agent) that is willing to relay from all IP addresses on the machine.
</ul>
<h3>Checklist for a properly installed ArsDigita Server Architecture</h3>
<ul>
<li> Web site responding
<li> Nightly database exports succeeding
<li> Machine backups are being performed
<li> KeepAlive sends alerts and restarts server when the virtual server is down or can't reach the database
<li> Uptime sends alerts when outside connectivity to machine fails
<li> Staff server is properly installed
<li> Reporte is producing log reports
<li> WatchDog is monitoring server log (not necessary if you use
Cassandracle inside ACS)
<li> Cassandrix is monitoring disk space conditions
<li> Cassandracle is monitoring the Oracle server (not necessary if you
use Cassandracle inside ACS)
<li> MTA Monitor is monitoring the mail server
<li> Development server is being utilized
</ul>
<h3>Installation steps</h3>
All ArsDigita services are given a unique name. Throughout the rest of this document, we will be using <i>service_name</i> as the name of the service. You will want to replace instances of <i>service_name</i> with the real name of the service you're creating.
<ol>
<li>Decide on the <b>physical machine</b> for your web service. <i>Your team leader should know this.</i>
<li>Setup a system account for your use when working on the system.
<p>
<i>This requires root access to the machine. One team member should be in charge of root commands.</i>
<ul>
<li>Connect to the machine as <code>root</code> using <code>ssh</code>:
<blockquote><code>ssh -l root machine_name</code></blockquote>
<li>Using <code>useradd</code>, create your account with primary group <code>arsdigit</code> and secondary group <code>dba</code>
<blockquote>
<code>
<pre>
useradd -g arsdigit -G dba -m username
</pre>
</code>
<p>
(see <code>man useradd</code> for more details about this step.)
</blockquote>
<li>Select a password for your account
<blockquote>
<code>
<pre>
passwd username
</pre>
</code>
</blockquote>
</ul>
<li>Prepare Oracle for ACS
<ul>
<li>Connect to oracle using <code>svrmgrl</code> <p>
<b>Note: you must be in the <code>dba</code> group to use <code>svrmgrl</code></b> (this should have been accomplished in step 2.)
<blockquote>
<code><pre>
$ svrmgrl
Oracle Server Manager Release 3.1.5.0.0 - Production
(c) Copyright 1997, Oracle Corporation. All Rights Reserved.
Oracle8i Enterprise Edition Release 8.1.5.0.0 - Production
With the Partitioning and Java options
PL/SQL Release 8.1.5.0.0 - Production
SVRMGR> connect internal
Connected.
</pre></code>
</blockquote>
<li>Find where the tablespaces are stored
<blockquote><code><pre>
SVRMGR> select file_name from dba_data_files;
</pre></code></blockquote>
Example results:
<blockquote><code><pre>
/ora8/m01/app/oracle/oradata/ora8/system01.dbf
/ora8/m01/app/oracle/oradata/ora8/oemrep01.dbf
/ora8/m01/app/oracle/oradata/ora8/rbs01.dbf
/ora8/m01/app/oracle/oradata/ora8/temp01.dbf
/ora8/m01/app/oracle/oradata/ora8/users01.dbf
/ora8/m01/app/oracle/oradata/ora8/indx01.dbf
/ora8/m01/app/oracle/oradata/ora8/drsys01.dbf
/ora8/m02/oradata/ora8/jsc01.dbf
</pre></code></blockquote>
<li>Using the above output, you should determine where to store your tablespace. As a general rule, you'll want to store your tablespace on a mount point (the <code>m0<i>n</i></code> subdirectories in the above listing) separate from the Oracle system data files. In this example, we'll use <code>/ora8/m02/oradata/ora8/</code>.
<li> Create a tablespace for the service. It is important that
the tablespace can <code>autoextend</code>. This allows the tablespace's storage capacity to grow as the size of the data grows.
<blockquote><pre><code>
SVRMGR> create tablespace <i>service_name</i> datafile '/ora8/m02/oradata/ora8/<i>service_name</i>01.dbf' size
50m autoextend on default storeage ( pctincrease 0);
</code></pre></blockquote>
<li> Create a database user for this service. Give the user access to the tablespace and rights to connect. (Substitute <code>database_password</code> with the appropriate password.)
<blockquote><code><pre>
SVRMGR> create user <i>service_name</i> identified by <i>database_password</i> default tablespace <i>service_name</i>
temporary tablespace temp quota unlimited on <i>service_name</i>;
SVRMGR> grant connect, resource, ctxapp to <i>service_name</i>;
</code></pre></blockquote>
<li> Run "sqlplus <i>service-name</i>" to make sure your Oracle user works correctly. (Very bad things can happen to Oracle if AOLServer repeated tries to connect with a misconfigured Oracle account).
<P>
<li> <b>Transfer data:</b> If you are moving the service from one location to another, export code for the old database and import into the new.
<blockquote>
To export:�
<code><pre>
exp <i>service_name</i>/<i>database_password</i> file=foo.dmp consistent=y full=y
</pre></code>
<p>
To import:
<code><pre>
imp <i>service_name</i>/<i>database_password</i> file=foo.dmp [fromuser=olduser touser=<i>service_name</i>]
</pre></code>
</blockquote>
<li>Should it become necessary to rebuild a tablespace from scratch, you can use the <code>drop user</code> command with the <code>cascade</code> option. This command will drop the user and every database object the user owns.
<blockquote><code>drop user <i>service_name</i> cascade </code>
<p>If this does not work because svrmgrl "cannot drop a user that is currently connected", make sure to kill the AOLserver using this user. If it still does not work, do:
<p>
<code> select username, sid, serial# from v$session where username='<i>service_name</i>'; </code> <p>
and then
<p>
<code>alter system kill session '<i>sid</i>,<i>serial#</i>';</code>
<p>
where <i>sid</i> and <i>serial#</i> are replaced with the corresponding values for the open session.
<p><b>Use with caution!</b><p>
If you feel the need to delete <i>everything</i> related to the service, you can also issue the following:
<code><pre>
drop tablespace <i>service_name</i> including contents ;
</pre></code>
</ul>
</ul>
<li>Set up <b>nightly Oracle exports</b>
<p>
<ul>
<li> Find out if oracle exports are running on the machine.
<blockquote>
<code>
crontab -l | grep export-oracle
</code>
</blockquote>
You should see a line similar to the following:
<blockquote>
<code>
0 23 * * * /usr/local/sbin/export-oracle
</code>
</blockquote>
If you don't see it, you should add it:
<blockquote><code><pre>EDITOR=emacs crontab -e</pre></code></blockquote>
<li>Make sure <code>/usr/local/sbin/export-oracle</code> has the correct environmental variables.<p>
Compare the settings in <code>/usr/local/sbin/export-oracle</code> with:
<blockquote>
<code><pre>
# su -l oracle
Sun Microsystems Inc. SunOS 5.6 Generic August 1997
$ env
HOME=/home/oracle
HZ=
LD_LIBRARY_PATH=/ora8/m01/app/oracle/product/8.1.5/lib:/usr/lib:/lib:/usr/openwin/lib
LOGNAME=oracle
ORACLE_BASE=/ora8/m01/app/oracle
ORACLE_HOME=/ora8/m01/app/oracle/product/8.1.5
ORACLE_SID=ora8
ORACLE_TERM=vt100
ORAENV_ASK=NO
PATH=/ora8/m01/app/oracle/product/8.1.5/bin:/ora8/m01/app/oracle/product/8.1.5/bin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/ccs/bin:/usr/ucb:/usr/dt/bin:/usr/openwin/bin:/usr/local/bin:/usr/sbin
SHELL=/bin/sh
TERM=vt100
TZ=US/Eastern
</pre></code>
</blockquote>
<li>Choose a location for the dump files and modify <code>/usr/local/sbin/export-oracle</code> accordingly. Your complete <code>export_oracle</code> file should look something like this:
<blockquote>
<code><pre>
#!/bin/sh
HOME=/home/oracle
HZ=
LD_LIBRARY_PATH=/ora8/m01/app/oracle/product/8.1.5/lib:/usr/lib:/lib:/usr/openwin/lib
LOGNAME=oracle
ORACLE_BASE=/ora8/m01/app/oracle
ORACLE_HOME=/ora8/m01/app/oracle/product/8.1.5
ORACLE_SID=ora8
ORACLE_TERM=vt100
ORAENV_ASK=NO
PATH=/ora8/m01/app/oracle/product/8.1.5/bin:/ora8/m01/app/oracle/product/8.1.5/bin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/ccs/bin:/usr/ucb:/usr/dt/bin:/usr/openwin/bin:/usr/local/bin:/usr/sbin
SHELL=/bin/sh
TERM=vt100
TZ=US/Eastern
exportdir=/<i>export_root</i>/oracle-exports
file=$exportdir/oraexport-<i>service_name</i>.dmp
mv -f $file.gz $file.old.gz
exp <i>service_name</i>/<i>database_password</i> file=$file owner=<i>service_name</i> consistent=Y
gzip $file
</pre></code>
</blockquote>
</ul>
<li>Set up the <b>Arsdigita Community System</b> on your virtual server
<ul>
<li>Alot an IP address
<ol>
<li> Find out which of the system's IP addresses aren't being used by another service (ignoring <code>localhost</code> (<code>127.0.0.1</code>):
<blockquote><code><pre>
netstat -in | awk '{ print $4 }' | xargs -i% sh -c 'if [ `grep % /home/nsadmin/*.ini > /dev/null` $? -eq "1" ]; then echo %; fi'
</pre></code></blockquote>
<li> Select the first of the available IP addresses for your service. Once you create the appropriate .ini file with the IP address, you should notify your syadmin that you have claimed this IP.
(ArsDigitans should do this by adding a ticket to the sysadmin ticket tracker on the intranet.)
</ol>
<li>Using <a href="/install/service-name.ini">service-name.ini</a> as a template, follow the steps under <a href="#installing">Installing a virtual server</a>. Follow the <a href="installation.html">Arsdigita Community System</a> where it says to install your service.
<li>To prepare for <b>high volume</b>.
<ul>
<li> Make sure your AOLServer is configured to use all the machines processors. For multi-processor Solaris boxes, include the following block in the <code>/home/nsadmin/service-name.ini</code> file:
<blockquote>
<pre>
[ns/threads]
; use more than 1 processor (Solaris)
SystemScope=on
</pre>
</blockquote>
<p>
Note: If you have a single processor machine, <code>SystemScope=off</code> is more efficient.
<p>
<li>If you use adp pages, configure AOLServer to store parsed versions.
<blockquote>
<pre>
[ns/server/service_name/modules]
...
nscache=nscache.so
[ns/server/service_name/module/nscache]
...
CacheADP=on
</pre>
</blockquote>
<li>It is tempting to increase the <code>MaxThreads</code>
parameter to get more performance. Increasing this beyond 50
causes more thrashing or more pages waiting for database
handles. (AOL does some tuning on this parameter and generally
settles on numbers between 8 and 75).
<li>It is tempting to increase the number of database handles
in your main pool. Increasing beyond 8-12 can do more harm than
good. The Oracle client library running inside the AOLServer
process does not handle large number of connections efficiently.
(It uses locks instead of more efficient multi-threading techniques).
<li>Look for handle-hogging pages. These usually appear
frequently and with long wait times on the bottom of monitor.tcl.
<p>
On handle-hogging pages (ie, intermedia searching), you don't
want to tie up all your Oracle handles and deny service to everyone else.
On these pages, you want to:
<ul>
<li> Obtain the handle as late as possible and release the handle
as early as possible.
<li> If possible, use only 1 handle from the pool.
<li> Refrain from sending email when holding a handle.
<li> Limit the number of times this page can run simultaneously.
For example:
<blockquote>
<pre>
if ![ad_return_if_another_copy_is_running 6] {
ns_log Notice "foo.tcl out of handles"
return
}
</pre>
</blockquote>
</ul>
<li>If you have done the above and are still limited by db
handles, you a limited by the ability of Oracle's client library
to manage connections. Symptoms:
<ul>
<li> monitor.tcl shows that all db-backed pages are backing up
<li> Oracle is not overloaded. The nsd processes might be taking
a large percentage of the machine's load or the CPU usage might
not be high.
</ul>
You can still get more hits out of the machine by
<ul>
<li>Adding another AOLServer process pointed at the same page root.
<li>Configure DNS to do a round robin.
</ul>
<li>Host graphics on a separate machine.
<li>Consider caching graphics
<blockquote>
<pre>
[ns/server/service_name/modules]
...
nscache=nscache.so
[ns/server/service_name/module/nscache]
...
CacheStatic=on
</pre>
</blockquote>
<font color=red>Careful!</font>:
On a db-backed web site, you want to make sure the entire
database is stored in RAM. Caching graphics can be
counterproductive if it interferes with query performance.
<li>Use multiple machines (see <a target=other href=http://photo.net/wtr/arsdigita-server-architecture.html>ArsDigita Server Architecture</a>)
</ul>
</ul>
<li>Set up <b><a href=http://arsdigita.com/free-tools/keepalive.html>Keepalive</a></b>.
<p>
Keepalive is a virtual server located on the same physical machine as your service. It watches your service, notifies you of problems, and restarts it should it go down. All the services on the same machine share the same <code>keepalive</code>.
<p>
If keepalive does not exist on your machine, you will have to install another virtual server on your machine.
<ul>
<li>Keepalive listens to port 1997. Because this should not conflict with any other service, you may re-use an IP address used by another service.
<li>Follow the directions for <a href="#installing">Installing a virtual server</a>. Use <a href="/install/keepalive.ini">keepalive.ini</a> as a template. Since keepalive does not use Oracle, you can run <code> /home/nsadmin/bin/nsd</code> instead of <code>/home/nsadmin/bin/nsd-oracle</code>
<li> To install the service, follow the <a href=http://arsdigita.com/free-tools/keepalive.html>Keepalive installation instructions</a>.
</ul>
<p>Once the Keepalive service is running, you need to test it:
<ul>
<li> Test that keepalive is monitoring your service by moving <code>/web/service_name/SYSTEM/dbtest.tcl</code>
<blockquote><code><pre>
mv /web/<i>service_name</i>/SYSTEM/dbtest.tcl /web/<i>serice_name</i>/SYSTEM/dbtest.tcl.moved
</pre></code></blockquote>
<li> Fix <code>/web/service_name/SYSTEM/dbtest.tcl</code> by moving it back
<blockquote><code><pre>
mv /web/<i>service_name</i>/SYSTEM/dbtest.tcl.moved /web/<i>serice_name</i>/SYSTEM/dbtest.tcl
</pre></code></blockquote>
</ul>
<li> Sign up for <b><a href=http://uptime.arsdigita.com/uptime/about.tcl>Uptime</a></b><br>
If the machine on which your service runs is down, the keepalive service on your machine will be down as well. Uptime resides on a separate server and sends alerts when your server can not be reached. You should use the forms at <a href="http://uptime.arsdigita.com/uptime/">Uptime</a> to register alerts for the URL <code>http://service_domain/SYSTEM/uptime.txt</code>. You'll want alerts to be sent to the following:
<ul>
<li> All the people involved with your service
<li> noc@arsdigita.com
</ul>
You should break your montoring page to make sure Uptime sends an alert.
Then return the page to normal.
<li>Set up <a href=http://photo.net/doc/glassroom.html>Glassroom</a> to maintain the information required to run the service and the ticket tracker.
<p>
The Glassroom for the service will be on a separate machine and has a domain staff.<i>service_name</i> domain. The ticket tracker and glassroom modules should be run on this server. Glassroom is built on top of an ACS installation.
<p>
<li> Put the domain name for the main and staff servers in the nameservers. One team member should be in charge of nameserver updates.
<li>Set up <a href=http://arsdigita.com/free-tools/watchdog.html>Watchdog</a> to monitor your logs for errors.
<p>Watchdog is virtual server located on the same machine as your service. It watches your log files and will notify you of tcl errors.
<p>
If watchdog does not exist on your machine, you will have to install another virtual server on your machine:
<ul>
<li>Watchdog listens to port 1998. Because this should not conflict with any other service, you may re-use an IP address used by another service.
<li>Follow the directions for <a href="#installing">installing a virtual server</a>. Use <a href="/install/watchdog.ini">watchdog.ini</a> as a template. Since watchdog does not use Oracle, you can run <code> /home/nsadmin/bin/nsd</code> instead of <code>/home/nsadmin/bin/nsd-oracle</code>
<li> To install the service, follow the <a href="http://arsdigita.com/free-tools/watchdog.html">Watchdog installation instructions</a>
</ul>
<p>
If Watchdog is installed on your machine, add your service:
<ul>
<li> Look at <code>/home/nsadmin/watchdog.ini</code> to locate the domain and port for Watchdog on this machine
<li> Visit Watchdog to add your service
<li> Create some tcl bugs and verify that you receive notifications. You can force Watchdog to act immediate by accessing <code>test.tcl</code> under the Watchdog server.
</ul>
<li>Set up <b>Reporte</b> to generate server log reports.
<ul>
<li> If Reporte does not exist for your server, use the <a href=http://arsdigita.com/free-tools/reporte.html>Installation Guide</a>
<li> Install a new service to Report
</ul>
<li>Set up <b>Rollover</b>, which will keep the size of the error logs under control.
<ul>
<li>Follow the directions for <a href="#installing">installing a virtual server</a> to create a virtual server for Rollover.
<li>Configure <code>/web/rollover/tcl/rollover-list-of-systems.tcl</code> as appropriate for your installation.
</ul>
<li>Set up <b>Cassandrix</b> to montor the Unix machine the server sits on. <a href="mailto:rolf@arsdigita.com">Rolf</a> insists this will be done shortly.
<li>Set up <b>Cassandracle</b> to monitor the Oracle tranactions your website performs. http://arsdigita.com/free-tools/cassandracle.html
<ul>
<li>Follow the directions for <a href="#installing">installing a virtual server</a> to create a virtual server for Cassandracle.
<li>Create a <code>cassandracle</code> user.
<li>Grant the <code>cassandracle</code> user <code>connect, resource</code> and <code>dba</code> privileges.
<blockquote><code><pre>
grant connect, resource, dba to cassandracle
</pre></code></blockquote>
</ul>
<li>Set up <b>ArsDigita MTA monitor</b> - this email monitor. Branimir is working on this.
<li><b>Development server </b>
<ul>
<li> Run the development server on the same machine/ip unless there is a reason not to. Use port 8080 for http, port 8443 for https.
<li> Set up an Oracle user (if needed)
<li> Set up another virtual server. This development server should be at the same IP as the running web service.
</ul>
</ol>
<a name=installing><h3> Installing a virtual server</h3></a>
<ol>
<li>Create /home/nsadmin/<i>service_name</i>.ini owned by <code>nsadmin</code>. Template ini files are listed in the above directions. Make the following substitutions as appropriate:
<ul>
<li> If your virtual server uses the database, substitute the service's database password with <i>database_password</i>.
<li> Substitute the email of the person in charge of the server for service_email
<li> Substitute the service's IP address for <code>service_ip_address</code>
<li> Substitute the service's domain (e.g., photo.net) for <code>service_domain</code>
<li>If you do not want to use ftp on your server, delete the <code>[ns/server/service_name/module/nsftp]</code> section and the line <code>nsftp=nsftp.so</code> in the <code>[ns/server/service_name/modules]</code> section.
</ul>
<li> Prepare the <code>servers/<i>service_name</i></code>.
This directory and all the files must be owned by nsadmin.
You can do this via one of the folowing methods:
<ul>
<li> Copy another service's directory or
<li> Copy the AOLserver template files
<blockquote><pre><code>
cp -pr /home/nsadmin/modules/nssetup/template1/ /home/nsadmin/servers/<i>service_name</i>
</code></pre></blockquote>
</ol>
<i>Note: If you start with the template, the nsadmin password for the service will be blank and you will have to aquire cert.pem and key.pem from another location.</i>
<li> Install your service in <code>/web/<i>service_name</i></code>
<li> Execute <code>/home/nsadmin/bin/nsd-oracle -c /home/nsadmin/servername.ini</code> to test out your server setup. View your error log to make sure the service is starting without any problems. Attempt to connect to the service from a web browser.
<li> Log into the /NS/Admin pages. Assign an nsadmin password if it is blank.
<li>Ensure that your service automatically starts on boot (or any other time the service dies):
<p>
<b>This requires root and deals with sensitive files that can break every service on the machine.</b> One member
from a team should be in charge of this step.
<p>
<ul>
<li>Open <code>/etc/inittab </code> for editing.
<li>Copy an existing nsd line for a web service. If one doesn't exist, use this example as a template:
<blockquote><code><pre>
nss:234:respawn:/home/nsadmin/bin/nsd-oracle -i -c /home/nsadmin/<i>service_name</i>.ini
</pre></code></blockquote>
<li>Make sure the first field is unique.
<li><font color=red>Important:</font> Make sure there is a newline at the end of the file. If there is not a newline at the end of the file, the system may suffer catastrophic failures.
<li> Use <code>/sbin/init q</code> to start the server.
<li> <font color=red>Important:</font> See if it works by killing your nsd process. The process should be restarted by the system. If this doesn't work, you have probabably destroyed all the services on the machine.
</ul>
</ul>
<hr>
<a href=\"mailto:gregh@arsdigita.com\"><address>gregh@arsdigita.com</address></a>
<a href=\"mailto:teadams@mit.edu\"><address>teadams@mit.edu</address></a>
</body>
</html>