<html>
<!--AD_DND-->
<head>
<title>Server Clustering</title>
</head>
<body bgcolor=white>
<h2>Server Clustering</h2>
part of the <a href="">ArsDigita Community System</a> by by <a href="jsalz@mit.edu">Jon Salz</a>
<hr>
<ul>
<li>Tcl: /tcl/ad-server-cluster.tcl
</ul>
<h2>The Problem</h2>
Many heavily-hit sites sit behind load balancers, which means that requests to a particular
site can be handled by one of several machine conspiring to appear as a single server.
For instance, requests to <tt>www.foobar.com</tt> might be routed to either <tt>www1.foobar.com</tt>,
<tt>www2.foobar.com</tt>, or <tt>www3.foobar.com</tt>,
three physically separate servers which share an Oracle tablespace (and
hence all the data in ACS).
<p>Many database queries are memoized in individual servers' local memory
(using the <tt><a href="/doc/proc-one.tcl?proc_name=util_memoize">util_memoize</a></tt> procedures) to minimize fetches from the database.
When a server updates an item in the database, the
old item needs to be removed from the server's local cache (using <tt><a href="/doc/proc-one.tcl?proc_name=util_memoize_flush">util_memoize_flush</a></tt>)
to force a database query the next time this item is accessed. But what happens when:
<ul>
<li><tt>www1.foobar.com</tt> does <tt><a href="/doc/proc-one.tcl?proc_name=util_memoize">util_memoize</a> "get_greeble_info 43"</tt> (incurring an actual
database lookup, <tt>SELECT * FROM greeble WHERE greeble_id = 43</tt>, and caching the result)
<li><tt>www2.foobar.com</tt> does <tt><a href="/doc/proc-one.tcl?proc_name=util_memoize">util_memoize</a> "get_greeble_info 43"</tt> (incurring a
database lookup and caching the result)
<li><tt>www1.foobar.com</tt> <tt>UPDATE</tt>s the info for greeble #43 and does
<tt><a href="/doc/proc-one.tcl?proc_name=util_memoize_flush">util_memoize_flush</a> "get_greeble_info 43"</tt>
<li><tt>www2.foobar.com</tt> does <tt><a href="/doc/proc-one.tcl?proc_name=util_memoize">util_memoize</a> "get_greeble_info 43"</tt> (returned a cached
value). The old info for greeble #43 hasn't been flushed from its local cache, so the result
is outdated!
</ul>
In general, if any of several servers can
update an item, the old version of the item can remain in other servers' local caches.
<a href="http://www.mshonline.com/how/homer/doh.wav">Doh!</a>
<h2>The Solution</h2>
We introduce the concept of a <i>server cluster</i>, a group of look-alike servers sharing an Oracle tablespace.
To set up a cluster, add the following to the ACS <tt>parameters/yourservername.ini</tt> file on each
of the servers in the cluster:
<blockquote><pre>; address information for a cluster of load-balanced servers (to enable
; distributed util_memoize_flushing, for instance). One entry per
; server; this machine's IP may be included as well
[ns/server/click/acs/server-cluster]
; 192.168.16.1 is www1.foobar.com
ClusterMachine=192.168.16.1
; 192.168.16.2 is www2.foobar.com
ClusterMachine=192.168.16.2
; 192.168.16.3 is www3.foobar.com
ClusterMachine=192.168.16.3</pre></blockquote>
Now when a server (say, <tt>www1.foobar.com</tt>) invokes
<tt><a href="/doc/proc-one.tcl?proc_name=util_memoize_flush">util_memoize_flush</a></tt> or <tt><a href="/doc/proc-one.tcl?proc_name=util_memoize_seed">util_memoize_seed</a></tt>, those routines use
<tt><a href="/doc/proc-one.tcl?proc_name=server_cluster_httpget_from_peers">server_cluster_httpget_from_peers</a></tt>
to issue an HTTP GET request to all machines in the cluster (omitting the local server):
<ul>
<li><tt>GET http://www2.foobar.com/SYSTEM/flush-memoized-statement.tcl?statement=<i>tcl-statement</i></tt>
<li><tt>GET http://www3.foobar.com/SYSTEM/flush-memoized-statement.tcl?statement=<i>tcl-statement</i></tt>
</ul>
causing the other machines (<tt>www2.foobar.com</tt> and <tt>www3.foobar.com</tt>) to flush the Tcl statement
from their local caches. This is transparent and works with all existing code.
<p>So don't think about it - just set up the <tt>server-cluster</tt> block in your <tt>yourservername.ini</tt> file,
and <tt>util_memoize</tt> and friends will be <a href="http://www.mshonline.com/how/homer/woohoo.wav">happy</a>.
<hr>
<a href="mailto:jsalz@mit.edu"><address>jsalz@mit.edu</address></a>
</body>
</html>