Activity OpenACS/openacs-4/packages/acs-tcl/tcl/cluster-procs.tcl

4:30 pm

gustafn

Fixed issue in cluster mode, denoted by Khy H in the OpenACS forum

A new command "ad_parameter_cache_flush_dict" was introduced to handle

the case, where a cluster node modifies a parameter value without

having it read before. This case could lead to a coherency problem for

parameter values.

Background:

This proc is necessary in cases, where a node writes a new

parameter value before it has read the old one.

Since a plain "nsv_dict unset ad_param $key $parameter_name"

raises an exception, when the pair does not exist, and we do

not want to allow in cluster requests arbitrary "catch"

commands, we allow "ad_parameter_cache_flush_dict" instead.

Probably, the best solution is to add support for

nsv_dict unset -nocomplain -- ad_param $key $parameter_nam

The existing nsv_dict was built after Tcl's "dict unset",

which does not have the "-nocomplain" option either. However,

an atomic operation would certainly be preferable over an exists/unset

pair, which is no acceptable solution.

For details, see https://openacs.org/forums/message-view?message_id=5822470

- -4
- +5
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 1 more file in changeset.

7:19 am

gustafn

Cluster updates: better support when running in containers

- generalized servers running with an INADDR_ANY address

- handling of running server with ephemeral ports

- streamline function and variable names

- added ./tcl/acs-container-procs.tcl

* make container definitions independent of cluster handling

* create container mapping based on a JSON file, which

can be obtained from docker to get information about

external addresses

* The final interface is a mapping file in Tcl providing at least

the Tcl dict ::docker::containerMapping containing the docker

mapping. A dict key like "8080/tcp" (internal port) will return a

dict containing the keys "host", "port" and "proto" (e.g. proto

https host 192.168.1.192 port 58115).

- -61
- +202
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 3 more files in changeset.

7:04 pm

gustafn

Permit "lang::message::cache" in acs::clusterwide operations

- -0
- +1
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl

8:28 pm

gustafn

Added support for automatic disconnect when a dynamic cluster node is shut down

When the dynamic cluster configuration has scale for certain

occasions, it makes sense to provide down scaling support when these

occasions are over, which does not rely on the configured cluster

disconnect timeout (ClusterAutodeleteInterval). The new code will send

automatically a disconnect request when a dynamic cluster node is

terminating gracefully.

Depending on the configuration, a new version of NaviServer will be

necessary to reliably execute disconnect requests. Appropriate changes

are in the NaviServer release/4.99 and main branches.

- Bumped version numbers:

* acs-tcl to 5.10.1b4

* acs-admin to 5.10.1b4

- -41
- +75
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 4 more files in changeset.

6:14 pm

gustafn

Adding auto-deletion of dynamic cluster nodes and small refactoring

- New kernel parameter "ClusterAutodeleteInterval" to specify, when a

dynamic cluster node is not regarded as temporarily unavailable but

as definitely gone. The default value for this parameter is 2m (2

minutes).

- small refactoring to reduce duplicated logic

- Bumped version numbers:

* acs-kernel to 5.10.1b4

* acs-tcl to 5.10.1b3

* acs-admin to 5.10.1b3

- -10
- +65
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 5 more files in changeset.

4:32 pm

gustafn

Small update for cluster support

- Improved visualization of cluster nodes that we lost contact to

- New kernel parameters to reduce hard-coded values and to make purpose more explicit

* new parameter ClusterHeartbeatInterval (default 20s),

was hardcoded before to the new default value

* renamed PreferredLocationRegexp -> ClusterPreferredLocationRegexp

* renamed EnableLoggingP -> ClusterEnableLoggingP

- Bumped version numbers:

* acs-kernel to 5.10.1b3

* acs-tcl to 5.10.1b2

* acs-admin to 5.10.1b2

acs-kernel acs-admin acs-tcl/

- -2
- +9
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 7 more files in changeset.

8:44 pm

gustafn

cluster setup: use qualified location as well for the local host

- -6
- +7
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl

4:55 pm

gustafn

added new flag to util::join_location to avoid removing the default port

this addresses the bug reported by Jonathan Kelley in the openacs.org Q&A forum

- -1
- +1
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 1 more file in changeset.

7:32 pm

gustafn

provide qualified_location for incoming join requests

Many thanks for Jonathan Kelley reporting this problem

- -1
- +1
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl

1:33 pm

antoniop

Tolerate the absence of the canonicalServer instance variable, complaining as if it was empty

- -2
- +2
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl

6:57 pm

gustafn

improve spelling

- -3
- +3
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 4 more files in changeset.

7:35 pm

gustafn

cluster reform: added admin page and minor updates

- -17
- +20
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 5 more files in changeset.

6:50 pm

gustafn

Cluster reform

- Support for dynamic cluster nodes:

Previous versions of OpenACS required to know the IP addresses of

the cluster nodes in advance, which is a show-stopper for many cloud

applications.

The new functionality allows arbitrary nodes to register as client

nodes at the canonical server, provided that these know a shared

secret. All messages of the intra-cluster talk are now

cryptographically signed using this shared secret.

In the current version, the shared secret key has to be specified in

the NaviServer configuration file (ClusterSecret). Later versions

will support the use of other measures such as generated keys, kept

as files.

Also in the new version it is still possible to use static peer

addresses (which do not have to register during startup).

- Support for multiple protocols for intra-cluster talk

Previous version of OpenACS required the communication of the

intra-cluster talk over HTTP. With these changes, multiple protocols

are supported. Most prominently HTTPS can be used, which is required

by some organization for all web communication, especially for cloud

applications. Other protocols might be added later (e.g. UDP).

- Nodes (such as the canonical server) can be specified via URL

locations. These location specifier have the following advantages

over classical IP address and port.

* support different protocols (http, https, udp)

* support of IP addresses or DNS names

* when DNS names map to multiple IP addresses

(e.g. multiple IPv4 and/or multiple IPv6 addresses)

the new parameter PreferredLocationRegexp can be use

to select the wanted. When this regexp does not match

any of the addresses, the first address is used.

* ports are now optional (except UDP)

* for location specifier: https://openacs.org

- Support for cluster communication statistics

(requires xotcl-request-monitor)

These statistics show the amount of intra-server

communication. The following statistics are collected

per node:

- number of broadcast operations

- number of send operations (active flushes)

- number of receive operation (passive flushes)

For N cluster nodes, the send operations are

basically the number of broadcast operations times

the number of nodes minus one.

These statistics can also be used prior to cluster

enabling, to obtain the number of potential broadcast

operations.

Kernel Parameters

CanonicalServer : location of the canonical server

ClusterAuthorizedIP : obsolete

ClusterEnabledP : unchanged

ClusterPeerIP : obsolete

EnableLoggingP : unchanged

PreferredLocationRegexp : new

DynamicClusterPeers : new, no not edit

Differences between old and new value for "CanonicalServer":

old:

- specify IP address with port

- example: 137.208.116.31:443

new:

- specify URL location

- support different protocols (http, https, udp)

- support of IP address or domain name

- when domain name maps to different IP addresses

(e.g. multiple IPv4 and/or multiple IPv6 addresses)

PreferredLocationRegexp can be use to select the wanted.

When the provided regexp does not match any of

the addresses, take the first address.

- port is optional

- example: https://openacs.org

Sample setup

Old:

CanonicalServer : 137.208.116.31:443

ClusterAuthorizedIP : 137.208.116.31

ClusterPeerIP : 137.208.116.31:443 137.208.116.31:8443

ClusterEnabledP : 1

EnableLoggingP : 1

New:

CanonicalServer : https://openacs.org

PreferredLocationRegexp : https://

ClusterEnabledP : 1

EnableLoggingP : 1

#

# Cluster secret for intra-cluster communications in NaviServer

# configuration file. Clustering will not be enabled if no value is

# provided.

#

ns_section ns/server/$server/acs {

# ...

ns_param ClusterSecret "please change me"

}

#-----------------------------------------------------------------------

# Support for munin statistics

#

# 1) install munin plugins for naviserver (munin-plugins-ns.git)

# 2) add link to the munin plugins (replace "openacs" by the name

# of your server in the munin configuration)

# /etc/munin/plugins/naviserver_openacs_count_cluster -> /usr/share/munin/plugins/naviserver_count

# 3) in the plugin plugin configuration (e.g., etc/munin/plugin-conf.d/naviserver)

# add a section like the following (again, replace "openacs" by the server name you used)

#

# [naviserver_openacs_count_cluster]

# env.title Cluster

# env.vars cluster:broadcast cluster:sent cluster:received

#

# 4) restart munin

- -245
- +639
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 4 more files in changeset.

3:17 pm

gustafn

make naming more consistent

- -1
- +1
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl

3:14 pm

gustafn

differentiate between sent and received intra-server messages

- -3
- +4
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl

2:02 pm

gustafn

added multiple delivery methods to intra-server talk

Here is some background information for my experiments with the delivery methods.

For this experiment, I compared 5 different means for this kind of

communications

- ns_http over HTTP (the standard setup, which is used in OpenACS 5.10)

- ns_http over HTTPS

- ns_conn over HTTP using persistent connections

- ns_conn over HTTPS using persistent connections

- ns_udp using UDP

I tested the is in 2-node cluster to make measurements simple consisting

of the canonical server and one node listening on the following protocols/ports:

- http://127.0.0.1:8101

- https://127.0.0.1:8444

- udp://127.0.0.1:8101

The first test sends per call 1000 intra-server commands from the canonical server

to the 2nd node over the various delivery methods:

set times 1000

lappend _ ns_http-[time {::acs::CS_127.0.0.1_8101 message set x ns_http} $times]

lappend _ ns_https-[time {::acs::CS_127.0.0.1_8444 message set x ns_https} $times]

lappend _ ns_connchan-http-[time {::acs::CS_127.0.0.1_8101 message -delivery connchan set x ns_http} $times]

lappend _ ns_connchan-https-[time {::acs::CS_127.0.0.1_8444 message -delivery connchan set x ns_https} $times]

lappend _ ns_udp-[time {::acs::CS_127.0.0.1_8101 message -delivery udp set x udp} $times]

join $_ \n

This leads to the following results:

ns_http 564.027083 microseconds per iteration

ns_https 1483.478916 microseconds per iteration

ns_connchan-http 147.688541 microseconds per iteration

ns_connchan-https 68.480875 microseconds per iteration

ns_udp 198.343416 microseconds per iteration

Since the commands are sent in sequence, the variant with the

persistent HTTP connection is the fastest, although this is Tcl

implemented. The slowest is the version with HTTPS via ns_http without

persistent connections. We see a factor of 20 in terms of performance.

When using ns_udp with the "-noreply" option, we have would have

a "fire and forget" solution, which might be ok when the packet loss

rate is low. That would lead to 54 microseconds.

Clearly, the numbers for persistent connections look the best, but it has

as well some disadvantages compared to other solutions:

- the server has to keep a socket open to every node (but no

connection thread)

- the keepalive setting of the server must set sufficiently long to

gain advantage of persistent connections (e.g. 5 sec keepalive,

heart beat frequency of 1s)

- Since the whole communication goes over a single connection, it is

necessary to serialize the requests to avoid that multiple

connection threads write concurrently to the same connection and

interfere with each other

- It is probably necessary to have a separate thread handling the

outgoing intra-server talk (implementing cmd queuing,

async-handling, heart-beat, etc.). Since this has to be a Tcl-thread

it will use up some memory (similar to a connection thread).

- This intra-server talk thread requires queuing and event handling we

have so far just in xotcl-core, so when implemented, it will require

the xotcl-core package (maybe this can be put later to acs-core).

As a second experiment, I've implemented a simple heart-beat service

inside the request monitor that checks the liveliness of the nodes

every second. So, in contrary to the back to back commands of the

first experiment, these are single calls. Here are some random

values for the 5 delivery methods:

[27/Dec/2022:20:29:34.171376][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_http sent total 2.907ms

[27/Dec/2022:20:29:34.182241][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x ns_https sent total 10.798ms

[27/Dec/2022:20:29:34.183475][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_connchan sent total 1.161m

[27/Dec/2022:20:29:34.183657][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x https-connchan sent total 0.086ms

[27/Dec/2022:20:29:34.188564][::throttle] Notice: -cluster: udp://127.0.0.1:8101 set x udp sent total 4.861ms

[27/Dec/2022:20:30:25.494080][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_http sent total 2.049ms

[27/Dec/2022:20:30:25.516306][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x ns_https sent total 21.903ms

[27/Dec/2022:20:30:25.517239][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_connchan sent total 0.814ms

[27/Dec/2022:20:30:25.522957][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x https-connchan sent total 0.33ms

[27/Dec/2022:20:30:25.534274][::throttle] Notice: -cluster: udp://127.0.0.1:8101 set x udp sent total 11.099ms

[27/Dec/2022:20:31:54.993455][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_http sent total 2.431ms

[27/Dec/2022:20:31:55.003036][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x ns_https sent total 9.499ms

[27/Dec/2022:20:31:55.010100][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_connchan sent total 6.981ms

[27/Dec/2022:20:31:55.010585][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x https-connchan sent total 0.322ms

[27/Dec/2022:20:31:55.017764][::throttle] Notice: -cluster: udp://127.0.0.1:8101 set x udp sent total 7.13ms

We see in essence the same pattern. The approach with the persistent

connections looks here the best as well. It is not clear to me, why

HTTPS over connchan is the best, but the communication seems ok. Maybe

some buffering/nagle algorithm is responsible for this. We see as well

that the round-trip takes typically single to double-digit

milliseconds. So when a single HTTP request to nsd triggers multiple

cache-flush operations to multiple nodes, this will take some

time. When e.g., the request issues 5 cash-flush operations, which are

sent to 5 nodes, and every request with take 1ms, the cache flushing

will make the original request about 25ms slower. This might also be

an argument for a separate thread doing these operations

asynchronously.

- -30
- +230
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 1 more file in changeset.

12:50 pm

gustafn

Added dynamic cluster configuration checking

- handle cases, where not all cluster nodes are available at startup

time (this is a problem for docker, since the IP addresses will not

be available as well)

- handle dynamic changing of configuration variables

- refactored code

- new methods "acs::Cluster register_nodes" and

"acs::Cluster refresh_blueprint" (called regulary via scheduled

procedure, running every minute)

- -0
- +294
./cluster-procs.tcl
- History
- Source
- Diff
./cluster-procs.tcl
… 2 more files in changeset.

cluster-procs.tcl

Line History

Stats

Commits this week: 0

Commit Activity

52 week commits volume

Commits by day

Commits by hour

Commit calendar

Most active committers (90 days)

gustafn

gustafn

gustafn

gustafn

gustafn

gustafn

gustafn

gustafn

gustafn

antoniop

gustafn

gustafn

gustafn

gustafn

gustafn

gustafn

gustafn