Index: openacs-4/packages/acs-mail-lite/acs-mail-lite.info =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/acs-mail-lite.info,v diff -u -N -r1.54 -r1.55 --- openacs-4/packages/acs-mail-lite/acs-mail-lite.info 7 Aug 2017 23:47:57 -0000 1.54 +++ openacs-4/packages/acs-mail-lite/acs-mail-lite.info 17 Feb 2018 17:08:30 -0000 1.55 @@ -9,19 +9,19 @@ f t - - Malte Sussdorff + Timo Hentschel + Malte Sussdorff Simplified reliable email transmission with bounce management. - 2017-08-06 + 2017-10-19 OpenACS - This package provides a service for sending messages, queueing messages in the database to ensure reliable sending and make sending a message 'transactional'. Replacement for acs-mail. + This package provides a service for sending and receiving messages, messages are queued in the database to ensure reliable sending and make sending a message 'transactional'. Replacement for acs-mail. 3 - - - - + + + + @@ -31,27 +31,38 @@ - + - + +Default for the empty value of this parameter will be to allow only files into the temporary folder of the system, as in '[ns_tmpdir]'" section_name="email"/> - + - + + + + + + + + + + + + + - Index: openacs-4/packages/acs-mail-lite/sql/postgresql/acs-mail-lite-create.sql =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/sql/postgresql/acs-mail-lite-create.sql,v diff -u -N -r1.13 -r1.14 --- openacs-4/packages/acs-mail-lite/sql/postgresql/acs-mail-lite-create.sql 27 Oct 2014 16:39:40 -0000 1.13 +++ openacs-4/packages/acs-mail-lite/sql/postgresql/acs-mail-lite-create.sql 17 Feb 2018 17:08:31 -0000 1.14 @@ -65,3 +65,360 @@ notification_time timestamptz default current_timestamp, notification_count integer default 0 ); + +-- +-- Add Incoming Mail Processing +-- +create sequence acs_mail_lite_in_id_seq; + +-- New tables + +-- table tracking incoming email +create table acs_mail_lite_from_external ( + aml_email_id integer primary key + not null + DEFAULT nextval ('acs_mail_lite_id_seq'), + -- Priority for processing incoming email in queue. + -- Lower number processed first. + priority integer, + -- using varchar instead of text for indexing + -- to and from email are defined according to headers. + -- See table acs_mail_lite_ie_headers + to_email_addrs varchar(1000), + from_email_addrs text, + subject text, + -- see acs_mail_lite_send_msg_id_map.msg_id + msg_id bigint, + -- used by prioritization calculations + -- For IMAP4 this is size defined by rfc822 + size_chars numeric, + -- time email received from server in seconds since tcl epoch + received_cs bigint, + -- Answers question: + -- Has all ACS Mail Lite processes finished for this email? + -- Processes like parsing email, bounced email, input validation + processed_p boolean, + -- Answers question: + -- Have all callbacks related to this email finished processing? + -- Upon release, delete all components of aml_email_id also from + -- tables acs_mail_lite_ie_headers, acs_mail_lite_ie_body_parts, and + -- acs_mail_lite_ie_files. + -- Release essentially means its available to be deleted. + release_p boolean +); + +create index acs_mail_lite_from_external_aml_email_id_idx + on acs_mail_lite_from_external (aml_email_id); +create index acs_mail_lite_from_external_processed_p_idx + on acs_mail_lite_from_external (processed_p); +create index acs_mail_lite_from_external_release_p_idx + on acs_mail_lite_from_external (release_p); + + + +-- Some services are offered between sessions of importing incoming email. +-- A unique ID provided by +-- acs_mail_lite_email_uid_id_map.uid_ext +-- is designed to +-- support UIDs for each email that are consistent between import sessions +-- from external source, such as specified by IMAP4 rfc3501 +-- https://tools.ietf.org/html/rfc3501 +-- It is also expected that each mailbox.host, mailbox and user are +-- consistent for duration of the service. +-- And yet, experience knows that sometimes email servers change +-- and UIDs for an email change with it. +-- Users switching email servers of an email account using a IMAP4 client +-- might hassle with moving email, but +-- in the process they generally know what is happening. They don't re-read +-- all the email. +-- We want to avoid this server re-reading and processing email +-- that has already been processed, when the UID of emails change. +-- The Questions become: + +-- What scenarios might we run into? +-- Another user reseting flags. +-- A server migration or restore with some conflicting UIDs. + +-- Can we recognize a change in server? +-- If so, can we signal ACS Mail Lite to ignore existing email +-- in a new environment? +-- Also, we should have a manual override to not ignore or ignore +-- in case of false positive and false negative triggers. + +-- Can we recognize if another user accesses the same email account +-- and arbitrarily selects some prior messages to unread? +-- Yes. The purpose of acs_mail_lite_email_uid_id_map is to act as a log +-- of prior processed messages. +-- If total new messages is a significant percentage of all messages +-- and service has been working for a week or more, +-- use statistics to generate a reasonable trigger. +-- Weekly produce a revised distribution curve of weekly counts of incoming. +-- If percentile is over 200%.. ie twice the weekly maximum.. +-- Also, test, for example, if new message count is less than +-- prior total, there's more of a chance that they are new messages; +-- Maybe check for one or two duplicates. +-- If new message count is over the total prior messsage count, flag a problem. +-- rfc3501 2.3.1.1. ..A client can only assume.. at the time +-- that it obtains the next unique identifier value.. that +-- messages arriving after that time will have a UID greater +-- than or equal to that value... + + +-- Can we recognize a change in server? +-- rfc3501 does not specify a unqiue server id +-- It specifies a unique id for a mailbox: UIDVALIDITY +-- UIDVALIDITY is optional, quite useful. +-- Rfc3501 specifies a unique id for each email: UID. +-- We can assign each email a more unique reference: +-- mailbox.host + mailbox.name + UIDVALIDITY (of mailbox) + UID. +-- We are more specific so that we detect more subtle cases of +-- server change, where checks by UID and UIDVALIDITY may not. + + +-- For example, when migrating email service and +-- and the new system initially restores the UIVALIDITY and message UID, +-- but references a different state of each email. The cause +-- of such cases are reasonable. For example, restoring +-- from backup to a new email host or restoring +-- before some batch event changed a bunch of things. So, +-- src_ext = mailbox.host + (user?) + mailbox.name + UIDVALIDITY +-- Leave user out for now.. +-- Priority is to have a robust way to ignore +-- prior messages recognized as 'new' messages. + +create table acs_mail_lite_email_uid_id_map ( + -- unqique internal inbound email id + -- src_ext_id identifies source, but is redundant + -- for identifying a unique email. + aml_email_id integer not null, + --uisng varchar instead of text for indexing purposes + -- Each UID externally defined such as from imap4 server + uid_ext varchar(3000) not null, + -- Each external source may apply a different uid. + -- This is essentialy an arbitrary constant frame reference between + -- connecting sessions with external server in most scenarios. + -- For IMAP4v1 rfc3501 2.3.1.1. item 4 ..combination of + -- mailbox.name, UIDVALIDITY, and UID must refer to a single + -- immutable message on that server forever. + -- default is: + -- ExternalSource parameter mailbox.name + -- and UIDVALIDITY with dash as delimiter + -- where ExternalSource parameter is + -- either blank or maybe mailbox.host for example. + -- external source reference id + -- see acs_mail_lite_email_src_ext_id_map.aml_src_id + src_ext_id integer +); + +create index acs_mail_lite_email_uid_id_map_uid_ext_idx + on acs_mail_lite_email_uid_id_map (uid_ext); +create index acs_mail_lite_email_uid_id_map_src_ext_id_idx + on acs_mail_lite_email_uid_id_map (src_ext_id); + +create table acs_mail_lite_email_src_ext_id_map ( + aml_src_id integer not null, + src_ext varchar(1000) not null +); + + + +-- Packages that are services, such as ACS Mail Lite, do not have a web UI. +-- Scheduled procs cannot read changes in values of package parameters +-- or get updates via web UI connections, or changes in tcl via apm. +-- Choices are updates via nsv variables and database value updates. +-- Choices via database have persistence across server restarts. +-- Defaults are set in acs_mail_lite::sched_parameters +-- These all are used in context of processing incoming email +-- unless stated otherwise. +-- Most specific flag takes precedence. +-- If an email is flagged high priority by package_id and +-- low priority by subject glob. It is assigned low priority. +-- Order of specificity. +-- medium default package_id party_id subject_id object_id +-- party_id can be group_id or user_id +-- If fast and low flag the same specificity for an email, low is chosen. +create table acs_mail_lite_ui ( + -- scan_replies_est_dur_per_cycle_s_override see www/doc/analysis-notes + sredpcs_override integer, + -- Answers question: Reprocess old email? + reprocess_old_p boolean, + -- Max number of concurrent threads for high priority processing + max_concurrent integer, + -- Any incoming email body part over this size is stored in file + -- instead of database. + max_blob_chars integer, + -- Minimum threashold for default medium (standard) priority + mpri_min integer, + -- Maximum value for default medium (standard) priority + mpri_max integer, + --space delimited list of package_ids to process at fast/high priority + hpri_package_ids text, + --space delimited list of package_ids to process at low priority + lpri_package_ids text, + --space delimited list of party_ids to process at fast/high priority + hpri_party_ids text, + --space delimited list of party_ids to process at low priority + lpri_party_ids text, + -- a glob for searching subjects to flag for fast/high priority + hpri_subject_glob text, + -- a glob for searching subjects to flag for low priority + lpri_subject_glob text, + --space delimited list of object_ids to process at fast/high priority + hpri_object_ids text, + --space delimited list of object_ids to process at low priority + lpri_object_ids text, + --filters to reject input as early as possible in processing inbound + --Each filter is a name value in standard tcl list format + --where name is a header name + reject_on_hit text, + reject_on_miss text +); + + +-- This table has similar requirements to acs_mail_lite_ui +-- proc acs_mail_lite_imap_conn_* needs to be able to update values +-- within scheduled procs without restarting server. +-- Port is ignored. Added because it is a common requirement of connections +-- that might one day be useful here, too. +create table acs_mail_lite_imap_conn ( + -- mailbox.host + ho text, + -- you guessed it + pa text, + -- port + po integer, + --timeout + ti integer, + -- user + us text, + -- mailbox.name See nsimap documentation for definition + na text, + -- space separated list of flags for imap related modifications + -- ssl means connect via ssl. + -- novalidatecert means accept a self-signed certificate + fl text +); + + +-- Following tables store parsed incoming email for processing by callbacks +-- defined in the rest of OpenACS + +-- incoming email headers +-- There should be a size limit per unit time from each source +-- to prevent DDOS attacks and such (at least to the imap system). +-- +create table acs_mail_lite_ie_headers ( + -- incoming email + -- only includes headers useful in processing the queue + -- Such as + -- size + -- from + aml_email_id integer, + -- header name, one header per row + -- For all headers together, see acs_mail_lite_ie_parts.c_type=headers + -- Special case: h_name = struct means + -- h_value contains entire value returned from ns_imap struct + -- as a tcl list + h_name text, + h_value text +); + +create index acs_mail_lite_ie_headers_aml_email_id_idx + on acs_mail_lite_ie_headers (aml_email_id); + +-- incoming email body parts +-- including email file attachments and file content +-- A part may be a filename. The filename data model is added +-- to the parts table to reduce code complexity. +-- An attached or inline file is a kind of part. +create table acs_mail_lite_ie_parts ( + aml_email_id integer, + section_id integer, + + -- In addition to content_type, there is a special case: + -- headers, which contains all headers for email + -- content_type = c_type + c_type text, + -- If type has a filename, this is original filename. + filename text, + -- If c_type is multipart, content is blank. part_id is branched. + content text, + -- An alternate filepathname for large blob, or + -- A local absolute filepath location + c_filepathname text +); + +create index acs_mail_lite_ie_parts_aml_email_id_idx + on acs_mail_lite_ie_parts (aml_email_id); + + +-- incoming email parts, name value pairs of +create table acs_mail_lite_ie_part_nv_pairs ( + aml_email_id integer, + -- Usage is same as acs_mail_lite_ie_parts.section_id + section_id integer, + -- name value pair + p_name text, + p_value text +); + +create index acs_mail_lite_ie_part_nv_pairs_aml_email_id_idx + on acs_mail_lite_ie_part_nv_pairs (aml_email_id); + +create table acs_mail_lite_ie_section_ref_map ( + -- 'Section' refers to usage with 'part' reference in 'ns_imap body' + -- Email parts can contain multiple parts. + -- Each multiple part can contain multiple parts. + + -- Section_ref is an absolute reference of a part + -- including the parts it is contained in, and + -- delimited by period. + -- It is defined by: + -- ns_imap body #s msgno part + -- And yet, this reference system holds for any email + -- storage, so is adopted for generic use as well. + + -- Default reference is value of 1. + -- A two part message has values 1 and 2. + -- Part 2 of a 3 part email (2/3) has reference '2' + -- If part 2 is also multiple parts, then + -- part 1 of part 2 of email has reference '2.1' and so on. + + -- Mapping is constant for each case. + -- For example, '1.2.2.1' will always point to the same integer. + -- So do not alter values as they are likely used by + -- multiple emails. + + + section_ref varchar(300), + section_id integer +); + +create index acs_mail_lite_ie_section_ref_map_section_ref_idx + on acs_mail_lite_ie_section_ref_map (section_ref); +create index acs_mail_lite_ie_section_ref_map_section_id_idx + on acs_mail_lite_ie_section_ref_map (section_id); + +-- +create table acs_mail_lite_send_msg_id_map ( + -- a randomized number unique to this table + -- unique not null + msg_id text primary key, + package_id integer + constraint aml_package_id_fk + references apm_packages, + party_id integer + constraint aml_from_external_party_id_fk + references parties (party_id), + object_id integer + constraint aml_from_external_obect_id_fk + references acs_objects (object_id), + -- Indicate approximate time when this email is created + datetime_cs integer, + -- other data or parameters to associate with email + other text +); + +create index acs_mail_lite_send_msg_id_map_msg_id_idx + on acs_mail_lite_send_msg_id_map (msg_id); + Index: openacs-4/packages/acs-mail-lite/sql/postgresql/acs-mail-lite-drop.sql =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/sql/postgresql/acs-mail-lite-drop.sql,v diff -u -N -r1.6 -r1.7 --- openacs-4/packages/acs-mail-lite/sql/postgresql/acs-mail-lite-drop.sql 18 Mar 2009 22:41:15 -0000 1.6 +++ openacs-4/packages/acs-mail-lite/sql/postgresql/acs-mail-lite-drop.sql 17 Feb 2018 17:08:31 -0000 1.7 @@ -6,7 +6,47 @@ -- drop table acs_mail_lite_queue; -drop sequence acs_mail_lite_id_seq; + drop table acs_mail_lite_mail_log; drop table acs_mail_lite_bounce; drop table acs_mail_lite_bounce_notif; + +-- inbound email data model +drop index acs_mail_lite_send_msg_id_map_msg_id_idx; +drop table acs_mail_lite_send_msg_id_map; + +drop index acs_mail_lite_ie_part_nv_pairs_aml_email_id_idx; +drop table acs_mail_lite_ie_part_nv_pairs; + +drop index acs_mail_lite_ie_section_ref_map_section_id_idx; +drop index acs_mail_lite_ie_section_ref_map_section_ref_idx; +drop table acs_mail_lite_ie_section_ref_map; + +drop index acs_mail_lite_ie_parts_aml_email_id_idx; +drop table acs_mail_lite_ie_parts; + +drop index acs_mail_lite_ie_headers_aml_email_id_idx; +drop table acs_mail_lite_ie_headers; + +drop table acs_mail_lite_ui; + +drop table acs_mail_lite_imap_conn; + +drop table acs_mail_lite_email_src_ext_id_map; + +drop index acs_mail_lite_email_uid_id_map_uid_ext_idx; +drop index acs_mail_lite_email_uid_id_map_src_ext_id_idx; + +drop table acs_mail_lite_email_uid_id_map; + +drop index acs_mail_lite_from_external_aml_email_id_idx; +drop index acs_mail_lite_from_external_processed_p_idx; +drop index acs_mail_lite_from_external_release_p_idx; + + + +drop table acs_mail_lite_from_external; + + +drop sequence acs_mail_lite_id_seq; +drop sequence acs_mail_lite_in_id_seq; Index: openacs-4/packages/acs-mail-lite/sql/postgresql/upgrade/upgrade-5.10.0d1-5.10.0d2.sql =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/sql/postgresql/upgrade/upgrade-5.10.0d1-5.10.0d2.sql,v diff -u -N --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ openacs-4/packages/acs-mail-lite/sql/postgresql/upgrade/upgrade-5.10.0d1-5.10.0d2.sql 17 Feb 2018 17:08:31 -0000 1.1 @@ -0,0 +1,356 @@ +-- acs-mail-lite/sql/postgresql/upgrade/upgrade-5.9.1b3-5.9.1b4.sql +-- +-- Add Incoming Mail Processing +-- +create sequence acs_mail_lite_in_id_seq; + +-- New tables + +-- table tracking incoming email +create table acs_mail_lite_from_external ( + aml_email_id integer primary key + not null + DEFAULT nextval ('acs_mail_lite_id_seq'), + -- Priority for processing incoming email in queue. + -- Lower number processed first. + priority integer, + -- using varchar instead of text for indexing + -- to and from email are defined according to headers. + -- See table acs_mail_lite_ie_headers + to_email_addrs varchar(1000), + from_email_addrs text, + subject text, + -- see acs_mail_lite_send_msg_id_map.msg_id + msg_id bigint, + -- used by prioritization calculations + -- For IMAP4 this is size defined by rfc822 + size_chars numeric, + -- time email received from server in seconds since tcl epoch + received_cs bigint, + -- Answers question: + -- Has all ACS Mail Lite processes finished for this email? + -- Processes like parsing email, bounced email, input validation + processed_p boolean, + -- Answers question: + -- Have all callbacks related to this email finished processing? + -- Upon release, delete all components of aml_email_id also from + -- tables acs_mail_lite_ie_headers, acs_mail_lite_ie_body_parts, and + -- acs_mail_lite_ie_files. + -- Release essentially means its available to be deleted. + release_p boolean +); + +create index acs_mail_lite_from_external_aml_email_id_idx + on acs_mail_lite_from_external (aml_email_id); +create index acs_mail_lite_from_external_processed_p_idx + on acs_mail_lite_from_external (processed_p); +create index acs_mail_lite_from_external_release_p_idx + on acs_mail_lite_from_external (release_p); + + + +-- Some services are offered between sessions of importing incoming email. +-- A unique ID provided by +-- acs_mail_lite_email_uid_id_map.uid_ext +-- is designed to +-- support UIDs for each email that are consistent between import sessions +-- from external source, such as specified by IMAP4 rfc3501 +-- https://tools.ietf.org/html/rfc3501 +-- It is also expected that each mailbox.host, mailbox and user are +-- consistent for duration of the service. +-- And yet, experience knows that sometimes email servers change +-- and UIDs for an email change with it. +-- Users switching email servers of an email account using a IMAP4 client +-- might hassle with moving email, but +-- in the process they generally know what is happening. They don't re-read +-- all the email. +-- We want to avoid this server re-reading and processing email +-- that has already been processed, when the UID of emails change. +-- The Questions become: + +-- What scenarios might we run into? +-- Another user reseting flags. +-- A server migration or restore with some conflicting UIDs. + +-- Can we recognize a change in server? +-- If so, can we signal ACS Mail Lite to ignore existing email +-- in a new environment? +-- Also, we should have a manual override to not ignore or ignore +-- in case of false positive and false negative triggers. + +-- Can we recognize if another user accesses the same email account +-- and arbitrarily selects some prior messages to unread? +-- Yes. The purpose of acs_mail_lite_email_uid_id_map is to act as a log +-- of prior processed messages. +-- If total new messages is a significant percentage of all messages +-- and service has been working for a week or more, +-- use statistics to generate a reasonable trigger. +-- Weekly produce a revised distribution curve of weekly counts of incoming. +-- If percentile is over 200%.. ie twice the weekly maximum.. +-- Also, test, for example, if new message count is less than +-- prior total, there's more of a chance that they are new messages; +-- Maybe check for one or two duplicates. +-- If new message count is over the total prior messsage count, flag a problem. +-- rfc3501 2.3.1.1. ..A client can only assume.. at the time +-- that it obtains the next unique identifier value.. that +-- messages arriving after that time will have a UID greater +-- than or equal to that value... + + +-- Can we recognize a change in server? +-- rfc3501 does not specify a unqiue server id +-- It specifies a unique id for a mailbox: UIDVALIDITY +-- UIDVALIDITY is optional, quite useful. +-- Rfc3501 specifies a unique id for each email: UID. +-- We can assign each email a more unique reference: +-- mailbox.host + mailbox.name + UIDVALIDITY (of mailbox) + UID. +-- We are more specific so that we detect more subtle cases of +-- server change, where checks by UID and UIDVALIDITY may not. + + +-- For example, when migrating email service and +-- and the new system initially restores the UIVALIDITY and message UID, +-- but references a different state of each email. The cause +-- of such cases are reasonable. For example, restoring +-- from backup to a new email host or restoring +-- before some batch event changed a bunch of things. So, +-- src_ext = mailbox.host + (user?) + mailbox.name + UIDVALIDITY +-- Leave user out for now.. +-- Priority is to have a robust way to ignore +-- prior messages recognized as 'new' messages. + +create table acs_mail_lite_email_uid_id_map ( + -- unqique internal inbound email id + -- src_ext_id identifies source, but is redundant + -- for identifying a unique email. + aml_email_id integer not null, + --uisng varchar instead of text for indexing purposes + -- Each UID externally defined such as from imap4 server + uid_ext varchar(3000) not null, + -- Each external source may apply a different uid. + -- This is essentialy an arbitrary constant frame reference between + -- connecting sessions with external server in most scenarios. + -- For IMAP4v1 rfc3501 2.3.1.1. item 4 ..combination of + -- mailbox.name, UIDVALIDITY, and UID must refer to a single + -- immutable message on that server forever. + -- default is: + -- ExternalSource parameter mailbox.name + -- and UIDVALIDITY with dash as delimiter + -- where ExternalSource parameter is + -- either blank or maybe mailbox.host for example. + -- external source reference id + -- see acs_mail_lite_email_src_ext_id_map.aml_src_id + src_ext_id integer +); + +create index acs_mail_lite_email_uid_id_map_uid_ext_idx + on acs_mail_lite_email_uid_id_map (uid_ext); +create index acs_mail_lite_email_uid_id_map_src_ext_id_idx + on acs_mail_lite_email_uid_id_map (src_ext_id); + +create table acs_mail_lite_email_src_ext_id_map ( + aml_src_id integer not null, + src_ext varchar(1000) not null +); + + + +-- Packages that are services, such as ACS Mail Lite, do not have a web UI. +-- Scheduled procs cannot read changes in values of package parameters +-- or get updates via web UI connections, or changes in tcl via apm. +-- Choices are updates via nsv variables and database value updates. +-- Choices via database have persistence across server restarts. +-- Defaults are set in acs_mail_lite::sched_parameters +-- These all are used in context of processing incoming email +-- unless stated otherwise. +-- Most specific flag takes precedence. +-- If an email is flagged high priority by package_id and +-- low priority by subject glob. It is assigned low priority. +-- Order of specificity. +-- medium default package_id party_id subject_id object_id +-- party_id can be group_id or user_id +-- If fast and low flag the same specificity for an email, low is chosen. +create table acs_mail_lite_ui ( + -- scan_replies_est_dur_per_cycle_s_override see www/doc/analysis-notes + sredpcs_override integer, + -- Answers question: Reprocess old email? + reprocess_old_p boolean, + -- Max number of concurrent threads for high priority processing + max_concurrent integer, + -- Any incoming email body part over this size is stored in file + -- instead of database. + max_blob_chars integer, + -- Minimum threashold for default medium (standard) priority + mpri_min integer, + -- Maximum value for default medium (standard) priority + mpri_max integer, + --space delimited list of package_ids to process at fast/high priority + hpri_package_ids text, + --space delimited list of package_ids to process at low priority + lpri_package_ids text, + --space delimited list of party_ids to process at fast/high priority + hpri_party_ids text, + --space delimited list of party_ids to process at low priority + lpri_party_ids text, + -- a glob for searching subjects to flag for fast/high priority + hpri_subject_glob text, + -- a glob for searching subjects to flag for low priority + lpri_subject_glob text, + --space delimited list of object_ids to process at fast/high priority + hpri_object_ids text, + --space delimited list of object_ids to process at low priority + lpri_object_ids text, + --filters to reject input as early as possible in processing inbound + --Each filter is a name value in standard tcl list format + --where name is a header name + reject_on_hit text, + reject_on_miss text +); + + +-- This table has similar requirements to acs_mail_lite_ui +-- proc acs_mail_lite_imap_conn_* needs to be able to update values +-- within scheduled procs without restarting server. +-- Port is ignored. Added because it is a common requirement of connections +-- that might one day be useful here, too. +create table acs_mail_lite_imap_conn ( + -- mailbox.host + ho text, + -- you guessed it + pa text, + -- port + po integer, + --timeout + ti integer, + -- user + us text, + -- mailbox.name See nsimap documentation for definition + na text, + -- space separated list of flags for imap related modifications + -- ssl means connect via ssl. + -- novalidatecert means accept a self-signed certificate + fl text +); + + +-- Following tables store parsed incoming email for processing by callbacks +-- defined in the rest of OpenACS + +-- incoming email headers +-- There should be a size limit per unit time from each source +-- to prevent DDOS attacks and such (at least to the imap system). +-- +create table acs_mail_lite_ie_headers ( + -- incoming email + -- only includes headers useful in processing the queue + -- Such as + -- size + -- from + aml_email_id integer, + -- header name, one header per row + -- For all headers together, see acs_mail_lite_ie_parts.c_type=headers + -- Special case: h_name = struct means + -- h_value contains entire value returned from ns_imap struct + -- as a tcl list + h_name text, + h_value text +); + +create index acs_mail_lite_ie_headers_aml_email_id_idx + on acs_mail_lite_ie_headers (aml_email_id); + +-- incoming email body parts +-- including email file attachments and file content +-- A part may be a filename. The filename data model is added +-- to the parts table to reduce code complexity. +-- An attached or inline file is a kind of part. +create table acs_mail_lite_ie_parts ( + aml_email_id integer, + section_id integer, + + -- In addition to content_type, there is a special case: + -- headers, which contains all headers for email + -- content_type = c_type + c_type text, + -- If type has a filename, this is original filename. + filename text, + -- If c_type is multipart, content is blank. part_id is branched. + content text, + -- An alternate filepathname for large blob, or + -- A local absolute filepath location + c_filepathname text +); + +create index acs_mail_lite_ie_parts_aml_email_id_idx + on acs_mail_lite_ie_parts (aml_email_id); + + +-- incoming email parts, name value pairs of +create table acs_mail_lite_ie_part_nv_pairs ( + aml_email_id integer, + -- Usage is same as acs_mail_lite_ie_parts.section_id + section_id integer, + -- name value pair + p_name text, + p_value text +); + +create index acs_mail_lite_ie_part_nv_pairs_aml_email_id_idx + on acs_mail_lite_ie_part_nv_pairs (aml_email_id); + +create table acs_mail_lite_ie_section_ref_map ( + -- 'Section' refers to usage with 'part' reference in 'ns_imap body' + -- Email parts can contain multiple parts. + -- Each multiple part can contain multiple parts. + + -- Section_ref is an absolute reference of a part + -- including the parts it is contained in, and + -- delimited by period. + -- It is defined by: + -- ns_imap body #s msgno part + -- And yet, this reference system holds for any email + -- storage, so is adopted for generic use as well. + + -- Default reference is value of 1. + -- A two part message has values 1 and 2. + -- Part 2 of a 3 part email (2/3) has reference '2' + -- If part 2 is also multiple parts, then + -- part 1 of part 2 of email has reference '2.1' and so on. + + -- Mapping is constant for each case. + -- For example, '1.2.2.1' will always point to the same integer. + -- So do not alter values as they are likely used by + -- multiple emails. + + + section_ref varchar(300), + section_id integer +); + +create index acs_mail_lite_ie_section_ref_map_section_ref_idx + on acs_mail_lite_ie_section_ref_map (section_ref); +create index acs_mail_lite_ie_section_ref_map_section_id_idx + on acs_mail_lite_ie_section_ref_map (section_id); + +-- +create table acs_mail_lite_send_msg_id_map ( + -- a randomized number unique to this table + -- unique not null + msg_id text primary key, + package_id integer + constraint aml_package_id_fk + references apm_packages, + party_id integer + constraint aml_from_external_party_id_fk + references parties (party_id), + object_id integer + constraint aml_from_external_obect_id_fk + references acs_objects (object_id), + -- Indicate approximate time when this email is created + datetime_cs integer, + -- other data or parameters to associate with email + other text +); + +create index acs_mail_lite_send_msg_id_map_msg_id_idx + on acs_mail_lite_send_msg_id_map (msg_id); Index: openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-callback-procs.tcl =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-callback-procs.tcl,v diff -u -N -r1.24 -r1.25 --- openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-callback-procs.tcl 7 Aug 2017 23:47:57 -0000 1.24 +++ openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-callback-procs.tcl 17 Feb 2018 17:08:31 -0000 1.25 @@ -91,11 +91,28 @@ } { upvar $array email - set to [acs_mail_lite::parse_email_address -email $email(to)] - ns_log Debug "acs_mail_lite::incoming_email -impl acs-mail-lite called. Recipient $to" + # for email_queue, header info is already parsed + if { [info exists email(aml_to_addrs)] } { + set to $email(aml_to_addrs) + } else { + set to [acs_mail_lite::parse_email_address -email $email(to)] + } + ns_log Debug "acs_mail_lite::incoming_email -impl acs-mail-lite called. Recepient $to" - lassign [acs_mail_lite::parse_bounce_address -bounce_address $to] user_id package_id signature - + if { ![info exists email(aml_user_id)] } { + # Traditional call parses here. Queue case is pre-parsed. + lassign [acs_mail_lite::parse_bounce_address -bounce_address $to] user_id package_id signature + } else { + set user_id $email(aml_user_id) + set package_id $email(aml_package_id) + # signature could come from a number of headers. Pre-parsing + # makes signature obsolete here. + set signature "" + } + # The above adaptions make this proc usable with newer versions of + # code in the legacy paradigm. + # Sadly, this bounces all cases with a user_id so it is not + # usable for the new inbound email callback scheme. # If no user_id found or signature invalid, ignore message if {$user_id eq ""} { ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: No equivalent user found for $to" @@ -105,7 +122,119 @@ } } +ad_proc -public -callback acs_mail_lite::email_inbound { + -headers_array_name:required + -parts_array_name:required + {-package_id ""} + {-object_id ""} + {-party_id ""} + {-other ""} + {-datetime_cs ""} +} { + Callback that is executed for inbound e-mails that are queued. + package_id, object_id, party_id, other, and datetime_cs are populated + only when information provided via a signed unique_id via + acs_mail_lite::unique_id_create +} - + +ad_proc -public -callback acs_mail_lite::email_inbound -impl acs-mail-lite { + -headers_array_name:required + -parts_array_name:required + {-package_id ""} + {-object_id ""} + {-party_id ""} + {-other ""} + {-datetime_cs ""} +} { + Example Implementation of acs_mail_lite::email_inbound. + This is where documentation for callback goes. + + @creation-date 2017-10-17 + + @param headers_array_name An array with all email headers. + @param parts_array_name An array with info on files and bodies. + @see acs_mail_lite::inbound_queue_pull_one + + @param package_id The package_id of package that sent the original email. + @param object_id + @param party_id + @param other + @param datetime_cs + + Not all inbound email are expected to be replies. +} { + upvar $headers_array_name headers_arr + upvar $parts_array_name parts_arr + set no_errors_p 1 + # ------------------- Do Not change code above this line in your copy --- + # Use this callback implementation as a template for other packages. + # Be sure to change 'impl acs-mail-lite' to a reference relevant to + # package implementation is used in. + # For example: -impl super-package-now-with-email + # + # This proc is called whenever an inbound email is pulled from the queue. + # + # System-wide bounces, vacation notices and other noise have already been + # filtered out. + # + # A package developer should just need to confirm input for their specific + # package. + # + # When supplied, package_id, object_id and party_id, other and datetime_cs + # are passed in headers via a signed unique_id. + # Values default to empty string. + + # headers_arr is an array of header values indexed by header name. + # header names are in original upper and lower case, which may + # have some significance in filtering cases. Although case should + # should not be relied on for obtaining a value. + # Some header indexes are created by ACS Mail Lite procs during + # processing. For example these indexes may be populated via + # a unique id header created using acs_mail_lite::unique_id_create : + # + # aml_package_id contains package_id + # + # aml_object_id contains object_id + # + # aml_party_id contains party_id (usually same as user_id) + # + # aml_other contains other data useful as input + # + # aml_datetime_cs contains approx time in seconds since epoch when sent. + # + # + # Other header names, and a description of their values, includes: + # + # aml_received_cs approx time in seconds since epoch when email received. + # aml_subject contains subject value. + # aml_to contents of 'to' header + # aml_to_addrs email address of 'to' header + # aml_from contents of 'from' header + # aml_from_addrs email address of 'from' header + + # For other created headers, see: acs_mail_lite::inbound_queue_pull_one + # Header indexes may not exist for all cases. + # + + # parts_arr is an array that contains all the information about attached + # or inline files and body contents. + # For details, see acs_mail_lite::inbound_queue_pull_one + # + + ns_log Debug "acs_mail_lite::email_inbound -impl acs-mail-lite called. Sender $headers_arr(aml_from_addrs)" + + # Important: If your implementation has an error, + # set no_errors_p to 0, so that the email remains + # in the queue for later examination, even though it is also + # marked as 'processed' so it will not be re-processed later. + # + + # ------------------- Do Not change code below this line in your copy --- + return $no_errors_p +} + + # Local variables: # mode: tcl # tcl-indent-level: 4 Index: openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-init.tcl =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-init.tcl,v diff -u -N -r1.15 -r1.16 --- openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-init.tcl 7 Aug 2017 23:47:57 -0000 1.15 +++ openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-init.tcl 17 Feb 2018 17:08:31 -0000 1.16 @@ -8,30 +8,75 @@ } +# +# outbound +# + # Default interval is about one minute (reduce lock contention with other jobs scheduled at full minutes) ad_schedule_proc -thread t 61 acs_mail_lite::sweeper -set queue_dir [parameter::get_from_package_key -parameter "BounceMailDir" -package_key "acs-mail-lite"] - -if {$queue_dir ne ""} { +#if {$queue_dir ne ""} { # if BounceMailDir is set then handle incoming mail - ad_schedule_proc -thread t 120 acs_mail_lite::load_mails -queue_dir $queue_dir -} - -# check every few minutes for bounces -#ad_schedule_proc -thread t [acs_mail_lite::get_parameter -name BounceScanQueue -default 120] acs_mail_lite::scan_replies - +# ad_schedule_proc -thread t 120 acs_mail_lite::load_mails -queue_dir $queue_dir +#} nsv_set acs_mail_lite send_mails_p 0 nsv_set acs_mail_lite check_bounce_p 0 -# ad_schedule_proc -thread t -schedule_proc ns_schedule_daily [list 0 25] acs_mail_lite::check_bounces - - # Redefine ns_sendmail as a wrapper for acs_mail_lite::send #ns_log Notice "acs-mail-lite: renaming acs_mail_lite::sendmail to ns_sendmail" #rename ns_sendmail _old_ns_sendmail #rename acs_mail_lite::sendmail ns_sendmail + + +# +# inbound +# +# acs_mail_lite::load_mails -queue_dir $queue_dir + +set inbound_queue_dir [file join [acs_root_dir] acs-mail-lite ] +file mkdir $inbound_queue_dir +# imap scan incoming = si_ +# maildir scan incoming = sj_ +# Scan incoming start time in clock seconds. +set si_start_time_cs [clock seconds] +# Scan incoming estimated duration pur cycle in seconds +#set scan_in_est_dur_per_cycle_s 120 +set si_dur_per_cycle_s [parameter::get_from_package_key -parameter "IncomingScanRate" -package_key "acs-mail-lite" -default 120] +# max_import_rate_per_s .5 +set si_mirps .16 +# Used by incoming email system +#nsv_set acs_mail_lite scan_in_start_t_cs $si_start_time_cs +nsv_set acs_mail_lite si_start_t_cs $si_start_time_cs +nsv_set acs_mail_lite si_dur_per_cycle_s $si_dur_per_cycle_s +nsv_set acs_mail_lite si_dur_per_cycle_s_override "" +nsv_set acs_mail_lite si_max_ct_per_cycle \ + [expr { int( $si_mirps * $si_dur_per_cycle_s ) } ] +if { [db_table_exists acs_mail_lite_ui] } { + acs_mail_lite::sched_parameters +} +ad_schedule_proc -thread t \ + $si_dur_per_cycle_s acs_mail_lite::imap_check_incoming + +# offset next cycle start +after 314 +ad_schedule_proc -thread t \ + $si_dur_per_cycle_s acs_mail_lite::maildir_check_incoming + +# offset next cycle start +after 314 +ad_schedule_proc -thread t \ + $si_dur_per_cycle_s acs_mail_lite::inbound_queue_pull + +ad_schedule_proc -thread t -schedule_proc ns_schedule_daily [list 1 41] acs_mail_lite::inbound_queue_release + + + +# acs_mail_lite::check_bounces +ad_schedule_proc -thread t -schedule_proc ns_schedule_daily [list 0 25] acs_mail_lite::check_bounces + + + # Local variables: # mode: tcl # tcl-indent-level: 4 Index: openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs-oracle.xql =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs-oracle.xql,v diff -u -N -r1.16 -r1.17 --- openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs-oracle.xql 20 Nov 2017 13:26:55 -0000 1.16 +++ openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs-oracle.xql 17 Feb 2018 17:08:31 -0000 1.17 @@ -50,6 +50,25 @@ + + + + update acs_mail_lite_mail_log + set last_mail_date = sysdate + where party_id = :user_id + + + + + + + + insert into acs_mail_lite_mail_log (party_id, last_mail_date) + values (:user_id, sysdate) + + + + select Index: openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs-postgresql.xql =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs-postgresql.xql,v diff -u -N -r1.15 -r1.16 --- openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs-postgresql.xql 20 Nov 2017 13:26:55 -0000 1.15 +++ openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs-postgresql.xql 17 Feb 2018 17:08:31 -0000 1.16 @@ -50,6 +50,26 @@ + + + + + update acs_mail_lite_mail_log + set last_mail_date = current_timestamp + where party_id = :user_id + + + + + + + + insert into acs_mail_lite_mail_log (party_id, last_mail_date) + values (:user_id, current_timestamp) + + + + select Index: openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs.tcl =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs.tcl,v diff -u -N -r1.98 -r1.99 --- openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs.tcl 29 Dec 2017 11:54:28 -0000 1.98 +++ openacs-4/packages/acs-mail-lite/tcl/acs-mail-lite-procs.tcl 17 Feb 2018 17:08:31 -0000 1.99 @@ -136,6 +136,11 @@ set smtppassword [parameter::get -parameter "SMTPPassword" \ -package_id $mail_package_id] + + # Consider adding code here to + # set orignator to acs-mail-lite parameter FixedSenderEmail + # if FixedSenderEmail is not empty, + # so as to be consistent for all cases calling this proc. set cmd [list smtp::sendmessage $multi_token -originator $originator] foreach header $headers { @@ -417,9 +422,31 @@ set reply_to $from_addr } + # Get any associated data indicating need to sign message-id + + # associate a user_id + set rcpt_id 0 + if { [llength $to_addr] eq 1 } { + set rcpt_id [party::get_by_email -email $to_addr] + } + set rcpt_id [ad_decode $rcpt_id "" 0 $rcpt_id] + + # Set the message_id - set message_id [mime::uniqueID] + # message-id gets signed if parameter defaults not passed + set message_id [acs_mail_lite::unique_id_create \ + -object_id $object_id \ + -package_id $package_id \ + -party_id $rcpt_id] + + # Build the originator address to be used as envelope sender + # and originator etc. + set orginator [bounce_address -user_id $rcpt_id \ + -package_id $package_id \ + -message_id $message_id] + + # Set the date set message_date [acs_mail_lite::utils::build_date] @@ -567,17 +594,8 @@ lappend headers_list [list DCC [join $bcc_addr ","]] } - # Build the originator address to be used as envelope sender - set rcpt_id 0 - if { [llength $to_addr] eq 1 } { - set rcpt_id [party::get_by_email -email $to_addr] - } - set rcpt_id [ad_decode $rcpt_id "" 0 $rcpt_id] + set originator $message_id - set originator [bounce_address -user_id $rcpt_id \ - -package_id $package_id \ - -message_id $message_id] - set errorMsg "" set status ok @@ -634,7 +652,7 @@ # could need to look at files for their own purposes. if {[string is true $delete_filesystem_files_p]} { foreach f $filesystem_files { - file delete -- $f + file delete $f } } if {$status ne "ok"} { @@ -672,7 +690,7 @@ {bcc {}} } { - Replacement for ns_sendmail for backward compatibility. + Replacement for ns_sendmail for backward compability. } { Index: openacs-4/packages/acs-mail-lite/tcl/bounce-procs-oracle.xql =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/Attic/bounce-procs-oracle.xql,v diff -u -N --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ openacs-4/packages/acs-mail-lite/tcl/bounce-procs-oracle.xql 17 Feb 2018 17:08:31 -0000 1.3 @@ -0,0 +1,56 @@ + + + + oracle8.1.6 + + + + + insert into acs_mail_lite_bounce_notif + (party_id, notification_count, notification_time) + (select user_id, 0 as notification_count, + trunc(sysdate-1-:notification_interval) as notification_time + from acs_mail_lite_bounce + where bounce_count >= :max_bounce_count) + + + + + + + + select u.user_id, u.email, u.first_names || ' ' || u.last_name as name + from cc_users u, acs_mail_lite_bounce_notif n + where u.user_id = n.party_id + and u.email_bouncing_p = 't' + and n.notification_time < sysdate - :notification_interval + and n.notification_count < :max_notification_count + + + + + + + + update acs_mail_lite_bounce_notif + set notification_time = trunc(sysdate), + notification_count = notification_count + 1 + where party_id = :user_id + + + + + + + + delete from acs_mail_lite_bounce + where party_id in (select party_id + from acs_mail_lite_mail_log + where last_mail_date < sysdate - :max_days_to_bounce) + + + + + + + Index: openacs-4/packages/acs-mail-lite/tcl/bounce-procs-postgresql.xql =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/Attic/bounce-procs-postgresql.xql,v diff -u -N --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ openacs-4/packages/acs-mail-lite/tcl/bounce-procs-postgresql.xql 17 Feb 2018 17:08:31 -0000 1.3 @@ -0,0 +1,56 @@ + + + + postgresql7.1 + + + + + insert into acs_mail_lite_bounce_notif (party_id, notification_count, notification_time) + select party_id, 0 as notification_count, + date_trunc('day', current_timestamp - to_interval(1 + :notification_interval, 'days')) + as notification_time + from acs_mail_lite_bounce + where bounce_count >= :max_bounce_count + + + + + + + + select u.user_id, u.email, u.first_names || ' ' || u.last_name as name + from cc_users u, acs_mail_lite_bounce_notif n + where u.user_id = n.party_id + and u.email_bouncing_p = 't' + and n.notification_time < current_timestamp - to_interval(:notification_interval, 'days') + and n.notification_count < :max_notification_count + + + + + + + + update acs_mail_lite_bounce_notif + set notification_time = date_trunc('day',current_timestamp), + notification_count = notification_count + 1 + where party_id = :user_id + + + + + + + + delete from acs_mail_lite_bounce + where party_id in (select party_id + from acs_mail_lite_mail_log + where last_mail_date < current_timestamp - to_interval(:max_days_to_bounce, 'days')) + + + + + + + Index: openacs-4/packages/acs-mail-lite/tcl/bounce-procs.tcl =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/Attic/bounce-procs.tcl,v diff -u -N -r1.16 -r1.17 --- openacs-4/packages/acs-mail-lite/tcl/bounce-procs.tcl 29 Dec 2017 12:16:02 -0000 1.16 +++ openacs-4/packages/acs-mail-lite/tcl/bounce-procs.tcl 17 Feb 2018 17:08:31 -0000 1.17 @@ -49,24 +49,46 @@ -package_id:required -message_id:required } { - Composes a bounce address + Composes a bounce address. If parameter FixedSenderEmail empty, + message_id is used. If message_id is empty, the legacy approach + for creating bounce_address is used. + @option user_id user_id of the mail recipient @option package_id package_id of the mail sending package (needed to call package-specific code to deal with bounces) @option message_id message-id of the mail @return bounce address } { - return "[bounce_prefix]-$user_id-[ns_sha1 $message_id]-$package_id@[address_domain]" + set mail_package_id [apm_package_id_from_key "acs-mail-lite"] + set fixed_sender [parameter::get -parameter "FixedSenderEmail" \ + -package_id $mail_package_id \ + -default "" ] + if { $fixed_sender ne "" } { + set ba $fixed_sender + } else { + if { $message_id ne "" } { + set ba $message_id + } else { + set ba [bounce_prefix] + append ba "-" $user_id "-" [ns_sha1 $message_id] \ + "-" $package_id "@" [address_domain] + ns_log Warning "acs_mail_lite::bounce_address is using \ + deprecated way. Supply message_id. Use acs_mail_lite::unique_id_create" + } + } + return $ba } #--------------------------------------- - ad_proc -public parse_bounce_address { + ad_proc -public -deprecated parse_bounce_address { -bounce_address:required } { This takes a reply address, checks it for consistency, and returns a list of user_id, package_id and bounce_signature found @option bounce_address bounce address to be checked @return tcl-list of user_id package_id bounce_signature + + @See acs_mail_lite::inbound_email_context } { set regexp_str "\[[bounce_prefix]\]-(\[0-9\]+)-(\[^-\]+)-(\[0-9\]*)\@" if {![regexp $regexp_str $bounce_address all user_id signature package_id]} { @@ -104,9 +126,24 @@ set max_days_to_bounce [parameter::get -package_id $package_id -parameter MaxDaysToBounce -default 3] set notification_interval [parameter::get -package_id $package_id -parameter NotificationInterval -default 7] set max_notification_count [parameter::get -package_id $package_id -parameter MaxNotificationCount -default 4] - set notification_sender [parameter::get -package_id $package_id \ - -parameter NotificationSender \ - -default "reminder@[address_domain]"] + set notification_sender [parameter::get -package_id $package_id -parameter NotificationSender -default "reminder@[address_domain]"] + if { $notification_sender eq "" } { + # Use the most specific default available + set fixed_sender [parameter::get -package_id $package_id -parameter "FixedSenderEmail"] + if { $fixed_sender ne "" } { + set notification_sender $fixed_sender + } elseif { [acs_mail_lite::utils::valid_email_p [ad_system_owner]] } { + set notification_sender [ad_system_owner] + } else { + # Set to an email address that is required to exist + # to avoid email loops and other issues + # per RFC 5321 section 4.5.1 + # https://tools.ietf.org/html/rfc5321#section-4.5.1 + # The somewhat unique capitalization may be useful + # for identifyng source in diagnostic context. + set notification_sender "PostMastER@[address_domain]" + } + } # delete all bounce-log-entries for users who received last email # X days ago without any bouncing (parameter) @@ -134,7 +171,10 @@ array set user $notification_list set user_id $user(user_id) set href [export_vars -base [ad_url]/register/restore-bounce {user_id}] - set body "Dear $user(name),\n\nDue to returning mails from your email account, we currently do not send you any email from our system. To reenable your email account, please visit\n$href" + set body "Dear $user(name),\n\n\ + Due to returning mails from your email account, \n \ + we currently do not send you any email from our system.\n\n \ + To re-enable your email notifications, please visit\n${href}" send -to_addr $notification_list -from_addr $notification_sender -subject $subject -body $body -valid_email ns_log Notice "Bounce notification send to user $user_id" Index: openacs-4/packages/acs-mail-lite/tcl/email-inbound-procs.tcl =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/email-inbound-procs.tcl,v diff -u -N --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ openacs-4/packages/acs-mail-lite/tcl/email-inbound-procs.tcl 17 Feb 2018 17:08:31 -0000 1.1 @@ -0,0 +1,2434 @@ +ad_library { + + Provides API for importing email under a varitey of deployment conditions. + + @creation-date 19 Jul 2017 + @cvs-id $Id: email-inbound-procs.tcl,v 1.1 2018/02/17 17:08:31 gustafn Exp $ + +} + +namespace eval acs_mail_lite {} + +# Although loose dependencies require imap procs right now, +# the inbound email procs are designed to integrate +# other inbound email paradigms with minimal amount +# of re-factoring of code. + +##code OpenACS Developers wanting to adapt code to other than IMAP: +# Use acs_mail_lite::imap_check_incoming +# as a template for creating a generic version: +# acs_mail_lite::check_incoming + +ad_proc -public acs_mail_lite::sched_parameters { + -sredpcs_override + -reprocess_old_p + -max_concurrent + -max_blob_chars + -mpri_min + -mpri_max + -hpri_package_ids + -lpri_package_ids + -hpri_party_ids + -lpri_party_ids + -hpri_subject_glob + -lpri_subject_glob + -hpri_object_ids + -lpri_object_ids + -reject_on_hit + -reject_on_miss +} { + Returns a name value list of parameters + used by ACS Mail Lite scheduled procs. + If a parameter is passed with value, the value is assigned to parameter. + + @option sched_parameter value + + @param sredpcs_override If set, use this instead of si_dur_per_cycle_s. See www/doc/analysis-notes + + @param reprocess_old_p If set, does not ignore prior unread email + + @param max_concurrent Max concurrent processes to import (fast priority) + + @param max_blob_chars Email body parts larger are stored in a file. + + @param mpri_min Minimum threshold integer for medium priority. Smaller is fast High priority. + + @param mpri_max Maximum integer for medium priority. Larger is Low priority. + + @param hpri_package_ids List of package_ids to process at fast priority. + + @param lpri_package_ids List of package_ids to process at low priority. + + @param hpri_party_ids List of party_ids to process at fast/high priority. + + @param lpri_party_ids List of party_ids to process at low priority. + + @param hpri_subject_glob When email subject matches, flag as fast priority. + + @param lpri_subject_glob When email subject matches, flag as low priority. + + @param hpri_object_ids List of object_ids to process at fast/high priority. + + @param lpri_object_ids List of object_ids to process at low priority. + + @param reject_on_hit Name/Value list. See acs_mail_lite::inbound_filters + + @param reject_on_miss Name/Value list. See acs_mail_lite::inbound_filters + +} { + # See one row table acs_mail_lite_ui + # sched_parameters sp + set sp_list [list \ + sredpcs_override \ + reprocess_old_p \ + max_concurrent \ + max_blob_chars \ + mpri_min \ + mpri_max \ + hpri_package_ids \ + lpri_package_ids \ + hpri_party_ids \ + lpri_party_ids \ + hpri_subject_glob \ + lpri_subject_glob \ + hpri_object_ids \ + lpri_object_ids \ + reject_on_hit \ + reject_on_miss ] + + foreach sp $sp_list { + if { [info exists $sp] } { + set new(${sp}) [set $sp] + } + } + set changes_p [array exists new] + set exists_p [db_0or1row acs_mail_lite_ui_r { + select sredpcs_override, + reprocess_old_p, + max_concurrent, + max_blob_chars, + mpri_min, + mpri_max, + hpri_package_ids, + lpri_package_ids, + hpri_party_ids, + lpri_party_ids, + hpri_subject_glob, + lpri_subject_glob, + hpri_object_ids, + lpri_object_ids, + reject_on_hit, + reject_on_miss + from acs_mail_lite_ui limit 1 + } ] + + if { !$exists_p } { + # set initial defaults + set sredpcs_override 0 + set reprocess_old_p "f" + set max_concurrent 6 + set max_blob_chars 32767 + set mpri_min "999" + set mpri_max "99999" + set hpri_package_ids "" + set lpri_package_ids "" + set hpri_party_ids "" + set lpri_party_ids "" + set hpri_subject_glob "" + set lpri_subject_glob "" + set hpri_object_ids "" + set lpri_object_ids "" + set reject_on_hit "" + set reject_on_miss "" + } + + + if { !$exists_p || $changes_p } { + set validated_p 1 + set new_pv_list [array names new] + if { $changes_p } { + foreach spn $new_pv_list { + switch -exact -- $spn { + sredpcs_override - + max_concurrent - + max_blob_chars - + mpri_min - + mpri_max { + set v_p [ad_var_type_check_integer_p $new(${spn})] + if { $v_p } { + if { $new(${spn}) < 0 } { + set v_p 0 + } + } + if { $v_p && $spn eq "mpri_min" } { + if { $new(${spn}) >= $mpri_max } { + set v_p 0 + ns_log Warning "acs_mail_lite::\ + sched_parameters mpri_min '$new(${spn})' \ + must be less than mpri_max '${mpri_max}'" + } + } + if { $v_p && $spn eq "mpri_max" } { + if { $new(${spn}) <= $mpri_min } { + set v_p 0 + ns_log Warning "acs_mail_lite::\ + sched_parameters mpri_min '${mpri_min}' \ + must be less than mpri_max '$new(${spn})'" + } + } + } + reprocess_old_p { + set v_p [string is boolean -strict $new(${spn}) ] + } + hpri_package_ids - + lpri_package_ids - + hpri_party_ids - + lpri_party_ids - + hpri_object_ids - + lpri_object_ids { + set v_p [ad_var_type_check_integerlist_p $new(${spn})] + } + hpri_subject_glob - + lpri_subject_glob { + if { $new(${spn}) eq "" } { + set v_p 1 + } else { + set v_p [regexp -- {^[[:graph:]\ ]+$} $new(${spn})] + if { $v_p && \ + [string match {*[\[;]*} $new(${spn}) ] } { + set v_p 0 + } + } + } + reject_on_hit - + reject_on_miss { + if { [f::even_p [llength $new(${spn}) ]] } { + set v_p 1 + } else { + set v_p 0 + } + } + defaults { + ns_log Warning "acs_mail_lite::sched_parameters \ + No validation check made for parameter '${spn}'" + } + } + if { !$v_p } { + set validated_p 0 + ns_log Warning "acs_mail_lite::sched_parameters \ + value '$new(${spn})' for parameter '${spn}' not allowed." + } + } + } + + if { $validated_p } { + foreach sp_n $new_pv_list { + set ${sp_n} $new($sp_n) + } + + db_transaction { + if { $changes_p } { + db_dml acs_mail_lite_ui_d { + delete from acs_mail_lite_ui + } + } + db_dml acs_mail_lite_ui_i { + insert into acs_mail_lite_ui + (sredpcs_override, + reprocess_old_p, + max_concurrent, + max_blob_chars, + mpri_min, + mpri_max, + hpri_package_ids, + lpri_package_ids, + hpri_party_ids, + lpri_party_ids, + hpri_subject_glob, + lpri_subject_glob, + hpri_object_ids, + lpri_object_ids, + reject_on_hit, + reject_on_miss) + values + (:sredpcs_override, + :reprocess_old_p, + :max_concurrent, + :max_blob_chars, + :mpri_min, + :mpri_max, + :hpri_package_ids, + :lpri_package_ids, + :hpri_party_ids, + :lpri_party_ids, + :hpri_subject_glob, + :lpri_subject_glob, + :hpri_object_ids, + :lpri_object_ids, + :reject_on_hit, + :reject_on_miss + ) + } + + # See acs_mail_lite::imap_check_incoming for usage of: + nsv_set acs_mail_lite si_configured_p 1 + } + } + + } + set s_list [list ] + foreach s $sp_list { + set sv [set ${s}] + lappend s_list ${s} $sv + } + return $s_list +} + +ad_proc -public acs_mail_lite::inbound_prioritize { + {-header_array_name ""} + {-size_chars ""} + {-received_cs ""} + {-subject ""} + {-package_id ""} + {-party_id ""} + {-object_id ""} +} { + Returns a prioritization integer for assigning priority to an inbound email. + Another proc processes in order of lowest number first. + Returns empty string if input values from email are not expected types. + Priority has 3 categories: high priority, normal priority, low priority + as specified in acs_mail_lite::sched_parameters + + Expects parameters to be passed within an array, or individually. + When passing via an array, parameter names have suffix "aml_". + For example, size_chars becomes aml_size_chars. + + Array values take precedence, if they exist. + + @param size_chars of email + + @param received_cs seconds since epoch when email received + + @param package_id associated with email (if any) + + @param party_id associated with email (if any) + + @param subject of email + + @param object_id associated with email (if any) + + @see acs_mail_lite::sched_parameters + +} { + if { $header_array_name ne "" } { + set hn_list [list \ + aml_size_chars \ + aml_received_cs \ + aml_subject \ + aml_package_id \ + aml_party_id \ + aml_object_id ] + upvar 1 $header_array_name h_arr + foreach hn $hn_list { + set vname [string range $hn 4 end] + if { [info exists h_arr(${hn}) ] } { + # set variable from array + set ${vname} $h_arr(${hn}) + } elseif { [set ${hn}] ne "" } { + # set array's index same as variable + set h_arr(${hn}) [set ${vname} ] + } + } + } + + set priority_fine "" + + set size_error_p 0 + # validate email inputs + if { ! ([string is wideinteger -strict $size_chars] \ + && $size_chars > 0) } { + set size_error_p 1 + ns_log Warning "acs_mail_lite::inbound_prioritize.283: \ + size_chars '${size_chars}' is not a natural number." + } + set time_error_p 0 + if { ! ([string is wideinteger -strict $received_cs] \ + && $received_cs > 0) } { + set time_error_p 1 + ns_log Warning "acs_mail_lite::inbound_prioritize.289: \ + received_cs '${received_cs}' is not a natural number." + } + + # *_cs means clock time from epoch in seconds, + # same as returned from tcl clock seconds + array set params_arr [acs_mail_lite::sched_parameters] + + set priority 2 + # Set general priority in order of least specific first + if { $package_id ne "" } { + if { $package_id in $params_arr(hpri_package_ids) } { + set priority 1 + } + if { $package_id in $params_arr(lpri_package_ids) } { + set priority 3 + } + } + + if { $party_id ne "" } { + if { $party_id in $params_arr(hpri_party_ids) } { + set priority 1 + } + if { $party_id in $params_arr(lpri_party_ids) } { + set priority 3 + } + } + + + if { [string match $params_arr(hpri_subject_glob) $subject] } { + set priority 1 + } + if { [string match $params_arr(lpri_subject_glob) $subject] } { + set priority 3 + } + + + if { $object_id ne "" } { + if { $object_id in $params_arr(hpri_object_ids) } { + set priority 1 + } + if { $object_id in $params_arr(lpri_object_ids) } { + set priority 3 + } + } + + # quick math for arbitrary super max of maxes + set su_max $params_arr(mpri_max) + append su_max "00" + set size_list [list $su_max] + set ns_section_list [list nssock nssock_v4 nssock_v6] + foreach section $ns_section_list { + lappend size_list [ns_config -int -min 0 $section maxinput] + } + set size_max [f::lmax $size_list] + # add granularity + switch -exact $priority { + 1 { + set pri_min 0 + set pri_max $params_arr(mpri_min) + } + 2 { + set pri_min $params_arr(mpri_min) + set pri_max $params_arr(mpri_max) + } + 3 { + set pri_min $params_arr(mpri_max) + set pri_max $size_max + } + default { + ns_log Warning "acs_mail_lite::inbound_prioritize.305: \ + Priority value not expected '${priority}'" + } + } + + ns_log Dev "inbound_prioritize: pri_max '${pri_max}' pri_min '${pri_min}'" + + set range [expr { $pri_max - $pri_min } ] + # deviation_max = d_max + set d_max [expr { $range / 2 } ] + # midpoint = mp + set mp [expr { $pri_min + $d_max } ] + ns_log Dev "inbound_prioritize: range '${range}' d_max '${d_max}' mp '${mp}'" + + # number of variables in fine granularity calcs: + # char_size, date time stamp + set varnum 2 + # Get most recent scan start time for reference to batch present time + set start_cs [nsv_get acs_mail_lite si_start_t_cs] + set dur_s [nsv_get acs_mail_lite si_dur_per_cycle_s] + ns_log Dev "inbound_prioritize: start_cs '${start_cs}' dur_s '${dur_s}'" + + # Priority favors earlier reception, returns decimal -1. to 0. + # for normal operation. Maybe -0.5 to 0. for most. + if { $time_error_p } { + set pri_t 0 + } else { + set pri_t [expr { ( $received_cs - $start_cs ) / ( 2. * $dur_s ) } ] + } + + # Priority favors smaller message size. Returns decimal 0. to 1. + # and for most, somewhere closer to perhaps 0. + if { $size_error_p } { + set pri_s [expr { ( $size_max / 2 ) } ] + } else { + set pri_s [expr { ( $size_chars / ( $size_max + 0. ) ) } ] + } + + set priority_fine [expr { int( ( $pri_t + $pri_s ) * $d_max ) + $mp } ] + ns_log Dev "inbound_prioritize: pri_t '${pri_t}' pri_s '${pri_s}'" + ns_log Dev "inbound_prioritize: pre(max/min) priority_fine '${priority_fine}'" + set priority_fine [f::min $priority_fine $pri_max] + set priority_fine [f::max $priority_fine $pri_min] + + if { $header_array_name ne "" } { + set h_arr(aml_priority) $priority_fine + } + return $priority_fine +} + + +ad_proc -public acs_mail_lite::email_type { + {-subject ""} + {-from ""} + {-headers ""} + {-header_arr_name ""} + {-reply_too_fast_s "10"} + {-check_subject_p "0"} +} { +

+ Scans email's subject, from and headers for actionable type. +

+ Returns actionable type and saves same type in header_arr_name(aml_type), + and saves some normalized header info + to reduce redundant processing downstream. See code comments for details. +

+ Actional types: \ + 'auto_gen' 'auto_reply', 'bounce', 'in_reply_to' or + empty string indicating 'other' type. +

+
  • + 'auto_reply' may be a Delivery Status Notification for example. +
  • + 'bounce' is a specific kind of Delivery Status Notification. +
  • + 'in_reply_to' is an email reporting to originate from local email, + which needs to be tested further to see if OpenACS needs to act on + it versus a reply to a system administrator email for example. +
  • + 'auto_gen' is an auto-generated email that does not qualify as 'auto_reply', 'bounce', or 'in_reply_to' +
  • + '' (Empty string) refers to email that the system does not recognize as a reply + of any kind. If not a qualifying type, returns empty string. +
+ Adds these index to headers array: +
  • + received_cs: the recevied time of email in tcl clock epoch time. +
  • + aml_type: the same value returned by this proc. +
+

+ If additional headers not calculated, they have value of empty string. +

+ If headers and header_arr_name provided, only header_arr_name will be used, if header_arr_name contains at least one value. +

+ If check_subject_p is set 1, \ + checks for common subjects identifying autoreplies. \ + This is not recommended to rely on exclusively. \ + This feature provides a framework for expaning classification of \ + emails for deployment routing purposes. +

+ If array includes keys from 'ns_imap struct', such as internaldate.*, \ + then adds header with epoch time quivilent to header index received_cs +

+ @param subject of email + @param from of email + @param headers of email, a block of text containing all headers and values + @param header_arr_name, the name of an array containing headers. + @param check_subject_p Set to 1 to check email subject. +} { + set ag_p 0 + set an_p 0 + set ar_p 0 + set as_p 0 + set dsn_p 0 + set irt_idx -1 + set or_idx -1 + set pe_p 0 + set ts_p 0 + set reject_p 0 + # header cases: {*auto-generated*} {*auto-replied*} {*auto-notified*} + # from: + # https://www.iana.org/assignments/auto-submitted-keywords/auto-submitted-keywords.xhtml + # and rfc3834 https://www.ietf.org/rfc/rfc3834.txt + + # Do NOT use x-auto-response-supress + # per: https://stackoverflow.com/questions/1027395/detecting-outlook-autoreply-out-of-office-emails + + # header cases: + # {*x-autoresponder*} {*autoresponder*} {*autoreply*} + # {*x-autorespond*} {*auto_reply*} + # from: + # https://github.com/jpmckinney/multi_mail/wiki/Detecting-autoresponders + # redundant cases are removed from list. + # auto reply = ar + set ar_list [list \ + {auto-replied} \ + {auto-reply} \ + {autoreply} \ + {autoresponder} \ + {x-autorespond} \ + ] + # Theses were in auto_reply, but are not specific to replies: + # {auto-generated} + # {auto-notified} + # See section on auto_gen types. (auto-submitted and the like) + + + if { $header_arr_name ne "" } { + upvar 1 $header_arr_name h_arr + } else { + array set h_arr [list ] + } + + if { $headers ne "" && [llength [array names h_arr]] < 1 } { + # To remove subject from headers to search, + # incase topic uses a reserved word, + # we rebuild the semblence of array returned by ns_imap headers. + # Split strategy from qss_txt_table_stats + set linebreaks "\n\r\f\v" + set row_list [split $headers $linebreaks] + foreach row $row_list { + set c_idx [string first ":" $row] + if { $c_idx > -1 } { + set header [string trim [string range $row 0 $c_idx-1]] + # following identifies multiline header content to ignore + if { ![string match {*[;=,]*} $header] } { + # list of email headers at: + # https://www.cs.tut.fi/~jkorpela/headers.html + # Suggests this filter for untrusted input: + if { [regsub -all -- {[^a-zA-Z0-9\-]+} $header {} h2 ] } { + ns_log Warning "acs_mail_lite:email_type.864: \ + Unexpected header '${header}' changed to '${h2}'" + set header $h2 + } + set value [string trim [string range $row $c_idx+1 end]] + # string match from proc safe_eval + if { ![string match {*[\[;]*} $value ] } { + # 'append' is used instead of 'set' in + # the rare case that there's a glitch + # and there are two or more headers with same name. + # We want to examine all values of specific header. + append h_arr(${header}) "${value} " + ns_log Dev "acs_mail_lite::email_type.984 \ + header '${header}' value '${value}' from text header '${row}'" + } + } + } + } + } + + set reject_p [acs_mail_lite::inbound_filters -headers_arr_name h_arr] + + + if { !$reject_p } { + + set hn_list [array names h_arr] + ns_log Dev "acs_mail_lite::email_type.996 hn_list '${hn_list}'" + # Following checks according to rfc3834 section 3.1 Message header + # https://tools.ietf.org/html/rfc3834 + + + # check for in-reply-to = irt + set irt_idx [lsearch -glob -nocase $hn_list {in-reply-to}] + # check for message_id = mi + # This is a new message id, not message id of email replied to + set mi_idx [lsearch -glob -nocase $hn_list {message-id}] + + # Also per rfc5436 seciton 2.7.1 consider: + # auto-submitted = as + + set as_idx [lsearch -glob -nocase $hn_list {auto-submitted}] + if { $as_idx > 1 } { + set as_p 1 + set as_h [lindex $hn_list $as_idx] + set an_p [string match -nocase $h_arr(${as_h}) {auto-notified}] + # also check for auto-generated + set ag_p [string match -nocase $h_arr(${as_h}) {auto-generated}] + } + + + + ns_log Dev "acs_mail_lite::email_type.1017 as_p ${as_p} an_p ${an_p} ag_p ${ag_p}" + + # If one of the headers contains {list-id} then email + # is from a mailing list. + + set i 0 + set h [lindex $ar_list $i] + while { $h ne "" && !$ar_p } { + #set ar_p string match -nocase $h $hn + + set ar_idx [lsearch -glob $hn_list $h] + if { $ar_idx > -1 } { + set ar_p 1 + } + + incr i + set h [lindex $ar_list $i] + } + + ns_log Dev "acs_mail_lite::email_type.1039 ar_p ${ar_p}" + + + # get 'from' header value possibly used in a couple checks + set fr_idx [lsearch -glob -nocase $hn_list {from}] + set from_email "" + if { $fr_idx > -1 } { + set fr_h [lindex $hn_list $fr_idx] + set from [ad_quotehtml $h_arr(${fr_h})] + set h_arr(aml_from) $from + set from_email [string tolower \ + [acs_mail_lite::parse_email_address \ + -email $from]] + set h_arr(aml_from_addrs) $from_email + set at_idx [string last "@" $from ] + } else { + set at_idx -1 + } + if { $at_idx > -1 } { + # from_email is not empty string + set from_host [string trim [string range $from $at_idx+1 end]] + set party_id [party::get_by_email -email $from_email] + if { $party_id ne "" } { + set pe_p 1 + } + } else { + set from_host "" + set party_id "" + } + + + + + if { !$ar_p && [info exists h_arr(internaldate.year)] \ + && $from ne "" } { + + # Use the internal timestamp for additional filters + set dti $h_arr(internaldate.year) + append dti "-" [format "%02u" $h_arr(internaldate.month)] + append dti "-" [format "%02u" $h_arr(internaldate.day)] + append dti " " [format "%02u" $h_arr(internaldate.hours)] + append dti ":" [format "%02u" $h_arr(internaldate.minutes)] + append dti ":" [format "%02u" $h_arr(internaldate.seconds)] " " + if { $h_arr(internaldate.zoccident) eq "0" } { + # This is essentially iso8601 timezone formatting. + append dti "+" + } else { + # Comment from panda-imap/src/c-client/mail.h: + # /* non-zero if west of UTC */ + # See also discussion beginning with: + # /* occidental *from Greenwich) timezones */ + # in panda-imap/src/c-client/mail.c + append dti "-" + } + append dti [format "%02u" $h_arr(internaldate.zhours)] + append dti [format "%02u" $h_arr(internaldate.zminutes)] "00" + if { [catch { + set dti_cs [clock scan $dti -format "%Y-%m-%e %H:%M:%S %z"] + } err_txt ] } { + set dti_cs "" + ns_log Warning "acs_mail_lite::email_type.1102 \ + clock scan '${dti}' -format %Y-%m-%d %H:%M:%S %z failed. Could not check ts_p case." + } + set h_arr(aml_received_cs) $dti_cs + # Does response time indicate more likely by a machine? + # Not by itself. Only if it is a reply of some kind. + + # Response is likely machine if it is fast. + # If the difference between date and local time is less than 10s + # and either from is "" or subject matches "return*to*sender" + + # More likely also from machine + # if size is more than a few thousand characters in a short time. + + # This is meant to detect more general cases + # of bounce/auto_reply detection related to misconfiguration + # of a system. + # This check is + # intended to prevent flooding server and avoiding looping + # that is not caught by standard MTA / smtp servers. + # An MTA likely checks already for most floods and loops. + # As well, this check providesy yet another + # indicator to intervene in uniquely crafted attacks. + + if { $pe_p && $dti_cs ne "" } { + # check multiple emails from same user + + nsv_lappend acs_mail_lite si_party_id_cs(${party_id}) $dti_cs + set max_ct [nsv_get acs_mail_lite si_max_ct_per_cycle] + set cycle_s [nsv_get acs_mail_lite si_dur_per_cycle_s] + set cs_list [nsv_get acs_mail_lite si_party_id_cs(${party_id})] + set cs_list_len [llength $cs_list] + if { $cs_list_len > $max_ct } { + set params_ul [acs_mail_lite::sched_parameters] + set lpri_pids [dict get $params_ul lpri_party_ids] + set lpri_pids_list [split $lpri_pids] + if { $party_id ni $lpri_pdis_list } { + # full check required + set start_cs [nsv_get acs_mail_lite si_start_t_cs] + set prev_start_cs [expr { $start_cs - $cycle_s } ] + set cs_list [lsort -integer -increasing -unique $cs_list] + set i 0 + set is_stale_p 1 + while { $is_stale_p && $i < $cs_list_len } { + set test_ts [lindex $cs_list $i] + if { $test_ts > $prev_start_cs } { + set is_stale_p 0 + } + incr i + } + if { $is_stale_p } { + set cs2_list [list ] + # Really? + # We just added dti_cs to si_party_id_cs(party_id) + # This happens when scaning email is delayed some + ns_log Warning "acs_mail_lite::email_type.655 \ + party_id '${party_id}' prev_start_cs '${prev_start_cs}' i '${i}' \ + cs_list_len '${cs_list_len}' cs_list '${cs_list}' cs2_list '${cs2_list}'" + } else { + set cs2_list [lrange $cs_list $i-1 end] + set cs2_list_len [llength $cs2_list] + if { $cs2_list_len > $max_ct } { + # si_max_ct_per_cycle reached for party_id + + # Flag as low priority if over count for cycle + # That is, add party_id to + # acs_mail_lite::sched_parameters -lpri_party_ids + # if it is not already + # Already checked at beginning of this check + lappend lpri_pids_list $party_id + acs_mail_lite::sched_parameters \ + -lpri_party_ids $lpri_pids_list + + } + } + nsv_set acs_mail_lite si_party_id_cs(${party_id}) $cs2_list + } + } + } + + # RFC 822 header required: DATE + set dt_idx [lsearch -glob -nocase $hn_list {date}] + # If there is no date. Flag it. + if { $dt_idx < 0 } { + set ts_p 1 + } else { + # Need to check received timestamp vs. when OpenACS + # or a system hosted same as OpenACS sent it. + + set dt_h [lindex $hn_list $dt_idx] + set dte_cs [ns_imap parsedate $h_arr(${dt_h})] + set diff 1000 + if { $dte_cs ne "" && $dti_cs ne "" } { + set diff [expr { abs( $dte_cs - $dti_cs ) } ] + } + # If too fast, set ts_p 1 + if { $diff < 11 } { + set ts_p 1 + } + + # check from host against acs_mail_lite's host + # From: header must show same OpenACS domain for bounce + # and subsequently verified not a user or system recognized + # user/admin address. + + # Examples of unrecognized addresses include mailer-daemon@.. + set host [dict get [acs_mail_lite::imap_conn_set] host] + if { $ts_p && [string -nocase "*${host}*" $from_host] } { + if { $from_email eq [ad_outgoing_sender] || !$pe_p } { + # This is a stray one. + set ag_p 1 + } + + } + + # Another possibility is return-path "<>" + # and Message ID unique-char-ref@bounce-domain + + # Examples might be a bounced email from + # a nonstandard web form on site + # or + # a loop where 'from' is + # a verified user or system recognized address + # and reply is within 10 seconds + # and a non-standard acs-mail-lite reply-to address + + + } + + } + + # Delivery Status Notifications, see rfc3464 + # https://tools.ietf.org/html/rfc3464 + # Note: original-envelope-id is not same as message-id. + # original-recipient = or + set or_idx [lsearch -glob -nocase $hn_list {original-recipient}] + if { $or_idx < 0 } { + # RFC3461 4.2 uses original-recipient-address + set or_idx [lsearch -glob \ + -nocase $hn_list {original-recipient-address}] + } + + # action = ac (required for DSN) + # per fc3464 s2.3.3 + set ac_idx [lsearch -glob -nocase $hn_list {action}] + if { $ac_idx > -1 } { + set ac_h [lindex $hn_list $ac_idx] + set status_list [list failed \ + delayed \ + delivered \ + relayed \ + expanded ] + # Should 'delivered' be removed from status_list? + # No, just set ar_p 1 instead of dsn_p 1 + + set s_i 0 + set status_p 0 + set stat [lindex $status_list $s_i] + while { $stat ne "" && !$status_p } { + # What if there are duplicate status values or added junk? + # Catch it anyway by wrapping glob with asterisks + if { [string match -nocase "*${stat}*" $h_arr(${ac_h})] } { + set status_p 1 + } + ns_log Dev "acs_mail_lite::email_type.1070 \ + status_p $status_p stat '${stat}' ac_h ${ac_h} h_arr(ac_h) '$h_arr(${ac_h})'" + + incr s_i + set stat [lindex $status_list $s_i] + } + if { $status_p } { + # status = st (required for DSN) + # per fc3464 s2.3.4 + set st_idx [lsearch -glob -nocase $hn_list {status}] + if { $st_idx > -1 } { + set st_h [lindex $hn_list $st_idx] + set dsn_p [string match {*[0-9][0-9][0-9]*} \ + $h_arr(${st_h}) ] + ns_log Dev "acs_mail_lite::email_type.1080 \ + dsn_p ${dsn_p} st_h ${st_h} h_arr(st_h) '$h_arr(${st_h})'" + if { $st_idx eq 2 || !$dsn_p } { + set ar_p 1 + } + } + } + } + + ns_log Dev "acs_mail_lite::email_type.1089 \ + ar_p ${ar_p} dsn_p ${dsn_p}" + + # if h_arr exists and.. + if { !$ar_p && $check_subject_p } { + # catch nonstandard cases + # subject flags + + # If 'from' not set. Set here. + if { $from eq "" } { + set fr_idx [lsearch -glob -nocase $hn_list {from}] + if { $fr_idx > -1 } { + set from $h_arr(${from}) + } + } + # If 'subject' not set. Set here. + if { $subject eq "" } { + set fr_idx [lsearch -glob -nocase $hn_list {subject}] + if { $fr_idx > -1 } { + set subject $h_arr(${subject}) + set h_arr(aml_subject) [ad_quotehtml $subject] + } + } + + set ps1 [string match -nocase {*out of*office*} $subject] + set ps2 [string match -nocase {*automated response*} $subject] + set ps3 [string match -nocase {*autoreply*} $subject] + set ps4 [string match {*NDN*} $subject] + set ps5 [string match {*\[QuickML\] Error*} $subject] + # rfc3834 states to NOT rely on 'Auto: ' in subject for detection. + #set ps6 \[string match {Auto: *} $subject\] + + # from flags = pf + set pf1 [string match -nocase {*mailer*daemon*} $from] + + set ar_p [expr { $ps1 || $ps2 || $ps3 || $ps4 || $ps5 || $pf1 } ] + } + + } + ns_log Dev "acs_mail_lite::email_type.1127 ar_p ${ar_p}" + + + # Return actionable types: + # 'auto_gen', 'auto_reply', 'bounce', 'in_reply_to' or '' (other) + + # a bounce also flags maybe auto_reply, in_reply_to, auto_gen + # an auto_reply also flags maybe auto_reply, auto_gen, in_reply_to + # an auto_gen does NOT include an 'in_reply_to' + # an in_reply_to does NOT include 'auto_gen'. + if { $dsn_p || $or_idx > -1 } { + set type "bounce" + } elseif { $ar_p || ( $irt_idx > -1 && \ + ( $ag_p || $as_p || $an_p || $ts_p ) ) } { + set type "auto_reply" + } elseif { $ag_p || $as_p || $an_p || $ts_p } { + set type "auto_gen" + } elseif { $irt_idx > -1 } { + set type "in_reply_to" + } else { + # other + set type "" + } + if { $header_arr_name ne "" } { + set h_arr(aml_type) $type + } + return $type +} + + +ad_proc -private acs_mail_lite::inbound_queue_insert { + -headers_arr_name + -parts_arr_name + {-priority ""} + {-aml_email_id ""} + {-section_ref ""} + {-struct_list ""} + {-error_p "0"} +} { + Adds a new, actionable incoming email to the queue for + prioritized processing. + + Returns aml_email_id if successful, otherwise empty string. +} { + upvar 1 $headers_arr_name h_arr + upvar 1 $parts_arr_name p_arr + + set id "" + # This should remain general enough to import + # email regardless of its source. + + # Email should already be parsed and in a transferable format + # in passed arrays + + # Array content corresponds to these tables: + + # h_arr($name) $value acs_mail_lite_ie_headers + # Some indexes match fields of table acs_mail_lite_from_external: + # h_arr(aml_email_id) + # h_arr(aml_to_addrs) to_email_addrs + # h_arr(aml_from_addrs) from_email_addrs + # h_arr(aml_priority) priority + # h_arr(aml_subject) email subject (normalized index reference). + # h_arr(aml_msg_id) email message-id or msg-id's cross-reference + # see acs_mail_lite_msg_id_map.msg_id + # h_arr(aml_size_chars) size_chars + # h_arr(aml_processed_p) processed_p + + # p_arr($section_id,) acs_mail_lite_ie_parts (content of a part) + # p_arr($section_id,nv_list) acs_mail_lite_part_nv_pairs + # p_arr(section_id_list) list of section_ids + # + # + # where index is section_id based on section_ref, and + # where top most section_ref is a natural number as + # there may be more than one tree. + # + # Specifically, + # for p_arr, content is p_arr($section_id,content) + # c_type is p_arr($section_id,c_type) + # filename is p_arr($section_id,filename) + # c_filepathname is p_arr($section_id,c_filepathname) + # + + + + if { !$error_p } { + + # email goes into queue tables: + + # This data is expected to be available at same moment + + db_transaction { + set id [db_nextval acs_mail_lite_in_id_seq] + + # acs_mail_lite_ie_headers + set h_names_list [array names h_arr] + set to_email_addrs "" + set from_email_addrs "" + set subject "" + set msg_id "" + set size_chars "" + set received_cs "" + # sub set of header names + foreach h_name $h_names_list { + set h_value $h_arr(${h_name}) + switch -nocase -- $h_name { + x-openacs-from - + aml_from_addrs - + from { + if { ![info exists h_arr(aml_from_addrs)] } { + set fr_addrs [acs_mail_lite::parse_email_address \ + -email $h_value ] + set h_arr(aml_from_addrs) $fr_addrs + } else { + set fr_addrs $h_arr(aml_from_addrs) + } + } + x-openacs-to - + aml_to_addrs - + to { + if { ![info exists h_arr(aml_to_addrs)] } { + set h_quoted [ad_quotehtml $h_value] + set h_arr(aml_to) $h_quoted + set to_addrs [acs_mail_lite::parse_email_address \ + -email $h_quoted ] + set h_arr(aml_to_addrs) $to_addrs + } else { + set to_addrs $h_arr(aml_to_addrs) + } + } + aml_msg_id { + set msg_id $h_value + } + x-openacs-subject - + aml_subject - + subject { + set subject $h_value + } + x-openacs-size - + aml_size_chars - + size { + if { ![info exists h_arr(aml_size_chars) ] } { + if { [string is wideinteger -strict $h_value] } { + set size_chars $h_value + } + } else { + set size_chars $h_arr(ams_size_chars) + } + } + aml_received_cs { + set received_cs $h_value + } + aml_priority { + set priority $h_value + } + } + + if { $priority eq "" } { + set priority [dict get \ + [acs_mail_lite::sched_parameters] mpri_max] + } + + db_dml acs_mail_lite_ie_headers_w1 { + insert into acs_mail_lite_ie_headers + (aml_email_id,h_name,h_value) + values (:id,:h_name,:h_value) + } + } + + # acs_mail_lite_from_external + set false 0 + #set processed_p 0 + #set release_p 0 + db_dml acs_mail_lite_from_external_w1 { + insert into acs_mail_lite_from_external + (aml_email_id, + priority, + to_email_addrs, + from_email_addrs, + subject, + msg_id, + size_chars, + received_cs, + processed_p, + release_p) + values (:id, + :priority, + :to_addrs, + :fr_addrs, + :subject, + :msg_id, + :size_chars, + :received_cs, + :false, + :false) + } + + + + set parts_list [list c_type filename content c_filepathname] + foreach section_id $p_arr(section_id_list) { + + # acs_mail_lite_ie_parts + foreach p $parts_list { + set $p "" + if { [info exists p_arr(${section_id},${p}) ] } { + set $p $p_arr(${section_id},${p}) + } + } + db_dml acs_mail_lite_ie_parts_w1 { + insert into acs_mail_lite_ie_parts + (aml_email_id, + section_id, + c_type, + filename, + content, + c_filepathname) + values + (:id, + :section_id, + :c_type, + :filename, + :content, + :c_filepathname) + } + + # acs_mail_lite_ie_part_nv_pairs + foreach {p_name p_value} $p_arr(${section_id},nv_list) { + db_dml acs_mail_lite_ie_part_mv_pairs_w1 { + insert into acs_mail_lite_ie_part_nv_pairs + (aml_email_id, + section_id, + p_name, + p_value) + values + (:id, + :section_id, + :p_name, + :p_value) + } + } + } + + + } on_error { + ns_log Error "acs_mail_lite::inbound_queue_insert \ + Unable to insert email. Headers: '[array get h_arr]' Error: ${errmsg}" + + } + } + return $id +} + + +ad_proc -private acs_mail_lite::inbound_queue_pull { +} { + Identifies and processes highest priority inbound email. +} { + + + # Get scheduling parameters + set start_cs [clock seconds] + # The value of si_dur_per_cycle_s is used + # to keep about 1 inbound_queue_pull active at a time. + # This is an artificial limit. + # For parallel processing of queue, remove this + # scheduling check, and query the queue with each iteration. + # That is, query the queue before processing + # each inbound email to avoid collision of attempts + # to process email more than once. + set si_dur_per_cycle_s \ + [nsv_get acs_mail_lite si_dur_per_cycle_s ] + set stop_cs [expr { $start_cs + int( $si_dur_per_cycle_s * .8 ) } ] + set aml_package_id [apm_package_id_from_key "acs-mail-lite"] + # ct = count + set pull_ct 0 + # sort only what we need. Process in 20 email chunks + set email_max_ct 20 + set pull_p 1 + while { $pull_p && [clock seconds ] < $stop_cs } { + + # ols = ordered lists + set chunk_ols [db_list acs_mail_lite_from_external_rN { + select aml_email_id from acs_mail_lite_from_external + where processed_p <>'1' + and release_p <>'1' + order by priority + limit :email_max_ct } ] + + set chunk_len [llength $chunk_ols] + if { $chunk_len < 1} { + set pull_p 0 + } + set i 0 + while { $i < $chunk_len && $pull_p && [clock seconds ] < $stop_cs } { + array unset h_arr + array unset p_arr + set error_p 0 + set aml_email_id [lindex $chunk_ols $i] + acs_mail_lite::inbound_queue_pull_one \ + -h_array_name h_arr \ + -p_array_name p_arr \ + -aml_email_id $aml_email_id + + set processed_p 0 + set bounced_p [acs_mail_lite::bounce_ministry] + if { !$bounced_p } { + + # following from acs_mail_lite::load_mail + set pot_object_id [lindex [split $h_arr(aml_to_addrs) "@"] 0] + ##code OpenACS Developers: + # object_id@domain is unconventional + # and may break if someone + # uses an email beginning with a number. + # Also, 'from' header could be spoofed.. + # This practice should be deprecated in favor of signed + # acs_mail_lite::unqiue_id_create. + # For emails originating elsewhere, another authentication + # method, such as a pre-signed unique-id in message + # content could be added as well. + # For now, we warn whenver this is used. + if { [ad_var_type_check_number_p $pot_object_id] } { + if { [acs_object::object_p -id h_arr(aml_object_id)] } { + ns_log Warning "acs_mail_lite::inbound_queue_pull \ + Accepted insecure email object_id '${pot_object_id}' \ + array get h_arr '[array get h_arr]'. See code comments." + callback -catch acs_mail_lite::incoming_object_email \ + -array h_arr \ + -object_id $pot_object_id + set processed_p 1 + } + } + if { !$processed_p } { + # Execute all callbacks for this email + + # Forums uses callbacks via notifications + # See callback + # acs_mail_lite::incoming_email -imp notifications + # in notifications/tcl/notification-callback-procs.tcl + # and + # notification::reply::get + # in forums/tcl/forum-reply-procs.tcl + # which is defined in file: + # notifications/tcl/notification-reply-procs.tcl + + #Callback acs_mail_lite::incoming_email bounces everything + # with a user_id. + # Its useful code has been added to + # acs_mail_lite::bounce_ministry. + # A new callback intended to be compatible with + # notification::reply::get (if possible) is invoked here + if { ![info exists h_arr(aml_package_id) ] } { + set h_arr(aml_package_id) $aml_package_id + } + set status [callback -catch acs_mail_lite::email_inbound \ + -header_array_name h_arr \ + -parts_array_name p_arr \ + -package_id $h_arr(aml_package_id) \ + -object_id $h_arr(aml_object_id) \ + -party_id $h_arr(aml_party_id) \ + -other $h_arr(aml_other) \ + -datetime_cs $h_arr(aml_datetime_cs)] + + if { [lsearch -exact $status "0"] > -1 } { + set error_p 1 + } + } + } + + # Email is removed from queue when + # set acs_mail_lite_from_external.processed_p 1. + # Do not release if there was an error. + # set acs_mail_lite_from_external.release_p !$error_p + set not_error_p [expr { ! $error_p } ] + db_dml acs_mail_lite_from_external_wproc { + update acs_mail_lite_from_external + set processed_p='1' + and release_p=:not_error_p + where acs_email_id=:acs_email_id + } + + incr i + } + + } + + return 1 +} + + + +ad_proc -private acs_mail_lite::inbound_queue_pull_one { + -h_array_name:required + -p_array_name:required + -aml_email_id:required + {-mark_processed_p "1"} + {-legacy_array_name ""} +} { + Puts email referenced by aml_email_id from the inbound queue into array + of h_array_name and p_array_name for use by registered callbacks. + + Arrays are repopulated with values in the same manner that + acs_mail_lite::inbounde_queue_insert recieves them. See below for details. + + When complete, marks the email in the queue as processed, + if mark_processed_p is 1. + + Array content corresponds to these tables: + 
+    h_arr($name) $value         acs_mail_lite_ie_headers
+    
+    Some indexes match fields of table acs_mail_lite_from_external:
+
+    h_arr(aml_email_id)     assigned by acs_mail_lite::inbound_queue_insert
+    h_arr(aml_to)           to email including any label
+    h_arr(aml_to_addrs)     to_email_addrs
+    h_arr(aml_from)         from email including any label
+    h_arr(aml_from_addrs)   from_email_addrs
+    h_arr(aml_priority)     priority    
+    h_arr(aml_subject)      email subject (normalized index reference).
+    h_arr(aml_msg_id)       email message-id or msg-id's cross-reference
+                            see acs_mail_lite_msg_id_map.msg_id
+    h_arr(aml_size_chars)   size_chars
+    
+    Some headers are transferred from the email generation process.
+    See acs_mail_lite::unique_id_create for details:
+
+    h_arr(aml_package_id)
+    h_arr(aml_party_id)
+    h_arr(aml_object_id)
+    h_arr(aml_other)
+    
+
+    Some headers are internally generated during input:
+    
+    h_arr(aml_type)         Type of email from acs_mail_lite::email_type
+    h_arr(aml_received_cs)  Time received in seconds since Tcl epoch 
+    h_arr(aml_datetime_cs)  Time unique_id generatd in seconds since Tcl epoch 
+    h_arr(aml_processed_p)  processed_p
+    h_arr(aml_priority)     a priority number assigned to email.
+
+    Email parts (of body) are kept in a separate array:
+
+    p_arr($section_ref,)  acs_mail_lite_ie_parts (content of a part)
+    p_arr($section_ref,nv_list)  acs_mail_lite_part_nv_pairs
+    p_arr(section_ref_list) list of section_refs
+    
+    
+    where index is section_ref based on section_ref, and
+    where top most section_ref is a natural number as
+    there may be more than one tree.
+      
+    Specifically, for p_arr array:
+
+    content        is  p_arr($section_ref,content)
+    c_type         is  p_arr($section_ref,c_type)
+    filename       is  p_arr($section_ref,filename)
+    c_filepathname is  p_arr($section_ref,c_filepathname)
+
+    where:
+    c_type is content-type header
+    filename is filename of an attachment in email
+    c_filepathname is the filepathname within the system.
+
+    Each section may have headers:
+    
+    To avoid any header-name collision with content, c_type etc,
+    headers are supplied in a name_value_list only:
+
+    list of headers by section is  p_arr($section_ref,name_value_list) 
+    list of section_refs       is  p_arr(section_ref_list) 
+
+    For direct compatibility with legacy email systems that used:
+

+ acs_mail_lite::email_parse, a compatible array is passed + to legacy_array_name, if parameter is used. +

+ @see acs_mail_lite::email_parse +} { + upvar 1 $h_array_name h_arr + upvar 1 $p_array_name p_arr + if { $legacy_array_name ne "" } { + upvar 1 $legacy_array_name l_arr + set build_l_arr_p 1 + # Save data in l_arr according to acs_mail_lite::parse_email + # in incoming-mail-procs.tcl + } else { + set build_l_arr_p 0 + } + + # This query may be redundant to some info in acs_mail_lite_ie_headers. + # acs_mail_lite_from_external + set x_list [db_list_of_lists acs_mail_lite_from_external_r1 { + select priority, to_email_addrs, from_email_addrs, + subject, msg_id, + size_chars, received_cs, processed_p, release_p + from acs_mail_lite_from_external + where aml_email_id=:aml_email_id + }] + lassign $x_list h_arr(aml_priority) \ + h_arr(aml_to_email_addrs) \ + h_arr(aml_from_email_addrs) \ + h_arr(aml_subject) \ + h_arr(aml_msg_id) \ + h_arr(aml_size_chars) \ + h_arr(aml_received_cs) \ + h_arr(aml_processed_p) \ + h_arr(aml_release_p) + + # collect from acs_mail_lite_ie_headers + set h_lists [db_list_of_lists acs_mail_lite_ie_headers_r1 { + select h_name, h_value + from acs_mail_lite_ie_headers + where aml_email_id=:aml_email_id } ] + set h_names_ul [list ] + foreach {n v} $h_lists { + set h_arr(${n}) "${v}" + lappend h_names_ul $n + } + + if { $build_l_arr_p } { + set l_headers_ul [array get h_arr] + lappend l_headers_ul message-id $h_arr(aml_msg_id) + lappend l_headers_ul subject $h_arr(aml_subject) + lappend l_headers_ul from $h_arr(aml_from_email_addrs) + lappend l_headers_ul to $h_arr(aml_to_email_addrs) + # provide lowercase of some headers if they exist + set to_lc_list [list date references in-reply-to return-path] + foreach tol $to_lc_list { + set tol_idx [lsearch -exact -nocase $h_names_ul $tol ] + if { $tol > -1 } { + set tol_ref [lindex $h_names_ul $tol_idx] + lappend l_headers_ul $tol $h_arr(${tol_ref}) + } + } + if { $h_arr(received_cs) ne "" } { + lappend l_headers_ul received [clock format $h_arr(received_cs) ] + } + set l_arr(headers) $l_headers_ul + } + + # collect from acs_mail_lite_ie_parts + set p_lists [db_list_of_lists acs_mail_lite_ie_parts_r1 { + select section_id,c_type,filename,content,c_filepathname + from acs_mail_lite_ie_parts + where aml_email_id=:aml_email_id } ] + foreach row $p_lists { + set section_ref [acs_mail_lite::seciton_ref_of [lindex $row 0]] + set p_arr(${section_ref},c_type) [lindex $row 1] + set p_arr(${section_ref},filename) [lindex $row 2] + set p_arr(${section_ref},content) [lindex $row 3] + set p_arr(${section_ref},c_filepathname) [lindex $row 4] + if { $section_ref ni $p_arr(section_ref_list) } { + lappend p_arr(section_ref_list) $section_ref + } + } + # collect from acs_mail_lite_ie_part_nv_pairs + set nvp_lists [db_list_of_lists acs_mail_lite_ie_part_nv_pairs_r1 { + select section_id, p_name, p_value + from acs_mail_lite_ie_part_nv_pairs + where aml_email_id=:aml_email_id } ] + set reserved_fields_ul [list content c_type filename c_filename] + foreach row $nvp_lists { + set section_ref [acs_mail_lite::section_ref_of [lindex $row 0]] + set name [lindex $row 1] + set value [lindex $row 2] + if { $name ni $reserved_fields_ul } { + lappend p_arr(${section_ref},name_value_list) $name $value + } + if { $section_ref ni $p_arr(section_ref_list) } { + lappend p_arr(section_ref_list) $section_ref + } + } + if { $build_l_arr_p } { + # Legacy approach assumes "application/octet-stream" + # for all attachments and + # base64 for encoding of all files. + # + # Encoding has already been handled for files before queing. + + # Legacy approach replaces nested parts with flat list + # from parse_email: + # The bodies consists of a list with two elements: + # content-type and content. + # The files consists of a list with three elements: + # content-type, filename and content. + + set bodies_list [list] + set files_list [list] + set default_encoding [encoding system] + foreach part $p_arr(section_ref_list) { + + lappend bodies_list [list \ + $p_arr(${section_ref},c_type) \ + $p_arr(${section_ref},content) ] + # check for local filename + if { $p_arr(${section_ref},c_filepathname) ne "" } { + # Since this is saved as a file and already decoded, + # guess content_type from file + # instead of assuming content type is same + # as type used in email transport. + set content_type [ns_guesstype $p_arr(${section_ref},c_filepathname)] + + lappend files_list [list \ + $content_type \ + $default_encoding \ + $p_arr(${section_ref},filename) \ + $p_arr(${section_ref},c_filepathname) ] + + } + } + set l_arr(bodies) $bodies_list + set l_arr(files) $files_list + } + + return 1 +} + +ad_proc -private acs_mail_lite::inbound_queue_release { +} { + Delete email from queue that have been flagged 'release'. + + This does not affect email via imap or other connections. + +} { + # To flag 'release', set acs_mail_lite_from_external.release_p 1 + + set aml_ids_list [db_list acs_mail_lite_from_external_rn { + select aml_email_id from acs_mail_lite_from_external + where release_p='1' }] + foreach aml_email_id $aml_ids_list { + db_transaction { + db_dml acs_mail_lite_from_external_dn { + delete from acs_mail_lite_from_external + where aml_email_id=:aml_email_id + } + db_dml acs_mail_lite_ie_headers_dn { + delete from acs_mail_lite_ie_headers + where aml_email_id=:aml_email_id + } + db_dml acs_mail_lite_ie_parts_dn { + delete from acs_mail_lite_ie_parts + where aml_email_id=:aml_email_id + } + db_dml acs_mail_lite_ie_part_nv_pairs_dn { + delete from acs_mail_lite_ie_part_nv_pairs + where aml_email_id=:aml_email_id + } + } on_error { + ns_log Error "acs_mail_lite::inbound_queue_release. \ + Unable to release aml_mail_id '${aml_email_id}'. Error is: ${errmsg}" + } + } + return 1 +} + + +ad_proc -private acs_mail_lite::inbound_filters { + -headers_arr_name +} { + Flags to ignore an inbound email that does not pass filters. + Returns 1 if email should be ignored, otherwise returns 0. + + Headers and values are not alphanumeric case sensitive. + + Inbound filters are dynamically updated via + acs_mail_lite::sched_parameters. + + Instead of rejecting, an email can be filtered to low priority + by using acs_mail_lite::inbound_prioritize parameters + + @see acs_mail_lite::sched_parameters + @see acs_mail_lite::inbound_prioritize +} { + upvar 1 $headers_arr_name h_arr + set reject_p 0 + set headers_list [array names h_arr] + + set p_lists [acs_mail_lite::sched_parameters] + + # For details on these filters, see tables: + # acs_mail_lite_ui.reject_on_hit + # .reject_on_miss + + # h = hit + set h_list [dict values $p_lists reject_on_hit] + set h_names_list [list ] + foreach {n v} $h_list { + set n_idx [lsearch -nocase -exact $headers_list $n] + if { $n_idx > -1 } { + set h [lindex $n_idx] + lappend h_names_list $h + set vh_arr(${h}) $v + } + } + set h_names_ct [llength $h_names_list] + set i 0 + while { !$reject_p && $i < $h_names_ct } { + set h [lindex $h_names_list $i] + if { [string match -nocase $vh_arr(${h}) $h_arr(${h})] } { + set reject_p 1 + } + + incr i + } + + + # m = miss + set m_list [dict values $p_lists reject_on_miss] + set m_names_list [list ] + foreach {n v} $m_list { + set n_idx [lsearch -nocase -exact $headers_list $n] + if { $n_idx > -1 } { + set h [lindex $n_idx] + lappend m_names_list $h + set vm_arr(${h}) $v + } + } + set m_names_ct [llength $m_names_list] + set i 0 + while { !$reject_p && $i < $m_names_ct } { + set h [lindex $m_names_list $i] + if { ![string match -nocase $vm_arr(${h}) $h_arr(${h})] } { + set reject_p 1 + } + + incr i + } + + return $reject_p +} + + +ad_proc -private acs_mail_lite::inbound_cache_clear { +} { + Clears table of all email uids for all history. + All unread input emails will be considered new and reprocessed. + To keep history, just temporarily forget it instead: + append a revision date to acs_mail_lite_email_src_ext_id_map.src_ext +

+ If you are not sure if this will do what you want, try setting + reprocess_old_p to '1'. + @see acs_mail_lite::sched_parameters + +} { + db_dml acs_mail_lite_email_uid_map_d { + update acs_mail_lite_email_uid_id_map { + delete from acs_mail_lite_email_uid_id_map + + } + } + return 1 +} + + +ad_proc -private acs_mail_lite::inbound_cache_hit_p { + email_uid + uidvalidity + mailbox_host_name +} { + Check email unqiue id (UID) against history in table. + If already exists, returns 1 otherwise 0. + Adds checked case to cache if not already there. + + uidvalidity is defined by imap rfc3501 2.3.1.1 + https://tools.ietf.org/html/rfc3501#section-2.3.1.1 + Other protocols have an analog mechanism, or one + can be made locally to be equivallent in use. +} { + set hit_p 0 + set src_ext $mailbox_host_name + append src_ext "-" $uidvalidity + set aml_src_id "" + db_0or1row -cache_key aml_in_src_id_${src_ext} \ + acs_mail_lite_email_src_ext_id_map_r1 { + select aml_src_id from acs_mail_lite_email_src_ext_id_map + where src_ext=:src_ext } + if { $aml_src_id eq "" } { + set aml_src_id [db_nextval acs_mail_lite_in_id_seq] + db_dml acs_mail_lite_email_src_ext_id_map_c1 { + insert into acs_mail_lite_email_src_ext_id_map + (aml_src_id,src_ext) + values (:aml_src_id,:src_ext) + } + } + set aml_email_id "" + db_0or1row acs_mail_lite_email_uid_id_map_r1 { + select aml_email_id from acs_mail_lite_email_uid_id_map + where uid_ext=:email_uid + and src_ext_id=:aml_src_id + } + if { $aml_email_id eq "" } { + set aml_email_id [db_nextval acs_mail_lite_in_id_seq] + db_dml acs_mail_lite_email_uid_id_map_c1 { + insert into acs_mail_lite_email_uid_id_map + (aml_email_id,uid_ext,src_ext_id) + values (:aml_email_id,:email_uid,:aml_src_id) + } + } else { + set hit_p 1 + } + return $hit_p +} + +ad_proc -private acs_mail_lite::section_ref_of { + section_id +} { + Returns section_ref represented by section_id. + Section_id is an integer. + Section_ref has format of counting numbers separated by dot. + First used here by ns_imap body and adopted for general email part refs. + + Defaults to empty string (top level reference and a log warning) + if not found. +} { + set section_ref "" + set exists_p 0 + if { [string is wideinteger -strict $section_id] } { + if { $section_id eq "-1" } { + set exists_p 1 + } else { + + set exists_p [db_0or1row acs_mail_lite_ie_section_ref_map_r_id1 { + select section_ref + from acs_mail_lite_ie_section_ref_map + where section_id=:section_id + } ] + } + } + if { !$exists_p } { + ns_log Warning "acs_mail_lite::section_ref_of '${section_id}' not found." + } + return $section_ref +} + +ad_proc -private acs_mail_lite::section_id_of { + section_ref +} { + Returns section_id representing a section_ref. + Section_ref has format of counting numbers separated by dot. + Section_id is an integer. + First used here by ns_imap body and adopted for general email part refs. +} { + set section_id "" + if { [regexp -- {^[0-9\.]*$} $section_ref ] } { + + if { $section_ref eq "" } { + set section_id -1 + } else { + set ckey aml_section_ref_ + append ckey $section_ref + set exists_p [db_0or1row -cache_key $ckey \ + acs_mail_lite_ie_section_ref_map_r1 { + select section_id + from acs_mail_lite_ie_section_ref_map + where section_ref=:section_ref + } ] + if { !$exists_p } { + db_flush_cache -cache_key_pattern $ckey + set section_id [db_nextval acs_mail_lite_in_id_seq] + db_dml acs_mail_lite_ie_section_ref_map_c1 { + insert into acs_mail_lite_ie_section_ref_map + (section_ref,section_id) + values (:section_ref,:section_id) + } + } + } + } + return $section_id +} + +ad_proc -private acs_mail_lite::unique_id_create { + {-unique_id ""} + {-package_id ""} + {-party_id ""} + {-object_id ""} + {-other ""} +} { + Returns a unique_id for an outbound email header message-id. + Signs unique_id when package_id, party_id, object_id, and/or other info are supplied. party_id is not supplied if its value is empty string or 0. + package_id not supplied when it is the default acs-mail-lite package_id. + If unique_id is empty string, creates a unique_id then processes it. + +} { + # remove quotes, adjust last_at_idx + if { [string match "<*>" $unique_id] } { + set unique_id [string range $unique_id 1 end-1] + } + set envelope_prefix [acs_mail_lite::bounce_prefix ] + if { ![string match "${envelope_prefix}*" $unique_id ] } { + set unique_id2 $envelope_prefix + append unique_id2 $unique_id + set unique_id $unique_id2 + } + set last_at_idx [string last "@" $unique_id] + if { $last_at_idx < 0 } { + set unique_id $envelope_prefix + append unique_id [string range [mime::uniqueID] 1 end-1] + set last_at_idx [string last "@" $unique_id] + } + + set bounce_domain [acs_mail_lite::address_domain] + if { [string range $unique_id $last_at_idx+1 end-1] ne $bounce_domain } { + # Use bounce's address_domain instead + # because message-id may also be used as originator + set unique_id [string range $unique_id 0 $last_at_idx] + append unique_id $bounce_domain + } + + set aml_package_id [apm_package_id_from_key "acs-mail-lite"] + if { ( $package_id ne "" && $package_id ne $aml_package_id ) \ + || ( $party_id ne "" && $party_id ne "0" ) \ + || $object_id ne "" \ + || $other ne "" } { + # Sign this message-id, and map message-id to values + set uid [string range $unique_id 0 $last_at_idx-1] + set domain [string range $unique_id $last_at_idx+1 end] + + set uid_list [split $uid "."] + if { [llength $uid_list] == 3 } { + # Assume this is a unique id from mime::uniqueID + + # Replace clock seconds of uniqueID with a random integer + # since cs is used to build signature, which defeats purpose. + set uid_partial [lindex $uid_list 0] + # Suppose: + # max_chars = at least the same as length of clock seconds + # It will be 10 for a while.. + # so use eleven 9's + # Some cycles are saved by using a constant + append uid_partial "." [randomRange 99999999999] + append uid_partial "." [lindex $uid_list 2] + + set uid $uid_partial + } + + # Just sign the uid part + set max_age [parameter::get -parameter "IncomingMaxAge" \ + -package_id $aml_package_id ] + ns_log Dev "acs_mail_lite::unique_id_create max_age '${max_age}'" + if { $max_age eq "" || $max_age eq "0" } { + # A max_age of 0 or '' expires instantly. + # User expects signature to not expire. + set signed_unique_id_list [ad_sign $uid] + set delim "-" + } else { + set signed_unique_id_list [ad_sign -max_age $max_age $uid] + set delim "+" + } + set signed_unique_id [join $signed_unique_id_list $delim] + + # Since signature is part of uniqueness of unique_id, + # use uid + signature for msg_id + set msg_id $uid + append msg_id "-" $signed_unique_id + + set datetime_cs [clock seconds] + db_dml acs_mail_lite_send_msg_id_map_w1 { + insert into acs_mail_lite_send_msg_id_map + (msg_id,package_id,party_id,object_id,other,datetime_cs) + values (:msg_id,:package_id,:party_id,:object_id,:other,:datetime_cs) + } + set unique_id "<" + append unique_id $msg_id "@" $domain ">" + } + return $unique_id +} + +ad_proc -private acs_mail_lite::unique_id_parse { + -message_id:required +} { + Parses a message-id compatible reference + created by acs_mail_lite::unique_id_create. + Returns package_id, party_id, object_id, other, datetime_cs in a name value list. + + datetime_cs is approximate system time in seconds from epoch when header was created. + + @see acs_mail_lite::unique_id_create +} { + if { [string match "<*>" $message_id] } { + # remove quote which is not part of message id according to RFCs + set message_id [string range $message_id 1 end-1] + } + set return_list [list ] + lassign $return_list package_id party_id object_id other datetime_cs + + set last_at_idx [string last "@" $message_id] + + set domain [string range $message_id $last_at_idx+1 end] + set unique_part [string range $message_id 0 $last_at_idx-1] + set first_dash_idx [string first "-" $unique_part] + + if { $first_dash_idx > -1 } { + # message-id is signed. + ns_log Dev "acs_mail_lite::unique_id_parse message_id '${message_id}'" + set unique_id [string range $unique_part 0 $first_dash_idx-1] + set signature [string range $unique_part $first_dash_idx+1 end] + set sign_list [split $signature "-+"] + + if { [llength $sign_list] == 3 } { + # signature is in good form + # Use the signature's delimiter instead of param IncomingMaxAge + # so that this works even if there is a change in param value + #set aml_package_id /apm_package_id_from_key "acs-mail-lite"/ + #set max_age /parameter::get -parameter "IncomingMaxAge" \ + # -package_id $aml_package_id / + #ns_log Dev "acs_mail_lite::unique_id_parse max_age '${max_age}'" + # if max_age is "" or "0" delim is "-". + # See acs_mail_lite::unique_id_create + if { [string first "-" $signature] } { + # A max_age of 0 or '' expires instantly. + # User expects signature to not expire. + set expiration_cs [ad_verify_signature $unique_id $sign_list] + } else { + + set expiration_cs [ad_verify_signature_with_expr $unique_id $sign_list] + } + if { $expiration_cs > 0 } { + set p_lists [db_list_of_lists \ + acs_mail_lite_send_msg_id_map_r1all { + select package_id, + party_id, + object_id, + other, + datetime_cs + from acs_mail_lite_send_msg_id_map + where msg_id=:unique_part } ] + set p_list [lindex $p_lists 0] + + lassign $p_list package_id party_id object_id other datetime_cs + } else { + ns_log Dev "acs_mail_lite::unique_id_parse unverified signature unique_id '${unique_id}' signature '${sign_list}' expiration_cs '${expiration_cs}'" + } + set bounce_domain [acs_mail_lite::address_domain] + if { $bounce_domain ne $domain } { + ns_log Warning "acs_mail_lite::unique_id_parse \ + message_id '${message_id}' is not from '@${bounce_domain}'" + } + } else { + ns_log Dev "acs_mail_lite::unique_id_parse \ + not in good form signature '${signature}'" + } + } else { + set unique_id $unique_part + set uid_list [split $unique_id "."] + if { [llength $uid_list] == 3 } { + # assume from a mime::uniqueID + set date_time_cs [lindex $uid_list 1] + } else { + set date_time_cs "" + } + + } + set r_list [list \ + package_id $package_id \ + party_id $party_id \ + object_id $object_id \ + other $other \ + datetime_cs $datetime_cs ] + return $r_list +} + + +ad_proc -private acs_mail_lite::inbound_email_context { + -header_array_name + {-header_name_list ""} + +} { + Returns openacs data associated with original outbound email in + the header_array_name and as an ordered list of values: + + package_id, party_id, object_id, other, datetime_cs + + datetime_cs is the time in seconds since tcl epoch. + + other can be most any data represented in sql text. + + By accessing all email headers, various scenarios of OpenACS sender + and replies can be checked to increase likelihood of retrieving + data in context of email. + + Array indexes have suffix aml_ added to index name: + aml_package_id, aml_party_id, aml_object_id, aml_other, aml_datetime_cs + + If a value is not found, an empty string is returned for the value. + + @see acs_mail_lite::unique_id_create + @see acs_mail_lite::unique_id_parse + +} { + upvar 1 $header_array_name h_arr + if { $header_name_list eq "" } { + set header_name_list [array names h_arr] + } + + # Here are some procs that help create a message-id or orginator + # or generated unique ids from inbound email headers + # that are of historical importance in helping + # shape this proc. + # acs_mail_lite::unique_id_create (current) + # acs_mail_lite::unique_id_parse (current) + # acs_mail_lite::generate_message_id + # acs_mail_lite::bounce_address + # acs_mail_lite::parse_bounce_address + # notification::email::reply_address_prefix + # notification::email::reply_address + # notification::email::address_domain + # notification::email::send + # acs_mail_lite::send + # mime::uniqueID + # acs_mail_lite::send_immediately + + + + # This proc should be capable of integrating with MailDir based service + # whether as a legacy support or current choice (instead of IMAP). + + + + # Note for imap paradigm: message-id should be in form: + # + # and unqiue_id should map to + # any package, party and/or object_id so + # as to not leak info unnecessarily. + # See table acs_mail_lite_send_msg_id_map + # and acs_mail_lite::unique_id_create/find/parse + + + # Bounce info needs to be placed in an rfc + # compliant header. Replies can take many forms. + # This could be a mess. + # If a service using MailDir switches to use IMAP, + # should we still try to make the MailDir work? + # Should this work with MailDir regardless of IMAP? + # Yes and yes. + # This should be as generic as possible and include legacy permutations. + + # General constraints: + # Header field characters limited to US-ASCII characters between 33 and 126 + # inclusive per rfc5322 2.2 https://tools.ietf.org/html/rfc5322#section-2.2 + # and white-space characters 32 and 9. + + # Per rfc6532 3.3 and 5322 2.1.1, "Each line of characters must be no more + # than 998 characters, and should be no more than 78 characters.." + # A domain name can take up to 253 characters. + + # Setting aside about 60 characters for a signature for a signed message-id + # should be okay even though it almost guarantees all cases of message_id + # will be over 78 characters. + + # Unique references are case sensitive per rfc3464 2.2.1 + # original email's envelope-id value is case sensitive per rfc3464 2.2.1 + # Angle brackets are used to quote a unique reference + + + # According to RFCs, + # these are the headers to check in a reply indicating original context: + + # original-message-id + # original-envelope-id + # message-id a unique message id per rfc2822 3.6.4 + # assigned by originator per rfc5598 3.4.1 + # https://tools.ietf.org/html/rfc5598#section-3.4.1 + # + # originator A special case alternate to 'From' header. + # Usually defined by first SMTP MTA. + # Notices may be sent to this address when + # a bounce notice to the original email's 'From' + # address bounces. + # See RFC5321 2.3.1 + # https://tools.ietf.org/html/rfc5321#section-2.3.1 + # and RFC5598 2.2.1 + # https://tools.ietf.org/html/rfc5598#section-2.1 + # msg-id + # In-Reply-to space delimited list of unique message ids per rfc2822 3.6.4 + # References space delimited list of unique message ids per rfc2822 3.6.4 + # + # original-recipient may contain original 'to' address of party_id + # original-recipient-address + # is an alternate to original-recipient + # used by rfc3461 4.2 + # https://tools.ietf.org/html/rfc3461#section-4.2 + # Recipient could be used as an extra layer + # of authentication after parsing. + # for example + # 'from' header is built as: + # party::email -party-id user_id + # in page: forums/www/message-email.tcl + # + + # check_list should be prioritized to most likely casees first. + set check_list [list \ + original-message-id \ + original-envelope-id \ + originator \ + message-id \ + msg-id \ + in-reply-to \ + references \ + ] + # + # + # + # existing oacs-5-9 'MailDir' ways to show context or authenticate origin: + # + + + # acs-mail-lite::send_immediately + # 'from' header defaults to acs_mail_lite parameter FixedSenderEmail + # 'Reply-to' defaults to 'from' header value. + # adds a different unique id to 'Return-Path'. + # example: + # address is built using acs_mail_lite::bounce_address + # Parsing is done with: + # acs_mail_lite::parse_bounce_address /acs_mail_lite::parse_email_address/ + # in callback acs_mail_lite::incoming_email -impl acs-mail-lite + # message-id + # Content-ID + # adds same unique id to 'message-id' and 'content-id'. + # example: <17445.1479806245.127@openacs.wu-wien.ac.at.wu-wien.ac.at> + + # Content-ID is added by proc: build_mime_message + # which relies on tcllib mime package + # in file acs-tcl/tcl/html-email-procs.tcl + # message-id is built by acs_mail_lite::generate_message_id + # or mime::uniqueID + # and used in acs_mail_lite::send_immediately + + # acs_mail_lite::generate_message_id: + # return "/clock clicks/./ns_time/.oacs@/address_domain/>" + # mime::uniqueID: + # return "" + # is defined in ns/lib/tcllib1.18/mime/mime.tcl + # mime(cid) is a counter that incriments by one each time called. + + lappend check_list content-id + + + # To make acs_mail_lite_send_msg_id_map more robust, + # should it be designed to import other references via a table map + # so external references can be used? No. + + # Replaced generic use of mime::uniqueID + # with acs_mail_lite::unique_id_create + # Don't assume acs_mail_lite::valid_signature works. It appears to check + # an unknown form and is orphaned (not used). + + + # + # Notifications package + # + # reply-to + # Mail-Followup-To + # parameter NotificationSender defaults to + # remainder@ acs_mail_lite::address_domain + # which defaults to: + # remainder@ parameter BounceDomain + # if set, otherwise to a driver hostname + # which.. + # adds the same unique id to 'reply-to' and 'mail-followup-to' + + # message-id is a way to generate a dynamic reply-to. + + # example: "openacs.org mailer" + # apparently built in notification::email::send + # located in file notifications/tcl/notification-email-procs.tcl + # reply_to built by calling local notification::email::reply_address + # where: + # if $object_id or $type_id is empty string: + #" /address_domain/ mailer \ + # " + # else + # "/address_domain/ mailer \ + # " + # where address_domain gets notifications package parameter EmailDomain + # and defaults to domain from ad_url + # and where reply_address_prefix gets + # notifications package parameter EmailReplyAddressPrefix + # Mail-Followup-To is set to same value, then calls acs_mail_lite::send + + lappend check_list mail-followup-to to + + # Contribute x-envelope-from from legacy case in + # acs_mail_lite::bounce_prefix? + # No. It's only referenced in a proc doc comment. + # lappend check_list x-envelope-from + + + # + # A legacy parameter from acs_mail_lite::parse_bounce_address + # + set bounce_prefix [acs_mail_lite::bounce_prefix] + set regexp_str "\[${bounce_prefix}\]-(\[0-9\]+)-(\[^-\]+)-(\[0-9\]*)\@" + + # + # setup for loop that checks headers + # + + set context_list [list ] + set check_list_len [llength $check_list] + set header_id 0 + set prefix "aml_" + set h_arr(aml_datetime_cs) "" + + # Check headers for signed context + while { $header_id < $check_list_len && $h_arr(aml_datetime_cs) eq "" } { + set header [lindex $check_list $header_id] + set h_idx [lsearch -exact -nocase $header_name_list $header] + if { $h_idx > -1 } { + set h_name [lindex $check_list $h_idx] + + # hv = header value + if { $header eq "references" } { + # references header may contain multiple message-ids + set hv_list [split $h_arr(${h_name}) ] + } else { + # header has one vale + set hv_list [list $h_arr(${h_name})] + } + set hv_list_len [llength $hv_list] + set hv_i 0 + while { $hv_i < $hv_list_len && $h_arr(aml_datetime_cs) eq "" } { + set hv [lindex $hv_list $hv_i] + # remove quoting angle brackets if any + if { [string match "<*>" $hv ] } { + set hv [string range $hv 1 end-1] + } + set context_list [acs_mail_lite::unique_id_parse \ + -message_id $hv] + if { $h_arr(aml_datetime_cs) eq "" \ + && [string match "${bounce_addrs}*" $hv] } { + + + ##code developers of OpenACS core: + # Legacy case should be removed for strict, secure + # handling of context info + + # Check legacy case + # Regexp code is from acs_mail_lite::parse_bounce_address + if { [regexp $regexp_str $hv \ + all user_id signature package_id] } { + set context_list [list \ + package_id $package_id \ + party_id $user_id \ + object_id "" \ + other "" ] + set sig_list [split $signature "."] + set sig_1 [lindex $sig_list 1] + if { [llength $sig_list ] == 3 \ + && [string is wideinteger -strict $sig_1] } { + lappend context_list datetime_cs $sig_1 + } else { + lappend context_list datetime_cs [clock seconds] + } + } + } + # prefix = "aml_" as in cname becomes: + # aml_package_id aml_party_id aml_object_id aml_other aml_datetime_cs + foreach {n v} $context_list { + set cname $prefix + append cname $n + set h_arr(${cname}) $v + } + + incr hv_i + } + } + + incr header_id + } + + return $context_list +} + +ad_proc acs_mail_lite::bounce_ministry { + -header_array_name:required +} { + Check if this email is notifying original email bounced. + If is a bounced notification, process it. + + Returns 1 if bounced or an auto generated reply that + should be ignored, otherwise returns 0 + + Expects header_array to have been previously processed by these procs: + + @see acs_mail_lite::email_type + @see acs_mail_lite::inbound_email_context +} { + upvar 1 $header_array_name h_arr + # This is called ministry, because it is expected to grow in complexity + # as bounce policy becomes more mature. + + # The traditional OpenACS MailDir way: + # code in acs_mail_lite::load_mails + # in which, if there is a bounce, calls: + # acs_mail_lite::record_bounce + # and later batches some admin via + # acs_mail_lite::check_bounces + # This approach likely does not work for + # standard email accounts where a FixedSenderEmail is expected and + # a dynamic (unstatic) email + # would bounce back again and therefore never be reported in system. + + # Specfics of the old way: + # acs_mail_lite::record_bounce which calls: + # acs_mail_lite::bouncing_user_p -user_id $h_arr(aml_user_id) + + # bounces are checked from the inbound queue + # before checking other cases that may trigger callbacks + + + set aml_list [list \ + aml_package_id \ + aml_party_id \ + aml_object_id \ + aml_other \ + aml_type \ + aml_to_addrs \ + aml_from_addrs \ + aml_datetime_cs ] + foreach idx $aml_list { + if { ![info exists h_arr(${idx})] } { + set h_arr(aml_package_id) "" + } + } + + set ignore_p 0 + if { $h_arr(aml_type) ne "" && $h_arr(aml_type) ne "in_reply_to" } { + set ignore_p 1 + # Record bounced email? + set party_id_from_addrs [party::get_by_email \ + -email $h_arr(aml_from_addrs)] + + if { $party_id_from_addrs ne "" } { + set user_id $party_id_from_addrs + if { ![acs_mail_lite::bouncing_user_p -user_id $user_id ] } { + + # Following literally from acs_mail_lite::record_bounce + ns_log Debug "acs_mail_lite::bounce_ministry.2264 \ + Bouncing email from user '${user_id}'" + # record the bounce in the database + db_dml record_bounce {} + if { ![db_resultrows]} { + db_dml insert_bounce {} + } + # end code from acs_mail_lite::record_bounce + + if { $h_arr(aml_party_id) ne $user_id \ + || $h_arr(aml_datetime_cs) eq "" } { + # Log it, because it might be a false positive. + # Existence of aml_datetime_cs means unique_id was signed. + # See acs_mail_lite::unique_id_parse + ns_log Warning "acs_mail_lite::bounce_ministry.2275 \ + Bounced email apparently from user_id '${user_id}' \ + with headers: '[array get h_arr]'" + + } + } + + } else { + # This is probably a bounce, but not from a recognized party + # Log it, because it might help with email related issues. + ns_log Warning "acs_mail_lite::bounce_ministry.2287 \ + email_type '$h_arr(aml_type)' ignored. headers: '[array get h_arr]'" + + } + } + + + return $ignore_p +} + +# +# Local variables: +# mode: tcl +# tcl-indent-level: 4 +# indent-tabs-mode: nil +# End: + Index: openacs-4/packages/acs-mail-lite/tcl/imap-inbound-procs.tcl =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/imap-inbound-procs.tcl,v diff -u -N --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ openacs-4/packages/acs-mail-lite/tcl/imap-inbound-procs.tcl 17 Feb 2018 17:08:31 -0000 1.1 @@ -0,0 +1,877 @@ +ad_library { + + Provides API for importing email via nsimap + + @creation-date 19 Jul 2017 + @cvs-id $Id: imap-inbound-procs.tcl,v 1.1 2018/02/17 17:08:31 gustafn Exp $ + +} + +#package require mime 1.4 ? (no. Choose ns_imap option if available +# at least to avoid tcl's 1024 open file descriptors limit[1]. +# 1. http://openacs.org/forums/message-view?message_id=5370874#msg_5370878 +# base64 and qprint encoding/decoding available via: +# ns_imap encode/decode type data + +namespace eval acs_mail_lite {} + + +ad_proc -private acs_mail_lite::imap_conn_set { + {-host ""} + {-password ""} + {-port ""} + {-timeout ""} + {-user ""} + {-name_mb ""} + {-flags ""} +} { + Returns a name value list of parameters + used by ACS Mail Lite imap connections + + If a parameter is passed with value, the value is assigned to parameter. + + @param name_mb See nsimap documentaion for mailbox.name. + @param port Ignored for now. SSL automatically switches port. +} { + # See one row table acs_mail_lite_imap_conn + # imap_conn_ = ic + set ic_list [list \ + host \ + password \ + port \ + timeout \ + user \ + name_mb \ + flags] + # ic fields = icf + set icf_list [list ] + foreach ic $ic_list { + set icf [string range $ic 0 1] + lappend icf_list $icf + if { [info exists $ic] } { + set new_arr(${ic}) [set $ic] + } + } + set changes_p [array exists new] + set exists_p [db_0or1row acs_mail_lite_imap_conn_r { + select ho,pa,po,ti,us,na,fl + from acs_mail_lite_imap_conn limit 1 + } ] + + if { !$exists_p } { + # set initial defaults + set mb [ns_config nsimap mailbox ""] + set mb_good_form_p [regexp -nocase -- \ + {^[{]([a-z0-9\.\/]+)[}]([a-z0-9\/\ \_]+)$} \ + $mb x ho na] + # ho and na defined by regexp? + set ssl_p 0 + if { !$mb_good_form_p } { + ns_log Notice "acs_mail_lite::imap_conn_set.463. \ + config.tcl's mailbox '${mb}' not in good form. \ + Quote mailbox with curly braces like: {{mailbox.host}mailbox.name} " + set mb_list [acs_mail_lite::imap_mailbox_split $mb] + if { [llength $mb_list] eq 3 } { + set ho [lindex $mb_list 0] + set na [lindex $mb_list 1] + set ssl_p [lindex $mb_list 2] + ns_log Notice "acs_mail_lite::imap_conn_set.479: \ + Used alternate parsing. host '${ho}' mailbox.name '${na}' ssl_p '${ssl_p}'" + } else { + set ho [ns_config nssock hostname ""] + if { $ho eq "" } { + set ho [ns_config nssock_v4 hostname ""] + } + if { $ho eq "" } { + set ho [ns_config nssock_v6 hostname ""] + } + set na "mail/INBOX" + set mb [acs_mail_lite::imap_mailbox_join -host $ho -name $na] + + ns_log Notice "acs_mail_lite::imap_conn_set.482: \ + Using values from nsd config.tcl. host '${ho}' mailbox.name '${na}'" + + } + } + + set pa [ns_config nsimap password ""] + set po [ns_config nsimap port ""] + set ti [ns_config -int nsimap timeout 1800] + set us [ns_config nsimap user ""] + if { $ssl_p } { + set fl "/ssl" + } else { + set fl "" + } + } + + if { !$exists_p || $changes_p } { + set validated_p 1 + set n_pv_list [array names new] + if { $changes_p } { + # new = n + foreach n $n_pv_list { + switch -exact -- $n { + port - + timeout { + if { $n_arr(${n}) eq "" } { + set v_p 1 + } else { + set v_p [string is digit -strict $n_arr(${n})] + if { $v_p } { + if { $n_arr(${n}) < 0 } { + set v_p 0 + } + } + } + } + name_mb - + flags - + host - + password - + user { + if { $n_arr(${n}) eq "" } { + set v_p 1 + } else { + set v_p [regexp -- {^[[:graph:]\ ]+$} $n_arr(${n})] + if { $v_p && \ + [string match {*[\[;]*} $n_arr(${n}) ] } { + set v_p 0 + } + } + } + defaults { + ns_log Warning "acs_mail_lite::imap_conn_set \ + No validation check made for parameter '${n}'" + } + } + if { !$v_p } { + set validated_p 0 + ns_log Warning "acs_mail_lite::imap_conn_set \ + value '$n_arr(${n})' for parameter '${n}' not allowed." + } + } + } + + if { $validated_p } { + foreach ic_n $n_pv_list { + set ${ic_n} $n_arr($ic_n) + } + + db_transaction { + if { $changes_p } { + db_dml acs_mail_lite_imap_conn_d { + delete from acs_mail_lite_imap_conn + } + } + db_dml acs_mail_lite_imap_conn_i { + insert into acs_mail_lite_imap_conn + (ho,pa,po,ti,us,na,fl) + values (:ho,:pa,:po,:ti,:us,:na,:fl) + } + } + } + } + set i_list [list ] + foreach i $ic_list { + set svi [string range $i 0 1] + set sv [set ${svi}] + lappend i_list ${i} $sv + } + return $i_list +} + +ad_proc -private acs_mail_lite::imap_conn_go { + {-conn_id ""} + {-host ""} + {-password ""} + {-port ""} + {-timeout ""} + {-user ""} + {-flags ""} + {-name_mb ""} + {-default_to_inbox_p "0"} + {-default_box_name "inbox"} +} { + Verifies connection (connId) is established. + Tries to establish a connection if it doesn't exist. + If mailbox doesn't exist, tries to find an inbox at root of tree + or as close as possible to it. + + If -host parameter is supplied, will try connection with supplied params. + Defaults to use connection info provided by parameters + via acs_mail_lite::imap_conn_set. + + @param port Ignored for now. SSL automatically switches port. + + @param default_to_inbox_p If set to 1 and name_mb not found, \ + assigns an inbox if found. + @param default_box_name Set if default name for default_to_inbox_p \ + should be something other than inbox. + + @return connectionId or empty string if unsuccessful. + @see acs_mail_lite::imap_conn_set +} { + # imap_conn_go = icg + # imap_conn_set = ics + if { $host eq "" } { + set default_conn_set_p 1 + set ics_list [acs_mail_lite::imap_conn_set ] + foreach {n v} $ics_list { + set $n "${v}" + ns_log Dev "acs_mail_lite::imap_conn_go.596. set ${n} '${v}'" + } + } else { + set default_conn_set_p 0 + } + + set fl_list [split $flags " "] + + set connected_p 0 + set prior_conn_exists_p 0 + + if { $conn_id ne "" } { + # list {id opentime accesstime mailbox} ... + set id "" + set opentime "" + set accesstime "" + set mailbox "" + + set sessions_list [ns_imap sessions] + set s_len [llength $sessions_list] + ns_log Dev "acs_mail_lite::imap_conn_go.612: \ + sessions_list '${sessions_list}'" + # Example session_list as val0 val1 val2 val3 val4 val5 val6..: + #'40 1501048046 1501048046 {{or97.net:143/imap/tls/user="testimap1"}} + # 39 1501047978 1501047978 {{or97.net:143/imap/tls/user="testimap1"}}' + set i 0 + while { $i < $s_len && $id ne $conn_id } { + set s_list [lindex $sessions_list 0] + set id [lindex $s_list 0] + if { $id eq $conn_id } { + set prior_conn_exists_p 1 + set opentime [lindex $s_list 1] + set accesstime [lindex $s_list 2] + set mailbox [lindex $s_list 3] + } + incr i + } + if { $prior_conn_exists_p eq 0 } { + ns_log Warning "acs_mail_lite::imap_conn_go.620: \ + Session broken? conn_id '${conn_id}' not found." + } + } + + if { $prior_conn_exists_p } { + # Test connection. + # status_flags = sf + if { [catch { set sf_list [ns_imap status $conn_id ] } err_txt ] } { + ns_log Warning "acs_mail_lite::imap_conn_go.624 \ + Error connection conn_id '${conn_id}' unable to get status. Broken? \ + Set to retry. Error is: ${err_txt}" + set prior_conn_exists_p 0 + } else { + set connected_p 1 + ns_log Dev "acs_mail_lite::imap_conn_go.640: fl_list '${fl_list}'" + } + } + + if { !$prior_conn_exists_p && $host ne "" } { + if { "ssl" in $fl_list } { + set ssl_p 1 + } else { + set ssl_p 0 + } + set mb [acs_mail_lite::imap_mailbox_join \ + -host $host \ + -name $name_mb \ + -ssl_p $ssl_p] + if { "novalidatecert" in $fl_list } { + if { [catch { set conn_id [ns_imap open \ + -novalidatecert \ + -mailbox "${mb}" \ + -user $user \ + -password $password] \ + } err_txt ] \ + } { ns_log Warning "acs_mail_lite::imap_conn_go.653 \ + Error attempting ns_imap open. Error is: '${err_txt}'" + } else { + set connected_p 1 + ns_log Dev "acs_mail_lite::imap_conn_go.662: \ + new session conn_id '${conn_id}'" + } + } else { + if { [catch { set conn_id [ns_imap open \ + -mailbox "${mb}" \ + -user $user \ + -password $password] \ + } err_txt ] \ + } { ns_log Warning "acs_mail_lite::imap_conn_go.653 \ + Error attempting ns_imap open. Error is: '${err_txt}'" + } else { + set connected_p 1 + ns_log Dev "acs_mail_lite::imap_conn_go.675: \ + new session conn_id '${conn_id}'" + } + } + + } + if { !$connected_p } { + set conn_id "" + } else { + # Check if mailbox exists. + set status_nv_list [ns_imap status $conn_id] + array set stat_arr $status_nv_list + set stat_n_list [array get names stat_arr] + set msg_idx [lsearch -nocase -exact $stat_n_list "messages"] + if { $msg_idx < 0 } { + set mb_exists_p 0 + ns_log Warning "acs_mail_lite::imap_conn_go.723 \ + mailbox name '${name_mb}' not found." + # top level = t + set t_list [ns_imap list $conn_id $host {%}] + ns_log Notice "acs_mail_lite::imap_conn_go.725 \ + available top level mailbox names '${t_list}'" + if { [llength $t_list < 2] && !$default_to_inbox_p } { + # Provide more hints. + set t_list [ns_imap list $conn_id $host {*}] + ns_log Notice "acs_mail_lite::imap_conn_go.727 \ + available mailbox names '${t_list}'" + } + } else { + set mb_exists_p 1 + } + + if { !$mb_exists_p && $default_to_inbox_p } { + set mb_default "" + set idx [lsearch -exact -nocase $t_list "${default_box_name}"] + if { $idx < 0 } { + set idx [lsearch -glob -nocase $t_list "${default_box_name}*"] + } + if { $idx < 0 } { + set idx [lsearch -glob -nocase $t_list "*${default_box_name}*"] + } + if { $idx < 0 } { + set t_list [ns_imap list $conn_id $mailbox_host {*}] + set idx_list \ + [lsearch -glob -nocase $t_list "*${default_box_name}*"] + set i_pos_min 999 + # find inbox closest to tree root + foreach mb_q_idx $idx_list { + set mb_q [lindex $tv_list $mb_q_idx] + set i_pos [string first ${default_box_name} \ + [string tolower $mb_q]] + if { $idx < 0 || $i_pos < $i_pos_min } { + set i_pos_min $i_pos + set idx $mb_q_idx + } + } + + } + # choose a box closest to tree root. + if { $idx > -1 } { + set mb_default [lindex $t_list $idx] + if { $default_conn_set_p } { + ns_log Notice "acs_mail_lite::imap_conn_go.775 \ + Setting default mailbox.name to '${mb_default}'" + acs_mail_lite::imap_conn_set -name_mb $mb_default + } + set mb [acs_mail_lite::imap_mailbox_join \ + -host $host \ + -name $name_mb \ + -ssl_p $ssl_p] + if { "novalidatecert" in $fl_list } { + set conn_id [ns_imap reopen \ + -novalidatecert \ + -mailbox "${mb}" \ + -user $user \ + -password $password] + } else { + set conn_id [ns_imap open \ + -mailbox "${mb}" \ + -user $user \ + -password $password] + } + } + } + + } + return $conn_id +} + + +ad_proc -public acs_mail_lite::imap_conn_close { + {-conn_id:required } +} { + Closes nsimap session with conn_id. + If conn_id is 'all', then all open sessions are closed. + + Returns 1 if a session is closed, otherwise returns 0. +} { + set sessions_list [ns_imap sessions] + set s_len [llength $sessions_list] + ns_log Dev "acs_mail_lite::imap_conn_close.716: \ + sessions_list '${sessions_list}'" + # Example session_list as val0 val1 val2 val3 val4 val5 val6..: + #'40 1501048046 1501048046 {{or97.net:143/imap/tls/user="testimap1"}} + # 39 1501047978 1501047978 {{or97.net:143/imap/tls/user="testimap1"}}' + set id "" + set i 0 + set conn_exists_p 0 + while { $i < $s_len && $id ne $conn_id } { + set id [lindex [lindex $sessions_list 0] 0] + if { $id eq $conn_id || $conn_id eq "all" } { + set conn_exists_p 1 + ns_log Dev "acs_mail_lite::imap_conn_close.731 session_id '${id}'" + if { [catch { ns_imap close $id } err_txt ] } { + ns_log Warning "acs_mail_lite::imap_conn_close.733 \ + session_id '${id}' error on close. Error is: ${err_txt}" + } + } + incr i + } + if { $conn_exists_p eq 0 } { + ns_log Warning "acs_mail_lite::imap_conn_close.732: \ + Session(s) broken? conn_id '${conn_id}' not found." + } + return $conn_exists_p +} + +ad_proc -public acs_mail_lite::imap_mailbox_join { + {-host ""} + {-name ""} + {-ssl_p "0"} +} { + Creates an ns_imap usable mailbox consisting of curly brace quoted + {mailbox.host}mailbox.name. +} { + # Quote mailbox with curly braces per nsimap documentation. + set mb "{" + append mb ${host} + if { [string is true -strict $ssl_p] && ![string match {*/ssl} $host] } { + append mb {/ssl} + } + append mb "}" ${name} + + return $mb +} + +ad_proc -public acs_mail_lite::imap_mailbox_split { + {mailbox ""} +} { + Returns a list: mailbox.host mailbox.name ssl_p, + where mailbox.host and mailbox.name are defined in ns_map documentation. + If mailbox.host has suffix "/ssl", suffix is removed and ssl_p is "1", + otherwise ssl_p is "0". + + If mailbox cannot be parsed, returns an empty list. +} { + set cb_idx [string first "\}" $mailbox] + if { $cb_idx > -1 && [string range $mailbox 0 0] eq "\{" } { + set ho [string range $mailbox 1 $cb_idx-1] + set na [string range $mailbox $cb_idx+1 end] + if { [string match {*/ssl} $ho ] } { + set ssl_p 1 + set ho [string range $ho 0 end-4] + } else { + set ssl_p 0 + } + set mb_list [list $ho $na $ssl_p] + } else { + # Not a mailbox + set mb_list [list ] + } + return $mb_list +} + +ad_proc -private acs_mail_lite::imap_check_incoming { +} { + Checks for new, actionable incoming email via imap connection. + Email is actionable if it is identified by acs_mail_lite::email_type. + + When actionable, email is buffered in table acs_mail_lite_from_external + and callbacks are triggered. + + @see acs_mail_lite::email_type + +} { + set error_p 0 + if { [nsv_exists acs_mail_lite si_configured_p ] } { + set si_configured_p [nsv_get acs_mail_lite si_configured_p] + } else { + set si_configured_p 1 + # Try to connect at least once + } + # This proc is called by ad_schedule_proc regularly + + # scan_in_ = scan_in_est_ = scan_in_estimate = si_ + if { $si_configured_p } { + set cycle_start_cs [clock seconds] + nsv_lappend acs_mail_lite si_actives_list $cycle_start_cs + set si_actives_list [nsv_get acs_mail_lite si_actives_list] + + set si_dur_per_cycle_s \ + [nsv_get acs_mail_lite si_dur_per_cycle_s] + set per_cycle_s_override [nsv_get acs_mail_lite \ + si_dur_per_cycle_s_override] + set si_quit_cs \ + [expr { $cycle_start_cs + int( $si_dur_per_cycle_s \ + * .8 ) } ] + if { $per_cycle_s_override ne "" } { + set si_quit_cs [expr { $si_quit_cs - $per_cycle_s_override } ] + # deplayed + } else { + set per_cycle_s_override $si_dur_per_cycle_s + } + + + set active_cs [lindex $si_actives_list end] + set concurrent_ct [llength $si_actives_list] + # pause is in seconds + set pause_s 10 + set pause_ms [expr { $pause_s * 1000 } ] + while { $active_cs eq $cycle_start_cs \ + && [clock seconds] < $si_quit_cs \ + && $concurrent_ct > 1 } { + + incr per_cycle_s_override $pause_s + nsv_set acs_mail_lite si_dur_per_cycle_s_override \ + $per_cycle_s_override + set si_actives_list [nsv_get acs_mail_lite si_actives_list] + set active_cs [lindex $si_actives_list end] + set concurrent_ct [llength $si_actives_list] + ns_log Notice "acs_mail_lite::imap_check_incoming.1198. \ + pausing ${pause_s} seconds for prior invoked processes to stop. \ + si_actives_list '${si_actives_list}'" + after $pause_ms + } + + if { [clock seconds] < $si_quit_cs \ + && $active_cs eq $cycle_start_cs } { + + set cid [acs_mail_lite::imap_conn_go ] + if { $cid eq "" } { + set error_p 1 + } + + if { !$error_p } { + + array set conn_arr [acs_mail_lite::imap_conn_set] + unset conn_arr(password) + set mailbox_host_name "{{" + append mailbox_host_name $conn_arr(host) "}" \ + $conn_arr(name_mb) "}" + + set status_list [ns_imap status $cid] + if { ![f::even_p [llength $status_list]] } { + lappend status_list "" + } + array set status_arr $status_list + set uidvalidity $status_arr(Uidvalidity) + if { [info exists status_arr(Uidnext) ] \ + && [info exists status_arr(Messages) ] } { + + set aml_package_id [apm_package_id_from_key "acs-mail-lite"] + set filter_proc [parameter::get -parameter "IncomingFilterProcName" \ + -package_id $aml_package_id] + # + # Iterate through emails + # + # ns_imap search should be faster than ns_imap sort + set m_list [ns_imap search $cid ""] + + foreach msgno $m_list { + set struct_list [ns_imap struct $cid $msgno] + + # add struct info to headers for use with ::email_type + # headers_arr = hdrs_arr + array set hdrs_arr $struct_list + set uid $hdrs_arr(uid) + + set processed_p [acs_mail_lite::inbound_cache_hit_p \ + $uid \ + $uidvalidity \ + $mailbox_host_name ] + + if { !$processed_p } { + set headers_list [ns_imap headers $cid $msgno] + array set hdrs_arr $headers_list + + set type [acs_mail_lite::email_type \ + -header_arr_name hdrs_arr ] + + + # Create some standardized header indexes aml_* + # with corresponding values + set size_idx [lsearch -nocase -exact \ + $headers_list size] + set sizen [lindex $headers_list $size_idx] + if { $sizen ne "" } { + set hdrs_arr(aml_size_chars) $hdrs_arr(${sizen}) + } else { + set hdrs_arr(aml_size_chars) "" + } + + if { [info exists hdrs_arr(received_cs)] } { + set hdrs_arr(aml_received_cs) $hdrs_arr(received_cs) + } else { + set hdrs_arr(aml_received_cs) "" + } + + set su_idx [lsearch -nocase -exact \ + $headers_list subject] + if { $su_idx > -1 } { + set sun [lindex $headers_list $su_idx] + set hdrs_arr(aml_subject) [ad_quotehtml $hdrs_arr(${sun})] + } else { + set hdrs_arr(aml_subject) "" + } + + set to_idx [lsearch -nocase -exact \ + $headers_list to] + if { ${to_idx} > -1 } { + set ton [lindex $headers_list $to_idx] + set hdrs_arr(aml_to) [ad_quotehtml $hdrs_arr(${ton}) ] + } else { + set hdrs_arr(aml_to) "" + } + + acs_mail_lite::inbound_email_context \ + -header_array_name hdrs_arr \ + -header_name_list $headers_list + + acs_mail_lite::inbound_prioritize \ + -header_array_name hdrs_arr + + set error_p [acs_mail_lite::imap_email_parse \ + -headers_arr_name hdrs_arr \ + -parts_arr_name parts_arr \ + -conn_id $cid \ + -msgno $msgno \ + -struct_list $struct_list] + + if { !$error_p && [string match {[a-z]*_[a-z]*} $filter_proc] } { + set hdrs_arr(aml_package_ids_list) [safe_eval ${filter_proc}] + } + if { !$error_p } { + + set id [acs_mail_lite::inbound_queue_insert \ + -parts_arr_name parts_arr + \ + -headers_arr_name hdrs_arr \ + -error_p $error_p ] + ns_log Notice "acs_mail_lite::imap_check_incoming \ + inserted to queue aml_email_id '${id}'" + } + + } + } + } else { + ns_log Warning "acs_mail_lite::imap_check_incoming.1274. \ + Unable to process email. \ + Either Uidnext or Messages not in status_list: '${status_list}'" + } + + if { [expr { [clock seconds] + 65 } ] < $si_quit_cs } { + # Regardless of parameter SMPTTimeout, + # if there is more than 65 seconds to next cycle, + # close connection + acs_mail_lite::imap_conn_close -conn_id $cid + } + + } + # end if !$error + + # remove active_cs from si_actives_list + set si_idx [lsearch -integer -exact $si_actives_list $active_cs] + # We call nsv_get within nsv_set to reduce chances of dropping + # a new list entry. + nsv_set acs_mail_lite si_actives_list \ + [lreplace \ + [nsv_get acs_mail_lite si_actives_list] $si_idx $si_idx] + + } else { + nsv_set acs_mail_lite si_configured_p 0 + } + # acs_mail_lite::imap_check_incoming should quit gracefully + # when not configured or there is error on connect. + + } + return $si_configured_p +} + +ad_proc -private acs_mail_lite::imap_email_parse { + -headers_arr_name + -parts_arr_name + -conn_id + -msgno + -struct_list + {-section_ref ""} + {-error_p "0"} +} { + Parse an email from an imap connection into array array_name + for adding to queue via acs_mail_lite::inbound_queue_insert + + Parsed data is set in headers and parts arrays in calling environment. + + struct_list expects output list from ns_imap struct conn_id msgno +} { + # Put email in a format usable for + # acs_mail_lite::inbound_queue_insert to insert into queue + + # for format this proc is to generate. + + # Due to the hierarchical nature of email and ns_imap struct + # this proc is recursive. + upvar 1 $headers_arr_name h_arr + upvar 1 $parts_arr_name p_arr + upvar 1 __max_txt_bytes __max_txt_bytes + set has_parts_p 0 + set section_n_v_list [list ] + if { ![info exists __max_txt_bytes] } { + set sp_list [acs_mail_lite::sched_parameters] + set __max_txt_bytes [dict get $sp_list max_blob_chars] + } + if { !$error_p } { + + if { [string range $section_ref 0 0] eq "." } { + set section_ref [string range $section_ref 1 end] + } + ns_log Dev "acs_mail_lite::imap_email_parse.706 \ +msgno '${msgno}' section_ref '${section_ref}'" + + # Assume headers and names are unordered + + foreach {n v} $struct_list { + if { [string match {part.[0-9]*} $n] } { + set has_parts_p 1 + set subref $section_ref + append subref [string range $n 4 end] + acs_mail_lite::imap_email_parse \ + -headers_arr_name h_arr \ + -parts_arr_name p_arr \ + -conn_id $conn_id \ + -msgno $msgno \ + -struct_list $v \ + -section_ref $subref + } else { + switch -exact -nocase -- $n { + bytes { + set bytes $v + } + disposition.filename { + regsub -all -nocase -- {[^0-9a-zA-Z\-.,\_]} $v {_} v + set filename $v + } + type { + set type $v + } + + default { + # do nothing + } + } + if { $section_ref eq "" } { + set h_arr(${n}) ${v} + } else { + lappend section_n_v_list ${n} ${v} + } + } + } + + if { $section_ref eq "" && !$has_parts_p } { + # section_ref defaults to '1' + set section_ref "1" + } + + set section_id [acs_mail_lite::section_id_of $section_ref] + ns_log Dev "acs_mail_lite::imap_email_parse.746 \ +msgno '${msgno}' section_ref '${section_ref}' section_id '${section_id}'" + + # Add content of an email part + set p_arr(${section_id},nv_list) $section_n_v_list + set p_arr(${section_id},c_type) $type + lappend p_arr(section_id_list) ${section_id} + + if { [info exists bytes] && $bytes > $__max_txt_bytes \ + && ![info exists filename] } { + set filename "blob.txt" + } + + if { [info exists filename ] } { + set filename2 [clock microseconds] + append filename2 "-" $filename + set filepathname [file join [acs_root_dir] \ + acs-mail-lite \ + $filename2 ] + set p_arr(${section_id},filename) $filename + set p_arr(${section_id},c_filepathname) $filepathname + if { $filename eq "blob.txt" } { + ns_log Dev "acs_mail_lite::imap_email_parse.775 \ + ns_imap body '${conn_id}' '${msgno}' '${section_ref}' \ + -file '${filepathname}'" + ns_imap body $conn_id $msgno ${section_ref} \ + -file $filepathname + } else { + ns_log Dev "acs_mail_lite::imap_email_parse.780 \ + ns_imap body '${conn_id}' '${msgno}' '${section_ref}' \ + -file '${filepathname}' -decode" + + ns_imap body $conn_id $msgno ${section_ref} \ + -file $filepathname \ + -decode + } + } elseif { $section_ref ne "" } { + # text content + set p_arr(${section_id},content) [ns_imap body $conn_id $msgno $section_ref] + ns_log Dev "acs_mail_lite::imap_email_parse.792 \ + text content '${conn_id}' '${msgno}' '${section_ref}' \ + $p_arr(${section_id},content)'" + + } else { + set p_arr(${section_id},content) "" + # The content for this case + # has been verified to be redundant. + # It is mostly the last section/part of message. + # + # If diagnostics urge examining these cases, + # Set debug_p 1 to allow the following code to + # to compress a message to recognizable parts without + # flooding the log. + set debug_p 0 + if { $debug_p } { + set msg_txt [ns_imap text $conn_id $msgno ] + # 72 character wide lines * x lines + set msg_start_max [expr { 72 * 20 } ] + set msg_txtb [string range $msg_txt 0 $msg_start_max] + if { [string length $msg_txt] \ + > [expr { $msg_start_max + 400 } ] } { + set msg_txte [string range $msg_txt end-$msg_start_max end] + } elseif { [string length $msg_txt] \ + > [expr { $msg_start_max + 144 } ] } { + set msg_txte [string range $msg_txt end-144 end] + } else { + set msg_txte "" + } + ns_log Dev "acs_mail_lite::imap_email_parse.818 IGNORED \ + ns_imap text '${conn_id}' '${msgno}' '${section_ref}' \n \ + msg_txte '${msg_txte}'" + } else { + ns_log Dev "acs_mail_lite::imap_email_parse.822 ignored \ + ns_imap text '${conn_id}' '${msgno}' '${section_ref}'" + } + } + + } + return $error_p +} + + +# +# Local variables: +# mode: tcl +# tcl-indent-level: 4 +# indent-tabs-mode: nil +# End: + + Index: openacs-4/packages/acs-mail-lite/tcl/incoming-mail-procs.tcl =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/Attic/incoming-mail-procs.tcl,v diff -u -N -r1.8 -r1.9 --- openacs-4/packages/acs-mail-lite/tcl/incoming-mail-procs.tcl 7 Aug 2017 23:47:57 -0000 1.8 +++ openacs-4/packages/acs-mail-lite/tcl/incoming-mail-procs.tcl 17 Feb 2018 17:08:31 -0000 1.9 @@ -15,24 +15,36 @@ #--------------------------------------- ad_proc -public address_domain {} { - @return domain address to which bounces are directed to + @return domain address to which bounces are directed to. + If empty, uses domain from FixedSenderEmail parameter, + otherwise the hostname in config.tcl is used. } { - set domain [parameter::get_from_package_key -package_key "acs-mail-lite" -parameter "BounceDomain"] + set domain [parameter::get_from_package_key \ + -package_key "acs-mail-lite" \ + -parameter "BounceDomain"] if { $domain eq "" } { - # - # If there is no domain configured, use the configured - # hostname as domain name - # - foreach driver {nsssl nssock} { - set driver_section [ns_driversection -driver $driver] - set configured_hostname [ns_config $driver_section hostname] - if {$configured_hostname ne ""} { - set domain $configured_hostname - break + # Assume a FixedSenderEmail domain, if it exists. + set email [parameter::get_from_package_key \ + -package_key "acs-mail-lite" \ + -parameter "FixedSenderEmail"] + if { $email ne "" } { + set domain [string range $email [string last "@" $email]+1 end] + } else { + # + # If there is no domain configured, use the configured + # hostname as domain name + # + foreach driver {nsssl nssock_v4 nssock_v6 nssock} { + set section [ns_driversection -driver $driver] + set configured_hostname [ns_config $section hostname] + if {$configured_hostname ne ""} { + set domain $configured_hostname + break + } } } - } - return $domain + } + return $domain } @@ -119,7 +131,7 @@ } #let's delete the file now - if {[catch {file delete -- $msg} errmsg]} { + if {[catch {file delete $msg} errmsg]} { ns_log Error "load_mails: unable to delete queued message $msg: $errmsg" } else { ns_log Debug "load_mails: deleted $msg" @@ -134,7 +146,7 @@ } { An email is splitted into several parts: headers, bodies and files lists and all headers directly. - The headers consists of a list with header names as keys and their corresponding values. All keys are lower case. + The headers consists of a list with header names as keys and their correponding values. All keys are lower case. The bodies consists of a list with two elements: content-type and content. The files consists of a list with three elements: content-type, filename and content. @@ -184,7 +196,7 @@ set content [read $stream] close $stream ns_log error $content - file delete -- $file + file delete $file return } @@ -280,6 +292,8 @@ @param from From address which will be checked if it is coming from a mailer daemon @return 1 if this is actually an autoreply + + @See acs_mail_lite::email_type } { set autoreply_p 0 if {$subject ne ""} { Index: openacs-4/packages/acs-mail-lite/tcl/maildir-inbound-procs.tcl =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/maildir-inbound-procs.tcl,v diff -u -N --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ openacs-4/packages/acs-mail-lite/tcl/maildir-inbound-procs.tcl 17 Feb 2018 17:08:31 -0000 1.1 @@ -0,0 +1,442 @@ +ad_library { + + Provides API for importing email via postfix maildir + + @creation-date 12 Oct 2017 + @cvs-id $Id: maildir-inbound-procs.tcl,v 1.1 2018/02/17 17:08:31 gustafn Exp $ + +} + +namespace eval acs_mail_lite {} + +ad_proc -private acs_mail_lite::maildir_check_incoming { +} { + Checks for new, actionable incoming email via Postfix MailDir standards. + Email is actionable if it is identified by acs_mail_lite::email_type. + + When actionable, email is buffered in table acs_mail_lite_from_external + and callbacks are triggered. + + @see acs_mail_lite::email_type + +} { + set error_p 0 + set mail_dir_fullpath [acs_mail_lite::mail_dir] + if { $mail_dir_fullpath ne "" } { + + + set newdir [file join $mail_dir_fullpath new "*"] + set curdir [file join $mail_dir_fullpath cur "."] + + set messages_list [glob -nocomplain $newdir] + + # only one of acs_mail_lite::maildir_check_incoming process at a time. + set cycle_start_cs [clock seconds] + nsv_lappend acs_mail_lite sj_actives_list $cycle_start_cs + set sj_actives_list [nsv_get acs_mail_lite sj_actives_list] + ns_log Notice "acs_mail_lite::maildir_check_incoming.37. start \ + sj_actives_list '${sj_actives_list}'" + + set active_cs [lindex $sj_actives_list end] + set concurrent_ct [llength $sj_actives_list] + # pause is in seconds + set pause_s 10 + set pause_ms [expr { $pause_s * 1000 } ] + while { $active_cs eq $cycle_start_cs \ + && $concurrent_ct > 1 } { + set sj_actives_list [nsv_get acs_mail_lite sj_actives_list] + set active_cs [lindex $sj_actives_list end] + set concurrent_ct [llength $sj_actives_list] + ns_log Notice "acs_mail_lite::maildir_check_incoming.1198. \ + pausing ${pause_s} seconds for prior invoked processes to stop. \ + sj_actives_list '${sj_actives_list}'" + after $pause_ms + } + + if { $active_cs eq $cycle_start_cs } { + + set aml_package_id [apm_package_id_from_key "acs-mail-lite"] + set filter_proc [parameter::get -parameter "IncomingFilterProcName" \ + -package_id $aml_package_id] + # + # Iterate through emails + # + foreach msg $messages_list { + set error_p [acs_mail_lite::maildir_email_parse \ + -headers_arr_name hdrs_arr \ + -parts_arr_name parts_arr \ + -message_fpn $msg] + if { $error_p } { + ns_log Notice "acs_mail_lite::maildir_check_incoming \ + could not process message file '${msg}'. Messaged moved to MailDir/cur/." + # Move the message into MailDir/cur for other mail reader + file copy $msg $curdir + file delete $msg + + } else { + # process email + + set uid $hdrs_arr(uid) + set uidvalidity [file mtime $mail_dir_fullpath] + set processed_p [acs_mail_lite::inbound_cache_hit_p \ + $uid \ + $uidvalidity \ + $mail_dir_fullpath ] + + if { !$processed_p } { + + set type [acs_mail_lite::email_type \ + -header_arr_name hdrs_arr ] + + set headers_list [array names hdrs_arr] + # Create some standardized header indexes aml_* + # with corresponding values + set size_idx [lsearch -nocase -exact \ + $headers_list size] + set sizen [lindex $headers_list $size_idx] + if { $sizen ne "" } { + set hdrs_arr(aml_size_chars) $hdrs_arr(${sizen}) + } else { + set hdrs_arr(aml_size_chars) "" + } + + if { [info exists hdrs_arr(received_cs)] } { + set hdrs_arr(aml_received_cs) $hdrs_arr(received_cs) + } else { + set hdrs_arr(aml_received_cs) "" + } + + set su_idx [lsearch -nocase -exact \ + $headers_list subject] + if { $su_idx > -1 } { + set sun [lindex $headers_list $su_idx] + set hdrs_arr(aml_subject) [ad_quotehtml $hdrs_arr(${sun})] + } else { + set hdrs_arr(aml_subject) "" + } + + set to_idx [lsearch -nocase -exact \ + $headers_list to] + if { ${to_idx} > -1 } { + set ton [lindex $headers_list $to_idx] + set hdrs_arr(aml_to) [ad_quotehtml $hdrs_arr(${ton}) ] + } else { + set hdrs_arr(aml_to) "" + } + + acs_mail_lite::inbound_email_context \ + -header_array_name hdrs_arr \ + -header_name_list $headers_list + + acs_mail_lite::inbound_prioritize \ + -header_array_name hdrs_arr + + if { [string match {[a-z]*_[a-z]*} $filter_proc] } { + set hdrs_arr(aml_package_ids_list) [safe_eval ${filter_proc}] + } + + set id [acs_mail_lite::inbound_queue_insert \ + -parts_arr_name parts_arr + \ + -headers_arr_name hdrs_arr \ + -error_p $error_p ] + ns_log Notice "acs_mail_lite::maildir_check_incoming \ + inserted to queue aml_email_id '${id}'" + } + + # Move the message into MailDir/cur for other mail handling + file copy $msg $curdir + file delete $msg + } + } + } + # remove active_cs from sj_actives_list + set sj_idx [lsearch -integer -exact $sj_actives_list $cycle_start_cs] + # We call nsv_get within nsv_set to reduce chances of dropping + # a new list entry. + nsv_set acs_mail_lite sj_actives_list \ + [lreplace [nsv_get acs_mail_lite sj_actives_list] $sj_idx $sj_idx] + ns_log Notice "acs_mail_lite::maildir_check_incoming.199. stop \ + sj_actives_list '${sj_actives_list}'" + ns_log Dev "acs_mail_lite::maildir_check_incoming.200. nsv_get \ + acs_mail_lite sj_actives_list '[nsv_get acs_mail_lite sj_actives_list]'" + } + # end if !$error + + return 1 +} + + +ad_proc -private acs_mail_lite::maildir_email_parse { + -headers_arr_name + -parts_arr_name + {-message_fpn ""} + {-part_id ""} + {-section_ref ""} + {-error_p "0"} +} { + Parse an email from a Postfix maildir into array array_name + for adding to queue via acs_mail_lite::inbound_queue_insert +

+ Parsed data is set in headers and parts arrays in calling environment. + @param message_fpn is absolute file path and name of one message +} { + # Put email in a format usable for + # acs_mail_lite::inbound_queue_insert to insert into queue + + # We have to generate the references for MailDir.. + + # 
+    # Most basic example of part reference:
+    # ref    # part
+    # 1    #   message text only
+
+    # More complex example. Order is not enforced, only hierarchy.
+    # ref    # part
+    # 1    #   multipart message
+    # 1.1    # part 1 of ref 1
+    # 1.2    # part 2 of ref 1
+    # 4    #   part 1 of ref 4
+    # 3.1    # part 1 of ref 3
+    # 3.2    # part 2 of ref 3
+    # 3.5    # part 5 of ref 3
+    # 3.3    # part 3 of ref 3
+    # 3.4    # part 4 of ref 3
+    # 2    #   part 1 of ref 2
+
+    # Due to the hierarchical nature of email, this proc is recursive.
+    # To see examples of struct list to build, see www/doc/imap-notes.txt
+    # and www/doc/maildir-test.tcl
+    # reference mime procs:
+    # https://www.tcl.tk/community/tcl2004/Tcl2003papers/kupries-doctools/tcllib.doc/mime/mime.html
+
+    upvar 1 $headers_arr_name h_arr
+    upvar 1 $parts_arr_name p_arr
+    upvar 1 __max_txt_bytes __max_txt_bytes
+    set has_parts_p 0
+    set section_n_v_list [list ]
+    # rfc 822 date time format regexp expression
+    set re822 {[^a-z]([a-z][a-z][a-z][ ,]+[0-9]+ [a-z][a-z][a-z][ ]+[0-9][0-9][0-9][0-9][ ]+[0-9][0-9][:][0-9][0-9][:][0-9][0-9][ ]+[\+\-][0-9]+)[^0-9]}
+
+    if { ![info exists __max_txt_bytes] } {
+        set sp_list [acs_mail_lite::sched_parameters]
+        set __max_txt_bytes [dict get $sp_list max_blob_chars]
+    }
+    if { $message_fpn ne "" } {
+        if {[catch {set m_id [mime::initialize -file ${message_fpn}] } errmsg] } {
+            ns_log Error "maildir_email_parse.71 could not parse \
+ message file '${message_fpn}' error: '${errmsg}'"
+            set error_p 1
+        } else {
+            # For acs_mail_lite::inbond_cache_hit_p, 
+            # make a uid if there is not one. 
+            set uid_ref ""
+            # Do not use email file's tail, 
+            # because tail is unique to system not email.
+            # See http://cr.yp.to/proto/maildir.html
+            
+            # A header returns multiple values in a list
+            # if header name is repeated in email.
+            set h_list [mime::getheader $m_id]
+            # headers_list 
+            set headers_list [list ]
+            foreach {h v} $h_list {
+                switch -nocase -- $h {
+                    uid {
+                        if { $h ne "uid" } {
+                            lappend struct_list "uid" $v
+                        }
+                        set uid_ref "uid"
+                        set uid_val $v
+                    }
+                    message-id -
+                    msg-id {
+                        if { $uid_ref ne "uid"} {
+                            if { $uid_ref ne "message-id" } {
+                                # message-id is not required
+                                # msg-id is an alternate 
+                                # Fallback to most standard uid
+                                set uid_ref [string tolower $h]
+                                set uid_val $v
+                            }
+                        }
+                    }
+                    received {
+                        if { [llength $v ] > 1 } {
+                            set v0 [lindex $v 0]
+                        } else {
+                            set v0 $v
+                        }
+                        if { [regexp -nocase -- $re822 $v0 match r_ts] } {
+                            set age_s [mime::parsedatetime $r_ts rclock]
+                            set dt_cs [expr { [clock seconds] - $age_s } ]
+                            lappend headers_list "aml_datetime_cs" $dt_cs
+                        }
+                    }
+                    default { 
+                        # do nothing
+                    }
+                }
+                lappend headers_list $h $v
+            }
+            lappend headers_list "aml_received_cs" [file mtime ${message_fpn}]
+            lappend headers_list "uid" $uid_val
+            
+            # Append property_list to to headers_list
+            set prop_list [mime::getproperty $m_id]
+            #set prop_names_list /mime::getproperty $m_id -names/
+            foreach {n v} $prop_list {
+                switch -nocase -exact -- $n {
+                    params {
+                        # extract name as header filename
+                        foreach {m w} $v {
+                            if { [string match -nocase "*name" $m] } {
+                                regsub -all -nocase -- {[^0-9a-zA-Z-.,\_]} $w {_} w
+                                if { $w eq "" } {
+                                    set w "untitled"
+                                } 
+                                set filename $w
+                                lappend headers_list "filename" $w
+                            } else {
+                                lappend headers_list $m $w
+                            }
+                        }
+                    }
+                    default {
+                        lappend headers_list $n $v
+                    }
+                }
+            }
+        }
+        if { $section_ref eq "" } {
+            set section_ref 1
+        }
+        set subref_ct 0
+        set type ""
+        
+        # Assume headers and names are unordered
+        foreach {n v} $headers_list {
+            if { [string match -nocase {parts} $n] } {
+                set has_parts_p 1
+                foreach part_id $v {
+                    incr subref_ct
+                    set subref $section_ref
+                    append subref "." $subref_ct
+                    acs_mail_lite::maildir_email_parse \
+                        -headers_arr_name h_arr \
+                        -parts_arr_name p_arr \
+                        -part_id $part_id \
+                        -section_ref $subref
+                }
+            } else {
+                switch -exact -nocase -- $n {
+                    size {
+                        set bytes $v
+                    }
+                    # content-type
+                    content {
+                        set type $v
+                    }
+                    default {
+                        # do nothing
+                    }
+                }
+                if { $section_ref eq "1" } {
+                    set h_arr(${n}) ${v}
+                } else {
+                    lappend section_n_v_list ${n} ${v}
+                }
+            }
+        }
+        
+        set section_id [acs_mail_lite::section_id_of $section_ref]
+        ns_log Dev "acs_mail_lite::maildir_email_parse.746 \
+message_fpn '${message_fpn}' section_ref '${section_ref}' section_id '${section_id}'"
+        
+        # Add content of an email part
+        set p_arr(${section_id},nv_list) $section_n_v_list
+        set p_arr(${section_id},c_type) $type
+        lappend p_arr(section_id_list) ${section_id}
+        
+        if { [info exists bytes] && $bytes > $__max_txt_bytes \
+                 && ![info exists filename] } {
+            set filename "blob.txt"
+        }
+        
+        if { [info exists filename ] } {
+            set filename2 [clock microseconds]
+            append filename2 "-" $filename
+            set filepathname [file join [acs_root_dir] \
+                                  acs-mail-lite \
+                                  $filename2 ]
+            set p_arr(${section_id},filename) $filename
+            set p_arr(${section_id},c_filepathname) $filepathname
+            if { $filename eq "blob.txt" } {
+                ns_log Dev "acs_mail_lite::maildir_email_parse.775 \
+ m_id '${m_id}' '${section_ref}' \
+ -file '${filepathname}'"
+                set txtfileId [open $filepathname "w"]
+                puts -nonewline $txtfileId [mime::getbody $m_id]
+                close $txtfileId
+            } else {
+                ns_log Dev "acs_mail_lite::maildir_email_parse.780 \
+ mime::getbody '${m_id}' '${section_ref}' \
+ -file '${filepathname}' -decode"
+                set binfileId [open $filepathname "w"]
+                chan configure $binfileId -translation binary
+                puts -nonewline $binfileId [mime::getbody $m_id -decode ]
+                close $binfileId
+            } 
+        } elseif { $section_ref ne "" } {
+            # text content
+            set p_arr(${section_id},content) [mime::buildmessage $m_id]
+            ns_log Dev "acs_mail_lite::maildir_email_parse.792 \
+ text m_id '${m_id}' '${section_ref}': \
+ $p_arr(${section_id},content)'"
+            
+        } else {
+            set p_arr(${section_id},content) ""
+            # The content for this case
+            # has been verified to be redundant.
+            # It is mostly the last section/part of message.
+            #
+            # If diagnostics urge examining these cases, 
+            # Set debug_p 1 to allow the following code to 
+            # to compress a message to recognizable parts without 
+            # flooding the log.
+            set debug_p 0
+            if { $debug_p } {
+                set msg_txt [mime::buildmessage $m_id]
+                # 72 character wide lines * x lines
+                set msg_start_max [expr { 72 * 20 } ]
+                set msg_txtb [string range $msg_txt 0 $msg_start_max]
+                if { [string length $msg_txt] \
+                         > [expr { $msg_start_max + 400 } ] } {
+                    set msg_txte [string range $msg_txt end-$msg_start_max end]
+                } elseif { [string length $msg_txt] \
+                               > [expr { $msg_start_max + 144 } ] } {
+                    set msg_txte [string range $msg_txt end-144 end]
+                } else {
+                    set msg_txte ""
+                }
+                ns_log Dev "acs_mail_lite::maildir_email_parse.818 IGNORED \
+ text '${message_fpn}' '${section_ref}' \n \
+ msg_txte '${msg_txte}'"
+            } else {
+                ns_log Dev "acs_mail_lite::maildir_email_parse.822 ignored \
+ text '${message_fpn}' '${section_ref}'"
+            }
+        }
+    }
+    return $error_p
+}
+
+
+#            
+# Local variables:
+#    mode: tcl
+#    tcl-indent-level: 4
+#    indent-tabs-mode: nil
+# End:
+
+
Index: openacs-4/packages/acs-mail-lite/tcl/test/email-inbound-procs.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/tcl/test/email-inbound-procs.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/tcl/test/email-inbound-procs.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,737 @@
+ad_library {
+    Automated tests for acs-mail-lite/tcl/email-inbound
+    @creation-date 2017-07-19
+}
+
+aa_register_case -cats {api smoke} acs_mail_lite_inbound_procs_check {
+    Test acs-mail-lite procs in email-inbound-procs.tcl 
+} {
+    aa_run_with_teardown \
+        -rollback \
+        -test_code {
+
+           ns_log Notice "aa_register_case:acs_mail_lite_inbound_procs_check"
+
+           set a_list [acs_mail_lite::sched_parameters]
+           array set params_def $a_list
+           
+           set bools_list [list reprocess_old_p]
+           set integer_list [list sredpcs_override max_concurrent \
+                                 max_blob_chars mpri_min mpri_max]
+           set ints_list [list hpri_package_ids lpri_package_idx hpri_party_ids lpri_party_ids hpri_object_ids lpri_object_ids]
+           set globs_list [list hpri_subject_glob lpri_subject_glob]
+           set lists_list [list reject_on_hit reject_on_miss]
+
+           set bools_v_list [list 0 1 t f true false]
+           set nv_list_list [list \
+                                 [list name1 value1 name2 value2 ] \
+                                 [list a-dashed "a value in quotes" \
+                                      b_unscored "b nother value in quotes" ] \
+                                 [list a-dash {a val in parens} \
+                                      b_underscore {value in parens} ] ]
+            foreach p [array names params_def] {
+                # test setting of each parameter separately
+                set param "-"
+                append param $p
+                if { $p in $bools_list } {
+                    set val_idx [randomRange 5]
+                    set val [lindex $bools_v_list $val_idx]
+                } elseif { $p in $integer_list } {
+                    set val [randomRange 32767]
+                } elseif { $p in $ints_list } {
+                    set nums_list [list]
+                    set up_to_10 [randomRange 10]
+                    for {set i 0} {$i < $up_to_10 } {incr i} {
+                        lappend nums_list [randomRange 32767]
+                    }
+                    set val [join $nums_list " "]
+                } elseif { $p in $lists_list } {
+                    set val_idx [randomRange 2]
+                    set val [lindex $nv_list_list $val_idx]
+                } 
+                aa_log "r41. Testing change of parameter '${p}' from \
+ '$params_def(${p})' to '${val}'"
+                
+                set b_list [acs_mail_lite::sched_parameters $param $val]
+                aa_log "param $param val $val b_list $b_list"
+                array unset params_new
+                array set params_new $b_list
+                foreach pp [array names params_def] {
+                    if { $pp eq $p } {
+                        if { $pp in $bools_list } {
+                            aa_equals "r48 Changed sched_parameter '${pp}' \
+  value '$params_def(${pp})' to '${val}' set" \
+                                [template::util::is_true $params_new(${pp})] \
+                                [template::util::is_true $val]
+                        } else {
+                            if { $params_new(${pp}) eq $params_def(${pp}) } {
+                                if { $pp eq "mpri_max" \
+                                         && $val < $params_def(mpri_min) } {
+                                    aa_log "r54a mpri_max $params_def(mpri_max) } {
+                                    aa_log "r54b mpri_min>mpri_max no change."
+                                } else {
+                                    aa_log "r55 '${pp}' no change."
+                                }
+                            } else {
+                                aa_equals "r56 Changed sched_parameter \
+ '${pp}' value '$params_def(${pp})' to '${val}' set" $params_new(${pp}) $val
+                                
+                            }
+                        }
+                    } else {
+                        if { $pp in $bools_list } {
+                            aa_equals "r62 Unchanged sched_parameter '${pp}' \
+  value '$params_def(${pp})' to '$params_new(${pp})' set" \
+                                [template::util::is_true $params_new(${pp})] \
+                                [template::util::is_true $params_def(${pp})]
+                        } else {
+                            aa_equals "r67 Unchanged sched_parameter '${pp}' \
+  value '$params_def(${pp})' to '$params_new(${pp})' set" \
+                                $params_new(${pp}) $params_def(${pp})
+                        }
+                    }
+                }
+                array set params_def $b_list
+            }
+
+            set instance_id [ad_conn package_id]
+            set sysowner_email [ad_system_owner]
+            set sysowner_user_id [party::get_by_email -email $sysowner_email]
+            set user_id [ad_conn user_id]
+            set package_ids [list $instance_id]
+            set party_ids [util::randomize_list \
+                               [list $user_id $sysowner_user_id]]
+            set object_ids [concat \
+                                $party_ids \
+                                $package_ids \
+                                $user_id \
+                                $sysowner_user_id]
+           # in order of least significant first, regarding prioritization:
+           # That is, object_id is more specific than glob_str or party_ids.
+            set priority_types [list \
+                                    package_ids \
+                                    party_ids \
+                                    glob_str \
+                                    object_ids]
+            set lh_list [list l h]
+            set subject [ad_generate_random_string]
+            set su_glob "*"
+            append su_glob [string range $subject [randomRange 8] end]
+ 
+           # priority_types are in order of least significant first.
+           set p_type_i 0
+            foreach p_type $priority_types {
+
+                # reset prameters
+                foreach {n v} $a_list {
+                    #set $n $v
+                    set p "-"
+                    append p $n
+                    aa_log "r106 resetting p '${p}' to v '${v}'"
+                    set b_list [acs_mail_lite::sched_parameters $p $v]
+                }
+
+                # set new case of parameters
+                set r [randomRange 10000]
+                set p_min [expr { $r + 999 } ]
+                set p_max [expr { $p_min * 1000 + $r } ]
+                set su_max $p_max
+                append su_max "00"
+
+                set c_list [acs_mail_lite::sched_parameters \
+                                -mpri_min $p_min \
+                                -mpri_max $p_max]
+                array set c_arr $c_list
+                set p_min $c_arr(mpri_min)
+                set p_max $c_arr(mpri_max)
+
+                aa_log "r115 p_min '${p_min}' p_max '${p_max}'"
+
+
+                set i 0
+                set p_i [lindex $priority_types $i]
+                while { $p_i ne $p_type && $i < $p_type_i } {
+                    # set a random value to be ignored 
+                    # because p_i is lower significance than
+                    # higher significance of p_type value
+
+                    # make low or high?
+                    set p [util::random_list_element $lh_list]
+                    set pa "-"
+                    append pa $p
+                    switch -exact -- $p_i {
+                        package_ids {
+                            append pa "pri_package_ids"
+                            set v $instance_id
+                        }
+                        party_ids {
+                            append pa "pri_party_ids"
+                            set v [join $party_ids " "]
+                        }
+                        glob_str {
+                            append pa "pri_subject_glob"
+                            set v $su_glob
+                        }
+                        object_ids {
+                            append pa "pri_object_ids"
+                            set v [join $object_ids " "]
+                        }
+                    }
+                    aa_log "r148: pa '${pa}' v '${v}' gets overridden"
+                    acs_mail_lite::sched_parameters ${pa} $v
+
+                    incr i
+                    set p_i [lindex $priority_types $i]
+                }
+                # What priority are we testing?
+                set p [util::random_list_element $lh_list]
+                aa_log "r163: Testing priority '${p}' for '${p_type}'"
+
+                set pa "-"
+                append pa $p
+                switch -exact -- $p_type {
+                    package_ids {
+                        append pa "pri_package_ids"
+                        set v $instance_id
+                    }
+                    party_ids {
+                        append pa "pri_party_ids"
+                        set v [join $party_ids " "]
+                    }
+                    glob_str {
+                        append pa "pri_subject_glob"
+                        set v $su_glob
+                    }
+                    object_ids {
+                        append pa "pri_object_ids"
+                        set v [join $object_ids " "]
+                    }
+                }
+                aa_log "r185: pa '${pa}' v '${v}'"
+                acs_mail_lite::sched_parameters ${pa} $v
+
+
+                # make four tests for each priority p_arr
+                # two vary in time, t1, t2
+                # two vary in size, s1, s2
+
+                set t0 [nsv_get acs_mail_lite si_start_t_cs]
+                set dur_s [nsv_get acs_mail_lite si_dur_per_cycle_s]
+
+                set size_list [list $su_max]
+                set ns_section_list [list nssock nssock_v4 nssock_v6]
+                foreach section $ns_section_list {
+                    lappend size_list [ns_config -int -min 0 $section maxinput]
+                }
+                set s0  [f::lmax $size_list]
+
+                aa_log "r161 given: t0 '${t0}' dur_s '${dur_s}'"
+                aa_log "r161b given: s0 '${s0}' su_max '${su_max}'"
+
+                set t1 [expr { int( $t0 - $dur_s * 1.9 * [random]) } ]
+                set t2 [expr { int( $t0 - $dur_s * 1.9 * [random]) } ]
+                set s1 [expr { int( $s0 * 0.9 * [random]) } ]
+                set s2 [expr { int( $s0 * 0.9 * [random]) } ]
+                aa_log "r167 priorities: t1 '${t1}' t2 '${t2}' s1 '${s1}' s2 '${s2}'"
+                if { $t1 < $t2 } {
+                    set t $t1
+                    # first in chronology = f1
+                    # second in chronology = f2
+                    set f1 t1
+                    set f2 t2
+                } else {
+                    set t $t2
+                    set f1 t2
+                    set f2 t1
+                }
+
+                if { $s1 < $s2 } {
+                    set s $s1
+                    # first in priority for size = z1
+                    # second in priority for size = z2
+                    set z1 s1
+                    set z2 s2
+                } else {
+                    set s $s2
+                    set z1 s2
+                    set z2 s1
+                }
+                
+                set p_arr(t1) [acs_mail_lite::inbound_prioritize \
+                                       -size_chars $s \
+                                       -received_cs $t1 \
+                                       -subject $subject \
+                                       -package_id $instance_id \
+                                       -party_id $user_id \
+                                       -object_id $instance_id]
+                aa_log "p_arr(t1) = '$p_arr(t1)'"
+
+                set p_arr(t2) [acs_mail_lite::inbound_prioritize \
+                                   -size_chars $s \
+                                   -received_cs $t2 \
+                                   -subject $subject \
+                                   -package_id $instance_id \
+                                   -party_id $user_id \
+                                   -object_id $instance_id]
+                aa_log "p_arr(t2) = '$p_arr(t2)'"
+
+                set p_arr(s1) [acs_mail_lite::inbound_prioritize \
+                                   -size_chars $s1 \
+                                   -received_cs $t \
+                                   -subject $subject \
+                                   -package_id $instance_id \
+                                   -party_id $user_id \
+                                   -object_id $instance_id]
+                aa_log "p_arr(s1) = '$p_arr(s1)'"
+
+                set p_arr(s2) [acs_mail_lite::inbound_prioritize \
+                                   -size_chars $s2 \
+                                   -received_cs $t \
+                                   -subject $subject \
+                                   -package_id $instance_id \
+                                   -party_id $user_id \
+                                   -object_id $instance_id]
+                
+                aa_log "p_arr(s2) = '$p_arr(s2)'"
+
+                # verify earlier is higher priority 
+                if { $p_arr(${f1}) < $p_arr(${f2}) } {
+                    set cron_p 1
+                } else {
+                    set cron_p 0
+                }
+                aa_true "earlier email assigned first \
+ ${f1} '$p_arr(${f1})' < ${f2} '$p_arr(${f2})' " $cron_p
+
+                # verify larger size has slower priority
+                if { $p_arr(${z1}) < $p_arr(${z2}) } {
+                    set size_p 1
+                } else {
+                    set size_p 0
+                }
+                aa_log "test r266 and r276 may fail when not testing \
+ a default clean system"
+                aa_true "smaller email assigned first \
+ ${z1} '$p_arr(${z1})' < ${z2} '$p_arr(${z2})' " $size_p
+
+                # verify that none hit or exceed the range limit
+                if { $p eq "l" } {
+                    foreach j [list t1 t2 s1 s2] {
+                        if { $p_arr($j) > $p_max && $p_arr($j) < $s0 } {
+                            set within_limits_p 1
+                        } else {
+                            set within_limits_p 0
+                        }
+                        aa_true "r266; prioirty for case '${j}' '${p_max}' < \
+  '$p_arr(${j})' < '${s0}' is within limits." $within_limits_p
+                    }
+                } elseif { $p eq "h" } {
+                    foreach j [list t1 t2 s1 s2] {
+                        if { $p_arr($j) > 0 && $p_arr($j) < $p_min } {
+                            set within_limits_p 1
+                        } else {
+                            set within_limits_p 0
+                        }
+                        aa_true "r276: prioirty for case '${j}' '0' < \
+  '$p_arr(${j})' < '${p_min}' is within limits." $within_limits_p
+                    }
+
+                }
+
+                incr p_type_i
+                # end foreach p_type loop
+            }
+
+           set ho "localhost"
+           set na "mail/INBOX"
+           set ssl_p 0
+           set t1 [acs_mail_lite::imap_mailbox_join \
+                       -host $ho -name $na -ssl_p $ssl_p]
+           set t2 {{localhost}mail/INBOX}
+           aa_equals "Test acs_mail_lite::imap_mailbox_join" $t1 $t2
+
+           set t2_list [acs_mail_lite::imap_mailbox_split $t2]
+           set t1_list [list $ho $na $ssl_p]
+           aa_equals "Test acs_mail_lite::imap_mailbox_split" $t1_list $t2_list
+
+
+           aa_log "Testing imap open/close via default connection params"
+           set conn_id [acs_mail_lite::imap_conn_close -conn_id "all"]
+           set es ""
+
+           aa_log "Following three tests pass when no imap sessions open."
+           aa_false "acs_mail_lite::imap_conn_close -conn_id 'all'" $conn_id
+
+           set conn_id [randomRange 1000]
+           set t3 [acs_mail_lite::imap_conn_close -conn_id $conn_id]
+           aa_false "acs_mail_lite::imap_conn_close -conn_id '${conn_id}'" $t3
+
+           set conn_id ""
+           set t3 [acs_mail_lite::imap_conn_close -conn_id $conn_id]
+           aa_false "acs_mail_lite::imap_conn_close -conn_id '${conn_id}'" $t3
+
+           aa_log "Following tests various session cases with open/close"
+           aa_log "Some will fail if a session cannot be established."
+
+
+           # see Example of an IMAP LIST in rfc6154: 
+           # https://tools.ietf.org/html/rfc6154#page-7
+           # ns_imap list $conn_id $mailbox pattern(* or %) substr
+
+
+           #set list [ns_imap list $conn_id $mailbox_host {}]
+           # returns: '{} noselect'  When logged in is not successful..
+           # set list [ns_imap list $conn_id $mailbox_host {*}]
+           # returns 'INBOX {} INBOX.Trash {} INBOX.sent-mail {}' when really logged in
+           # and mailbox_name part of mailbox is "", and mailbox is in form {{mailbox_host}}
+           # set list [ns_imap list $conn_id $mailbox_host {%}]
+           # returns 'INBOX {}' when really logged in
+           # and mailbox_name part of mailbox is ""
+           # If mailbox_name exists and is included in mailbox_host, returns '' 
+           # If mailbox_name separate from mailbox_host, and exists and in place of %, returns 'mailbox {}'
+           # for example 'INBOX.Trash {}'
+
+
+           set sid [acs_mail_lite::imap_conn_go]
+           set sid_p [ad_var_type_check_integer_p $sid]
+           aa_true "acs_mail_lite::imap_conn_go" $sid_p
+
+           set sid2 [acs_mail_lite::imap_conn_close -conn_id $sid]
+           aa_true "acs_mail_lite::imap_conn_close -conn_id '${sid}'" $sid2
+
+           set sid3 [acs_mail_lite::imap_conn_go -conn_id $sid]
+           set sid3_p [ad_var_type_check_integer_p $sid3]
+           aa_false "acs_mail_lite::imap_conn_go -conn_id '${sid}'" $sid3_p
+
+           set sid4 [acs_mail_lite::imap_conn_go -conn_id ""]
+           set sid4_p [ad_var_type_check_integer_p $sid4]
+           aa_true "acs_mail_lite::imap_conn_go -conn_id ''" $sid4_p
+
+           set sid5 "all"
+           set closed_p [acs_mail_lite::imap_conn_close -conn_id $sid5]
+           aa_true "acs_mail_lite::imap_conn_close -conn_id '${sid5}'" $closed_p
+
+           aa_log "Testing for auto replies"
+
+           # load example headers
+           set files_list [glob -directory [file join [acs_root_dir] \
+                                                packages \
+                                                acs-mail-lite \
+                                                www \
+                                                doc ] \
+                               -- {headers-example-[0-9]*.txt} ]
+
+           #NOTE: number 24 is example of auto-generated
+
+           set i ""
+           foreach f $files_list {
+               
+               if { [regexp {([0-9]+)} [file tail $f] i ] } {
+                   set fid [open $f r ]
+                   # headers-example = he
+                   set he_arr(${i}) [read $fid ]
+                   
+                   switch -exact -- $i {
+                       3 {
+                           set type_arr(${i}) "in_reply_to"
+                       }
+                       24 {
+                           set type_arr(${i}) "auto_gen"
+                       }
+                       default {
+                           set type_arr(${i}) ""
+                       }
+                   }
+
+
+                   ns_log Notice "test/email-inbound-procs.tcl.394 i $i f $f"
+                   
+                   close $fid
+               } else {
+                   ns_log Warning "test/email-inbound-procs.tcl.401 f ${f} not processed"
+               }
+           }
+
+           
+
+           aa_log "Test using full headers in text of default cases."
+           set csp 0
+           set su ""
+           set from ""
+           set i [llength $files_list]
+           for {set ii 1} {$ii <= $i} {incr ii } {
+               set type [acs_mail_lite::email_type \
+                             -subject $su \
+                             -from $from \
+                             -headers $he_arr(${ii}) \
+                             -check_subject_p $csp ]
+               #aa_log "r401. headers '$he_arr(${ii})'"
+               aa_equals "r402. unmodified headers-example-${ii}.txt of \
+ type '$type_arr(${ii})'. type from acs_mail_lite::email_type" \
+                   $type $type_arr(${ii})
+           }
+           
+           aa_log "Test using full headers in modified cases, including
+ false flags for subject and from fields that should be ignored."
+           set csp 0
+           set su "out of office"
+           set from "mailer daemon"
+           # ordered list, highest priority first.
+           # See last if in acs_mail_lite::email_type
+           set t_olist [list bounce auto_reply auto_gen in_reply_to ""]
+           set s_list [list failed delayed relayed expanded]
+           set ar_list [list auto-notified \
+                            auto-replied \
+                            auto-reply \
+                            autoreply \
+                            autoresponder \
+                            x-autorespond ]
+           for {set ii 1} {$ii <= $i} {incr ii } {
+               # send garbage to try to confuse proc
+               set t [randomRange 4]
+               set h ""
+               # Some examples already have header types that limit 
+               # test type.
+               if { $type_arr(${ii}) eq "auto_gen" && $t > 2 } {
+                   set t [randomRange 2]
+               }
+
+               if { $type_arr(${ii}) eq "in_reply_to" && $t > 1 } {
+                   set t [randomRange 1]
+               }
+               set type_test [lindex $t_olist $t]
+               
+               if { $t == 3 || $t < 2  } {
+                   # add in_reply_to headers
+                   append h "in-reply-to : " [ad_generate_random_string 30]
+                   append h "\n"
+               }
+               if { $t < 3 } {
+                   # add auto_gen headers
+                   append h "auto-submitted : " [ad_generate_random_string]
+                   append h "\n"
+                   append h "auto-generated : " [ad_generate_random_string]
+                   append h "\n"
+               }
+               if { $t < 2 } {
+                   # add auto_reply headers
+                   switch [randomRange 2] {
+                       0 { 
+                           append h [lindex $ar_list [randomRange 5]]
+                           append h " : " [ad_generate_random_string]
+                       }
+                       1 {
+                           append h "action : delivered"
+                       }
+                       2 {
+                           set h2 [lindex $s_list [randomRange 3]]
+                           append h "action : " $h2 "\n"
+                           append h "status : thisis a test" 
+                       }
+                   }                       
+                   append h "\n"
+               }
+               if { $t < 1 } {
+                   # add bounce headers
+                   if { [randomRange 1] } {
+                       # test original-recipient (diverted, reply)
+                       append h "original-recipient : "
+                       append h [ad_system_owner] "\n"
+                   } else {
+                       # test delivery status notification
+                       append h action
+                       append h " : " [lindex $s_list [randomRange 3]]
+                       append h "\n" status " : " 
+                       append h [expr { 99 + [randomRange 900] } ] " "
+                       append h [ad_generate_random_string [randomRange 9]]
+                       append h "\n"
+                   }
+               }
+               # maybe mess up capitalization
+               set c [randomRange 3]
+               switch -exact -- $c {
+                   0 {
+                       set h [string tolower $h]
+                   }
+                   1 {
+                       set h [string toupper $h]
+                   }
+                   2 {
+                       set h [string totitle $h]
+                   }
+                   default {
+                       # do nothing
+                   }
+               }
+               aa_log "t ${t} type_test '${type_test}' h '${h}'"
+               ns_log Dev "t ${t} type_test '${type_test}' h '${h}'"
+               set he $he_arr(${ii})
+               append he "\n" $h
+               set type [acs_mail_lite::email_type \
+                             -subject $su \
+                             -from $from \
+                             -headers $he \
+                             -check_subject_p $csp ]
+               aa_equals "r501 headers-example-${ii}.txt \
+ ($type_arr(${ii})) to '${type_test}'. Matches acs_mail_lite::email_type" \
+                   $type $type_test
+               ns_log Dev "r501n headers-example-${ii}.txt \
+ ($type_arr(${ii})) to '${type_test}'. acs_mail_lite::email_type '${type}'"
+           }
+
+
+           aa_log "r600 test acs_mail_lite::section_id_of "
+           aa_log "r601 test empty case ''"
+           set section ""
+           set sect_id1 [acs_mail_lite::section_id_of $section]
+           set sect_id2 [acs_mail_lite::section_id_of $section]
+           set sect_arr(${sect_id1}) $section
+           aa_equals "r601 test empty case section '${section}'" \
+               $sect_id2 $sect_id1
+           set section [ad_generate_random_string]
+           # Some random strings are integers.
+           append section A
+           set sect_id1 [acs_mail_lite::section_id_of $section]
+           set sect_id2 [acs_mail_lite::section_id_of $section]
+           aa_equals "r602 test bad ref case section '${section}'" \
+               $sect_id2 $sect_id1
+           set sect_arr(${sect_id1}) $section
+           aa_equals "r603 test bad ref case section '${section}' returns ''" \
+               $sect_id1 ""
+
+
+           set section [randomRange 100]
+           set sect_id1 [acs_mail_lite::section_id_of $section]
+           set sect_id2 [acs_mail_lite::section_id_of $section]
+           aa_equals "r605 test case section '${section}'" \
+               $sect_id2 $sect_id1
+           set sect_arr(${sect_id1}) $section
+           for {set i 0} {$i < 6} {incr i} {
+               append section "." [randomRange 100]
+               set sect_id1 [acs_mail_lite::section_id_of $section]
+               set sect_id2 [acs_mail_lite::section_id_of $section]
+               aa_equals "r606 test case section '${section}'" \
+                   $sect_id2 $sect_id1
+               set sect_arr(${sect_id1}) $section
+           }
+
+           aa_log "r610 test acs_mail_lite::section_ref_of "
+           aa_log "r611 test empty case ''"
+           set sect_ref1 ""
+           set sect_ref2 [acs_mail_lite::section_ref_of ""]
+           aa_equals "r616 test case section '${sect_ref1}'" \
+               $sect_id2 $sect_id1
+           
+           foreach sect_id [array names sect_arr] {
+
+               set sect_ref1 $sect_arr(${sect_id})
+               if { $sect_id ne "" } {
+
+                   set sect_ref2 [acs_mail_lite::section_ref_of $sect_id]
+                   aa_equals "r616 test case section '${sect_id}'" \
+                       $sect_ref2 $sect_ref1
+               }
+           }
+
+
+           aa_log "r700 test acs_mail_lite::unique_id_create/parse paradigm"
+
+           set integer_max 2147483647
+           incr integer_max -2
+           set fields_list [list package_id party_id object_id other]
+           set package_id_list [db_list apm_package_ids_rall { select 
+               distinct package_id from apm_packages } ]
+           set aml_package_id [apm_package_id_from_key "acs-mail-lite"]
+           set party_id_list [db_list parties_rall { select
+               distinct party_id from parties }]
+           set object_id_list [db_list acs_objects_rall { select
+               distinct object_id from acs_objects} ]
+           set package_ct [llength $package_id_list]
+           set party_ct [llength $party_id_list]
+           set object_ct [llength $object_id_list]
+           for {set i 0} {$i < 12} {incr i } {
+               set package_id [lindex $package_id_list \
+                                   [randomRange $package_ct]]
+               set party_id [lindex $party_id_list \
+                                 [randomRange $party_ct]]
+               set object_id [lindex $object_id_list \
+                                  [randomRange $object_ct]]
+               set other [ad_generate_random_string]
+               set blank_id [randomRange 3]
+               set blank_field [lindex $fields_list $blank_id]
+               set $blank_field ""
+               # if package_id = aml_package_id, it still is signed here
+               set m_arr(package_id,${i}) $package_id
+               set m_arr(party_id,${i}) $party_id
+               set m_arr(object_id,${i}) $object_id
+               set m_arr(other,${i}) $other
+               set m_arr(msg_id,${i}) [acs_mail_lite::unique_id_create \
+                                           -package_id $package_id \
+                                           -party_id $party_id \
+                                           -object_id $object_id \
+                                           -other $other ]
+           }
+           for {set i 0} {$i < 12} {incr i } {
+               array unset e_arr
+               aa_log "r701 test message-id '$m_arr(msg_id,${i})'"
+               set e_list [acs_mail_lite::unique_id_parse \
+                               -message_id $m_arr(msg_id,${i}) ]
+               array set e_arr $e_list
+               foreach field $fields_list {
+                       aa_equals "r703 test acs_mail_lite::unique_id \
+ i '${i}' field '${field}'" $e_arr(${field}) $m_arr(${field},${i})
+
+               }
+
+           }
+           aa_log "test default case"
+           set msg_id [acs_mail_lite::unique_id_create ]
+           set e_list [acs_mail_lite::unique_id_parse -message_id $msg_id]
+           foreach {n v} $e_list {
+               switch -- $n {
+                   object_id -
+                   package_id -
+                   party_id -
+                   datetime_cs -
+                   other {
+                       aa_equals "r710 test acs_mail_lite::unqiue_id $n has val ''" $v ""
+                   }
+                   datetime_not {
+                       set is_integer_p [string is wideinteger -strict $v]
+                       aa_true "r711 test acs_mail_lite::unique_id $n is integer" $is_integer_p
+
+                   }
+               }
+           }
+           aa_log "test passing blank case as external"
+               set msg_id [acs_mail_lite::unique_id_create -unique_id [mime::uniqueID]]
+               set e_list [acs_mail_lite::unique_id_parse -message_id $msg_id]
+           foreach {n v} $e_list {
+               switch -- $n {
+                   object_id -
+                   package_id -
+                   party_id -
+                   datetime_cs -
+                   other {
+                       aa_equals "r710 test acs_mail_lite::unqiue_id $n has val ''" $v ""
+                   }
+                   datetime_not {
+                       set is_integer_p [string is wideinteger -strict $v]
+                       aa_true "r711 test acs_mail_lite::unique_id $n is integer" $is_integer_p
+
+                   }
+               }
+           }
+       }
+}
+
+
+
+
+# Local variables:
+#    mode: tcl
+#    tcl-indent-level: 4
+#    indent-tabs-mode: nil
+# End:
Index: openacs-4/packages/acs-mail-lite/www/doc/analysis-notes.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/analysis-notes.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/analysis-notes.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,170 @@
+
+  @title;noquote@
+  @context;noquote@
+  
@title@

+
Notes in preparation for adding IMAP to legacy bounce MailDir paradigm

+  
New procs

+  
For imap, each begin of a process should not assume a connection exists or doesn't exist. Check connection using 'imap ping' before login.
+    This should help re-correct any connection drop-outs due to intermittent or one-time connection issues.
+  

+  
Each scheduled event should quit in time for next process, so that imap info being processed is always nearly up-to-date.
+    This is important in case a separate manual imap process is working in tandem and changing circumstances.
+    This is equally important to quit in time, because imap references relative sequences of emails.
+    Two concurrent connections would likely have different and overlapping references.
+    The overlapping references would likely cause issues, since each connection would expect to process
+    the duplicates as if they are not duplicates.
+  

+
variables useful while exploring new processes like forecasting and scheduling

+  

+    
scan_in_active_p

+    
(don't use. See si_active_cs). Answers question. Is a proc currently scanning replies?

+    
si_active_cs

+    
(don't use. See si_actives_list.) The clock scan of the most recently started cycle. If a cycle's poll doesn't match, it should not process any more email.

+    
si_actives_list

+    
A list of start clock seconds of active imap_checking_incoming procs

+    
scan_incoming_configured_p

+    
Is set to 0 if there is an error trying to connect. OTherwise is set to 1 by acs_mail_lite::imap_check_incoming

+
+    
replies_est_next_start

+    
Approx value of [clock seconds] next scan is expected to begin

+
+    
duration_ms_list

+    
Tracks duration of processing of each email in ms of most recent process, appended as a list.
+      When a new process starts processing email, the list is reset to only include the last 100 emails. That way, there is always rolling statistics for forecasting process times.

+
+    
scan_in_est_dur_per_cycle_s

+    
Estimate of duration of current cycle

+
+    
scan_in_est_quit_cs

+    
When the current cycle should quit based on [clock seconds]

+
+    
scan_in_start_cs

+    
When the current cycle started scanning based on [clock seconds]

+
+    
cycle_start_cs

+    
When the current cycle started (pre IMAP authorization etc) based on [clock seconds]

+
+    
cycle_est_next_start_cs

+    
When the next cycle is to start (pre IMAP authorization etc) based on [clock seconds]

+
+    
parameter_val_changed_p

+    
If related parameters change, performance tuning underway. Reset statistics.

+
+    
scan_in_est_dur_per_cycle_s_override

+    
If this value is set, use it instead of the scan_in_est_dur_per_cycle_s

+
+    
accumulative_delay_cycles

+    
Number of cycles that have been skipped 100% due to ongoing process (in cycles).

+      
+    
+  

+  

+    Check scan_incoming_active_p when running new cycle.
+    Also set replies_est_next_start to clock seconds for use with time calcs later in cycle.
+    If already running, wait a second, check again.. until 90% of duration has elapsed.
+    If still running, log a message and quit in time for next event.
+  

+  

+    Each scheduled procedure should also use as much time as it needs up to the cut-off at the next scheduled event.
+    Ideally, it needs to forecast if it is going to go overtime with processing of the next email, and quit just before it does.
+  

+  

+    Use duration_ms_list to determine a time adjustment for quiting before next cycle:
+    scan_in_est_dur_per_cycle_s + scan_repies_start_time =
+    scan_in_est_quit_cs
+  

+  

+    And yet, predicting the duration of the future process is difficult.
+    What if the email is 10MB and needs parsed, whereas all prior emails were less then 10kb?
+    What if one of the callbacks converts a pdf into a png and annotates it for a web view and takes a few minutes?
+    What if the next 5 emails have callbacks that take 5 to 15 minutes to process each waiting on an external service?
+  

+  
The process needs to be split into at least two to handle all cases. 
+  

+    The first process collects incoming email and puts it into a system standard format with a minimal amount of effort sufficient for use by callbacks. The goal of this process is to keep up with incoming email to all mail available to the system at the earliest possible moment.
+  

+    The second process should render a prioritized queue of imported email that have not been processed. First prioritizing new entries, perhaps re-prioritizing any callbacks that error or sampling re-introducing prior errant callbacks etc. then continuing to process the stack. 
+  

+Using this paradigm, parallel processes could be invoked for the queue without significantly changing the paradigm.
+

+  
To reduce overhead on low volume systems, these processes should be scheduled to minimize concurrent operation.
+  

+  
Priorities should offer 3 levels of performance. Colors designate priority to discern from other email priority schemes:

+  
  • 
+      High (abbrev: hpri, Fast Priority, a priority value 1 to mpri_min  (default 999): allow concurrent processes. That is, when a new process starts, it can also process unprocessed cases. As the stack grows, processes run in parallel to reduce stack up to acs_mail_lite_ui.max_concurrent.
+    
  • 
+      Med (abbrev: mpri, Standard Priority, a priority mpri_min to mpri_max (default 9999)): Process one at a time with casual overlap. (Try to) quit before next process starts. It's okay if there is a little overlapping.
+    
  • 
+      Low (abbrev: lpri, Low Priority, a priority value over mpri_max): Process one at a time only. If a new cycle starts and the last is still running, wait for it to quit (or quit before next cycle).
+  

+
+
Priority is calculated based on timing and file size

+
 
+set range priority_max - priority_min
+set deviation_max { ($range / 2 }
+set midpoint { priority_min + $deviation_max }
+time_priority =  $deviation_max (  clock seconds of received datetime - scan_in_start_cs ) / 
+            ( 2 * scan_in_est_dur_per_cycle_s )
+
+size_priority = 
+   $deviation_max * ((  (size of email in characters)/(config.tcl's max_file_upload_mb *1000000) ) - 0.5)
+
+set equation = int( $midpoint + ($time_priority + size_priority) / 2)
+

+
Average of time and file size priorities. 

+
hpri_package_ids and lpri_package_ids and hpri_party_ids and lpri_party_ids and mpri_min and mpri_max and hpri_subject_glob and lpri_subject_glob are defined in acs_maile_lite_ui, so they can be tuned without restarting server. ps. Code should check if user is banned before parsing any further.

+
A proc should be available to recalculate existing email priorities. This means more info needs to be added to table acs_mail_lite_from_external (including size_chars)

+  
Import Cycle

+  
This scheduling should be simple.  Maybe check if a new process wants to take over. If so, quit.

+  
+  
Prioritized stack processing cycle

+  

+    If next cylce starts and current cycle is still running,
+    set scan_in_est_dur_per_cycle_s_override to actual wait time the current cycle has to wait including any prior cycle wait time --if the delays exceed one cycle (accumulative_delay_cycles.
+  

+  
From acs-tcl/tcl/test/ad-proc-test-procs.tcl
+    # This example gets list of implimentations of a callback: (so they could be triggered one by one)
+     ad_proc -callback a_callback { -arg1 arg2 } { this is a test callback } -
+    set callback_procs [info commands ::callback::a_callback::*]
+    
+  

+  

+    Each subsquent cycle moves toward renormalization by adjusting
+    scan_in_est_dur_per_cycle_s_override toward value of
+    scan_in_est_dur_per_cycle_s by one
+    replies_est_dur_per_cycle with minimum of
+    scan_in_est_dur_per_cycle_s.
+    Changes are exponential to quickly adjust to changing dynamics.
+  

+  

+    For acs_mail_lite::scan_in,
+  

+    Keep track of email flags while processing.

+    Mark /read when reading.

+    Mark /replied if replying.
+  

+  

+    When quitting current scheduled event, don't log out if all processes are not done.
+    Also, don't logout if imaptimeout is greater than duration to cycle_est_next_start_cs.
+   
+    Stay logged in for next cycle.
+  

+  

+    Delete processed messages when done with a cycle?
+    No. What if message is used by a callback with delay in processing?
+    Move processed emails in a designated folder ProcessFolderName parameter.
+    Designated folder may be Trash.
+    Set ProcessFolderName by parameter If empty, Default is hostname of ad_url ie:
+    [util::split_location [ad_url] protoVar ProcessFolderName portVar]
+    If folder does not exist, create it. ProcessFolderName only needs checked if name has changed.
+    

+    MailDir marks email as 'read' by moving from '/new' dir to '/cur' directory. ACS Mail Lite implementations should be consistent as much as possible, and so mark emails in IMAP as 'read' also.
+
+  

+    Email attachments
+    

+  

+   Since messages are not immediately deleted, create a table of attachment url references. Remove attachments older than AttachmentLife parameter seconds.
+   Set default to 30 days old (2592000 seconds).
+   Unless ProcessFolderName is Trash, email attachments can be recovered by original email in ProcessFolderName.
+No. Once callbacks are processed, assume any transfer of attachments has occurred, so that processed email can be purged.    

Index: openacs-4/packages/acs-mail-lite/www/doc/analysis-notes.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/analysis-notes.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/analysis-notes.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,2 @@
+set title "Analysis, annotations, planning for adding IMAP feature"
+set context [list [list index "Documentation"] $title]
Index: openacs-4/packages/acs-mail-lite/www/doc/config-nsimap-part.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/config-nsimap-part.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/config-nsimap-part.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,15 @@
+#---------------------------------------------------------------------
+# nsimap
+#---------------------------------------------------------------------
+
+   ns_section      ns/server/${server}/module/nsimap
+      ns_param     idle_timeout    1800
+      # Timeout is in seconds, it defines inactivity period after which sessions will close.
+
+       ns_param     debug       0
+
+       # Optional: Set default values in all sessions for mailbox, user and/or password.
+       #ns_param    mailbox     ""
+       #ns_param    user        ""
+       #ns_param    password    ""
+
Index: openacs-4/packages/acs-mail-lite/www/doc/devel-notes-change-log.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/devel-notes-change-log.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/devel-notes-change-log.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,47 @@
+Development notes and change log
+
+
+Documentation changes are too numerous to deliniate.
+General strategy is to separate into topics of:
+setup, installation, inbound email, outbound email, legacy info and notes.
+
+
+Portions of the following sources have been added to ACS Mail Lite docs:
+
+http://openacs.org/xowiki/incoming_email
+http://openacs.org/forums/message-view?message_id=543694
+
+Some suggestions have been incorporated into new code.
+
+A collection of example email headers are included.
+Unit tests in acs-mail-lite/tcl/test/email-inbound-procs.tcl use these.
+Ones from other sources on web are compatible
+with OpenACS' distribution license and include attribution.
+Each set of example email headers is in a separate file in this package:
+ acs-mail-lite/www/doc/headers-example-N.txt
+where N is a natural number.
+
+
+New parameters added to new section: IMAP
+
+Parameters
+        IMAPHost
+        IMAPPassword
+        IMAPPort (always same until.. it isn't)
+        IMAPTimeout
+        IMAPUser
+        IncomingFilterProcName
+        ExternalSource default is blank. 
+
+Utility pages (requires admin permission)
+
+        imap-test              For testing nsimap API
+        maildir-test           For testing maildir API
+        maildir-actives-reset  For reseting concurrency check if
+                               acs_mail_lite::maildir_check_incoming errors.
+
+IMAP
+        imap-notes.txt contains notes to help with imap implementation and 
+         to be added to nsimap package 
+
+other files:
Index: openacs-4/packages/acs-mail-lite/www/doc/glossary.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/glossary.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/glossary.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,43 @@
+
+  @title;noquote@
+  @context;noquote@
+  
@title@

+
+
+

+
+  
Header fields

+  

+    Name/value pairs. A core set provide email parameters.
+    See RFC5322 2.2 https://tools.ietf.org/html/rfc5322#section-2.2
+    


+    A list of registered headers and their usage is at
+    RFC4021 https://tools.ietf.org/html/rfc4021
+    


+    Custom headers can also be added using 
+    the exension-field prefix "x-" or "X-"
+    See RFC822 4.7.4 https://tools.ietf.org/html/rfc822
+  

+  
+  
Reply-To

+  

+    Replies of email with Repy-To are addressed to address in Reply-To header.
+  

+
+  
Return-Path

+  

+    The address of replies when there are message delivery issues. 
+    The final SMTP delivery agent inserts a header "Return-Path". 
+    This should not be generated by the original sender. 
+    New Return-Paths overwrite an existing one, 
+    so that there is never more than one. 
+    See RFC5321 4.4: https://tools.ietf.org/html/rfc5321#section-4.4
+  
+  
Sender

+  

+    Sender usually refers to the email of the entity sending the email. 
+    This addressed is used in the 'From' header of outbound email.
+  

+
+
+

Index: openacs-4/packages/acs-mail-lite/www/doc/glossary.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/glossary.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/glossary.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,3 @@
+set title "Glossary"
+set context [list [list index "Documentation"] $title]
+
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-1.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-1.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-1.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,26 @@
+Return-Path: 
+Received: from dipole.node ([unix socket])
+           by dipole.node (Cyrus 2.5.11) with LMTPA;
+            Tue, 01 Aug 2017 06:00:30 +0000
+X-Sieve: CMU Sieve 2.4
+Received: from localhost (unknown [12.27.188.27.186.44])
+          by dipole.node (Postfix) with ESMTP id CA0551638C04
+          for ; Tue,  1 Aug 2017 06:00:30 +0000 (UTC)
+Received: from dipole.node ([12.27.188.27.186.70])
+ by localhost (positive.propul.voyager13 [12.27.188.27.186.44]) (maiad, port 10024)
+ with ESMTP id 05590-02 for ;
+ Tue,  1 Aug 2017 06:00:30 +0000 (UTC)
+Received: by dipole.node (Postfix, from userid 0)
+          id 960341638BFC; Tue,  1 Aug 2017 06:00:30 +0000 (UTC)
+From: terminal3@dipole.node (Cron Daemon)
+To: terminal3@dipole.node
+Subject: Cron  /usr/local/bin/vacuumdb --all --full --analyze -U pgsql
+X-Cron-Env: 
+X-Cron-Env: 
+X-Cron-Env: 
+X-Cron-Env: 
+X-Cron-Env: 
+Message-Id: <20170801060030.960341638BFC@dipole.node>
+Date: Tue,  1 Aug 2017 05:59:00 +0000 (UTC)
+
+
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-10.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-10.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-10.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,21 @@
+Content-Type: text/plain; charset="us-ascii"
+Date: Sun, 29 Jul 2007 13:08:52 +0200
+From: Username 
+Message-ID: <20070729110852.111111@gmx.net>
+MIME-Version: 1.0
+Subject: header test
+To: Username 
+X-Authenticated: #33333333
+X-Mailer: WWW-Mail 6100 (Global Message Exchange)
+X-Priority: 3
+X-Provags-ID: G11D2DsdDDkF13DrwAc8wFFGyftuH1D+5jsXgvVepqK7XQlVHKHiu
+ SekD1HHRJ3gH+9uX8afy+MN4KLoamn6RTgjE== 
+Content-Transfer-Encoding: 7bit
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/GMX_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-11.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-11.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-11.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,18 @@
+Message-ID: <1186073825.46b20ce1ad3cd@sendinghost.com>
+Date: Thu,  2 Aug 2007 20:57:05 +0400
+From: username@sendinghost.com
+To: username@receivinghost.com
+Subject: Header test
+MIME-Version: 1.0
+Content-Type: text/plain; charset=ISO-8859-5
+Content-Transfer-Encoding: 8bit
+User-Agent: Internet Messaging Program (IMP) 3.2.2
+X-Originating-IP: 255.255.255.255
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Horde_IMP_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-12.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-12.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-12.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,19 @@
+Message-ID: 
+Content-Type: multipart/alternative;
+              boundary="_123eccb8-5ca2-67e8-a450-1234c16e36a3_"
+X-Originating-IP: [255.255.255.255]
+From: Username 
+To: Username 
+Subject: header test
+Date: Sat, 28 Jul 2007 10:35:46 +0100
+Importance: Normal
+MIME-Version: 1.0
+X-OriginalArrivalTime: 28 Jul 2007 09:35:47.0016 (UTC) FILETIME=[AD3C4C80:01C7D0FA]
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Hotmail_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-13.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-13.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-13.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,19 @@
+X-Mailer: iPhone Mail (3B48b)
+Mime-Version: 1.0 (Apple Message framework v752.2)
+Content-Transfer-Encoding: 7bit
+Message-Id: <38D1C1FD-3C35-4568-925C-FC46CAC0DE8A@sendinghost.com>
+Content-Type: text/plain;
+              charset=US-ASCII;
+              format=flowed
+To: Other User 
+From: User Name 
+Subject: Subject Line
+Date: Mon, 26 Feb 2007 20:21:57 -0500
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/IPhone_Mail_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-14.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-14.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-14.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,20 @@
+Message-ID: <8A1F11BBD7D7413EAE9E2792741BC6A4@hostname>
+From: "Username" 
+To: "Username" 
+Subject: header test
+Date: Fri, 3 Aug 2007 18:22:27 +0200
+MIME-Version: 1.0
+Content-Type: multipart/alternative;
+              boundary="----=_NextPart_000_000B_01C7D5FB.3ED1C170"
+X-Priority: 3
+X-MSMail-Priority: Normal
+X-Mailer: Microsoft Windows Mail 6.0.6000.16386
+X-MimeOLE: Produced By Microsoft MimeOLE V6.0.6000.16386
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Microsoft_Mail_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-15.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-15.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-15.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,17 @@
+Date: Sat, 28 Jul 2007 18:29:15 +0200
+To: Username 
+Subject: header test
+Message-ID: <20070728162915.GA2046@localhost>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=us-ascii
+Content-Disposition: inline
+User-Agent: Mutt/1.5.13 (2006-08-11)
+From: Username 
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Mutt_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-16.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-16.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-16.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,17 @@
+From: "Username" 
+To: "Username" 
+Subject: Header test
+Date: Thu, 2 Aug 2007 20:54:36 +0400
+Message-Id: <20070802165414.M43479@sendinghost.com>
+X-Mailer: Open WebMail 2.50 20050106
+X-OriginatingIP: 255.255.255.255 (username@sendinghost.com)
+MIME-Version: 1.0
+Content-Type: text/plain; charset=koi8-r
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Open_WebMail_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-17.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-17.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-17.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,18 @@
+From: "Username" 
+To: "'Username'" 
+Subject: header test
+Date: Sat, 28 Jul 2007 22:02:14 +0200
+MIME-Version: 1.0
+Content-Type: multipart/alternative;
+              boundary="----=_NextPart_000_0003_01C7D162.F589D6C0"
+X-Mailer: Microsoft Office Outlook, Build 11.0.5510
+X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1081
+Thread-Index: DcfRUi9X1F1oqQo8TgRTgfuoq+BT/A==
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Outlook_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-18.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-18.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-18.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,22 @@
+From: "Username" 
+To: "Username" 
+Subject: header test
+Date: Sat, 28 Jul 2007 14:11:13 +0200
+Message-ID: 
+MIME-Version: 1.0
+Content-Type: text/plain;
+              charset="iso-8859-1"
+Content-Transfer-Encoding: 7bit
+X-Priority: 3 (Normal)
+X-MSMail-Priority: Normal
+X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
+Importance: Normal
+X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Outlook_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-19.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-19.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-19.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,18 @@
+Received: by localhost with Microsoft Mail
+          id <01C7D126.F90D2860@localhost>; Sat, 28 Jul 2007 14:52:52 +0200
+Message-ID: <01C7D126.F90D2860@localhost>
+From: Username 
+To: "'Username'" 
+Subject: header test
+Date: Sat, 28 Jul 2007 14:52:46 +0200
+MIME-Version: 1.0
+Content-Type: text/plain; charset="us-ascii"
+Content-Transfer-Encoding: 7bit
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Outlook_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-2.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-2.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-2.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,20 @@
+Return-Path: 
+Received: from peratt.node ([unix socket])
+           by peratt.node (Cyrus 2.5.11) with LMTPA;
+            Tue, 01 Aug 2017 05:43:56 +0000
+X-Sieve: CMU Sieve 2.4
+Received: from localhost (unknown [23.77.3.4.186.44])
+          by peratt.node (Postfix) with ESMTP id EEBE81638F0E
+          for ; Tue,  1 Aug 2017 05:43:55 +0000 (UTC)
+Received: from peratt.node ([23.77.3.4.186.70])
+ by localhost (maia-lon.uk.plasma.xtcvr.station [23.77.3.4.186.44]) (maiad, port 10024)
+ with ESMTP id 98595-01 for ;
+ Tue,  1 Aug 2017 05:43:55 +0000 (UTC)
+Received: by peratt.node (Postfix, from userid 0)
+          id CF9CD1638AAE; Tue,  1 Aug 2017 05:43:33 +0000 (UTC)
+To: plasma-dipole-3@peratt.node
+Subject: peratt.node monthly run output
+Message-Id: <20170801054337.CF9CD1638AAE@peratt.node>
+Date: Tue,  1 Aug 2017 05:43:33 +0000 (UTC)
+From: plasma-dipole-3@peratt.node (Charlie Plasma-Dipole-3)
+
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-20.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-20.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-20.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,21 @@
+Message-ID: <000f10c7183d$abe4d510$6031a8c0@hostname>
+From: "Username" 
+To: "Username" 
+Subject: Testing
+Date: Wed, 4 Apr 2007 14:11:45 +0100
+MIME-Version: 1.0
+Content-Type: text/plain;
+              charset="iso-8859-1"
+Content-Transfer-Encoding: 7bit
+X-Priority: 3
+X-MSMail-Priority: Normal
+X-Mailer: Microsoft Outlook Express 6.00.2800.1807
+X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1807
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Outlook_Express_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-21.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-21.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-21.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,14 @@
+Date: Tue, 6 Mar 2007 11:10:36 -0500 (EST)
+From: Sender Name 
+To: Getter Name 
+cc: Other Person 
+Subject: The subject text
+Message-ID: 
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Pine_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-22.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-22.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-22.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,18 @@
+Date: Mon, 6 Aug 2007 00:41:44 +0200
+From: Username 
+X-Mailer: The Bat! (v3.95.8) Professional
+X-Priority: 3 (Normal)
+Message-ID: <1398886086.20070806004144@sendinghost.com>
+To: Username 
+Subject: header test. the bat
+MIME-Version: 1.0
+Content-Type: text/plain; charset=us-ascii
+Content-Transfer-Encoding: 7bit
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/The_Bat!_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-23.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-23.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-23.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,23 @@
+DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
+  s=s1024; d=yahoo.com;
+  h=Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
+  b=ql3kRKrhner1LTFFVBgCYI1uqK4+8hrb6d/Fefr/HkLuObQwIrIpEXA1OiagbuFZU+H+ue1anFvm1cHQ4hjpdUcjpIIPL7ldNL9YnOxauugdVW+
+  OpbTvAu0XaGf2t7eBqOWJF0Y5gM7TE27WdElgVRikunfCQca1VFV6KSuQP0o=;
+Received: from [a.b.c.d] by web53409.mail.re2.yahoo.com via HTTP; Sat, 14 Feb 2009 05:42:03 PST
+X-Mailer: YahooMailWebService/0.7.260.1
+Date: Sat, 14 Feb 2009 05:42:03 -0800 (PST)
+From: Sender Name 
+Reply-To: sender@yahoo.com
+Subject: Test Message
+To: recipient@domain.com
+MIME-Version: 1.0
+Content-Type: text/plain; charset=us-ascii
+Message-ID: <695976.86300.qm@web53409.mail.re2.yahoo.com>
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Yahoo!_Mail_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-24.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-24.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-24.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,22 @@
+Return-Path: 
+Received: from blue.light ([unix socket])
+           by blue.light (Cyrus 2.5.11) with LMTPA;
+            Sat, 29 Jul 2017 07:39:07 +0000
+X-Sieve: CMU Sieve 2.4
+Received: from localhost (unknown [147.67.119.44])
+          by blue.light (Postfix) with ESMTP id 2C14C15B9A66
+          for ; Sat, 29 Jul 2017 07:39:07 +0000 (UTC)
+Received: from blue.light ([147.67.119.70])
+ by localhost (maia.lux.europa.eu [147.67.119.44]) (maiad, port 999)
+ with ESMTP id 30818-09 for ;
+ Sat, 29 Jul 2017 07:39:07 +0000 (UTC)
+Received: by blue.light (Postfix, from userid 0)
+          id EAB5515B9A62; Sat, 29 Jul 2017 07:39:06 +0000 (UTC)
+To: der_kommissar@blue.light
+From: der_inspector@blue.light
+Auto-Submitted: auto-generated
+Subject: *** SECURITY information for blue.light ***
+Message-Id: <20170729073906.EAB5515B9A62@blue.light>
+Date: Sat, 29 Jul 2017 07:39:06 +0000 (UTC)
+
+blue.light : Jul 29 07:39:06 : der_inspector : user NOT in sudoers ; TTY=pts/2 ; PWD=/usr/www/der_inspector ; USER=der_kommissar ; COMMAND=/usr/tmp/bin/su -
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-3.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-3.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-3.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,67 @@
+Return-Path: 
+Received: from base2.tranquility.moon ([unix socket])
+           by base2.tranquility.moon (Cyrus 2.5.11) with LMTPA;
+            Sun, 30 Jul 2017 04:48:17 +0000
+X-Sieve: CMU Sieve 2.4
+Received: from localhost (unknown [123.12.188.227.186.44])
+          by base2.tranquility.moon (Postfix) with ESMTP id 995F118B3D5B
+          for ; Sun, 30 Jul 2017 04:48:17 +0000 (UTC)
+Received: from base2.tranquility.moon ([123.12.188.227.186.70])
+ by localhost (maia-lon.uk.x-servatory-2.node.moon [123.12.188.227.186.44]) (maiad, port 10024)
+ with ESMTP id 94027-09 for ;
+ Sun, 30 Jul 2017 04:48:17 +0000 (UTC)
+X-Filterlist: domain auto-accepted by SQLfilter-1.8.0
+Received: from mail-pf0-f178.imperial-planet.com (mail-pf0-f178.imperial-planet.com [209.85.192.178])
+          (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
+          (No client certificate requested)
+          by base2.tranquility.moon (Postfix) with ESMTPS id 24D0B18B3D47
+          for ; Sun, 30 Jul 2017 04:48:15 +0000 (UTC)
+Received: by mail-pf0-f178.imperial-planet.com with SMTP id d67so41277311pfc.0
+        for ; Sat, 29 Jul 2017 21:48:15 -0700 (PDT)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
+        d=spacemail.com; s=20161025;
+        h=from:subject:to:references:message-id:date:user-agent:mime-version
+         :in-reply-to:content-transfer-encoding;
+        bh=+1InCnu+/H07zCd70FnSXWIiaWyNpw6ChtEw8srUzVE=;
+        b=slyfmcEntEwQjiaktV+00DLhfqv05HOhurmn6HoszBv2yYUMyhYLI+7euXSFOdT0wm
+         4pLsiSSf0zqKRwNq0p2D9GuKrcohw7lewyTUv84RxIW+k27Et5W/Z9J5f2jSQ8eaAz2Z
+         WWfZKx1NpPp3zOAI5Zcy6STdwveXt7NeT+8/jL0/IBMORlMOlEY7IzrSxwCCa+l8kxLT
+         t9n64V9UHfqMfdb00rmZK/vjnonVfSzBgpTzhvSyCpzRTi6dRoXArlr6sAnYaCmkog+b
+         VZXqEV7ghjtL2gJLV0FK7U48/8FSR+OLEWKEOnnspTrUA3ew59VlkaVbZ1PAnFE5bcRS
+         AwHg==
+X-Imperial-Planet-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
+        d=1e100.node; s=20161025;
+        h=x-sm-message-state:from:subject:to:references:message-id:date
+         :user-agent:mime-version:in-reply-to:content-transfer-encoding;
+        bh=+1InCnu+/H07zCd70FnSXWIiaWyNpw6ChtEw8srUzVE=;
+        b=UV6ze+PdY0fhoqL2Z9zhg9FYgsYmHyaTxZQa/GfUR3GuA6EYvTzqLBF7cRxttx6rwU
+         qnUtV9FjEbu8Wj7KKFYDTo39OkzZIWJSPsfRcH13CnMjn2f/6XJtiukDCgQZQnm9TE/g
+         Z2lYCoT0kOrH8bepDr+mAZVB/LWfdFM1Eep7yzTwMrRDddsl7CbCrZUsWsGSOL8dtZtz
+         ECdNqGrcQ1QqB4AL250n7chbPwkjE5ofgdpj6jGpNCOb6jHKx4Id2LC3IHq6H7DxHEN7
+         SilI7PR926kRnx01XDS7IPN7vJxaBAwzXIvOoMWfbhkco/3cZYTPdQDduMKwNVNTchAg
+         ABbw==
+X-Sm-Message-State: AIVw112enRFCietlhpx5qREqf/9uwAFRVVIYFzjI+W1o6VxVJTlgailE
+                    COcqLn2g81yUJr5/
+X-Received: by 192.168.110.101.89.6 with SMTP id f6mr11804924pgu.270.1501390093690;
+        Sat, 29 Jul 2017 21:48:13 -0700 (PDT)
+Received: from kbh.local (167.7-67-5-197-54.ptld.explorer.node. [167.7.67.5.197.54])
+        by smtp.imperial-planetmail.com with ESMTPSA id r14sm20432425pfb.70.2017.07.29.21.48.09
+        for 
+        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
+        Sat, 29 Jul 2017 21:48:11 -0700 (PDT)
+From: Spawnof Anu 
+X-Imperial-Planet-Original-From: Spawnof Anu 
+Subject: Fwd: Anu, see 30 new updates from Co.Design, FabLab
+ Hub, and more
+To: admin@base2.tranquility.moon
+References: <71.CB.24428.D995D795@twitter.com>
+X-Forwarded-Message-Id: <71.CB.24428.D995D795@twitter.com>
+Message-ID: <78d4f963-298b-1712-7e28-640bf5892600@unit3.n.mars>
+Date: Sat, 29 Jul 2017 21:48:04 -0700
+User-Agent: Mozilla/5223.0 (Macintosh; Intel Mac OS X 10.995223; rv:5223.0)
+ Gecko/20100101 Thunderbird/5223.2.1
+MIME-Version: 1.0
+In-Reply-To: <71.CB.24428.D995D795@twitter.com>
+Content-Type: text/plain; charset=UTF-8; format=flowed
+Content-Transfer-Encoding: 7bit
+
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-4.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-4.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-4.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,38 @@
+Received: from lists.securityfocus.com (lists.securityfocus.com [205.206.231.19])
+        by outgoing2.securityfocus.com (Postfix) with QMQP
+        id 7E9971460C9; Mon,  9 Jan 2006 08:01:36 -0700 (MST)
+Mailing-List: contact forensics-help@securityfocus.com; run by ezmlm
+Precedence: bulk
+List-Id: 
+List-Post: 
+List-Help: 
+List-Unsubscribe: 
+List-Subscribe: 
+Delivered-To: mailing list forensics@securityfocus.com
+Delivered-To: moderator for forensics@securityfocus.com
+Received: (qmail 20564 invoked from network); 5 Jan 2006 16:11:57 -0000
+From: YJesus 
+To: forensics@securityfocus.com
+Subject: New Tool : Unhide
+User-Agent: KMail/1.9
+MIME-Version: 1.0
+Content-Disposition: inline
+Date: Thu, 5 Jan 2006 16:41:30 +0100
+Content-Type: text/plain;
+  charset="iso-8859-1"
+Content-Transfer-Encoding: quoted-printable
+Message-Id: <200601051641.31830.yjesus@security-projects.com>
+X-HE-Spam-Level: /
+X-HE-Spam-Score: 0.0
+X-HE-Virus-Scanned: yes
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Email_Headers#Sample_Header
+Status: RO
+Content-Length: 586
+Lines: 26
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-5.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-5.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-5.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,18 @@
+Message-ID: <41B5F981.5040504@sendinghost.com>
+Date: Tue, 07 Dec 2004 13:42:09 -0500
+From: User Name 
+User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206)
+X-Accept-Language: en-us, en
+MIME-Version: 1.0
+To: username@receivinghost.com
+Subject: Testing
+Content-Type: text/plain; charset=ISO-8859-1; format=flowed
+Content-Transfer-Encoding: 7bit
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Thunderbird_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-6.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-6.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-6.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,18 @@
+Mime-Version: 1.0 (Apple Message framework v752.2)
+Content-Transfer-Encoding: 7bit
+Message-Id: <38D1C1FD-3C35-4568-925C-FC46CAC0DE8A@sendinghost.com>
+Content-Type: text/plain;
+              charset=US-ASCII;
+              format=flowed
+To: Other User 
+From: User Name 
+Subject: Subject Line
+Date: Mon, 26 Feb 2007 20:21:57 -0500
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Apple_Mail_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-7.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-7.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-7.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,17 @@
+Message-Id: <6.0.0.22.0.20070728180447.02342558@sendinghost.com>
+X-Sender: username@pop.sendinghost.com
+X-Mailer: QUALCOMM Windows Eudora Version 6.0.0.22
+Date: Sat, 28 Jul 2007 18:05:07 +0200
+To: Username 
+From: Username 
+Subject: header test
+Mime-Version: 1.0
+Content-Type: text/plain; charset="us-ascii"; format=flowed
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Eudora_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-8.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-8.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-8.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,17 @@
+Subject: header test
+From: Username 
+To: Username 
+Content-Type: text/plain
+Date: Sat, 28 Jul 2007 11:52:35 +0200
+Message-Id: <1185616355.19231.0.camel@localhost>
+Mime-Version: 1.0
+X-Mailer: Evolution 2.10.1 
+Content-Transfer-Encoding: 7bit
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Evolution_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/headers-example-9.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/headers-example-9.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/headers-example-9.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,24 @@
+DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed;
+        d=gmail.com; s=beta;
+        h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type;
+        b=OITvzFGKQQUjywUQB7U8dQypDAeOGqBIhfcb8VKioP2UU5P2aJL3l2adoyRqSp9h/Fo9A6wY5EIRsfaCWM9ge+EzCob/
+4p85jcEn3uW8dpRyBFQXMuK2q0RMIk3FznrXAM4W5FvoJIPP04qgXErar+/hZq03vEUIErV1v6p2Fy4=
+DomainKey-Signature: a=rsa-sha1; c=nofws;
+        d=gmail.com; s=beta;
+        h=received:message-id:date:from:to:subject:mime-version:content-type;
+        b=oC+hlWhBboQ+RlsKCL4r2pQxpgKRM9iUgCBmw9wZqlEcxj+A3q+fJkDXgLKmI1twfvTHj7GQ3HDzSLzw982UD
++CPh1bPQxkhNbylUBRtwpoFeixIk7OmR2YE1iYrYpQXf3dEcXNfKs7ffoeY18plJNJG0S8RRmXLaR6XqXFVUoo=
+Message-ID: 
+Date: Mon, 5 Mar 2007 09:10:41 -0800
+From: UserName 
+To: OtherUserName 
+Subject: Subject Line
+MIME-Version: 1.0
+x-cc-source-doc: Forensics Wiki
+x-cc-license: Attribution-ShareAlike 2.5 Generic (CC BY-SA 2.5)
+x-cc-license-url: https://creativecommons.org/licenses/by-sa/2.5/
+x-cc-source: http://www.forensicswiki.org/wiki/Gmail_Header_Format
+
+Hello,
+This header content is from Forensics Wiki (CC BY-SA 2.5)
+See x-cc-* header content for attribution.
Index: openacs-4/packages/acs-mail-lite/www/doc/imap-install.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/imap-install.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/imap-install.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,84 @@
+
+  @title;noquote@
+  @context;noquote@
+  
@title@

+  

+    These notes augment nsimap documentation at https://bitbucket.org/naviserver/nsimap.
+  

+  
Get imap from https://github.com/jonabbey/panda-imap

+  
Build errors

+  
If there are errors building panda-imap mentioning to use -fPIC. See its use in following OS specific examples.
+  

+  
nsimap.so installed?

+  
nsimap.so may not be automatically added to the NAVISERVER install directory after a build.
+  

+  
Copy it to the NaviServer install directory's bin directory:

+  cp nsimap.so /usr/local/ns/bin/.
+  
Replace '/usr/local/ns' with the value given in the build for the flag NAVISERVER=
+  

+  
Add nsimap section to NaviServer's config.tcl file

+  
Instead of copy/pasting the nsimap parameters for the config.tcl file from the web instructions,
+    insert this text snip along other module configurations in the config.tcl file:
+    config-nsimap-part.txt
+  

+ 
+  
In the ${server}/modules section of the config.tcl file near the comment "These modules aren't used in standard OpenACS installs",
+    have nsimap loaded by adding this line:
+  

+  ns_param  nsimap    ${bindir}/nsimap.so
+    
+  
Tcl quoting mailbox value

+  
For parsing to work in 'ns_imap open' avoid wrapping 
+    the mailbox value with double quotes. Quote with curly braces only.

+  
This works:

+  
+    set mailbox {{localhost}/mail/INBOX}
+    

+  
These may not parse as expected:

+  
+    set mailbox "{localhost}/mail/INBOX"
+    set mailbox "{{localhost}/mail/INBOX}"
+  

+
+  
Notes on installing nsimap on FreeBSD 10.3-STABLE

+  

+    Build panda-imap with:
+  

+  gmake bsf EXTRACFLAGS=-fPIC
+  

+    Then build nsimap with:
+  

+  
+    gmake NAVISERVER=/usr/local/ns IMAPFLAGS=-I../../panda-imap/c-client/ "IMAPLIBS=../../panda-imap/c-client/c-client.a -L/usr/local/ns/lib -lpam -lgssapi_krb5 -lkrb5"
+  
+  
Note that NaviServer library is referenced in two places in that line,
+    in case your local system locates NaviServer's installation directory elsewhere.

+  
If there are errors during startup related to FD_SETSIZE and nsd crashing, try this to get nsd to not quit unexpectedly during startup:

+  
In the startup script for nsd, add the following before invoking nsd:

+  
+    # aolserver4 recommends descriptors limit (FD_SETSIZE) to be set to 1024, 
+    # which is standard for most OS distributions
+    # For freebsd systems, uncomment following line:
+    ulimit -n 1024
+  

+  
Note: This does not fix any problem associated with a crash, only makes problem evaporate for low volume traffic sites.

+    
+  
Notes on installing nsimap on Ubuntu 16.04 LTS

+  
Install some development libraries:

+  apt-get install libssl-dev libpam-unix2 libpam0g-dev libkrb5-dev
+  
Build panda-imap with:

+  make ldb EXTRACFLAGS=-fPIC
+  
If your system requires ipv4 only, add the flags:
+    IP=4 IP6=4 SSLTYPE=nopwd like this:

+  make ldb EXTRACFLAGS=-fPIC IP=4 IP6=4 SSLTYPE=nopwd
+  
Some of these are defaults, but the defaults weren't recognized on the test system,
+    so they had to be explicitely invoked in this case.
+  

+  

+    Then build nsimap with:
+  

+  
+    make NAVISERVER=/usr/local/ns IMAPFLAGS=-I../../panda-imap/c-client "IMAPLIBS=../../panda-imap/c-client/c-client.a -L/usr/local/ns/lib -lpam -lgssapi_krb5 -lkrb5"
+  
+  
Note that NaviServer library is referenced in two places in that line,
+    in case your local system locates NaviServer's installation directory elsewhere.

Index: openacs-4/packages/acs-mail-lite/www/doc/imap-install.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/imap-install.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/imap-install.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,2 @@
+set title "Install nsimap"
+set context [list [list index "Documentation"] $title]
Index: openacs-4/packages/acs-mail-lite/www/doc/imap-notes.txt
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/imap-notes.txt,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/imap-notes.txt	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,152 @@
+
+Clarifications of nsimap API
+
+command               comment
+ns_imap check #s      Requires a mailbox to be selected first.
+ns_imap list #s ref pattern ?substr?     (ditto)
+
+
+Complete list of nsimap API:
+
+# ns_imap status #s 
+# ns_imap error #s  ???
+# ns_imap expunge #s   Deletes email in trash folder
+# ns_imap ping #s
+# ns_imap check #s
+
+# ns_imap n_msgs #s  Returns total count of messages in mailbox
+# ns_imap n_recent #s Returns count of messages since last ping???
+# For the task of incoming, get value of "Unseen" from ns_imap status
+# 
+
+# ns_imap list #s  list of mailbox using reference and pattern.
+#                    glob with * for all mailboxes or % for * w/o ones in tree
+# ns_imap lsub #s is ns_imap list for only subscribed mailboxes
+
+
+# Options with #session and mailbox or other params
+# ns_imap append #s mailbox text
+# ns_imap copy #s sequence mailbox
+# ns_imap move #s sequence mailbox
+# ns_imap m_create #s mailbox
+# ns_imap m_delete #s mailbox
+# ns_imap m_rename #s mailbox newname
+# ns_imap search # searchCriteria (IMap2 criteria only)
+# ns_imap subscribe #s mailbox
+# ns_imap unsubscribe #s mailbox
+# ns_imap sort #s criteria reverse -flags
+
+#other 
+# ns_imap parsedate datestring
+# ns_imap getquote #s root
+# ns_imap setquota #s root size
+# ns_imap setacl #s mailbox user value
+
+#email specific
+# ns_imap uid #s msgno   (gets UID of msgno)
+# ns_imap struct #s msgno  Returns UID of msgno, internal time info etc. For example:
+'uid 4 flags {} size 1634 internaldate.day 16 internaldate.month 8 internaldate.year 2017 internaldate.hours 21 internaldate.minutes 3 internaldate.seconds 45 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type text encoding 7bit subtype PLAIN lines 10 bytes 169 \
+body.charset utf-8 \
+body.format flowed \
+msgno 3'
+
+another example:
+
+ 'uid 6 flags {} size 3226 internaldate.day 17 internaldate.month 8 internaldate.year 2017 internaldate.hours 9 internaldate.minutes 25 internaldate.seconds 9 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype REPORT \
+body.report-type delivery-status \
+body.boundary 1F5D214C96DE.1502961909/or97.net \
+part.1 {type text encoding 7bit subtype PLAIN description Notification lines 15 bytes 589 body.charset us-ascii} \
+part.2 {type message encoding 7bit subtype DELIVERY-STATUS description {Delivery report} bytes 449} \
+part.3 {type message encoding 7bit subtype RFC822 description {Undelivered Message} lines 28 bytes 1134 message {type text encoding 7bit subtype PLAIN lines 3 bytes 10 body.charset utf-8 body.format flowed}} \
+part.count 3 msgno 5'
+
+some more examples with varying attachments and content. uid 12 has nested example.
+
+'uid 2 flags {} size 2929 internaldate.day 2 internaldate.month 8 internaldate.year 2017 internaldate.hours 8 internaldate.minutes 17 internaldate.seconds 49 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type text encoding 7bit subtype PLAIN lines 1 bytes 6 body.charset utf-8 body.format flowed msgno 1'
+
+'uid 3 flags {} size 1268 internaldate.day 16 internaldate.month 8 internaldate.year 2017 internaldate.hours 20 internaldate.minutes 49 internaldate.seconds 8 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type text encoding 7bit subtype PLAIN lines 1 bytes 6 body.charset utf-8 body.format flowed msgno 2'
+
+'uid 4 flags {} size 1634 internaldate.day 16 internaldate.month 8 internaldate.year 2017 internaldate.hours 21 internaldate.minutes 3 internaldate.seconds 45 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type text encoding 7bit subtype PLAIN lines 10 bytes 169 body.charset utf-8 body.format flowed msgno 3'
+
+'uid 5 flags {} size 3295 internaldate.day 17 internaldate.month 8 internaldate.year 2017 internaldate.hours 9 internaldate.minutes 24 internaldate.seconds 53 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype REPORT body.report-type delivery-status body.boundary 064B414C913C.1502961893/or97.net \
+part.1 {type text encoding 7bit subtype PLAIN description Notification lines 16 bytes 629 body.charset us-ascii} \
+part.2 {type message encoding 7bit subtype DELIVERY-STATUS description {Delivery report} bytes 480} \
+part.3 {type message encoding 7bit subtype RFC822 description 'uid 4 flags {} size 1634 internaldate.day 16 internaldate.month 8 internaldate.year 2017 internaldate.hours 21 internaldate.minutes 3 internaldate.seconds 45 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type text encoding 7bit subtype PLAIN lines 10 bytes 169 body.charset utf-8 body.format flowed msgno 3'
+
+'uid 6 flags {} size 3226 internaldate.day 17 internaldate.month 8 internaldate.year 2017 internaldate.hours 9 internaldate.minutes 25 internaldate.seconds 9 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype REPORT body.report-type delivery-status body.boundary 1F5D214C96DE.1502961909/or97.net \
+part.1 {type text encoding 7bit subtype PLAIN description Notification lines 15 bytes 589 body.charset us-ascii} \
+part.2 {type message encoding 7bit subtype DELIVERY-STATUS description {Delivery report} bytes 449} \
+part.3 {type message encoding 7bit subtype RFC822 description {Undelivered Message} lines 28 bytes 1134 message {type text encoding 7bit subtype PLAIN lines 3 bytes 10 body.charset utf-8 body.format flowed}} \
+part.count 3 msgno 5'
+
+'uid 7 flags A size 4452633 internaldate.day 13 internaldate.month 8 internaldate.year 2017 internaldate.hours 18 internaldate.minutes 7 internaldate.seconds 46 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype MIXED body.boundary ------------EA9B7FF02CD92F6E7BE79B91 \
+part.1 {type text encoding 7bit subtype PLAIN lines 7 bytes 238 body.charset windows-1252 body.format flowed} \
+part.2 {type image encoding base64 subtype JPEG bytes 4447046 disposition ATTACHMENT disposition.filename IMG_5951.JPG body.name IMG_5951.JPG} \
+part.3 {type text encoding 7bit subtype PLAIN lines 4 bytes 27 disposition ATTACHMENT disposition.filename {Attached Message Part} body.charset UTF-8 body.name {Attached Message Part}} \
+part.count 3 msgno 6'
+
+'uid 8 flags {} size 2317061 internaldate.day 10 internaldate.month 8 internaldate.year 2017 internaldate.hours 6 internaldate.minutes 12 internaldate.seconds 14 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype MIXED body.boundary Apple-Mail-83B5FB6A-752A-457E-B193-3A9DFD99C5FB \
+part.1 {type text encoding 7bit subtype PLAIN lines 2 bytes 4 body.charset us-ascii} \
+part.2 {type image encoding base64 subtype JPEG bytes 2312000 disposition INLINE disposition.filename IMG_0725.JPG body.name IMG_0725.JPG body.x-apple-part-url 060021DF-68C0-4C1F-BDC8-FDEE8363B2DF} \
+part.3 {type text encoding 7bit subtype PLAIN lines 3 bytes 25 body.charset us-ascii} \
+part.count 3 msgno 7'
+
+'uid 9 flags {} size 91768 internaldate.day 2 internaldate.month 8 internaldate.year 2017 internaldate.hours 10 internaldate.minutes 33 internaldate.seconds 48 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype MIXED body.boundary 94eb2c11522ea41fb70555c2cc9f \
+part.1 {type multipart encoding 7bit subtype ALTERNATIVE body.boundary 94eb2c11522ea41fb20555c2cc9d \
+part.1 {type text encoding 7bit subtype PLAIN lines 35 bytes 821 body.charset UTF-8 body.format flowed body.delsp yes} \
+part.2 {type text encoding qprint subtype HTML lines 138 bytes 4294 body.charset UTF-8} \
+part.count 2} \
+part.2 {type application encoding base64 subtype PDF bytes 80754 disposition ATTACHMENT disposition.filename 5070312172586741-12.pdf body.name 5070312172586741-12.pdf} \
+part.count 2 msgno 8'
+
+ 'uid 10 flags {} size 11267 internaldate.day 27 internaldate.month 3 internaldate.year 2015 internaldate.hours 11 internaldate.minutes 0 internaldate.seconds 21 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype MIXED body.boundary 001a11c1f5eeecc5f4051243090a \
+part.1 {type text encoding 7bit subtype PLAIN lines 4 bytes 272 body.charset UTF-8} \
+part.2 {type application encoding base64 subtype X-DIA-DIAGRAM bytes 2696 disposition ATTACHMENT disposition.filename xdcpm-openacs-core-github-3.dia body.name xdcpm-openacs-core-github-3.dia} \
+part.3 {type application encoding base64 subtype X-DIA-DIAGRAM bytes 3074 disposition ATTACHMENT disposition.filename xdcpm-openacs-core-github-4.dia body.name xdcpm-openacs-core-github-4.dia} \
+part.count 3 msgno 9'
+
+'uid 11 flags A size 15479 internaldate.day 3 internaldate.month 4 internaldate.year 2015 internaldate.hours 9 internaldate.minutes 59 internaldate.seconds 8 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype MIXED body.boundary ------------080507060504090603040608 \
+part.1 {type text encoding 7bit subtype PLAIN lines 150 bytes 5716 body.charset utf-8 body.format flowed} \
+part.2 {type text encoding base64 subtype PLAIN lines 61 bytes 4542 disposition ATTACHMENT disposition.filename xdcpm-parallel-dev-process1.txt body.charset UTF-8 body.name xdcpm-parallel-dev-process1.txt} \
+part.3 {type application encoding base64 subtype OCTET-STREAM bytes 3000 disposition ATTACHMENT disposition.filename xdcpm-parallel-dev-process1.dia body.x-mac-type 0 body.x-mac-creator 0 body.name xdcpm-parallel-dev-process1.dia} \
+part.count 3 msgno 10'
+
+ 'uid 12 flags {} size 33487 internaldate.day 28 internaldate.month 1 internaldate.year 2017 internaldate.hours 4 internaldate.minutes 15 internaldate.seconds 7 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype MIXED body.boundary ----=_Part_22057419_699298704.1485580507727 \
+part.1 {type multipart encoding 7bit subtype ALTERNATIVE body.boundary ----=_Part_22057420_472295197.1485580507727 \
+part.1 {type text encoding qprint subtype PLAIN lines 87 bytes 2182 disposition INLINE body.charset UTF-8} \
+part.2 {type text encoding qprint subtype X-WATCH-HTML lines 11 bytes 286 disposition INLINE body.charset UTF-8} \
+part.3 {type text encoding qprint subtype HTML lines 703 bytes 26358 disposition INLINE body.charset UTF-8} \
+part.count 3} \
+part.2 {type text encoding base64 subtype CALENDAR lines 13 bytes 1046 disposition ATTACHMENT disposition.filename Apple_Support_Appt.ics} \
+part.count 2 msgno 11'
+
+
+
+
+# ns_imap headers #s msgno ?-array arr_name
+# ns_imap header #s msgno hdrname
+# ns_imap text #s msgno -flags UID/PEEK/INTERNAL (peek doesn't set \Seen flag)
+# ns_imap body #s msgno part -flags UID/PEEK/INTERNAL
+# ns_imap bodystruct #s msgno part -flags 
+# ns_imap delete #s sequence -flags
+# ns_imap undelete #s sequence flags
+
+
+Other useful lib procs provided by ns_map:
+
+  ns_imap parsedate datestring
+        parses date/time string and returns seconds since epoch if date is
+        correct or empty string if not
+
+  ns_imap uid #s msgno
+        returns UID for specified message number
+
+  ns_imap striphtml text ?tags?
+        strips dangerous HTML tags from the given HTML text, 
+        by default it removes body/title/div/object/frame tags.
+        If tags are specified it will use them instead of internal list.
+
+  ns_imap encode type data
+  ns_imap decode type data
+        performs encodeing/decoding given text according given format.
+        type may be one of the following: base64 qprint
Index: openacs-4/packages/acs-mail-lite/www/doc/imap-test.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/imap-test.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/imap-test.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,3 @@
+

+@content;noquote@
+

Index: openacs-4/packages/acs-mail-lite/www/doc/imap-test.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/imap-test.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/imap-test.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,248 @@
+ad_page_contract {
+    Provies a framework for manually testing acs_mail_lite procs
+    A dummy mailbox value provided to show example of what is expected.
+} {
+    {user ""}
+    {password ""}
+    {mailbox {{or97.net}inbox}}
+}
+set user_id [ad_conn user_id]
+set package_id [ad_conn package_id]
+set admin_p [permission::permission_p \
+                 -party_id $user_id \
+                 -object_id $package_id \
+                 -privilege admin ]
+if { !$admin_p } {
+    set content "Requires admin permission"
+    ad_script_abort
+}
+
+#proc email_parse {struct_list {ref ""} } {
+#    foreach {n v} $struct_list {
+#        if { [string match {part.[0-9]*} $n] } {
+#            set subref $ref
+#            append subref [string range $n 4 end]
+#            #puts "\n test1 '${v}' '${subref}'"          
+#            email_parse $v $subref
+# 
+#        } else {
+#            #puts "\t ${ref} > ${n} : ${v}"
+#        }
+#    }
+#    return 1
+#}
+#
+set content "acs-mail-lite/www/doc/imap.tcl start \n\n"
+#ns_log Notice "$content"
+# acs_mail_lite::imap_conn_go
+set fl_list [list ]
+ns_log Notice "test ns_imap open\n"
+set mailbox_host [string range $mailbox 1 [string first "\}" $mailbox]-1]
+
+ns_log Notice "0. mailbox '${mailbox}'"
+
+set conn_id [ns_imap open -mailbox ${mailbox} -user $user -password $password ]
+ns_log Notice "conn_id '${conn_id}'"
+
+# ACL RIGHTS by hub cyrus: kxten 
+# from: https://tools.ietf.org/html/rfc4314.html#page-5
+# k = create mailboxes
+# x delete mailbox
+# t delete messages
+# e perform expunge as a part of close
+# n (obsolete) write shared annotations
+
+
+# Session only Options are:
+set status_list [ns_imap status $conn_id]
+ns_log Notice "ns_imap status $conn_id = '${status_list}'"
+append content "\n status_list '${status_list}'"
+
+set ping [ns_imap ping $conn_id]
+ns_log Notice "ns_imap ping $conn_id = '${ping}'"
+
+set check_ct [ns_imap check $conn_id]
+ns_log Notice "ns_imap check $conn_id = '${check_ct}'"
+
+set n_msgs_ct [ns_imap n_msgs $conn_id]
+ns_log Notice "ns_imap n_msgs $conn_id = '${n_msgs_ct}'"
+append content "\nn_msgs_ct '${n_msgs_ct}'"
+
+set n_recent_ct [ns_imap n_recent $conn_id]
+ns_log Notice "ns_imap n_recent $conn_id = '${n_recent_ct}'"
+
+if { ![f::even_p [llength $status_list] ] } {
+    lappend status_list ""
+}
+array set status_arr $status_list
+#set unseen_idx [lsearch -nocase -exact $status_list "unseen"]
+#set unseen_ct [lindex $status_list $unseen_idx+1]
+append content "\narray get status_arr '[array get status_arr]'"
+set last [expr { $status_arr(Uidnext) - 1 }]
+set first [expr { $last - $status_arr(Messages) + 1 } ]
+set range $first
+append range ":" $last
+append content "\nfirst $first last $last range $range"
+ns_log Notice "first $first last $last range $range"
+
+#set check [ns_imap check $conn_id]
+#ns_log Notice "ns_imap check $conn_id = '${check}'"
+
+#should be:
+#ns_imap list #s ref pattern ?substr?
+# see example:  https://apple.stackexchange.com/questions/105145/what-are-the-default-special-folder-names-for-imap-accounts-in-mail-app-like-dr
+
+# a session from osx terminal:
+
+# A1 LIST "" "%"
+# * LIST (\HasNoChildren) "." "Sent Messages"
+# * LIST (\HasNoChildren) "." "Junk"
+# * LIST (\HasNoChildren) "." "Archive"
+# * LIST (\HasNoChildren) "." "Deleted Messages"
+# * LIST (\HasNoChildren) "." "Notes"
+# * LIST (\HasNoChildren) "." "Drafts"
+# * LIST (\HasNoChildren) "." "INBOX"
+# A1 OK List completed.
+
+# also Example of an IMAP LIST in rfc6154: 
+# https://tools.ietf.org/html/rfc6154#page-7
+# ns_imap list $conn_id $mailbox pattern(* or %) substr
+
+#set list [ns_imap list $conn_id $mailbox_host {}]
+# returns: '{} noselect'  When logged in is not successful..
+# set list [ns_imap list $conn_id $mailbox_host {*}]
+# returns 'INBOX {} INBOX.Trash {} INBOX.sent-mail {}' when really logged in
+# and mailbox_name part of mailbox is "", and mailbox is in form {{mailbox_host}}
+# set list [ns_imap list $conn_id $mailbox_host {%}]
+# returns 'INBOX {}' when really logged in
+# and mailbox_name part of mailbox is ""
+# If mailbox_name exists and is included in mailbox_host, returns '' 
+# If mailbox_name separate from mailbox_host, and exists and in place of %, returns 'mailbox {}'
+# for example 'INBOX.Trash {}'
+
+
+
+#set expunge [ns_imap expunge $conn_id]
+#ns_log Notice "ns_imap expunge $conn_id = '${expunge}'"
+
+
+# ns_imap status #s 
+# ns_imap error #s  ???
+# ns_imap expunge #s ???
+# ns_imap ping #s
+# ns_imap check #s
+# ns_imap list #s  list of mailbox using reference and pattern.
+#                    glob with * for all mailboxes or % for * w/o ones in tree
+# ns_imap lsub #s is ns_imap list for only subscribed mailboxes
+
+
+# Options with #session and mailbox or other params
+# ns_imap append #s mailbox text
+# ns_imap copy #s sequence mailbox
+# ns_imap move #s sequence mailbox
+# ns_imap m_create #s mailbox
+# ns_imap m_delete #s mailbox
+# ns_imap  m_rename #s mailbox newname
+# ns_imap search # searchCriteria (IMap2 criteria only)
+# ns_imap subscribe #s mailbox
+# ns_imap unsubscribe #s mailbox
+# ns_imap sort #s criteria reverse -flags
+
+#other 
+# ns_imap parsedate datestring
+# ns_imap getquote #s root
+# ns_imap setquota #s root size
+# ns_imap setacl #s mailbox user value
+
+set messages1_list [ns_imap search $conn_id ""]
+ns_log Notice "messages1_list' '${messages1_list}'"
+#set messages2_lists [ns_imap sort $conn_id "date" 0 ]
+#ns_log Notice "messages2_lists' '${messages2_lists}'"
+
+
+#set test [ns_imap body $conn_id 1 1]
+#ns_log Notice "imap-test.tcl: test '${test}'"
+#ad_script_abort
+
+foreach msgno $messages1_list {
+
+    set struct_list [ns_imap struct $conn_id $msgno]
+    ns_log Notice "ns_imap struct $msgno: '${struct_list}'"
+    append content "

"
+    append content "struct_list $struct_list 
"
+    # example value:
+    #  'uid 6 flags {} size 3226 internaldate.day 17 internaldate.month 8 internaldate.year 2017 internaldate.hours 9 internaldate.minutes 25 internaldate.seconds 9 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype REPORT body.report-type delivery-status body.boundary 1F5D214C96DE.1502961909/or97.net part.1 {type text encoding 7bit subtype PLAIN description Notification lines 15 bytes 589 body.charset us-ascii} part.2 {type message encoding 7bit subtype DELIVERY-STATUS description {Delivery report} bytes 449} part.3 {type message encoding 7bit subtype RFC822 description {Undelivered Message} lines 28 bytes 1134 message {type text encoding 7bit subtype PLAIN lines 3 bytes 10 body.charset utf-8 body.format flowed}} part.count 3 msgno 5'
+    array unset hh_arr
+    array unset pp_arr
+    acs_mail_lite::imap_email_parse \
+        -headers_arr_name hh_arr \
+        -parts_arr_name pp_arr \
+        -conn_id $conn_id \
+        -msgno $msgno \
+        -struct_list $struct_list
+
+    #    append content "\n hh_arr [array get hh_arr] \n"
+    #    append content "\n pp_arr [array get pp_arr] \n"
+
+    
+    #    set bodystruct_list [ns_imap bodystruct $conn_id $msgno]
+    #    ns_log Notice "ns_imap bodystruct $msgno: '${bodystruct_list}'"
+    # have we read it before?  check again uid's processed
+    
+    #array set headers_arr $struct_list
+    #    ns_imap headers $conn_id $msgno -array headers_arr
+    ##    ns_log Notice "array names headers_arr '[array names headers_arr]'"
+    ##    ns_log Notice "array get headers_arr '[array get headers_arr]'"
+    #    array set headers_arr $struct_list
+    #    set type [acs_mail_lite::email_type -header_arr_name headers_arr]
+    #    ns_log Notice "type '${type}'"
+    # ns_imap headers #s msgno ?-array arr_name
+    # ns_imap header #s msgno hdrname
+    #ns_imap text $conn_id $msgno -flags UID/PEEK/INTERNAL (peek doesn't set \Seen flag)
+
+
+    #    set msg_txt [ns_imap text $conn_id $msgno]
+    #    set msg_start_max [expr { 72 * 15 } ]
+    #    set msg_txtb [string range $msg_txt 0 $msg_start_max]
+    #    if { [string length $msg_txt] > [expr { $msg_start_max + 400 } ] } {
+    #        set msg_txte [string range $msg_txt end-$msg_start_max end]
+    #    } elseif { [string length $msg_txt] > [expr { $msg_start_max + 144 } ] } {
+    #        set msg_txte [string range $msg_txt end-144 end]
+    #    } else {
+    #        set msg_txte ""
+    #    }
+    #    ns_log Notice "ns_imap text $conn_id $msgno msg_txt: \
+        # ${msg_txtb} ...  ${msg_txte}"
+
+    # ns_imap body #s msgno part -flags UID/PEEK/INTERNAL
+    #    set msg_list [ns_imap body $conn_id $msgno part -flags UID/PEEK/INTERNAL
+
+
+
+}
+
+#email specific
+#ns_imap uid #s msgno   (gets UID of msgno)
+# ns_imap headers #s msgno arr_name
+# ns_imap header #s msgno hdrname
+# ns_imap text #s msgno -flags UID/PEEK/INTERNAL (peek doesn't set \Seen flag)
+# ns_imap body #s msgno part -flags UID/PEEK/INTERNAL
+
+
+# ns_imap bodystruct #s msgno part -flags  (a subset of ns_imap struct)
+# ns_imap delete #s sequence -flags
+# ns_imap undelete #s sequence flags
+
+
+
+
+ns_log Notice "0. test ns_imap close"
+set conn_id [acs_mail_lite::imap_conn_close -conn_id $conn_id]
+
+#append content [ns_imap open -mailbox {{hub.org}mail/INBOX} -testdummyparam -novalidatecert -user support -password "" ]
+#set content [acs_mail_lite::imap_conn_go -host ${mailbox} -password $password -user $user]
+append content \n \n [clock seconds]
+
+#set struct_list [list uid 12 flags {} size 33487 internaldate.day 28 internaldate.month 1 internaldate.year 2017 internaldate.hours 4 internaldate.minutes 15 internaldate.seconds 7 internaldate.zoccident 0 internaldate.zhours 0 internaldate.zminutes 0 type multipart encoding 7bit subtype MIXED body.boundary ----=_Part_22057419_699298704.1485580507727 part.1 {type multipart encoding 7bit subtype ALTERNATIVE body.boundary ----=_Part_22057420_472295197.1485580507727 part.1 {type text encoding qprint subtype PLAIN lines 87 bytes 2182 disposition INLINE body.charset UTF-8} part.2 {type text encoding qprint subtype X-WATCH-HTML lines 11 bytes 286 disposition INLINE body.charset UTF-8} part.3 {type text encoding qprint subtype HTML lines 703 bytes 26358 disposition INLINE body.charset UTF-8} part.count 3} part.2 {type text encoding base64 subtype CALENDAR lines 13 bytes 1046 disposition ATTACHMENT disposition.filename Apple_Support_Appt.ics} part.count 2 msgno 11 ]
+
+regsub -all -- {\n} $content {
} content
Index: openacs-4/packages/acs-mail-lite/www/doc/inbound-message-bot.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/inbound-message-bot.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/inbound-message-bot.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,60 @@
+
+  @title;noquote@
+  @context;noquote@
+  
@title@

+
+  

+    Inbound Message Bot is designed to handle more email faster
+    while making available more email content in a consistent, useful way.
+    It prioritizes incoming messages using a variety of indicators
+    into a queue for processing and triggering callbacks.
+  

+  

+    Inbound Message Bot works with the latest version of acs-mail-lite
+    in a general fashion using callbacks.
+  

+

+    The code is general enough to adapt to any message sources,
+    such as social network apps for example.
+  

+  
The first implementation of Message Bot handles email imported from IMAP or MailDir. These can operate concurrently.

+
+
Overview of operation

+  

+    New messages can be processed by setting the package parameter 
+    IncomingFilterProcName to
+    the name of a custom filter that examines headers 
+    of each email and assigns a
+    package_id or modifies other flags based on custom criteria.
+  

+  

+    Incoming attachments are placed in folder acs_root_dir/acs-mail-lite
+    since emails are queued. 
+    Attachments might need to persist passed a system reset, 
+    which may clear a standard system tmp directory used by ad_tmpdir.
+    Note that this is different than the value provided by parameter
+    FilesystemAttachmentsRoot. 
+    FilesystemAttachmentsRoot is for outbound attachments.
+  

+  

+    A callback is subsequently triggered. Packages with a 
+    registered callbacks process the email.
+  

+  

+    When callbacks are finished, email is marked as 'read' by
+    the importing procedure, and deleted from the import queue
+    at a regular interval. An error in one of the callbacks will
+    prevent an email from being deleted without concern that
+    the email will be re-processed by other callback implementations.
+  

Index: openacs-4/packages/acs-mail-lite/www/doc/inbound-message-bot.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/inbound-message-bot.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/inbound-message-bot.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,2 @@
+set title "Inbound Message Bot"
+set context [list [list index "Documentation"] $title]
Index: openacs-4/packages/acs-mail-lite/www/doc/inbound.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/inbound.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/inbound.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,75 @@
+
+  @title;noquote@
+  @context;noquote@
+  
@title@

+
+  

+    Scheduled procs collect the email. 
+    Current methods include:
+
+ acs_mail_lite::imap_check_incoming 
+ for IMAP and 
+ acs_mail_lite::maildir_check_incoming 
+   for MailDir.
+

+    Both schedules procs can be run simultaneously without intefering with
+    each other.
+    In both cases, email is added to the same queue for processing via callbacks.
+  

+  
These procs provide an overview of inbound email processing in general.

+    

+     Legacy Incoming E-mail up to oacs-5-9 for package developers
+
  
Overview

+  

+    A scheduled procedure begins by checking for new incoming email.
+    The interval is set by package parameter IncomingScanRate.
+  

+    ACS Mail Lite detects if an
+    email is responding to email sent from OpenACS (via ACS Mail Lite).
+    It verifies 
+    message_id or originator hashkey via
+    acs_mail_lite::unique_id_parse and identifies
+     bounced or reply emails.
+    

+    Then, the message-id is cross-referenced 
+    for any parameters passed to the email, such as package_id or party_id. 
+  

+    The email is 
+    prioritized
+    and 
+    placed in a queue, where it is pulled from the queue in sequence.
+

+    When pulled, an email is processed by callbacks.
+    
+    A package can be associated with replies to outbound email generated by the same package.
+    This enables each package to deal with incoming email in its own way.
+    For example, a social networking site could log an event 
+    in a special table about subscribers.
+    The package-key is determined from the package_id that sent the email.
+  

+    For bounced email, 
+    the procedure logs a bounced mail event for the user associated 
+    with the bounced email address.
+  

+    A separate process checks if an email account
+    needs to inactivate notifications due to chronic bounce errors:
+  

+  
    
+    
  • If a user's last mail bounced more than
+      MaxDaysToBounce days ago without any further
+      bounced mail then the bounce-record counter gets reset (deleted).
+      ACS Mail Lite assumes the user's email account is working again 
+      and no longer refuses emails.
+      MaxDaysToBounce is a package parameter.
    • 
+    
  • If more than MaxBounceCount emails are returned
+      for a particular user then
+      the account associated with the email stops receiving email
+      notifications channeled through ACS Mail Lite.
+      The email_bouncing_p flag is set to 't'.
    • 
+    
  • To notify users that they will not receive any more email and to
+      tell them how to re-enable the email account in the system,
+      a notification email is sent every
+      NotificationInterval days up to
+      MaxNotificationCount times.
+      It contains a link to re-enable email notifications.
    • 
+  

Index: openacs-4/packages/acs-mail-lite/www/doc/inbound.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/inbound.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/inbound.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,2 @@
+set title "Incoming E-mail"
+set context [list [list index "Documentation"] $title]
Index: openacs-4/packages/acs-mail-lite/www/doc/incoming-email-legacy-notes.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/incoming-email-legacy-notes.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/incoming-email-legacy-notes.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,497 @@
+
+  @title;noquote@
+  @context;noquote@
+  
@title@

+  
The information on this page is volatile and likely changed.
+    Refer to API documentation for specifics on existing system.

+
An incoming email system in general:

+  
    
+	
  1. scans for e-mails on a regular basis.
    2. 
+	
  2. checks if any email came from an auto mailer.
    3. 
+	
  3. Parses new ones, and
    4. 
+	
  4. processes them by firing off callbacks.
    5. 
+  

+ 
+
+
+
Legacy implementation notes for OpenACS 5.9

+  
email filters

+  

+    Vinod has made a check for auto mailers by using procmail as follows.
+    Maybe we could get this dragged into Tcl code (using regexp or a Procmail recipe parser) instead, thereby removing the need for setting up procmail in the first place.
+  

+  

+Revised procmail filters:
+

+
+
+    :0 w * ^subject:.*Out of Office AutoReply /dev/null 
+    :0 w * ^subject:.*Out of Office /dev/null :0 w * ^subject:.*out of the office /dev/null 
+    :0 w * ^subject:.*NDN /dev/null :0 w * ^subject:.*[QuickML] Error: /dev/null 
+    :0 w * ^subject:.*autoreply /dev/null :0 w * ^from.*mailer.*daemon /dev/null
+
+

+
+  

+To make things granular a separate parsing procedure should deal with loading the e-mail into the Tcl interpreter and setting variables in an array for further processing.
+

+
+  
+    ad_proc parse_email { 
+    -file:required
+    -array:required
+    } { 
+    ...
+    }
+
+

+
parsing email

+  

+    An email is split into several parts: headers, bodies and files.
+  

+  
+  

+    The headers consists of a list with header names as keys and their corresponding values.
+    All keys are lower case.
+  

+  
+  

+    The bodies consists of a list with two elements: content-type and content.
+  

+
+  

+    The files consists of a list with three elements: content-type, filename and content.
+  

+
+  

+    An array with all the above data is upvarred to the caller environment.
+  

+
+  

+    Processing an email should result in an array like this:
+  

+
+  
HEADERS

+
+  
    
+	
  • message_id
    • 
+	
  • subject
    • 
+	
  • from
    • 
+	
  • to
    • 
+	
  • date
    • 
+	
  • received
    • 
+	
  • references
    • 
+	
  • in-reply-to
    • 
+	
  • return-path
    • 
+	
  • ...
    • 
+  

+
+  

+    X-Headers:
+  

+
+  
    
+	
  • X-Mozilla-Status
    • 
+	
  • X-Virus Scanned
    • 
+	
  • ...
    • 
+  

+
+  

+    We do not know which headers are going to be available in the e-mail.
+    We set all headers found in the array.
+    The callback implementation then checks if a certain header is present or not.
+  

+
+
+    #get all available headers
+    set keys [mime::getheader $mime -names]
+    
+    set headers [list]
+
+    # create both the headers array and all headers directly for the email array
+    foreach header $keys {
+    set value [mime::getheader $mime $header]
+    set email([string tolower $header]) $value
+    lappend headers [list $header $value]
+    }
+    set email(headers) $headers
+  
+

+
+  
Bodies 

+
+  

+    An e-mail usually consists of one or more bodies.
+    With the advent of complex_send, OpenACS supports sending of multi-part e-mails.
+    Use complex_send when you want to send out and e-mail in text/html and text/plain.
+    Some email clients only recognize text/plain.
+  

+
+
+    switch [mime::getproperty $part content] {
+    "text/plain" {
+    lappend bodies [list "text/plain" [mime::getbody $part]]
+    }
+    "text/html" {
+    lappend bodies [list "text/html" [mime::getbody $part]]
+    }
+    }
+  
+

+
+  
Files

+
+  

+    OpenACS supports tcllib mime functions.
+    Getting incoming files to work is a matter of looking for a part where there exists a "Content-disposition" part.
+    All these parts are file parts.
+    Together with scanning for email bodies, code looks something like this:
+

+
+
+    set bodies [list]
+    set files [list]
+    
+    #now extract all parts (bodies/files) and fill the email array
+    foreach part $all_parts {
+
+    # Attachments have a "Content-disposition" part
+    # Therefore we filter out if it is an attachment here
+    if {[catch {mime::getheader $part Content-disposition}]} {
+    switch [mime::getproperty $part content] {
+    "text/plain" {
+    lappend bodies [list "text/plain" [mime::getbody $part]]
+    }
+    "text/html" {
+    lappend bodies [list "text/html" [mime::getbody $part]]
+    }
+    }
+    } else {
+    set encoding [mime::getproperty $part encoding]
+    set body [mime::getbody $part -decode]
+    set content  $body
+    set params [mime::getproperty $part params]
+    if {[lindex $params 0] == "name"} {
+    set filename [lindex $params 1]
+    } else {
+    set filename ""
+    }
+
+    # Determine the content_type
+    set content_type [mime::getproperty $part content]
+    if {$content_type eq "application/octet-stream"} {
+    set content_type [ns_guesstype $filename]
+    }
+
+    lappend files [list $content_type $encoding $filename $content]
+    }
+    }
+    set email(bodies) $bodies
+    set email(files) $files
+

+
+  

+    Note that the files ie attachments are actually stored in the /tmp directory from where they can be processed further.
+    It is up to the callback to decide if to import the file into OpenACS or not.
+    Once all callbacks have been fired files in /tmp will have to be deleted again though.
+

+
+  
Firing off callbacks 

+
+  

+    Now that we have the e-mail parsed and have an array with all the information, we can fire off the callbacks.
+    The firing should happen in two stages.
+

+
+  

+    The first stage is where we support a syntax like "object_id@yoursite.com".
+

+
+  

+    Second, incoming e-mail could look up the object_type, and then call the callback implementation specific to this object_type.
+    If object_type = 'content_item', use content_type instead. 
+

+
+  
+    ad_proc -public -callback acs_mail_lite::incoming_object_email { -array:required -object_id:required } { }
+
+
+  
+    callback acs_mail_lite::incoming_object_email -impl $object_type -array email -object_id $object_id
+
+
+
+    ad_proc -public -callback acs_mail_lite::incoming_object_email -impl user {
+
+    -array:required
+
+    -object_id:required
+
+    } {
+
+    Implementation of mail through support for incoming emails
+
+    } {
+
+    # get a reference to the email array
+
+    upvar $array email
+
+    # make the bodies an array
+
+    template::util::list_of_lists_to_array $email(bodies) email_body
+
+    if {[exists_and_not_null email_body(text/html)]} {
+
+    set body $email_body(text/html)
+
+    } else {
+
+    set body $email_body(text/plain)
+
+    }
+
+    set reply_to_addr "[party::get_by_email $email(from)]@[ad_url]"
+
+    acs_mail_lite::complex_send \
+
+    -from_addr $from_addr \
+
+    -reply_to $reply_to_addr \
+
+    -to_addr $to_addr \
+
+    -subject $email(subject) \
+
+    -body $body \
+
+    -single_email \
+
+    -send_immediately
+
+    }
+
+

+
+  

+    Object id based implementations are useful for automatically generating "reply-to" addresses.
+    With ProjectManager and Contacts object_id is also handy, because Project / TaskID is prominently placed on the website.
+    If you are working on a task and you get an e-mail by your client that is related to the task, just forward the email to "$task_id@server.com" and it will be stored along with the task.
+    Highly useful :).
+  

+
+  

+    Obviously you could have implementations for:
+  

+
+  
    
+	
  • 
+	  
    
+        forums_forum_id: Start a new topic
+      
    
+	
    • 
+	
  • 
+	  
    
+        forums_message_id: Reply to an existing topic
+      
    
+	
    • 
+	
  • 
+	  
    
+        group_id: Send an e-mail to all group members
+      
    
+	
    • 
+	
  • 
+	  
    
+        pm_project_id: add a comment to a project
+      
    
+	
    • 
+	
  • 
+	  
    
+        pm_task_id: add a comment to a task and store the files in the projects folder (done)
+      
    
+	
    • 
+  

+
+  

+    
+  

+
+  

+    Once the e-mail is dealt with in an object oriented approach we are either done with the message (an object_id was found in the to address) or we need to process it further.
+  

+
+    ad_proc -public -callback acs_mail_lite::incoming_email {
+    -array:required
+    -package_id
+    } {
+    }
+  
+
+

+
+    array set email {}
+    
+    parse_email -file $msg -array email
+    set email(to) [parse_email_address -email $email(to)]
+    set email(from) [parse_email_address -email $email(from)]
+
+    # We execute all callbacks now
+    callback acs_mail_lite::incoming_email -array email
+  
+
+

+
+  

+
+    For this a general callback should exist which can deal with every leftover e-mail and each implementation will check if it wants to deal with this e-mail.
+    How is this check going to happen? As an example, a package could have a prefix, as is the case with bounce e-mails as handled in acs_mail_lite::parse_bounce_address (see below):
+

+
+
+    ad_proc -public -callback acs_mail_lite::incoming_email -impl acs-mail-lite {
+    -array:required
+    -package_id:required
+    } {
+    @param array        An array with all headers, files and bodies. To access the array you need to use upvar.
+    @param package_id   The package instance that registered the prefix
+    @return             nothing
+    @error
+    } {
+    upvar $array email
+
+    set to [acs_mail_lite::parse_email_address -email $email(to)]
+    ns_log Debug "acs_mail_lite::incoming_email -impl acs-mail-lite called. Recepient $to"
+
+    util_unlist [acs_mail_lite::parse_bounce_address -bounce_address $to] user_id package_id signature
+    
+    # If no user_id found or signature invalid, ignore message
+    # Here we decide not to deal with the message anymore
+
+
+
+    if {[empty_string_p $user_id]} {
+    if {[empty_string_p $user_id]} {
+    ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: No equivalent user found for $to"
+    } else {
+    ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: Invalid mail signature $signature"
+    }
+    } else {
+    ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: Bounce checking $to, $user_id"
+    
+    if { ![acs_mail_lite::bouncing_user_p -user_id $user_id] } {
+    ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: Bouncing email from user $user_id"
+    # record the bounce in the database
+    db_dml record_bounce {}
+    
+    if {![db_resultrows]} {
+    db_dml insert_bounce {}
+    }
+    }
+    }
+    }
+    
+
+  
+
+

+
+  

+    Alternatively we could just check the whole to address for other things, e.g. if the to address belongs to a group (party).
+

+
+
+    ad_proc -public -callback acs_mail_lite::incoming_email -impl contacts_group_mail {
+    -array:required
+    {-package_id ""}
+    } {
+    Implementation of group support for incoming emails
+    
+    If the to address matches an address stored with a group then send out the email to all group members
+
+    @author Malte Sussdorff (malte.sussdorff@cognovis.de)
+    @creation-date 2005-12-18
+
+    @param array        An array with all headers, files and bodies. To access the array you need to use upvar.
+    @return             nothing
+    @error
+    } {
+
+    # get a reference to the email array
+    upvar $array email
+
+    # Now run the simplest mailing list of all
+    set to_party_id [party::get_by_email -email $email(to)]
+    
+    if {[db_string group_p "select 1 from groups where group_id = :to_party_id" -default 0]} {
+    # make the bodies an array
+    template::util::list_of_lists_to_array $email(bodies) email_body
+    
+    if {[exists_and_not_null email_body(text/html)]} {
+    set body $email_body(text/html)
+    } else {
+    set body $email_body(text/plain)
+    }
+    
+    acs_mail_lite::complex_send \
+    -from_addr [lindex $email(from) 0] \
+    -to_party_ids [group::get_members -group_id $to_party_id] \
+    -subject $email(subject) \
+    -body $body \
+    -single_email \
+    -send_immediately
+
+    }
+    } 
+  
+
+

+
+  

+    Or check if the to address follows a certain format.
+

+
+
+    ad_proc -public -callback acs_mail_lite::incoming_email -impl contacts_mail_through {
+    -array:required
+    {-package_id ""}
+    } {
+    Implementation of mail through support for incoming emails
+    
+    You can send an e-amil through the system by sending it to user#target.com@yoursite.com
+    The email will be send from your system and if mail tracking is installed the e-mail will be tracked.
+
+    This allows you to go in direct communication with a customer using you standard e-mail program instead of having to go to the website.
+
+    @author Malte Sussdorff (malte.sussdorff@cognovis.de)
+    @creation-date 2005-12-18
+    
+    @param array        An array with all headers, files and bodies. To access the array you need to use upvar.
+    @return             nothing
+    @error
+    } {
+    # get a reference to the email array
+    upvar $array email
+
+    # Take a look if the email contains an email with a "#"
+    set pot_email [lindex [split $email(to) "@"] 0]
+    if {[string last "#" $pot_email] > -1} {
+    ....
+    }
+    }
+
+

+
+  

+    Alternatives to this are:
+  

+
+  
    
+	
  • ${component_name}-bugs@openacs.org (where component_name could be openacs or dotlrn or contacts or whatever), to store a new bug in bug-tracker.
    • 
+	
  • username@openacs.org (to do mail-through using the user name, which allows you to hide the actual e-mail of the user whom you are contacting).
    • 
+  

+
+  
Cleanup

+
+  

+Once all callbacks have been fired off,  e-mails need to be deleted from the Maildir directory and files which have been extracted need to be deleted as well from the /tmp directory. 
+

Index: openacs-4/packages/acs-mail-lite/www/doc/incoming-email-legacy-notes.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/incoming-email-legacy-notes.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/incoming-email-legacy-notes.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,2 @@
+set title "Incoming E-mail legacy notes"
+set context [list [list index "Documentation"] $title]
Index: openacs-4/packages/acs-mail-lite/www/doc/incoming-email.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/incoming-email.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/incoming-email.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,31 @@
+
+  @title;noquote@
+  @context;noquote@
+  
@title@

+
+  

+    Incoming E-Mail in OpenACS works with the latest version of acs-mail-lite in a general fashion using callbacks to interface with individual package features.
+  

+
+  
Processing incoming e-mail

+  

+    A scheduled like acs_mail_lite::inbound_queue_pull processes incoming email by loading each email and triggering callbacks.
+  

+
+  
notifications package

+  
Alternately, return email can be processed via notifications package.
+    Forums package uses this method. See usage of notification::reply::get
+    in forums/tcl/forum-reply-procs.tcl. notification::reply::get is
+    defined in notifications/tcl/notification-reply-docs.tcl.
+    To use this method, install the notifications package, and 
+    read documentation in notifications package.
+  

+    Note: Notifications package requires ACS-Mail-Lite package.
+    ACS-Mail-Lite is the most direct way of interfacing with email
+    send and receive.
+  

+
+
Release Notes

+
+
Please file bugs in the Bug Tracker.

+
Index: openacs-4/packages/acs-mail-lite/www/doc/incoming-email.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/incoming-email.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/incoming-email.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,2 @@
+set title "Incoming E-mail for package developers"
+set context [list [list index "Documentation"] $title]
Index: openacs-4/packages/acs-mail-lite/www/doc/index.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/index.adp,v
diff -u -N -r1.2 -r1.3
--- openacs-4/packages/acs-mail-lite/www/doc/index.adp	7 Aug 2017 23:47:57 -0000	1.2
+++ openacs-4/packages/acs-mail-lite/www/doc/index.adp	17 Feb 2018 17:08:31 -0000	1.3
@@ -1,74 +1,42 @@
-
-{/doc/acs-mail-lite {ACS Mail Services Lite}} {User Documentation for ACS Mail Lite}
-User Documentation for ACS Mail Lite
 
-
User Documentation for ACS Mail Lite

-
-Acs Mail Lite handles sending of email via sendmail or smtp and
-includes a bounce management system for invalid email accounts.
-
When called to send a mail, the mail will either get sent
-immediately or placed in an outgoing queue (changeable via
-parameter) which will be processed every few minutes.

-
ACS Mail Lite uses either sendmail (you have to provide the
-location of the binary as a parameter) or SMTP to send the mail. If
-the sending fails, the mail will be placed in the outgoing queue
-again and be given another try a few minutes later when processing
-the queue again.

-
Each email contains an X-Envelope-From address constructed as
-follows:

-The address starts with "bounce" (can be changed by a
-parameter) followed by the user_id, a hashkey and the package_id of
-the package instance that sent the email, separated by
-"-". The domain name of this address can be changed with
-a parameter.

-
The system checks every 2 minutes (configurable) in a certain
-maildirectory (configurable) for newly bounced emails, so the
-mailsystem will have to place every mail to an address beginning
-with "bounce" (or whatever the appropriate parameter
-says) in that directory. The system then processes each of the
-bounced emails, strips out the message_id and verifies the hashkey
-in the bounce-address. After that the package-key of the package
-sending the original mail is found out by using the package_id
-provided in the bounce address. With that, the system then tries to
-invoke a callback procedure via a service contract if one is
-registered for that particular package-key. This enables each
-package to deal with bouncing mails on their own - probably logging
-this in special tables. ACS Mail Lite then logs the event of a
-bounced mail of that user.

-
Every day a procedure is run that checks if an email account has
-to be disabled from receiving any more mail. This is done the
-following way:

+  @title;noquote@
+  @context;noquote@
+  
@title@

+  

+    Acs Mail Lite provides a standard way for OpenACS packages to send and receive email.
+  

+  
features

 
    
-
  • If a user received his last mail X days ago without any further
-bounced mail then his bounce-record gets deleted since it can be
-assumed that his email account is working again and no longer
-refusing emails. This value can be changed with the parameter
-"MaxDaysToBounce".
  • If more then Y emails were returned by a particular user then
-his email account gets disabled from receiving any more mails from
-the system by setting the email_bouncing_p flag to t. This value
-can be changed with the parameter "MaxBounceCount".
  • To notify users that they will not receive any more mails and
-to tell them how to reenable the email account in the system again,
-a notification email gets sent every 7 days (configurable) up to 4
-times (configurable) that contains a link to reenable the email
-account.
    • 
+  
  • API for sending email
    • 
+  
  • Outgoing email processed via SMTP or system's sendmail agent.
    • 
+  
  • Bounce detection for outgoing email
    • 
+  
  • Incoming email processed via IMAP4 or system's MailDir facility.
    • 
+  
  • API for receiving email via custom method
    • 
+  
  • Callback hooks to other packages triggered by replies to sent email
    • 
+  
  • Callback hooks to other packages triggered by incoming email events
    • 
 

 
-To use this system here is a quick guide how to do it with postfix.
-
    
-
  • Edit /etc/postfix/main.cf
-
      
-
    • Set "recipient_delimiter" to " - "
    • Set "home_mailbox" to "Maildir/"
    • Make sure that /etc/postfix/aliases is hashed for the alias
-database
      • 
-
    
-
  • Edit /etc/postfix/aliases. Redirect all mail to
-"bounce" (if you leave the parameter as it was) to
-"nsadmin" (in case you only run one server).
    • 
-

-
-In case of multiple services on one system, create a bounce email
-for each of them (e.g. changeing "bounce" to
-"bounce_service1") and create a new user that runs the
-aolserver process for each of them. You do not want to have
-service1 deal with bounces for service2.
-
Release Notes

-
Please file bugs in the Bug Tracker.

+  
Contents

+  
+  
+  
Release Notes

+  
+  

+    A new IMAP4 feature. See planning notes and development and change notes for change details and reasoning.
+  

+  

+    Please file bugs in the Bug Tracker.
+  

Index: openacs-4/packages/acs-mail-lite/www/doc/index.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/index.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/index.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,2 @@
+set title "ACS Mail Lite Documentation"
+set context [list $title]
Index: openacs-4/packages/acs-mail-lite/www/doc/maildir-actives-reset.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/maildir-actives-reset.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/maildir-actives-reset.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,3 @@
+

+@content;noquote@
+

Index: openacs-4/packages/acs-mail-lite/www/doc/maildir-actives-reset.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/maildir-actives-reset.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/maildir-actives-reset.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,22 @@
+ad_page_contract {
+    Provies a framework for manually testing acs_mail_lite procs
+    A dummy mailbox value provided to show example of what is expected.
+} {
+    {mail_dir ""}
+}
+set user_id [ad_conn user_id]
+set package_id [ad_conn package_id]
+set admin_p [permission::permission_p \
+                 -party_id $user_id \
+                 -object_id $package_id \
+                 -privilege admin ]
+if { !$admin_p } {
+    set content "Requires admin permission"
+    ad_script_abort
+}
+
+set content "www/doc/maildir-actives-reset"
+#nsv_set acs_mail_lite sj_actives_list /lrange /nsv_get acs_mail_lite sj_actives_list/ end end/
+nsv_set acs_mail_lite sj_actives_list [list]
+append content ".. done."
+
Index: openacs-4/packages/acs-mail-lite/www/doc/maildir-install.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/maildir-install.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/maildir-install.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,128 @@
+
+  @title;noquote@
+  @context;noquote@
+  
@title@

+
+
+  
Postfix MailDir installation on Linux OS

+
+  
For dynamic originator address handling of replies,
+    one must have an understanding of postfix configuration basics. See http://www.postfix.org/BASIC_CONFIGURATION_README.html.
+  

+  
A default Postfix installation tends to work with a fixed email address without many modifications to configuration. 
+  

+  

+    These instructions use the following example values:
+  

+
+  
    
+	
  • hostname: www.yourserver.com
    • 
+	
  • oacs server user: service0
    • 
+	
  • OS: Linux
    • 
+	
  • email user: service0
    • 
+	
  • email's home dir: /home/service0
    • 
+	
  • email user's mail dir: /home/service0/MailDir
    • 
+  

+
+  

+    Important: 
+    The email user service0 does not have a ".forward" file. 
+    This user is only used for running the OpenACS website. 
+    Follow strict configuration guidelines to avoid email looping back unchecked.
+  

+  
The oacs server user needs to have read and write permissions to email user's mail dir. 
+For test cases, one may use a common OS user for oacs server user and email user. 
+For more strict server configurations, use a different OS user for each, and grant permission for oacs server user to access files and directories in email user's mail dir.
+  

+    For Postfix, the email user and oacs user do not have to be the same. 
+    Furthermore, Postfix makes distinctions between virtual users and user aliases.
+  

+    Future versions of this documentation should use examples with different names to help distinguish between standard configuration examples and the requirements of ACS Mail Lite package.
+  

+
+  

+    Postfix configuration parameters:
+  

+
+  
+    myhostname=www.yourserver.com
+
+    myorigin=$myhostname
+
+    inet_interfaces=$myhostname, localhost
+
+    mynetworks_style=host
+
+    virtual_alias_domains = www.yourserver.com
+
+    virtual_maps=regexp:/etc/postfix/virtual
+
+    home_mailbox=MailDir/

+
+  

+    Here is the sequence to follow when installing email service on system for first time. If your system already has email service, adapt these steps accordingly:
+  

+
+  
    
+	
  1. Install Postfix
    2. 
+	
  2. Install SMTP (for Postfix)
    3. 
+	
  3. Edit /etc/postfix/main.cf
+      
    • Set "recipient_delimiter" to " - "
      • 
+        
    • Set "home_mailbox" to "Maildir/"
+        
      • 
+        
    • Make sure that /etc/postfix/aliases is hashed for the alias database
+        
      • 
+      
    
+    
    4. 
+    
  4. Edit /etc/postfix/aliases. Redirect all mail to "bounce". 
+      If you're only running one server, 
+      using user "nsadmin" maybe more convenient.
+      In case of multiple services on one system, 
+      create a bounce email for each of them by changing "bounce" to "bounce_service1", bounce_service2 et cetera.
+      Create a new user for each NaviServer process.
+      You do not want to have service1 deal with bounces for service2.
+    
    5. 
+    
  5. Edit /etc/postfix/virtual.
+      Add a regular expression to filter relevant incoming emails for processing by OpenACS. 
+	  @www.yourserver.com service0
+	
    6. 
+	
  6. Edit /etc/postfix/master.cf
+      Uncomment this line so Postfix listens to emails from internet:
+	  smtp inet n - n - - smtpd
+	
    7. 
+	
  7. Create a mail directory as service0
+	  mkdir /home/service0/mail
+	
    8. 
+	
  8. Configure ACS Mail Lite parameters
+	  BounceDomain: www.yourserver.com
    
+	    BounceMailDir: /home/service0/MailDir
    
+	    EnvelopePrefix: bounce
    
+	    
    
+	    The EnvelopePrefix is for bounce e-mails only.
+	
    9. 
+	
  9. Configure Notifications parameters
    
+	  EmailReplyAddressPrefix: notification
    
+	    EmailQmailQueueScanP: 0
    
+	    
    
+	    We want acs-mail-lite incoming handle the Email Scanning, not each package separately.    
+	  Configure other packages likewise
    
+	
    10. 
+	
  10. Invoke postmap in OS shell to recompile virtual db:
+	  postmap /etc/postfix/virtual
+	
    11. 
+	
  11. Restart Postfix. 
    
+	  /etc/init.d/postfix restart
+	
    12. 
+	
  12. Restart OpenACS
    13. 
+  

+  

+    Developer note: Parameters should be renamed: 

+    BounceDomain to IncomingDomain

+    BounceMailDir to IncomingMaildir

+    EnvelopePrefix to BouncePrefix

+    ..to reflect that acs-mail-lite is capable of dealing with other types of incoming e-mail.
+  

+	Furthermore, 
+    setting IncomingMaildir parameter clarifies that incoming email handling is setup. 
+    This is useful for other packages to determine if they can rely on incoming e-mail working (e.g. to set the reply-to email to an  e-mail address which actually works through a callback if the IncomingMaildir parameter is enabled).
+  

Index: openacs-4/packages/acs-mail-lite/www/doc/maildir-install.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/maildir-install.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/maildir-install.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,2 @@
+set title "Install Postfix MailDir"
+set context [list [list index "Documentation"] $title]
Index: openacs-4/packages/acs-mail-lite/www/doc/maildir-test.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/maildir-test.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/maildir-test.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,3 @@
+

+@content;noquote@
+

Index: openacs-4/packages/acs-mail-lite/www/doc/maildir-test.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/maildir-test.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/maildir-test.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,152 @@
+ad_page_contract {
+    Provies a framework for manually testing acs_mail_lite procs
+    A dummy mailbox value provided to show example of what is expected.
+} {
+    {mail_dir ""}
+}
+set user_id [ad_conn user_id]
+set package_id [ad_conn package_id]
+set admin_p [permission::permission_p \
+                 -party_id $user_id \
+                 -object_id $package_id \
+                 -privilege admin ]
+if { !$admin_p } {
+    set content "Requires admin permission"
+    ad_script_abort
+}
+
+set content "www/doc/maildir-test start
"
+
+if { $mail_dir eq "" } {
+    # assume Ubuntu linux install based on installation script
+    set mail_dir "/home/nsadmin/Maildir"
+}
+if { ![string match {/new/*} $mail_dir] } {
+    append mail_dir "/new/*"
+}
+set messages_list [glob -nocomplain $mail_dir]
+set s " : "
+
+
+set var_list [list msg m_id c_type header_names_list headers_list part_ids_list body_parts_list datetime_cs property_names_list property_list body_parts_list ]
+
+set var2_list [list part_id p_header_names_list p_headers_list p_property_names_list p_property_list p_body_parts_list ]
+
+foreach msg $messages_list {
+    
+    set m_id [mime::initialize -file $msg]
+
+    set header_names_list [mime::getheader $m_id -names]
+    # a header returns multiple values in a list, if header element is repeated in email.
+    set headers_list [mime::getheader $m_id]
+
+    set r_idx [lsearch -nocase $header_names_list "received"]
+    if { $r_idx > -1 } {
+        set r_nam [lindex $header_names_list $r_idx]
+        array set h_arr $headers_list
+        if { [llength $h_arr(${r_nam}) ] > 1 } {
+            set received_str [lindex $h_arr(${r_nam}) 0 ]
+        } else {
+            set received_str $h_arr(${r_nam})
+        }
+        if { [regexp -nocase -- {([a-z][a-z][a-z][ ,]+[0-9]+ [a-z][a-z][a-z][ ]+[0-9][0-9][0-9][0-9][ ]+[0-9][0-9][:][0-9][0-9][:][0-9][0-9][ ]+[\+\-][0-9]+)[^0-9]} $received_str match received_ts] } {
+            set age_s [mime::parsedatetime $received_ts rclock]
+            ns_log Notice "maildir-test.tcl.30 rclock $age_s"
+            set datetime_cs [expr { [clock seconds] - $age_s } ]
+        }
+    }
+
+    set property_names_list [mime::getproperty $m_id -names]
+    set property_list [mime::getproperty $m_id]
+
+
+    # following group are redundant to mime::getproperty $m_id
+    set params_list [mime::getproperty $m_id params]
+    set encoding_s [mime::getproperty $m_id encoding]
+    set content_s [mime::getproperty $m_id content]
+    ns_log Notice "maildir-test.tcl.22 m_id '${m_id}' content_s '${content_s}'"
+    set size_s [mime::getproperty $m_id size]
+
+    if { [string match "multipart/*" $content_s] \
+             || [string match -nocase "inline*" $content_s ] } {
+        # or 'parts' exists in property_list
+
+        set part_ids_list [mime::getproperty $m_id parts]
+
+    } else {
+        # this is a leaf
+#        set body_parts_list [mime::getbody $m_id]
+        set body_parts_list [mime::buildmessage $m_id]
+        set bpl "
"
+        append bpl [string range $body_parts_list 0 240]
+        append bpl "
.. ..
" [string range $body_parts_list end-120 end]
+        append bpl "
"
+        set body_parts_list $bpl
+    }
+
+
+
+
+
+
+
+    append content "

New message

"
+    foreach var $var_list {
+        if { [info exists $var] } {
+            append content $var $s [set $var] " 

"
+        }
+
+    }
+
+    if { [info exists part_ids_list ] } {
+        foreach part_id $part_ids_list {
+            set p_header_names_list [mime::getheader $part_id -names]
+            set p_headers_list [mime::getheader $part_id]
+            set p_property_names_list [mime::getproperty $part_id -names]
+            set p_property_list [mime::getproperty $part_id ]
+            # includes params size content encoding
+            # params is like flags in IMAP
+            set p_params_list [mime::getproperty $part_id params]
+            set p_content_s [mime::getproperty $part_id content]
+            ns_log Notice "maildir-test.tcl.63 part_id '${part_id}' p_content_s '${p_content_s}'"
+            set p_size_s [mime::getproperty $part_id size]
+            if { [string match "multipart/*" $p_content_s] \
+                     || [string match -nocase "inline*" $p_content_s ] } {
+
+                set p_part_ids_list [mime::getproperty $part_id parts]
+
+            } else {
+                # this is a leaf
+                set p_encoding_s [mime::getproperty $part_id encoding]
+#                set p_body_parts_list [mime::getbody $part_id]
+                set p_body_parts_list [mime::buildmessage $part_id]
+                set bpl "
"
+                append bpl [string range $p_body_parts_list 0 240]
+                append bpl "
.. ..
" [string range $p_body_parts_list end-120 end]
+                append bpl "
"
+                set p_body_parts_list $bpl
+            }
+            
+#            append content "part_id '${part_id}'
"
+            foreach var $var2_list {
+                if { [info exists $var] } {
+                    append content $var $s [set $var] " 

"
+                }
+            }
+        }
+    }
+    # cleanup current message
+    foreach var $var_list {
+        if { [info exists $var] && $var ne "msg" && $var ne "m_id" } {
+            unset $var
+        }
+    }
+    foreach var $var2_list {
+        if { [info exists $var] } {
+            unset $var
+        }
+    }
+    mime::finalize $m_id -subordinates all
+
+
+}
Index: openacs-4/packages/acs-mail-lite/www/doc/outbound.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/outbound.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/outbound.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,88 @@
+
+  @title;noquote@
+  @context;noquote@
+  
@title@

+
+  

+    Email is sent via sendmail or SMTP. If SMTP is not configured,
+    sendmail is assumed.
+  

+    A bounce management system is available for tracking accounts with
+    email that have issues receiving email. 
+  

+    Email can be sent immediately
+    or placed in an outgoing queue that
+    is processed at regular intervals.
+    Package parameter sendImmediatelyP sets the default.
+  

+    If sending fails, mail to send is put in the outgoing queue
+    again. The queue is processed every few minutes.
+  

+  
A legacy description of the process

+

+Acs Mail Lite handles sending of email via sendmail or smtp
+and includes a bounce management system for invalid email
+accounts.
+

+When called to send a mail, the mail will either get sent immediately
+or placed in an outgoing queue (changeable via parameter) which
+will be processed every few minutes.
+

+    Each outbound email contains an
+    "X-Envelope-From <address@IncomingDomain>" header.
+    The address part consists of values from package parameter
+    EvenlopePrefix
+    followed by the email sender's user_id, a hashkey,
+    and the package_id of the
+    package instance that is sending the email.
+    The address components are separated by a dash ("-").
+    IncomingDomain refers to the value of package parameter IncomingDomain.
+  

+ACS Mail Lite uses either sendmail (you have to provide the
+location of the binary as a parameter) or SMTP to send the mail.
+If the sending fails, the mail will be placed in the outgoing queue
+again and be given another try a few minutes later when processing
+the queue again.
+

+Each email contains an X-Envelope-From address constructed as
+follows:

+The address starts with "bounce" (can be changed by a parameter)
+followed by the user_id, a hashkey and the package_id of the
+package instance that sent the email, separated by "-". The
+domain name of this address can be changed with a parameter.
+

+The system checks every 2 minutes (configurable) in a certain
+maildirectory (configurable) for newly bounced emails, so the
+mailsystem will have to place every mail to an address beginning
+with "bounce" (or whatever the appropriate parameter says) in that
+directory. The system then processes each of the bounced emails,
+strips out the message_id and verifies the hashkey in the bounce-address.
+After that the package-key of the package sending the original mail
+is found out by using the package_id provided in the bounce
+address. With that, the system then tries to invoke a callback
+procedure via a service contract if one is registered for that
+particular package-key. This enables each package to deal with
+bouncing mails on their own - probably logging this in special tables.
+ACS Mail Lite then logs the event of a bounced mail of that
+user.
+

+Every day a procedure is run that checks if an email account
+has to be disabled from receiving any more mail. This is done
+the following way:
+

+
    
+
  • If a user received his last mail X days ago without any further
+bounced mail then his bounce-record gets deleted since it can
+be assumed that his email account is working again and no longer
+refusing emails. This value can be changed with the parameter
+"MaxDaysToBounce".
    • 
+
  • If more then Y emails were returned by a particular user then
+his email account gets disabled from receiving any more mails
+from the system by setting the email_bouncing_p flag to t. This
+value can be changed with the parameter "MaxBounceCount".
    • 
+
  • To notify users that they will not receive any more mails and to
+tell them how to reenable the email account in the system again,
+a notification email gets sent every 7 days (configurable)
+up to 4 times (configurable) that contains a link to reenable
+the email account.
    • 
+

Index: openacs-4/packages/acs-mail-lite/www/doc/outbound.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/outbound.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/outbound.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,2 @@
+set title "Outgoing E-mail"
+set context [list [list index "Documentation"] $title]
Index: openacs-4/packages/acs-mail-lite/www/doc/setup.adp
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/setup.adp,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/setup.adp	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,38 @@
+
+  @title;noquote@
+  @context;noquote@
+  
@title@

+  

+    Messages can come from a variety of sources.
+    Handling replies to outbound email is a common case.
+  

+  Currently there are these distinct paradigms for setting up return email:
+

+  
    
+    
  1. 
+      A fixed outbound email address. FixedSenderEmail parameter defines
+      the fixed email address used. Each package sending email can create
+      and set its own FixedSenderEmail parameter. The default
+      is to use ACS-Mail-Lite's parameter. 
+      As an originating SMTP agent, orignator is set to the
+      ACS-Mail-Lite's parameter, if it is not empty.
+      The replying email's message-id is used to reference any mapped
+      information about the email, such as package_id or object_id.
+      The message-id includes a signed signature to detect and reject
+      a tampered message-id, and prevents publishing of system ids.
+    
  2. 
+      A dynamic originator address that results in 
+      a custom return email address for each outbound email. 
+      This provides an alternate way to supply the original message_id key, 
+      if the message_id key is altered. 
+      For this to work, the email system must be configured to
+      accept email to any account at the domain specified by the
+      BounceDomain parameter.
+    
    3. 
+  

+
+  
IMAP

+  
After installing nsimap, setup consists of filling out the relevant parameters in the acs-mail-lite package, mainly: BounceDomain, FixedSenderEmail and the IMAP section.
+

+  
Postfix MailDir on Linux OS

+  
After installing and configuring Postifx, a setup consists of filling out the relevant parameters in the acs-mail-lite package.
Index: openacs-4/packages/acs-mail-lite/www/doc/setup.tcl
===================================================================
RCS file: /usr/local/cvsroot/openacs-4/packages/acs-mail-lite/www/doc/setup.tcl,v
diff -u -N
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ openacs-4/packages/acs-mail-lite/www/doc/setup.tcl	17 Feb 2018 17:08:31 -0000	1.1
@@ -0,0 +1,2 @@
+set title "Setup and configuration"
+set context [list [list index "Documentation"] $title]