%myvars; ]> By Bryan Quinn and Todd Nightingale The following is a requirements document for the OpenACS Package Manager (APM), version 4.0 (APM4). APM4 offers a superset of APM v3.3 functionality with the following specific enhancements: A public procedural API. (v 3.3 only has web-based UI) Support for dependency checking. Support for compound packages (to support installation chaining). Support for on-line parameter setting. Support for sub-site level configuration (requires revised parameter and /admin pages at sub-site level; deprecation of site-wide parameter file). To differentiate these new requirements from the requirements of version 3.3, all requirements new in v4 are prefaced with the number 4. We gratefully acknowledge the authors of APM 3 for their original design documentation which suggested these features, as well as the influence of the design and open-source implementation of the Red Hat Package manager, the Debian packaging system, and PERL's CPAN in the development of the ideas behind this document. A typical website will tend to offer its users a number of web-based services or applications, e.g. a bulletin board, calendaring, classified ads, etc. A website may also have underlying subsystems, such as a permission system, content management system, etc. For such applications and subsystem components, modularity - or the degree to which a component can be encapsulated and decoupled from the rest of the system - is of great value. Thus the OpenACS Package Manager (APM) was created to allow website components, or packages, to be added, removed, and upgraded easily, with minimum disturbance to the rest of the system. This allows site owners to steadily offer users new and improved services, and also allows programmers to quickly and easily distribute their OpenACS components in a standardized manner to other OpenACS sites. In general, a package is a unit of software that serves a single well-defined purpose. The OpenACS Package Manager (APM) provides a mechanism for packaging, installing, and configuring OpenACS software in a consistent, user-friendly, and subsite-aware manner. The OpenACS Package Manager (APM) consists of: A standard format for APM packages including: Version numbering, independent of any other package and the OpenACS as a whole Specification of the package interface Specification of dependencies on other packages (if any) Attribution (who wrote it) and ownership (who maintains it) Web-based tools for package management: Obtaining packages from a remote distribution point Installing packages, if and only if: All prerequisite packages are installed No conflicts will be created by the installation Configuring packages (obsoleting the monolithic OpenACS configuration file) Upgrading packages, without clobbering local modifications Uninstalling unwanted packages A registry of installed packages, database-backed and integrated with filesystem-based version control Web-based tools for package development: Creating new packages locally Releasing new versions of locally-created packages Uploading packages to a global package repository on the web Use of these tools should be safe, i.e. installing or removing a package should never break an OpenACS installation Web-based tools for package configuration: The ability to change package parameter values on-line through a simple web interface. A new ad_parameter which does not require a monolithic site-wide parameter's file or server restarts for changes to take effect. The ability to manage multiple package instances at the sub-site level. The APM is intended for the following classes of users, which may or may not overlap: Developers (referred to as 'the developer') use the APM to create a software package for distribution and use the procedural API for direct control of the APM system. Site-wide administrators (referred to as 'the administrator') use the APM to install packages for their OpenACS instance, and optionally make them available to sub-sites. Sub-site administrators (referred to as 'the sub-admin') use an administration interface to configure and enable packages for their sub-site. Initial Package Development David Developer writes a piece of software used to do knowledge management (km) for the OpenACS. He distributes his data model, procedure code, UI pages, and his documentation according to the APM specification. He splits the documentation and the code into sub-packages, and creates a KM installation-chain to install both with the APM developer UI. Noting that his software was built with Patricia Programmer's Super Widget toolkit, he specifies that as a dependency. Moreover, since this package is capable of being used at the sub-site level, David configures this option in the package. When the package development is complete, David uses the APM developer UI to construct a distribution file. He assigns it a version number, 1.0, and makes the package available for download at the OpenACS package repository. Initial Package Installation Annie Admin learns of David's KM system by browsing the OpenACS package repository. Annie Admin uses the APM administrator UI on her system. She selects to install a package from a URL and types the URL displayed on the system. The APM automatically downloads the package. The dependency checker notices that Patricia's Super Widget toolkit is required, so it warns Annie of this. Annie selects an option to find a package that satisfies the dependency at the OpenACS APM repository. The APM informs Annie that it can download and install Jim's Super Widget toolkit. Annie confirms this option. After successfully installing Jim's toolkit, Annie proceeds to install David's KM system. The data model is loaded and all of the files necessary for the software are installed. Because installation was successful, the package is available for use. Since the package is available for use, its initialization routines are set to run automatically on server startup. Annie is warned that since there are initialization routines, she must restart the server for the package to be ready for use. Annie restarts the server. Initial Subsite Use of Package Annie Admin decides to make the KM module available only to a particular sub-site type on her OpenACS system, and not others. She specifies this option using the Sub-site type UI (not part of APM). Annie Admin notifies Sally SubAdmin by e-mail that a new package is now available for use. Sally goes to her sub-site /admin page and sees that a new entry, KM, is available. Sally clicks on it and finds links to the installed KM documentation and to the web based configuration utility. Then, Sally configures the package using an automatically generated web interface and enables KM for use on her sub-site. After some initial use of the package, Sally decides to change some parameters using the SubAdmin UI. These changes take effect immediately, without any server restarts. Upgrade Process Sally SubAdmin finds a bug in the KM system and sends a report to David Developer. David reads the bug report and verifies that the bugs are present in the current version. Because the bugs are present in the shared procedure file, David assigns a watch to the file. David makes the necessary modifications to the source code and saves the file. Because a watch was assigned to the file, the APM automatically reloads the updated code. David tests the program and confirms that the bug is fixed. He increments the minor version number and makes km v 1.1 available for download at the repository. Sally SubAdmin asks Annie Administrator to upgrade the package using the APM UI. This upgrade supersedes the old version of KM at the site-wide level. Once Annie upgrades the package, the new version starts working immediately in Sally's sub-site. Procedural API Danielle Developer wants her software to perform different actions depending on what version of another package is installed. She uses the APM procedural API to check if KM version 1.0 is installed or version 1.1. Based on the results of this procedural call, the software exhibits different behavior. APM 3.3 Design document Five minute guide to packaging a module Sub-sites 4.500.0 Package Identification (All of these items are entered by the developer using the developer UI.) 4.500.1 A human readable package key that is guaranteed to be unique to the local OpenACS site must be maintained by the APM. For example, "apm." 4.500.5 A package id (primary key) that is guaranteed to be unique to the local site must be maintained by the APM. For example, "25." 4.500.10 A package URL that is guaranteed to be unique across all sites must be maintained by the APM. The package URL should point to a server that allows download of the latest version of the package. For example, "http://openacs.org/software." 4.505.0 Version Identification (All of these items are entered by the developer using the developer UI.) 4.505.1 A version id (primary key) that is guaranteed to be unique to the local site must be maintained by the APM. 4.505.5 A version URL that is guaranteed to be unique across all sites must be maintained by the APM. The version URL should point to a server that allows download of a specific version of the package. The API for APM v3 is explicitly a private API. However, it would be useful to obtain information from the APM through a procedural API. Implementing the API specified below is quite easy given that there are pages that already do all of the below in raw SQL. 4.400.0 Packages Status Predicates 4.400.1 Given defining information such as a package URL, the APM API can return the status of the package on the local OpenACS instance. 4.405.0 Package Information Procedures 4.405.1 The APM API can return information for any locally installed packages, including the version number, paths and files, and package key. 4.410.0 Sub-site Procedures 4.410.1 After a package has been installed at the site-wide level, the system API will provide means to check for package presence, creation, enabling, disabling, and destruction on a subsite. 4.415.0 Parameter Values (replaces ad_parameter) 4.415.1 The system API shall allow subsite parameters for an installed package to be set by either site-wide administrators or sub-site admins. The subsite parameter can be set to be non-persistent (but default is to survive server restarts). The subsite parameter can also be set to only take effect after a server restart (default is immediate). 4.415.5 Parameters for a given subsite and package can be returned by the system API. Provisions will be made to assure that packages are securely identified. 4.600.1 Each package will have a PGP signature and there will be MD5 timestamps for each file within the package. 4.600.5 The APM will provide a facility to validate both the PGP signature and MD5 stamps information before a package install. The user interface is a set of HTML pages that are used to drive the underlying API. It is restricted to site-wide administrators because the actions taken here can dramatically affect the state of the running OpenACS. The intent of the developer's interface is to enable the developer to construct and maintain APM packages. It will be possible to disable the developer's interface for production sites to help reduce the chance of site failure; much of the functionality here can have cascading effects throughout the OpenACS and should not be used on a production site. 10.0 Define a package. The developer must be able to create a new package by specifying some identifying information for the package. This includes a package name, a package key, version information, owner information, and a canonical URL. 10.1 The APM must maintain the state of all locally generated packages. 10.50 If the developer fails to provide the required information, the package cannot be created. 10.55 All of the package information should be editable after creation, except for the package key. 4.10.60 The package creator must specify whether the package is capable of being used in sub-sites, or if only a single, global instance of the package is permitted. 4.10.65 If the developer fails to provide unique information for unique fields specified in the data model requirements, the package cannot be created. 20.0 Add files to a package 20.1 The developer must be able to add files to the package. This is done by copying the files into the package directory in the host OS's filesystem. Files can be added at any point after package creation. 20.3 Once a package has been versioned and distributed, no new files should be added to the package without incrementing the version number. 20.5 The APM's UI should facilitate the process of adding new files, by scanning the filesystem for new files automatically, and allowing the developer to confirm adding them. 20.10 The developer cannot add files to a given package via the UI that do not exist in the filesystem already. 20.15 Package file structure must follow a specified convention. Please see the design document for what we do currently. 30.0 Remove files from a package The developer must be able to remove files from a package. This can be done in two ways. 30.1 Access the APM UI, browse the file list, and remove files. 30.1.1If a file is removed from the package list, but not from the filesystem, an error should be generated at package load time. 30.5 Remove the file from filesystem. 30.5.1 The APM UI should take note of the fact that the file is gone and offer the developer an option to confirm the file's deletion. 40.0 Modify files in a package. 40.1 The developer should be able to modify files in the filesystem. The APM UI should not interfere with this. 40.5 However, if the developer modifies files containing procedural definitions, APM UI should allow a means to watch those files and automatically reload them if changed. See requirement 50.0 for more detail. 40.10 Also, although a change in files implies that the package distribution file is out of date, it is the developer's responsibility to update it. 4.45.0 Manage Package Dependency Information. 4.45.1 The developer should be able to specify which interfaces the package requires. 4.45.5 The developer should be able to specify which interfaces the package provides. 4.45.10 Circular dependencies are not allowed. 50.0 Watch a file 4.50.1 The developer should be able to assign a watch to any Tcl procedure file, whether in /packages or /tcl. 50.5 If a watched file is locally modified, then it will be automatically reloaded, thus allowing for any changes made to take affect immediately. 4.50.10 The setting of a watch should be persistent across server restarts. 60.0 Display an XML package specification 60.1 The developer should be able to view the XML package specification that encodes all package information. 70.0 Write an XML package specification to the filesystem 70.1 The developer should be able to write an up-to-date XML specification to disk. 70.5 The developer should be able to request the current XML specification for all installed, locally generated packages. 130.0 Distribution file generation 130.1 The developer should be able to generate a .APM distribution file for the package with just one click. 130.5 Generating a distribution file implies doing an "up-to-date" check on all of the files. If any of the files have changed since package installation, then a new version of the package is created. 140.0 Access CVS information 140.1 The developer should be able to determine the CVS status of a package, or all packages, with a single click. 4.400.0 Compound Package Construction 4.400.1 The developer can include .APM packages (sub-packages) within a package (the compound package) like any other file. 4.400.5 The recommended usage for this feature is to allow for separation of optional and required components from the installation as well as better organization of files once installed. For example, all documentation for the community-core can be packages as community-core-doc.apm. It is legal to include sub-packages with dependencies that are not satisfied by the packages in the compound package, but this is discouraged. In such a case, the sub-package should really be a separate package that is required by the compound package. 4.400.10 If a sub-package is required for the installation of the compound package, the compound package should have a registered dependency on the sub-package. The requirement of the administrator's interface is to enable the administrator to install, enable, upgrade, disable, deinstall, and delete packages. 80.0 Package Enable/Disable 4.80.1 The administrator should be able mark an installed package as enabled. This means that the package is activated and its functionality is delivered through the Request Processor. As of OpenACS 4, this is done through the sub-site system. 4.80.5 Moreover, the administrator must be able to disable a package, thereby removing the functionality provided to a sub-site. As of OpenACS 4, this is done through the sub-site system. 90.0 Package Install 90.1 The administrator must be able to install new packages either from locally maintained .APM files or from URLs. 90.5 In the case of a URL, the APM transparently downloads the APM file off the web, proceeds with a file based installation, and then optionally removes the .APM file just downloaded. 90.10.1 If .APM files are present in a package, then it is considered a compound package (use 4.410.0). 90.15.0 Installation requires these steps: 90.15.1The package dependencies are scanned. If some dependencies are not present, the system warns the administrator that installation cannot proceed until those packages are installed. 90.15.2 Assuming all dependencies are present, APM extracts the contents of the APM file into the /packages directory. 90.15.3 The administrator is offered the option of importing directly into CVS. 90.15.4 The administrator is given a list of data model scripts found in the package and can select which ones to be executed. 90.15.5 If no errors are recorded during this process, the package is enabled. 4.410.0 Compound package Install 4.410.1 If .APM files are present in a package, then it is considered a compound package. 4.410.5.0 Installation of a compound package proceeds according to the following sequence: 4.410.5.1 Identify the set of all sub-packages within the compound package by scanning for all files with .APM. 4.410.5.2 Identify which sub-packages are required by checking the dependencies of the compound package. If there dependencies not satisfied by the current system or the packages included with the compound package, halt installation and inform user to install these packages first. 4.410.5.3 Present Administrator with the ability to choose which sub-packages to install. Required sub-packages must be installed. 4.410.5.4 Proceed with the installation of each sub-package, starting with required packages. If the sub-package is already installed, then do nothing. Else, If the sub-package is a normal package, proceed according to 90.15.0, otherwise if it is a compound package, proceed according to 4.410.5.0. 4.410.5.5 If all required sub-packages are installed, proceed to install non-required sub-packages. If there was a failure during the installation of a required sub-package, then the installation of the compound package is also a failure. 4.410.5.6 Any attempt to install a compound package in the future involves a choice presented to the admin of installing any uninstalled sub-packages. 4.420.0 Recovering from failed package installation 4.420.1 If any error is generated during package installation, the package is not considered installed. To recover from this failure, the package should be selected for installation again. 100.0 Version Upgrade 100.1 The administrator can upgrade to a new version of a package. This entails 100.1.1 Running any necessary and included upgrade scripts. 100.1.5 Replacing any old files with new versions. 100.1.10 Marking the old version of the package as 'superseded' and disabling it. 100.1.15 Assuming no errors from above, the new package is enabled. 110.0 Package Deinstall 110.1 The administrator must be able to deinstall a package that has already been installed. Deinstallation entails: 110.1.1 Running any data model scripts necessary to drop the package. 110.1.5 Moving all of the files into a separate location in the filesystem from the installed packages. 4.110.1.10 If the package is a compound package, then the administrator must confirm removing all sub-packages. Optionally, some sub-packages can be kept. 110.5 Deinstalled packages can be re-installed at a later date. 4.110.10 If deinstalling a package or any of its sub-packages breaks a dependency, then deinstallation cannot proceed until the package registering the dependency is removed. 120.0 Package Deletion 120.1 The administrator should be able to completely erase all records of the package. This involves removing all instances of the package, all related database tables and content. 120.5 This option can only be used if all package instances are deleted or marked as disabled. This is purposefully cumbersome because deleting all instances of a package can have far-sweeping consequences throughout a site and should almost never be done. 150.0 Scan for new or modified packages 150.1 The administrator should be able to scan the filesystem for any changes made in any of the installed package files. 150.5 The administrator should be able to scan the filesystem for any newly installed packages. If the developer is in charge of creating packages and the administrator for installing them, then the sub-site administrator is responsible for configuring and enabling packages. In order for a package to be available for a sub-site it must be associated with the sub-site's type specification. This interface is part of the sub-site /admin interface. 4.300 Creating a package instance. 4.300.1 From the sub-site /admin interface, there should be an option to view all packages available in the system as well as an option to add a package to the subsite. 4.300.5 From the "add" option, the sub-admin can select from a list of packages registered as available in the sub-site type to which the sub-site belongs. 4.300.19 Once a package instance is added, it is available on the list of the subsite's available packages. 4.305 Configuring a package instance. 4.305.1 An automatic web interface that lists all parameters with current values must be available. 4.305.5 Changing the values for the parameters is accomplished simply by submitting an HTML form. 4.310 Enabling a package instance. 4.310.1 The sub-admin should be able to enable a package with a single click. Enabling a package means that the OpenACS will serve its URLs properly. 4.315 Disabling a package instance. 4.315.1 The sub-admin should be able to disable a package with a single click. Disabling a package means that the OpenACS will no longer serve those URLs. 4.320 Deleting a package instance. 4.320.1 Deleting a package instance involves deleting not only the package instance, but any and all content associated with it. It is questionable whether this option should even be available due to its drastic consequences. Reviewer comments appreciated. Despite the fact that requirements are meant to be design/implementation neutral, the following thoughts were in our head when specifying these requirements. You must be familiar with the new object design for this to be comprehensible. When a package is installed system-wide, a corresponding acs_object_type is created for it. All parameters registered for the package are registered for that acs_object_type. When a package instance is created, it is an acs_object. Its parameters are set using the acs_attribute_values table. The automatic web interface for setting package parameters should be one and the same with the interface for setting acs object attribute values. Consequently, the implementation of these features should be quite straightforward. Document Revision # Action Taken, Notes When? By Whom? 0.1 Creation 8/10/2000 Bryan Quinn, Todd Nightingale Reviewed 8/11/2000 John Prevost, Mark Thomas, and Pete Su 0.2 Revised and updated 8/12/2000 Bryan Quinn 0.3 Reviewed, revised, and updated - conforms to requirements template. 8/18/2000 Kai Wu 0.4 Minor edits before ACS 4 Beta. 9/30/2000 Kai Wu