{/doc/assessment/ {Assessment}} {Assessment functional requirements} Assessment functional requirements

Introduction

The assessment module provides OpenACS with capabilities to conduct surveys, tests and dynamic information gathering in general, as can be seen in the use cases.

Vision Statement

The motivation behind the Assessment package is to extend the functionality of the existing Survey package in both depth and breadth:

The current Survey package is a very capable piece of engineering that provides stand-alone data collection functions. It is subsite-aware and has been integrated to some extent with portlets. It also is just being integrated into user registration processes. These efforts point the path down which the Assessment package intends to proceed to its logical conclusion.

Development efforts for Assessment thus involve two tracks:

The measure of success of the Assessment package is the ease with which it can rapidly be deployed into some high-profile implementations, notably dotLRN and a clinical trials management system under development.

Use Cases

The assessment module in it's simplest form is a dynamic information gathering tool. This can be clearly seen in the first group of use cases, which deal with surveys (one form of assessment, e.g. for quality assurance or clinical trials). An extension of this information gathering the possibility to conduct an evaluation on the information given, as we show in the second group of use cases (testing scenarios). Last but not least, the assessment tool should be able to provide it's information gathering features to other packages within the OpenACS framework as well.

It is very important to note that not all parameters and features mentioned in this use case should be displayed to the user at all times. Depending on the use case, a good guess with pre determined parameters should be made for the user (e.g. no need to let the user fill out correct answers to questions, if the question is not used in a test). Some use cases like elections require special parameters not necessary anywhere else (like counting system).

Survey scenario

The survey scenarios are the basic use cases for the use of the assessment system.

Simple survey

An editor wants to conduct surveys on his site. For this purpose he creates questions which are stored in a question catalogue. From this question catalogue, the editor choose the questions he wants to use in his current survey along with the style the survey should be presented to the user. Once satisfied he can make the survey public or test it first. Once the survey is public subjects (users) of the site can take the survey by filling out the generated form with all the questions the author added to the survey.

Quality Assurance

A company wants to get feedback from users about it's product. It creates a survey which offers branching (to prevent users from filling out unnecessary data, e.g. if you answered you have never been to Europe the question "Have you seen Rome" should not show up) and multi-dimensional likert scales (To ask for the quality and importance of a part of the product in conjunction).

Professional data entry

A clinic wants to conduct a trial. For this research assistants are asked to interview the patients and store the answers in the assessment on behalf of the client. For meeting FDA requirements it is mandatory to prove exactly who created any datum, when, whether it is a correct value, whether anyone has looked at it or edited it and when along with other audit trails. As mistakes might happen, it is important that the system runs checks on the plausibility of the entered data and the validity of it (area code should be five digits, if the age of the patient is below 10, no need to ask for credit card information, ...).

University survey

A Professor wants to create a test by searching through the question database and selecting old questions. He searches the database for a specific keyword or browses by category. The System presents him all questions which have the keyword and/or category in it. The Professor is able to preview every question and may then decide which question he will transfer into the survey.

Internal Evaluation

An institution wants to survey students to compare the quality of specific courses, teachers, or other factors effecting the quality of their education and level of happiness. It should be possible for the person who takes the survey to submit the survey anonymously and only be able to take the survey once.

It should also be able to show the results of a survey to a group of users (e.g. a specific department evaluated). The results should be able to be displayed in a way that give a department a ranking compared with other departments.

Reuse of questions

The author of multiple choice question decides that the provided answers are not good for differentiating the knowledge of the subjects and changes some of them. All editors using this question should be informed and asked, if they want to use the changed version or the original one. If the decision is made to switch, it has to be guaranteed that a distinction between subjects that answered the original and the new version is kept. In addition the editor should be able to inform all subjects that have taken the question already, that it has changed (and that they might (have to) re-answer).

Multiple languages

The quality assurance team of the company mentioned above realizes that the majority of it's user base is not native English speakers. This is why they want to add additional translations to the questions to broaden the response base. For consistency, the assessment may only be shown to the subject if all questions used have been translated. Furthermore, it is necessary to store the language used along with the response (as a translation might not be as good as the original).

The poll

An editor wants to conduct a poll on the site with immediate publication of the result to get a feeling how users like the new design of the website. The result can be displayed in an includelet (see the below for details) on any page the editor wants.

The election

The OpenACS community wants to conduct a new election on the OCT. On creation the names of the contestants have to be available along with a list of all users allowed to vote. Depending on the election system, the users have one or multiple votes (ranked or not), which are calculated in a certain way. Once the election is over the result is published.

Collective Meeting planing

The sailing club needs to find meeting time for all skippers to attend. Given a number of predefined choices, each skipper can give his/her preference for the time slots. The slot with the highest approval wins and is automatically entered into the calendar of all skippers and a notification send out.

Testing scenario

Especially in the university environment it is important to be able to conduct tests. These help the students to prepare for exams but also allow Professors to conduct exams. In addition to the data collection done in a survey scenario testing adds checks and instant evaluation to assessment.

Proctored Exam

A Professor wants to have a proctored test in a computer room. He wants to create the test using question that he has added and are already in the database. The only people allowed to take the test are the people that have actually showed up in the room (e.g. restricting the exam to specific IP-subnet and/or an exam password which he will give the students in the room at the time of the test that gives them access to the exam). Additional security measures include:

The Mistake

A Professor has created a test from the question pool and have administered the exam to a group of students. The test has been taken by some of his students already. He discovers that the answer to one of the questions is not correct. He modifies the test and should be given the option to change the results of exams that have already been completed and the option to notify students who have taken the test and received a grade that their results have changed.

Discriminatory power

A Professor has created a test which is taken by all of his students. The test results should be matched with the individual results to create the discriminatory power and the reliability of the questions used in the test. The results should be stored in the question database and be accessible by every other professor which has the privileges to access the database of this professor.

[A Question improves the test in reliability if it differentiates in the context of the test. This is happening if it has discriminatory power. The Question has discriminatory power if it is splitting good from bad students within the question in the same way they passes the test as good and bad students. The discriminatory power tells the professor if the question matches the test. Example: A hard question with a high mean value should be answered by good students more often right than by bad students. If the questions is answered same often by good and bad students the discriminatory power tells the professor that the question is more to guess than to know]

The vocabulary test

A student wants to learn a new language. While attending the class, he enters the vocabulary for each section into the assessment system. If he wants to check his learned knowledge he takes the vocabulary test which will show him randomized words to be translated. Each word will have a ranking stating how probable it is for the word to show up in the test. With each correct answer the ranking goes down, with each wrong answer it goes up. Once a section has been finished and all words have been translated correctly, the student may proceed to the next section. Possible types of questions: To determine the correct answer it is possible to do a char-by-char compare and highlight the wrong parts vs. just displaying the wrong and correct answer (at the end of the test or once the answer is given).

The quizz

To pep up your website you offer a quiz, which allows users to answer some (multiple choice) questions and get the result immediately as a percentage score in a table comparing that score to other users. Users should be able to answer only a part of the possible questions each time. If the user is in the top 2%, offer him the contact address of "Mensa", other percentages should give encouraging text.

Scoring

The computer science department has a final exam for the students. The exam consists of 3 sections. The exam is passed, if the student achieves at least 50% total score. In addition the student has to achieve at least 40% in each of the sections. The first section is deemed more important, therefore, it gets a weight of 40%, the other two sections only 30% towards the total score. Each section consists of multiple questions that have a different weight (in percent) for the total score of the section. The sum of the weights has to be 100%, otherwise the author of the section gets a warning. Some of the questions are multiple choice questions, that get different percentages for each answer. As the computer science department wants to discourage students from giving wrong answers, some wrong answers have a negative percentage (thereby reducing the total score in the section).

Reuse in other packages

The information gathering capabilities of the assessment system should be able to be reused by other packages.

User profiling

In order to join a class at the university the student has to fill out some questions. The answers can be viewed by the administrator but also by other students (pending the choice of the user). This latter functionality should not be part of assessment itself, but of a different module, making use of assessment. The GPI user-register is a good example for this.

Includes

Using a CMS the editor wants to include the poll on the first page on the top right corner. The result should be shown on a separate page or be included in the CMS as well.

Information gathering for developers

A developer needs functionality for gathering dynamic information easily. For this he should be able to easily include an assessment instead of using ad_form directly in his code. This gives the administrator of the site the option to change the questions at a later stage (take the questions in the user sign-up process as an example).

Database questions

Some answers to questions should be stored directly in database tables of OpenACS in addition to the assessment system. This is e.g. useful if your questions ask for first_names and last_name. When answering the question, the user should see the value currently stored in the database as a default.

Action driven questions

The company conducting the QA wants to get more participants to it's survey by recommendation. For this each respondee is asked at the end of the survey if he would recommend this survey to other users (with the option to give the email address of these users). The answer will be processed and an email send out to all given emails inviting them to take the survey.

User Types

There are several types of administrative users and end-users for the Assessment package which drive the functional requirements. Here is a brief synopsis of their responsibilities in this package.

Package-level Administrator

Assigns permissions to other users for administrative roles.

Editor

Has permissions to create, edit, delete and organize in repositories Assessments, Sections and Items. This includes defining Item formats, configuring data validation and data integrity checks, configuring scoring mechanisms, defining sequencing/navigation parameters, etc.

Editors could thus be teachers in schools, principal investigators or biostatisticians in clinical trials, creative designers in advertising firms -- or OpenACS developers incorporating a bit of data collection machinery into another package.

Scheduler

Has permissions to assign, schedule or otherwise map a given Assessment or set of Assessments to a specific set of subjects, students or other data entry personnel. These actions potentially will involve interfacing with other Workflow management tools (e.g. an "Enrollment" package that would handle creation of new Parties (aka clinical trial subjects) in the database.

Schedulers could also be teachers, curriculum designers, site coordinators in clinical trials, etc.

Analyst

Has permissions to search, sort, review and download data collected via Assessments.

Analysts could be teachers, principals, principal investigators, biostatisticians, auditors, etc.

Subject

Has permissions to complete an Assessment providing her own responses or information. This would be a Student, for instance, completing a test in an educational setting, or a Patient completing a health-related quality-of-life instrument to track her health status. Subjects need appropriate UIs depending on Item formats and technological prowess of the Subject -- kiosk "one-question-at-a-time" formats, for example. May or may not get immediate feedback about data submitted.

Subjects could be students, consumers, or patients.

Data Entry Staff

Has permissions to create, edit and delete data for or about the "real" Subject. Needs UIs to speed the actions of this trained individual and support "save and resume" operations. Data entry procedures used by Staff must capture the identity if both the "real" subject and the Staff person entering the data -- for audit trails and other data security and authentication functions. Data entry staff need robust data validation and integrity checks with optional, immediate data verification steps and electronic signatures at final submission. (Many of the tight-sphinctered requirements for FDA submissions center around mechanisms encountered here: to prove exactly who created any datum, when, whether it is a correct value, whether anyone has looked at it or edited it and when, etc etc...)

Staff could be site coordinators in clinical trials, ensurance adjustors, accountants, tax preparation staff, etc.

System / Application Overview

Editing of Assessments

Scheduling of Assessments

Analysis of Assessments

Performance of Assessments