The purpose of this experiment is to test the effectiveness
+and efficiency of the current ported OpenACS calendar and from the result of
+the test, to find out what improvement can be made to the calendar. In other
+words, this usability test is performed to improve the quality of the current
+calendar application.�
The targeting web application for this usability test is the
+calendar from OpenACS, which can be accessed from http://ebt.ee.usyd.edu.au:9013.
+This is an online calendar, which can be personalized and can be easily
+accessed with any web browsers, that allows one to enter and keep track of
+anything that could reasonably be written on an ordinary paper calendar. It
+also provides different ways of viewing the calendar information; for example,
+day view, week view, month view and year view.
+
+
+
+
This package could be used with any
+application where tracking items in time is important.
+
+
+
+
There are a number of features, which
+any application must bring to the table to be reasonably considered a complete
+calendaring application. The OpenACS calendar package currently supports all
+but one of these features:
+
+
�
+Multiple
+different ways of viewing the calendar.�
+At a minimum this must include day, week, month, and year views. The
+OpenACS calendar also includes two list views.
+
+
�
+A rich set of
+navigation features for easily selecting and changing both the views and the
+time spans viewed.
+
+
�
+The ability to
+easily add, edit and delete items.
+
+
�
+The ability to
+easily set up recurring items.
+
+
The only feature that is missing is the
+alarm and reminder function.
The majority of the user population consists of those who
+use Internet frequently, as they are comfortable with the idea of using
+web-based application. The OpenACS calendar is designed to target primarily
+those who, to oneself, tracking items in time is important.� This would include most business settings
+for meetings and appointments, educational settings, clubs, friends� birthdays
+and other community organizations; and those who want to please and attract
+customers or Internet users by including a calendar as an extra functionality
+for their web site.
+
+
+
+
Users often use the OpenACS calendar to put down a small
+note for their schedule for the day, week, month or year, or to view or edit
+what they have already entered into the calendar. OpenACS calendar users often
+want to open up the calendar, use the least amount of time and in their current
+working environment, put down the events or activities to be done on a
+particular date and time, or to recall the upcoming events which they have
+pre-entered.
+
+
Thus using information obtained from
+the above analysis, the following description-user matrix is formed:
+
+
+
+
+
+
+
+
+
+
Primary User
+
+
+
Secondary User
+
+
+
+
+
Main Occupation
+
+
+
Business
+ people / Students
+
+
+
Website
+ developers
+
+
+
+
+
Age Group
+
+
+
18 � 50
+
+
+
21 � 45
+
+
+
+
+
Aim
+
+
+
To enter and
+ keep track of events or items in the personalized online calendar quickly and
+ easily.
+
+
+
To add the
+ calendar as an extra functionality on their website to attract customers.
+
+
+
+
+
Time spend on site
+
+
+
Short �
+ Medium
+
+
+
Medium �
+ Long
+
+
+
+
+
Computer/Internet skill
+
+
+
Medium �
+ High
+
+
+
Medium �
+ High
+
+
+
+
+
Main thing they look for
+
+
+
A
+ user-friendly private online calendar.
+
+
+
Easy to
+ install and ready to use.
+
+
+
+
+
Most common task
+
+
+
Add and view
+ events.
+
+
+
Install the
+ calendar software and make the calendar software available to use by
+ customers.
The main goal of this usability test is
+to improve the way user can use the OpenACS calendar to access an event (this
+includes add, retrieve, edit and delete) in the shortest amount of time and complete their time management using
+the calendar without doubt and difficulties.
+This will eliminate the possibility that the user gives up the use of online
+calendar because the procedure is too complicated.
+
+
+
+
The critical, typical and potentially
+troublesome tasks to be chosen for evaluation will be the task of adding a new event, retrieving and editing an existing event,and deleting an existing
+event completing the calendar event management cycle.
+
+
+
+
They are defined as critical and
+typical tasks as they are the core functionality of a calendar, which are also
+designed to be the most frequently used functions. They are potentially
+troublesome tasks due to the fact that managing your daily, weekly or monthly
+schedule can become quite difficult when one have too many events happening on
+the same day, week or month, and user might possibly give up in the middle if
+they find it too complicated.
+
+
+
+
The main benchmarks selected for this
+evaluation are as follows:
+
+
1.
+Number of
+mouse clicks used in each step
+
+
2.
+Amount of time
+spent in each step
+
+
3.
+Number of
+pages visited in each step
+
+
The above benchmarks are quantified
+usability measures. These can reflect the efficiency and the difficulty or the
+ease of using the calendar application.
+
+
+
+
This test is designed to be carried out
+as soon as the implementation and testing cycle has completed, and requires one
+day evaluation and one day for analyzing the statistics of the outcome of the
+evaluation test, and produce a report containing the facts that is derived from
+the analysis. This report will then be sent to the development team to assist
+them to improve their design. A need for holding a meeting between the
+developing team and the evaluation team might be possible to discuss on the way
+the website can be improved.
+
+
To carry out this usability test, 3
+users and 1 tester will be required. The duration of each test is approximately
+30 minutes and the tester is required for recording the statistics of the users
+as they carry out the test and preparation of test equipment and
+materials.�
+
+
+
+
As soon as the result of the evaluation
+test formed, a meeting should immediately be held by the evaluation team and
+the development team to make decisions as to which of the corrections should be
+prioritized. Depending on the result, the amount of functionality within a page
+can either be increased or decreased; the style of the interface can be
+changed.
o�� Daily����� o�� Weekly����� o�� Monthly����� o��� Every 3 months���� �o��� Never used a computer
+
+
How
+ often do you use the internet?
+
+
o�� Daily����� o�� Weekly����� o�� Monthly����� o��� Every 3 months����� o��� Never used a computer
+
+
Have
+ you used an electronic or online calendar before?
+
+
o�� YES���������������������� o��� NO���������������������� o�� Never heard of it
+
+
+
+
+
+
+
+
+Grouping all
+the above derived requirements and analytical outcomes, the following test
+mateial is produced:
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
�
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Figure1
+��������������� OpenACS calendar Usability Test material
+
+
+
+
+
+
The usability test (see Appendix C) for the OpenACS calendar
+was conducted on 6th October 2002. Results from the usability test
+reflects the following need for improvements:
+
+
+
+
�
+Allow event adding by
+clicking on a particular date on the month-view calendar
+
+
�
+Add a function that
+will provide easier and faster date access. This can be three drop-down lists
+that allow users to select the day, month and year of the specific date and a
+GO button to retrieve the date
+
+
�
+Enable the view of
+mini calendars of each month on the year-view calendar
+
+
�
+Allow events on the
+same day to display in start time ordering in the calendar
+
+
�
+Provide instructions
+and help on calendar usage
+
+
�
+Highlight on the mini
+calendar the dates that has an event
+
+
�
+Provide distinction
+between events of different calendars in use
+
+
�
+Bullet points to
+separate out different events in the month-view calendar
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Appendix C �
+
+
+
+
Usability Test Results
+
+
+
+
+
+
+
+
+
+
+
About the User:
+
+
+
+
Please collect the following information from the
+user:
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
User
+No:�������� o� 1���������������� o� 2���������������� o� 3
o� Daily��
+o� Weekly��
+o� Monthly��
+o� Every 3 months�� o� What is a computer?
+
+
�����������
+
+
How
+often do you use the internet?
+
+
+
+
+
+
o� Daily��
+o� Weekly��
+o� Monthly��
+o� Every 3 months�� o� What is a computer?
+
+
+
+
Have
+you used an electronic or online calendar before?
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
o�� YES��������������������� o��� NO���������������������� o�� Never heard of it
+
+
+
+
�
+
+
About
+the Test:
+
+
+
+
Explain to the user that
+today he/she is going to pretend to manage (add, view, edit and delete) some
+events on particular dates using the OpenACS calendar. Each time his/her action
+will be statistically be recorded with his performance and explain to him/her
+that this information is used only for the evaluation for the design of the
+web. Explain the whole procedure of each round, step by step, and start timing
+after recording down the target recipient.
+
+
+
+
+
+
Task Zero � Getting Started
+
+
+
+
Before using the OpenACS calendar, users are
+required to login. On the top right hand corner of the web browser, user will
+see a box for entering email address and password to the system. Please enter
+the following login details:
+
+
E-mail:����������� user1@openacs.org
+
+
+
Password:������ user1
+
+
After entering the above, click on calendar on the bottom of the page.
+
+
+
+
Goal: To login as a
+user to the calendar.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Task One � Adding Events
+
+
+
+
Scenario:
+
+
+
+
I
+have a few appointments, which I�d like to add to my OpenACS calendar before I
+forget. My appointments are as follows:
+
+
�
+Dinner appointment on 22nd of November
+2001, 7:30pm at Sydney Hilton Hotel.
+
+
�
+Best friend�s birthday on 1st December 2001
+
+
�
+House warming party of a friend, Amy, on next Sunday
+(from today) 3:00pm.
+
+
Address: 7 Seven Street,
+Seventh Coast, 2000 NSW
+
+
Phone: 1234-5678
+
+
�
+
+
Goal:������������� To add the above 3 events into the
+calendar.
+
+
+
+
Time started: _4:40_
+am/pm
+
+
+
+
Number of mouse clicks/double-clicks applied
+(appointment 1):
Any
+observations of difficulties or comment in this task:
+
+
+
+
User has no specific
+difficulties. When user trying to add an event to the calendar, his immediate
+action is to click on the date on the mini calendar rather than using the �add
+an event� link provided just below the mini calendar.
+
+
+
+
+
+
+
+
+
Task
+Two � Viewing Events
+
+
+
+
+
+
Scenario:
+
+
+
+
First I would like to find out today�s date and if I
+have any appointments today.
+
+
After finding out about
+today, I would like to find out the exact date, time and duration of my dental
+appointment, which is in mid January 2002. This dental appointment was made 2
+months ago and I have forgotten the exact date and time and would like to find
+it out.
+
+
+
+
Goal 1:����������� To
+find out today�s date and what appointments I have today.
+
+
Goal 2:���������� To find out the exact date, time and duration for my dental
+appointment in mid January 2002.
+
+
+
+
Time started: __4:
+45_ am/pm
+
+
+
+
Number
+of mouse clicks/double-clicks applied (Goal 1):
Any
+observations of difficulties or comment in this task:
+
+
+
+
User finishes
+this task with no difficulty. User realized that the �today� link will bring
+him to the current date. Goal 2 was reached fast, user firstly clicked on the
+�year� link to change the year, then via the �month� link, found the
+appointment easily.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Task
+Three � Editing Events
+
+
+
+
+
+
Scenario:
+
+
+
+
I have realized that I
+cannot make the dental appointment which I had made earlier on. I would like to
+amend it to 22nd February 2002, at 4:00pm.�
+
+
+
+
Goal:�������������� To
+edit the time of the appointment to 22nd February 2002, 4:00pm.
User didn�t
+realize that he could adjust the date and time on the retrieved event,
+therefore the user went the long way around. He clicked on the �year� link,
+adjusted the year to 2002, then selected �month� and realized that he cannot
+add items via the month-view. When the user clicked on the �day�, the year
+turned back to 2001 instead of 2002. This got the user confused.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Task
+Four � Deleting Events
+
+
+
+
+
+
Scenario:
+
+
+
+
Amy called me last night
+that the house warming party has been cancelled. I would like to delete this
+from my OpenACS calendar.
+
+
+
+
Goal :������������� To
+delete Amy�s house warming party entry (see part one) from the OpenACS
+calendar.
User had no
+trouble looking for the house warming party event entered earlier on, and the
+delete button was obvious to the user.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
User Feedback:
+
+
+
+
Did you encounter any
+difficulties/hesitations when adding events in this OpenACS calendar?
+
+
+
+
Yes, a bit.
+Some paths to dates are very difficult to go to and are somewhat misleading. It
+is slow to access a particular date, too many steps required.
+
+
+
+
+
+
+
+
Any recommendations on how
+this OpenACS calendar can be improved?
+
+
+
+
Add a field for
+selection so that users can select a certain date quickly.
+
+
Allow input from the month
+calendar.
+
+
Add bullet points in month
+view calendar.
+
+
+
+
+
+
+
+
Would you want to use this
+OpenACS calendar to manage your calendar events in the future? If no, why?
+
+
+
+
Yes.����
+
+
+
+
+
+
+
+
Any extra functions that you
+would like to see from this OpenACS calendar application if you use it again
+later?
+
+
+
+
No. Other than the improvements written above.
+
+
+
+
+
+
+
+
+
�About the User:
+
+
+
+
+
+
Please collect the following information from the
+user:
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
User
+No:�������� o� 1���������������� o� 2���������������� o� 3
o� Daily��
+o� Weekly��
+o� Monthly��
+o� Every 3 months�� o� What is a computer?
+
+
�����������
+
+
How
+often do you use the internet?
+
+
+
+
+
+
o� Daily��
+o� Weekly��
+o� Monthly��
+o� Every 3 months�� o� What is a computer?
+
+
+
+
Have
+you used an electronic or online calendar before?
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
o�� YES��������������������� o��� NO���������������������� o�� Never heard of it
+
+
+
+
�
+
+
About
+the Test:
+
+
+
+
Explain to the user that
+today he/she is going to pretend to manage (add, view, edit and delete) some
+events on particular dates using the OpenACS calendar. Each time his/her action
+will be statistically be recorded with his performance and explain to him/her
+that this information is used only for the evaluation for the design of the
+web. Explain the whole procedure of each round, step by step, and start timing
+after recording down the target recipient.
+
+
+
+
+
+
Task Zero � Getting Started
+
+
+
+
Before using the OpenACS calendar, users are
+required to login. On the top right hand corner of the web browser, user will
+see a box for entering email address and password to the system. Please enter
+the following login details:
+
+
E-mail:����������� user2@openacs.org
+
+
+
Password:������ user2
+
+
After entering the above, click on calendar on the bottom of the page.
+
+
+
+
Goal: To login as a
+user to the calendar.
+
+
+
+
+
+
+
+
+
+
+
Task One � Adding Events
+
+
+
+
Scenario:
+
+
+
+
I
+have a few appointments, which I�d like to add to my OpenACS calendar before I
+forget. My appointments are as follows:
+
+
�
+Dinner appointment on 22nd of November
+2001, 7:30pm at Sydney Hilton Hotel.
+
+
�
+Best friend�s birthday on 1st December 2001
+
+
�
+House warming party of a friend, Amy, on next Sunday
+(from today) 3:00pm.
+
+
Address: 7 Seven Street,
+Seventh Coast, 2000 NSW
+
+
Phone: 1234-5678
+
+
�
+
+
Goal:������������� To add the above 3 events into the
+calendar.
+
+
+
+
Time started: _5:17_
+am/pm
+
+
+
+
Number
+of mouse clicks/double-clicks applied (appointment 1):
Any
+observations of difficulties or comment in this task:
+
+
+
+
User encountered some
+trouble and got confused while adding events. User was expecting a �Save� or
+�Go� or something similar after adding typing in the event, rather than �Submit
+Query�. User does not understand the �Submit Query� word. User did not use the
+mini calendar, but approached the �add an event� link.
+
+
User did not realize the
+�today� link, which can bring her to the current date.
+
+
+
+
+
+
+
+
+
+
+
+
+
Task
+Two � Viewing Events
+
+
+
+
Scenario:
+
+
+
+
First I would like to find out today�s date and if I
+have any appointments today.
+
+
After finding out about
+today, I would like to find out the exact date, time and duration of my dental
+appointment, which is in mid January 2002. This dental appointment was made 2
+months ago and I have forgotten the exact date and time and would like to find
+it out.
+
+
+
+
Goal 1:����������� To
+find out today�s date and what appointments I have today.
+
+
Goal 2:���������� To find out the exact date, time and duration for my
+dental appointment in mid January 2002.
+
+
+
+
Time started: __5:
+25_ am/pm
+
+
+
+
Number
+of mouse clicks/double-clicks applied (Goal 1):
Any
+observations of difficulties or comment in this task:
+
+
+
+
User did not
+realize the �today� link, which is placed under the mini calendar. User was
+stuck at the point when she needs to change the year of the current year because
+there is no display at the year-view and no instructions telling her what to
+do. User got confused and found that the calendar is a little misleading. Goal
+2 was difficult to achieve by the user.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Task Three � Editing Events
+
+
+
+
Scenario:
+
+
+
+
I have realized that I
+cannot make the dental appointment which I had made earlier on. I would like to
+amend it to 22nd February 2002, at 4:00pm.�
+
+
+
+
Goal:�������������� To
+edit the time of the appointment to 22nd February 2002, 4:00pm.
User had
+trouble retrieving the event. After deleting the event, the user wanted to
+double check if the event had been deleted, but after the delete is done,
+calendar switched itself to display the current date instead of the date the
+user was once at. User found this very inconvenience.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
User Feedback:
+
+
+
+
Did you encounter any
+difficulties/hesitations when accessing events in this OpenACS calendar?
+
+
+
+
Yes. The
+functions are not very well done. Dates and events are not simple enough to
+access.
+
+
+
+
+
+
+
+
Any recommendations on how
+this OpenACS calendar can be improved?
+
+
+
+
Should add short cuts to
+access dates easier. Should allow people to add things by clicking on the month
+calendar.Should
+use more icon or diagrams instead of written English, for exampleuse
+a button with �X� instead of the �delete� word would be good. The calendar
+should also make things / items entered more obvious. What I meant is it should
+allow user to distinguish the dates with events and dates without events from
+the small calendar.
+
+
+
+
+
+
Would you want to use this
+OpenACS calendar to manage your calendar events in the future? If no, why?
+
+
+
+
No. Access to
+the calendar events involves too many steps and the process is too slow. Using
+calendars like this should be made as easy as ABC. Non Internet users will have
+a lot of trouble using this calendar.
+
+
+
+
+
+
Any extra functions that you
+would like to see from this OpenACS calendar application if you use it again
+later?
+
+
+
+
Have a back
+link to return to the previous step or page. In the month calendar, it should
+allow people to change the year instead of clicking the year link and change it
+from there. The month calendar should allow adding events by clicking on the
+dates displayed in it.
+
+
As mentioned
+before in question 2, should make the date containing events into other colours
+or perhaps bolded.
+
+
+
+
+
+
+
+
+
About
+the User:
+
+
+
+
+
+
Please collect the following information from the
+user:
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
User
+No:�������� o� 1���������������� o� 2���������������� o� 3
o� Daily��
+o� Weekly��
+o� Monthly��
+o�Every 3 months�� o� What is a computer?
+
+
�����������
+
+
How
+often do you use the internet?
+
+
+
+
+
+
o� Daily��
+o� Weekly��
+o� Monthly��
+o� Every 3 months�� o� What is a computer?
+
+
+
+
Have
+you used an electronic or online calendar before?
+
+
+
+
+
+
o�� YES��������������������� o��� NO���������������������� o�� Never heard of it
+
+
+
+
�
+
+
About
+the Test:
+
+
+
+
Explain to the user that
+today he/she is going to pretend to manage (add, view, edit and delete) some
+events on particular dates using the OpenACS calendar. Each time his/her action
+will be statistically be recorded with his performance and explain to him/her
+that this information is used only for the evaluation for the design of the
+web. Explain the whole procedure of each round, step by step, and start timing
+after recording down the target recipient.
+
+
+
+
+
+
Task Zero � Getting Started
+
+
+
+
Before using the OpenACS calendar, users are
+required to login. On the top right hand corner of the web browser, user will
+see a box for entering email address and password to the system. Please enter
+the following login details:
+
+
E-mail:����������� user3@openacs.org
+
+
+
Password:������ user3
+
+
After entering the above, click on calendar on the bottom of the page.
+
+
+
+
Goal: To login as a
+user to the calendar.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Task One � Adding Events
+
+
+
+
Scenario:
+
+
+
+
I
+have a few appointments, which I�d like to add to my OpenACS calendar before I
+forget. My appointments are as follows:
+
+
�
+Dinner appointment on 22nd of November
+2001, 7:30pm at Sydney Hilton Hotel.
+
+
�
+Best friend�s birthday on 1st December 2001
+
+
�
+House warming party of a friend, Amy, on next Sunday (from
+today) 3:00pm.
+
+
Address: 7 Seven Street,
+Seventh Coast, 2000 NSW
+
+
Phone: 1234-5678
+
+
�
+
+
Goal:������������� To add the above 3 events into the
+calendar.
+
+
+
+
Time started: _9:30_
+am/pm
+
+
+
+
Number
+of mouse clicks/double-clicks applied (appointment 1):
Any
+observations of difficulties or comment in this task:
+
+
+
+
User has no specific
+difficulties.
+
+
+
+
+
+
+
+
+
Task Two � Viewing Events
+
+
+
+
Scenario:
+
+
+
+
First I would like to find out today�s date and if I
+have any appointments today.
+
+
After finding out about
+today, I would like to find out the exact date, time and duration of my dental
+appointment, which is in mid January 2002. This dental appointment was made 2
+months ago and I have forgotten the exact date and time and would like to find
+it out.
+
+
+
+
Goal 1:����������� To
+find out today�s date and what appointments I have today.
+
+
Goal 2:���������� To find out the exact date, time and duration for my
+dental appointment in mid January 2002.
+
+
+
+
Time started: __9:
+36_ am/pm
+
+
+
+
Number
+of mouse clicks/double-clicks applied (Goal 1):
User had no
+trouble looking for the house warming party event entered earlier on, and the
+delete button was obvious to the user.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
User
+Feedback:
+
+
+
+
+
+
Did you encounter any
+difficulties/hesitations when accessing events in this OpenACS calendar?
+
+
+
+
No. I managed
+to get the tasks done without much trouble.
+
+
+
+
+
+
+
+
Any recommendations on how
+this OpenACS calendar can be improved?
+
+
+
+
Allow adding events in the
+month-view calendar.
+
+
Should include a field to
+go to a particular date so it will be easier to open a date.
+
+
Should have some kind of
+formatting to distinguish the group events and personal events.
+
+
+
+
+
+
Would you want to use this
+OpenACS calendar to manage your calendar events in the future? If no, why?
+
+
+
+
No. I have used
+other calendar programs before, they are much more user-friendly and faster to
+use. The OpenACS calendar is too slow, this include the page load time and the
+steps and page load to view a particular event.�������
+
+
+
+
+
+
+
+
Any extra functions that you
+would like to see from this OpenACS calendar application if you use it again
+later?
+
+
+
+
Make a field to
+allow date accessing, so you can select date from there rather than loading all
+the unnecessary pages.
