After a careful reading on Steve Krug’s “Don’t Make Me Think“, Jay Melton’s “LMS Moodle: A Usability Evaluation” andUsability.gov, I have finally finished a usability testing plan. The idea is to test how easy Angel, Moodle, Blackboard and eCollege are to use but it’s not confirmed yet. I just thought I would share this idea. Also in the usability plan is Accessibility compliance testing which guidelines I got from Section 508 [§ 1194.22 Web-based intranet and internet information and applications.] and W3 Web Content Accessibility. This should provide access to all users with most of special needs in accordance to the Section 508 of the Disabilities Act.
The resulting usability testing plan is a compilation of Krug’s book, Melton’s research methodology and Usability.gov guidelines. If they are copy they are referenced in the document.
1. Overall goal
Through usability testing [we] will assess “how easy user interfaces are to use” (Nielsen, 2003) and how compliant to section 508 the LMS/CMS being trialed by the committee. For that purpose, a small number of currently enrolled students will be selected as participants and assigned certain tasks to be performed in the LMS/CMS. Our goal is to help the CMS committee perceive the extent of helpfulness an LMS/CMS can contribute to the learning process.
2. Specific Questions
· How easy is the LMS to use?
· What is the overall impression of the GUI [interface]?
· Does this LMS/CMS contribute or enrich, in any fashion, the learning process?
3. Tasks rubric and techniques
Participants of the usability test will be assigned certain tasks that represent the most performed activities students currently do in [current LMS]. These tasks are of high frequency, priority and of great importance to the learning process. A list of eight tasks to be performed by participants is composed below:
1. Join a course in [LMS being tested] [find it, Enroll in course using enrollment key]
2. What’s this course about? Find syllabus, handouts, projects.
3. Discuss a topic in forum
4. Submit an assignment with attachment
5. Take a test
6. Email your instructor
7. Check your grades
8. Find help [specific topic in help info]
Accessibility
Section 508 of the Disabilities Act and W3 provide content guidelines that GUI’s (user interfaces) should follow in order to provide accessibility to all users whether they have special needs or not.
· Section 508
· W3 – Web Content Accessibility
In addition to the guidelines above, the LMS/CMS should follow the accessibility rules below:
· Enable online content to be read without requiring style sheets.
· All images should contain “alt” tags
· Pick a LMS that doesn’t require installation or use of applets, plug-ins, or active X
· All files and forms should be accessed by Assistive Technology users – such as information, field elements, and functionality required for completion and submission of the form.
· Links should be carefully described to allowing special needs users to navigate through online content.
· Whenever a timed activity is present let the user know that there is a time limit / frame for this activity to be completed.
4. Practical issues
Number of Participants
If 1 or 2 systems are tested, 4 participants should work well in which 2 have to be regularly matriculated and 2 matriculated students with special needs that utilize 508-compliant software.
If 3 or 4 systems are being tested, then you’ll need at least 5 or 6 participants where 2 of them are of special needs. Be careful on the time each student spends on Student selection criteria:
Participant A
· Previous experience with word processing
· Have past experience with [current LMS]
· Taken at least one course in [current LMS]
· Submitted one assignment in [current LMS]
Participant B
· Have as little [current LMS] experience as possible if none
Participant C
· Special needs user (use one type of specific software) – the special need is to be determined by the Office of Special Services of your institution.
· Have past experience with [current LMS] (minimum)
Participant D
· Special needs user (use another different type of specific software) – the special need is to be determined by the Office of Special Services of your institution.
· Have past experience with [current LMS] (little minimum)
Recruitment
[Discuss with the Dean of the College or your department on how participants can be recruited. Whether public posting, mass emails, reaching out to student governments]
Compensation for participants
[Discuss with your department or your supervisor on some options for participant’s compensation]
Testing Environment (Melton, 2006)
· No interruptions
· One Video camera
· Adequate lighting
· Adequate temperature
· Low noise
· One computer connected to the Internet
Evaluator (Melton, 2006)
The evaluator will take notes on participants’ success in completing the tasks, rating task completion by using the following terminology:
( + ) easy
( √ ) medium
( – ) difficult
Trial run
Prior to test day, Evaluators will be called for a usability test trial run. At this event, a usability test will be simulated where each evaluator receives a script they can follow to run the test.
5. Ethical Issues
Participant’s identity and privacy will be protected. The focus of this test is the system and not the student. Participants may terminate the test at anytime. Their image will not be recorded. The camera’s point of view should be on screen,
mouse and keyboard. (Melton, 2006)
6. Evaluation, interpretation and presentation of Data
On test day
Each participant will be provided with three documents: Two consent forms (to be written) and one demographic questionnaire.
Consent form – two copies, one for the student and one for the project
Questionnaire:
· Name
· E-mail
· Year (1st / 2nd)
· Do you have experience with [current LMS] (Yes / NO)
· If yes, in what context did you experience [current LMS] (online / Hybrid / I don’t know / NA)
· In these experiences with [current LMS] what activities have you performed (email, upload assignments, view grades, find information about the course, do a test)
· Do you have experience with other Content Management Systems (LMS/CMS) (Yes / No)
· If yes, in what context did you experience CMS (online / Hybrid / webenhanced / I don’t know / NA)
· In these previous experiences with CMS what activities have you performed (email, upload assignments, view grades, find information about the course, do a test.
Duration of test (to test 1 system) (Melton, 2006, p. 348)
Time Range between 35 to 45 minutes
Test steps (copied from Krug, 2006 and adapted)
· Step 1. Read Instructions aloud
Introduce yourself and explain your role in the project. Describe the purpose of the participant’s in the project. Make sure the participant knows that he/she is not being tested. Ask participant to think out loud and that you might not be able to answer all questions – so the data collected isn’t contaminated. ASk for participant’s permission to record thehis/her computer screen and audio.
Do you have any questions before we begin?
· Step 2. Are you ready?
· Step 3. Start recording
· Step 4. Begin test
Show the CMS/LMS home page and ask the participant for some feedback on his/her impression of what strikes the most.
Ask the participant not to click but just observe and tell.
Tell the student that you’re moving onto the tasks testing per se. and that the student should think aloud. (Krug, 2006)
· Evaluator worksheets – one evaluator fills the tables below during the test timing each completion and determining the success and satisfaction of the user on using the LMS.
Tasks sheet for usability testing
