next up previous
Next: Post-test Analysis Up: User Testing Previous: Ethical Considerations

The Test Itself

As the use of a pen and tablet was new to most users and most of the users had not used, or even seen, this system before, each testing session started with an initial training phase. The users were able to become familiar with the use of a pen and tablet and become familiar with using the system. The training was conducted by first giving them a short demonstration of the system, then the observer guiding them through a set of practice formulae. These formulae were similar in complexity and structure to those that were used in the main testing phase. When the users felt that they were ready, or had worked through all the formulae, they moved onto the main testing phase.

During the training the possible additional pressure of being observed was minimised by not having the video camera running.

To guide the testing session, users were given a set of five formulae to work through, starting with a simple one and progressively increasing in complexity. Having a set of predetermined formulae meant that all the participants were doing the same things. As a result, it was possible to make comparisons and statistics across the users who are used. It also means that if, in the future, further testing is done, comparisons can be made across users using the system under different conditions.

The five formulae that were entered by the users for the unaided part of the user testing are shown below. These formula are representative of the complexity of formulae that the formula processor's current grammar can handle.

& (1) \hspace{1cm} x^{2}+4 \\
& (2) \hspace{1cm} \int{x^2+4...
...) \hspace{1cm} \int^{8}_{3}{\frac{(2^{x}+4x)}{-\sqrt{z}}}dx=8 \\

As the participants used the system, their activities were recorded by a video camera pointed at the computer's screen for later analysis. Dumas and Redish  give floor plans of a number of professional user testing suites, all having a number of video cameras trained on the user and computer, and often additional observers watching through one-way glass.

An additional way to monitor participants' activities is to modify the application, or run an additional program, so that an electronic record of the user's actions is kept. Individual keystrokes and mouse activity can be recorded. Automatically generated reports on various aspects of their use of the user interface can then be produced, as well as providing logs for later analysis. The user testing of this system relied solely on the video recordings of the user's screen and any observations made by the observer watching the users use the system.

On the conclusion of each testing session, the participants were asked to respond to both an oral and written questions about the system. The advantage of a written or online questionnaire is that nobody has to transcribe the participant's answers, and it also offers the opportunity for anonymity: the participant can freely express themselves without fear of retribution. Having a verbal question session means that the questioner is able to ask additional questions to get more information in relation to answers that the participant has given, or as a result of something the participant did during the testing. It is important to design the questions so that they are non-leading. It is also suggested to have some redundancy in the questions that you are asking: designing questions with overlap.

For the user testing of this system, the participants filled out an anonymous written questionnaire, then answered a number of questions that were asked verbally by the observer. The anonymous questionnaire, included in Appendix [*], gathered basic information to gauge the user's experience with computers and formula-entry type systems. It then asked questions to get the user's overall opinion of the system, and how they found it in comparison to other systems that they may have already used. The verbal questions, included in Appendix E, sought the participant's opinions of the new interface concepts developed. Some overlap did exist as a number of comments received in the verbal questioning were duplicated from their answers in the anonymous questionnaire.

next up previous
Next: Post-test Analysis Up: User Testing Previous: Ethical Considerations
Steve Smithies