Conclusion

This thesis has described and evaluated a system that allows the freehand entry of formulae using a pen and tablet. After entering the formula, it is easy to arbitrarily manipulate it: rearranging, adding or deleting symbols. Once the user has entered the formula to their satisfaction, the system can then generate the corresponding L^ATEX command string. This command string can then be copied and pasted into the user's L^ATEX document.

In addition to combining an existing character recogniser and formula parser with new user interface ideas, this thesis has made a number of contributions:

• An algorithm for automatically grouping the user's strokes into symbols, using the character recogniser to evaluate different possibilities.

• New user interface concepts for identifying misgrouped and misrecognised symbols using annotated bounding boxes, and lagging the recognition so as not to distract the user.

• A ``squiggle select'' for correcting stroke grouping errors, allowing the user to draw a line through the strokes of symbols that should belong together.

• Showing that a graph rewriting parser has the potential to work well for handwritten input, although this particular implementation did not.

• Showing that a pen-based user interface for the entry and editing of mathematical formulae is desirable and preferable to existing command-string or template based entry systems.

A pen-based approach provides a more natural and familiar interaction method than existing command-string or template-based equation editors. When entering a formula using this system, a user does not have to learn a special language or notation; more importantly, they do not need to determine the overall structure of a formula before entering it. With the user not having to spend time searching for special symbols in menus, maneuvering the cursor into boxes in templates, or referring to manuals to find the correct commands for the operations they wish to do, entering and editing a formula is much easier.

This system does not offer a fast alternative to existing formula entry systems, even for expert users, but it is much more intuitive, less complex, and easier to learn. Its strengths are in the entry of large, complex formulae, and the editing of these formulae after entry.

The system is an interface overlaid on modules for handwriting recognition, equation parsing, and typesetting. The performance of the system, in terms of both speed and accuracy, can be improved by improving the performance of these elements. However, since there will always be some ambiguity in handwritten input, simple methods for correcting errors, such as the ones presented in this thesis, will always be necessary. With this in mind, this system offers a pop up menu offering character recognition alternatives, and the means to regroup strokes into symbols is provided with a ``squiggle select''.

The underlying formula processor uses graph rewriting to process the initial formula and determine what the corresponding L^ATEX representation is. This type of parser is able to cope well with the position and size variations inherent in handwritten input. The only problems with the formula processor implementation are its lack of speed and occasional tendency to misparse formulae. The use of context rules and advanced graph searching techniques would address both these issues.

The user testing of the system provided very encouraging feedback. While the entry times for the formulae tested were slower than the corresponding entry times for L^ATEX and Microsoft's Equation Editor, all the users who tested the system were very positive about the concept of a pen-based formula entry system, saying that it was preferable to, and much more intuitive to use, than existing command-string based or template-based systems. The new concepts in the user interface also received positive comments.

The user testing also highlighted a number of issues that need to be addressed, and provided a large number of suggestions for future directions that this system could take. Some of the most significant are:

• improving the graph rewriting formula processor's speed and accuracy, perhaps through the use of context rules or advanced techniques for graph matching.

• improving the feedback to the user when their formula is unparsable, possibly offering an indication of what is wrong with it.

• eliminating the need to enter a special mode, by pressing a button on a toolbar or choosing a menu option, in order to modify the character recognition or stroke groupings. Hot-spots on each symbol's bounding box could be used for this.

• expansion of the set of formulae that the system can understand, so that it would be of more use to mathematicians and physicists. The current set of formulae that the system can process is too small for such users.

• creation of a GUI tool to enable end users to easily add to and change the grammar used by the formula processor. This would let users add their own notations and customise the system to understand their mathematical layout style.

• introduction of capabilities for mathematical manipulation, perhaps initiated through gestures. Operations such as the evaluation, expansion, or factorisation of formulae could be accomplished through a link to an external mathematics package, such as Mathematica. However, if this interface was to be expanded to include formula processing operations initiated by a gesture, there are usually no intuitive gestures available for a user. Kajler and Soiffer warn that a system can be hard to learn and confusing for a user who is faced with a large number of arbitrary gestures that need to be learnt. This is a problem common to all gesture based interfaces.

Ultimately, the creation of a fully featured pen-based formula entry system should be possible, ideally as a front end to symbolic manipulation or mathematics packages. A virtual piece of paper that cannot only record writing, but also interpret it on demand and allow gesture-based algebraic manipulation could one day be created. This prototype system is a step in this direction.

warn that a system can be hard to learn and confusing for a user who is faced with a large number of arbitrary gestures that need to be learnt. This is a problem common to all gesture based interfaces.

Ultimately, the creation of a fully featured pen-based formula entry system should be possible, ideally as a front end to symbolic manipulation or mathematics packages. A virtual piece of paper that cannot only record writing, but also interpret it on demand and allow gesture-based algebraic manipulation could one day be created. This prototype system is a step in this direction.