PART II
http://tinyurl.com/39bb8a
enter 09631101
Look for the Tab "Published Documents"
FIELD OF THE INVENTION
The present invention relates generally to computer-assisted data entry and more particularly to a method, system, and apparatus for computer-assisted text generation and entry using a pointing device with a personal computing device, and to computer-readable media having executable instructions for supporting text generation and entry using a pointing device.
BACKGROUND OF THE INVENTION
The wide-spread adoption of miniaturized personal computing devices, such as hand-held devices and personal digital assistants (PDAs), has led to an increasing use of devices to send and receive text and data. One example of this trend is pen-based computing, wherein users enter text and commands into hand-held personal computers via a touch-sensitive screen. While such pen-based computing is popular, especially with the increasing power of miniature computing devices, it does present challenges to a user entering data in an application running on the hand-held device. For instance, many hand-held computers and personal digital assistants require that the user enter data according to a predetermined scripting style, such as with the PalmPilot.TM. series of PDAs. Other hand-held devices provide a handwriting recognition system which requires that the computer learn the user's handwriting style. While such data entry mechanisms are useful, they are relatively difficult to use and complex to learn and can be prone to error in the event the user deviates from the predetermined scripting style or the user's traditional handwriting style.
Many pen-based computing systems, both large and small, offer the user the option to enter text using an on-screen digital keyboard. On-screen digital keyboards are typically miniaturized replica of conventional full-sized physical keyboards, such as QWERTY keyboards. Many on-screen keyboards have shown themselves to be less than efficient for entering text. When using a pointing device such as a pen, a user is typically required to enter text one character at a time by tapping out individual character selections from the on-screen keyboard. This "hunt-and-peck" method of typing with a single pointing device is time-consuming, especially when a user is entering large amounts of data.
Another common challenge when entering data into a personal computing device with a single pointing device such as a pen or stylus, and in particular when entering text, is that each letter making up the word or phrase must be entered manually. The longer the word or phrase, the greater the amount of manual entry required.
Text completion systems have been developed in an effort to assist users with text entry. In general, these systems predict and suggest a complete word completion based on a partial text entry entered by a user. These systems allow a user to type in the partial text entry and then accept a predicted text completion for the partial text entry. This avoids the keystrokes that would otherwise be required to type the complete text desired by a user. While such text completion systems provide some basic assistance for users to more rapidly enter text than would be required if every character of the desired text had to be typed in independently, there remains a need in the art for a more flexible text completion system for use with a single pointing device. It would also be desirable for such a text completion system to employ a convenient selection technique which would reduce the amount of movement of the pointing device required to enter text into a computer. It would further be desirable if such a system were applicable to both large and small personal computing devices.
Another problem in the art is that soft, or digital, keyboards have tended to be continually displayed so as to permanently consume screen space or have needed to be manually invoked and dismissed by the user. It would be desirable if a digital keyboard could automatically appear and disappear as required.
SUMMARY OF THE INVENTION
The above and related desires are addressed in the present invention by providing a novel and non-obvious method, system and computer-readable instructions for computer-assisted text generation and entry using a pointing device with a personal computing device.
In one aspect of the present invention the user can rapidly enter and search for text using a data entry system through a combination of entering one or more characters on a digitally displayed keyboard with a pointing device and using a search list to obtain a list of completion candidates. The user can activate the search list to obtain a list of completion candidates at any time while entering a partial text entry with the data entry system. When the search list is active, a list of completion candidates is displayed on a graphical user interface for the user to select from and the user can perform one of several actions. The user can deactivate the search list and return to modifying the current partial text entry and other text. The user can select one of the completion candidates in the search list and use the selected completion candidate to replace the partial text entry which the user is currently entering. When the user deactivates the interactive search list, the user can immediately continue adding to or modifying the current partial text entry being entered, and may re-invoke the search list to further search for completion candidates based on the modified partial text entry. In the second case, the selected completion candidate is used to replace the partial text entry that the user is currently entering, and the data entry system begins monitoring for a new partial text entry from the user.
In one embodiment, when the search list is active the user may use one of the completion candidates in the search list to initiate a further automated search to obtain a more refined list of completion candidates. In this embodiment, multi-level search lists and searching are available to help accelerate completion of a partial text entry. As a result, the user can automatically initiate an iterative search wherein a completion candidate listed in the search list is used as the new partial text entry to dynamically obtain a new list of completion candidates, which is then displayed in the search list. The automated ability to use the search list to obtain a refined list of completion candidates allows the user to quickly make good use of search results that are only partially successful. When the search list is revised with a new list of completion candidates, the user can then choose one of the completion candidates in the new list, or the user can repeat the iterative search process once again by choosing one of the completion candidates in the new list and activating a further iterative search. In addition, the user may return to keyboard entry with the last completion candidate selected by the user in the previous iteration of the search list. This latter feature provides the user with the convenience of being able to automatically and seamlessly continue entering the desired word, phrase, or character sequence using the last completion candidate selected by the user in the previous iteration of the interactive search list. Thus, the user can quickly and easily replace the user's current partial text entry with a partially successful completion candidate and continue building upon or modifying this partial completion candidate using the keyboard while at the same time allowing the user the flexibility to re-enter the search list at any time.
In another aspect of the present invention, there is provided a method of processing text entered into a personal computing device with a pointing device. With this method, a partial text entry is received and used to obtain a dynamically generated list of completion candidates. The list of completion candidates is displayed in a search list within a graphical user interface. A user input signal associated with the pointing device is received. If the user input signal corresponds to a first type of user selection with the pointing device, then the search list is deactivated. If the user input signal corresponds to a second type of user selection with the pointing device, then the partial text entry is replaced with a completion candidate from the search list.
In one embodiment, if the user input signal corresponds to a third type of user selection with the pointing device, then a refined list of completion candidates is dynamically obtained based on one of the completion candidates from the search list. The refined list is displayed in the search list for further user selection.
In another aspect of the present invention, there is provided a method of processing an input string at least partially entered into a personal computing device with a pointing device. This aspect includes performing a search of a set of completion candidates to locate a plurality of possible completion candidates for completing the input string in response to a prior located possible completion candidate or a character selectable by a user. At least one of the plurality of possible completion candidates and characters selectable by the user are displayed.
In another aspect of the present invention, a method is provided of user-based text entry. With this aspect, a set of position coordinates for a pointing device is monitored relative to a user interface and a digital keyboard is displayed on the user interface at a last known set of coordinates for the pointing device whenever the digital keyboard is activated for user input.
In another aspect of the present invention, a method is provided for interchanging the display of a digital keyboard and a search list. The digital keyboard is displayed on a user interface when a user is entering text a keystroke at a time. As the digital keyboard is displayed, user input is monitored. If the user input corresponds to activating the search list, then the digital keyboard is replaced with the search list. If the user input corresponds to terminating use of the search list once activated, then the search list is replaced with the digital keyboard.
In another aspect of the present invention, a digital keyboard is configured to include a plurality of characters assigned to predetermined locations within a layout for the digital keyboard according to a predetermined frequency distribution associated with the plurality of characters. The plurality of characters includes less commonly used characters and more commonly used characters based on the predetermined frequency distribution. The digital keyboard is displayed on a graphical user interface with the less commonly used characters displayed substantially further from a center of the digital keyboard than the more commonly used characters.
In another aspect of the present invention there is provided a system for computer-assisted text generation and entry. The system includes an input interface, a processing unit and a computer-readable medium. The input interface receives user input signals based on actions with a pointing device. The computer-readable medium contains computer-readable instructions for directing the processing unit to assist with text generation and entry based on user input received via the input interface with the pointing device. The computer-readable medium includes instructions for receiving a partial text entry; for obtaining a dynamically generated list of completion candidates based on the partial text entry; for displaying the list of completion candidates in a search list on a display device; and for receiving a user input signal associated with the pointing device from the input interface. If the user input signal corresponds to a first type of user selection with the pointing device, the computer-readable instructions are programmed to deactivate the search list. If the user input signal corresponds to a second type of user selection with the pointing device, the computer-readable instructions are programmed to replace the partial text entry with a completion candidate from the search list.
Other aspects and features of the present invention will become apparent to those ordinarily skilled in the art upon review of the following description of specific embodiments of the invention in conjunction with the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
In the accompanying drawings which illustrate embodiments of the invention,
FIG. 1 is a schematic diagram of a personal computing device loaded with a data entry system, according to a first embodiment of the invention;
FIG. 2 is a schematic diagram of the data entry system of the first embodiment;
FIG. 3 is a schematic representation illustrating the display of a digital keyboard on a graphical user interface within the personal computing device of the first embodiment;
FIG. 4 is a schematic representation of a data structure for a dictionary according to the first embodiment;
FIG. 5 is a schematic representation illustrating the interchangeable display of the digital keyboard and an interactive search list according to the first embodiment;
FIG. 5A is a schematic representation illustrating potential completion candidates for retrieval and display in the interactive search list according to an example of the use of the first embodiment;
FIGS. 6 and 6A are flow diagrams illustrating, by way of example from the user's perspective, the use of the data entry system of the first embodiment;
FIGS. 7 to 9 are flow diagrams illustrating the flow of operation of the data entry system according to the first embodiment;
FIG. 10 is a schematic representation of an alternative embodiment of a digital keyboard layout according to the present invention;
FIG. 11 is a schematic representation of another alternative embodiment of a digital keyboard layout according to the present invention;
FIG. 12 is a schematic representation of another alternative embodiment of a digital keyboard layout according to the present invention;
FIG. 13 is a schematic representation of another configuration of a digital keyboard according to an embodiment of the present invention;
FIG. 14 is a schematic representation of the layout for the interactive search list according to the first embodiment;
FIGS. 15 to 18 are schematic representations of alternative layouts for the interactive search list according to alternate embodiments of the present invention;
FIG. 19 is a schematic representation of a configuration for the digital keyboard and the interactive search list according to an embodiment of the present invention;
FIG. 20 is a schematic representation of an alternative configuration for the digital keyboard according to an embodiment of the present invention;
FIGS. 21 to 22 are flow diagrams illustrating the flow of operation of the data entry system according to an alternative embodiment of the present invention;
FIGS. 23 to 24 are flow diagrams illustrating the flow of operation of the data entry system according to an alternative embodiment of the present invention;
FIGS. 25 to 26 are flow diagrams illustrating the flow of operation of the data entry system according to an alternative embodiment of the present invention;
FIG. 27 is a schematic representation of an alternative configuration for the digital keyboard according to an embodiment of the present invention.
It will be appreciated that for simplicity and clarity of illustration, elements illustrated in the accompanying drawings have not necessarily been drawn to scale. For example, the dimensions of some of the elements are exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals and labels have been repeated among the drawings to indicate corresponding or analogous elements and features.
DETAILED DESCRIPTION
Reference will now be made in detail to implementations and embodiments of the invention, examples of which are illustrated in the accompanying drawings.
Introduction
In one aspect of the present invention the user can rapidly enter and search for text using a data entry system through a combination of entering one or more characters on a digitally displayed keyboard with a pointing device and using an interactive search list to dynamically obtain a list of completion candidates. The user can activate the interactive search list to obtain a dynamically generated list of completion candidates at any time while entering a partial text entry with the data entry system. In this specification "partial text entry" means a sequence of one or more characters making up a leading portion of a word, phrase or character sequence. When the interactive search list is active, a dynamically generated list of completion candidates is displayed on a graphical user interface for the user to select from. The list of completion candidates is retrieved from at least one dictionary by a candidate prediction system which retrieves completion candidates that are most likely to contain the completion candidate desired by the user. Candidate prediction is based on statistical measures ranking the entries within the dictionary relative to each other. When the interactive search list is active, the user can perform one of several actions with the data entry system including any of the following: (1) the user can deactivate the interactive search list and return to modifying the current partial text entry and other text; or (2) the user can select one of the completion candidates in the interactive search list and use the selected completion candidate to replace the partial text entry which the user is currently entering.
In the first case, when the user deactivates the interactive search list, the user can immediately continue adding to or modifying the current partial text entry being entered, and may re-invoke the interactive search list to further search for completion candidates based on the modified partial text entry. In the second case, the selected completion candidate is used to replace the partial text entry that the user is currently entering, and the data entry system begins monitoring for a new partial text entry (of a word, phrase, or character sequence) from the user.
In certain embodiments of the present invention, as illustrated further below, a third action available to the user when the interactive search list is activated is to: (3) use one of the completion candidates in the interactive search list to initiate a further automated search to obtain a more refined dynamic list of completion candidates from the dictionary.
In the third case, multi-level search lists and searching are available to help accelerate completion of a partial text entry. In this latter case, the user can automatically initiate an iterative search wherein a completion candidate listed in the interactive search list is used as the new partial text entry to dynamically obtain a new list of completion candidates, which is then displayed in an updated interactive search list. The automated ability to use the interactive search list to dynamically obtain a refined list of completion candidates allows the user to quickly make good use of search results that are only partially successful. When the interactive search list is updated with a new list of completion candidates, the user can then choose one of the completion candidates in the new list, or the user can repeat the iterative search process once again by choosing one of the completion candidates in the new list and activating a further iterative search. In addition, the user may return to keyboard entry with the last completion candidate selected by the user in the previous iteration of the interactive search list. This latter feature provides the user with the convenience of being able to automatically and seamlessly continue entering the desired word, phrase, or character sequence using the last completion candidate selected by the user in the previous iteration of the interactive search list. Thus, the user can quickly and easily replace the user's current partial text entry with a partially successful completion candidate and continue building upon or modifying this partial completion candidate using the keyboard while at the same time allowing the user the flexibility to re-enter the interactive search list at any time.
As discussed further in this specification, other actions may also be carried out by the user when the interactive search list is active.
Operation Environment
FIG. 1 shows a schematic diagram of a personal computing device 10 for text entry according to a first embodiment of the invention. The personal computing device 10 shown in FIG. 1 contains at least one processing unit 12 (such as a CPU or a similar processor or multiprocessor) connected by a bus to a computer-readable medium 16. The computer-readable medium 16 provides a memory store for software and data residing within the personal computing device 10. The computer-readable medium 16 can include one or more types of computer-readable media including volatile memory such as Random Access Memory (RAM), and non-volatile memory, such as a hard disk or Read Only Memory (ROM). Preferably, the computer-readable medium 16 includes a combination of volatile and non-volatile memory. In the first embodiment, the computer-readable medium 16 contains an operating system, a data entry system 26 and an application 27 receptive to user-based text entry such as a word processor. The computer-readable medium 16 may also store alternative or other applications such as a browser or micro-browser, an e-mail application, and/or other end-user applications.
The operating system can be any of several well-known operating systems depending on the personal computing device used. For example, for hand-held devices, the operating system can be PalmOS.TM., Windows CE.TM., or an equivalent operating system. For larger systems, such as with work stations or desktop computers, a more robust operating system may be used such as, for example, Windows 95 .TM., Windows 98.TM., Windows NT.TM., Windows 2000.TM., MacOS.TM., UNIX, Linux or the like. For the purposes of the first embodiment, the operating system is PalmOS.TM..
The data entry system 26 is implemented as software that runs on the processing unit 12 to support computer-assisted text generation and entry for the user, although in other alternatives the data entry system 26 can be implemented as computer-readable instructions in firmware or embedded in hardware components. In the first embodiment, electronic text and documents are generated and maintained by the application 27 and the user authors and edits the electronic text and documents with the data entry system 26 which communicates with the application 27 through an application programming interface (API). This allows the data entry system 26 to be portable so that it can be used by one or more applications to accept text and data entry from the user. As an alternative, the data entry system 26 may be integrated into part of an application.
The personal computing device 10 includes a graphical display device 15 and a hardware input interface 17 receptive to user input from a pointing device. In this specification, the term "pointing device" means an input device that allows a user to select one choice amongst one or many choices (a user-based selection). Some pointing devices enable a user to make user-based selections by pointing to a desired choice and include, by way of example, a pen, stylus, or finger. More generally, pointing devices capable of supporting user-based selections include, by way of example, the pointing devices above capable of pointing, as well as other input devices such as a mouse, trackball or the like.
The graphical display device 15 is connected to and controlled by the processing unit 12 via a video display circuit 13. The graphical display device 15 may be a CRT, a liquid crystal display, or an equivalent computer display.
In the first embodiment, the personal computing device 10 is a personal digital assistant wherein the graphical display device 15 and the hardware input interface 17 are combined in the form of a touch-sensitive screen 14 that serves both as a graphical display 15 and as an input interface receptive to generating coordinate position signals in response to contact from a pointing device such as a pen or stylus. It will be appreciated by those skilled in the art that the personal computing device 10 is represented in the following discussion as a personal digital assistant for illustration purposes only, and that the invention may be practised with other personal computing devices including hand-held devices, personal computers and other microprocessor-based electronic devices, mobile telephones, internet appliances, and embedded devices, having a suitable graphical display and an input interface receptive to user input via a pen, stylus, finger, mouse, or an equivalent pointing device that allows the user to select one choice from many. Other types of equivalent personal computing devices to which the features and aspects of the present invention are applicable include, by way of example, an internet appliance controlled via a remote control (for instance, running an Internet service through a television via a set top box and a remote control). In other embodiments, the hardware input interface 17 may be a digitising tablet, a pressure-sensitive input surface or a proximity sensing input surface. It will also be appreciated that the personal computing device 10 may be powered by an internal (18) or external power source.
As shown in FIG. 2, the data entry system 26 includes computer-readable instructions for a digital keyboard 28, a candidate prediction system 32, a dictionary 20, and an interactive search list 30. The digital keyboard 28 provides an interface for the user to enter text and data into the personal computing device 10 (FIG. 1). As the user enters in characters via the digital keyboard 28 to construct a word, phrase, or character sequence, the characters entered by the user are stored in a search string as a partial text entry. The search string is used by the candidate prediction system 32 to search the dictionary 20 for completion candidates that begin with the current partial text entry being stored in the search string. The interactive search list 30 is used to display for user selection the list of completion candidates retrieved by the candidate prediction system 32. In the first embodiment, the data entry system 26 supports gesture-based user input for the selection of completion candidates from the interactive search list 30 as will be described in further detail below.
Digital Keyboard
As illustrated in FIG. 3, an image of the digital keyboard 28 is displayed on a graphical user interface 34 within the screen area of the touch-sensitive screen 14 once the data entry system 26 is initialised and ready to receive input from the user. The digital keyboard 28 contains a plurality of keys each of which is associated with at least one character from a set of characters. For example, when the English alphabet or a character set containing the English alphabet is used, each key on the digital keyboard 28 can contain one or more letters from the English alphabet. It should be noted that reference to the English alphabet is by way of example only, and that the digital keyboard 28 can be configured to contain and display any set of characters which the user may then select and use to enter text into the personal computing device 10. The terms "character set" and "set of characters" refer in this specification to a set containing a plurality of letters, numbers and/or other typographic symbols. Examples of character sets include, but are not limited to, one or more alphabets of a written language (e.g. English, French, German, Spanish, Chinese, or Japanese), and binary-coded character sets such as ASCII (American Standard Code for Information Interchange), EBCDIC (Extended Binary Coded Decimal Interexchange Code), and BCD (Binary Coded Decimal).
In the first embodiment, the digital keyboard 28 displays digital keys containing characters from the English alphabet along with special characters selected from the ASCII character set. Words, phrases, and character sequences can be typed into an electronic document or electronic text by simply using the pointing device to tap or select in sequence each key for the desired word, phrase, or character sequence. As will be discussed further below, the user can also use the digital keyboard 28 to initiate an automated search for completion candidates to more rapidly and flexibly enter words, phrases, and/or character sequences in an automated manner. As will also be discussed later in this specification, several enhancements to the digital keyboard 28 may be implemented to further enhance the user's ability to quickly, efficiently, and flexibly enter text using a pointing device.
Dictionary
FIG. 4 shows a sample data structure for the dictionary 20. Preferably, as in the first embodiment, the dictionary 20 contains completion candidates with weight values for ranking completion candidates relative to each other. The dictionary 20 contains a plurality of entries, with each entry having a completion candidate field 22 for storing a completion candidate and a weight field 24 for storing a numeric value associated with the completion candidate stored in a corresponding completion candidate field 22. Each completion candidate stored in the dictionary 20 represents a word, phrase, or character sequence according to a particular language. Character sequences may include, but are not limited to, word continuations. Word continuations represent a leading part of a word, but not an entire word. In the first embodiment where the dictionary is based on American english, word continuations may include, by way of example, such common word prefixes as "com", "con", "dis", "expl", "inter", "mis", "para", "pre", "syn", "tele", "trans" and "univers". Obtaining a list of completion candidates which can include common word prefixes reduces user effort in terms of the number of steps required to generate a desired text entry. Furthermore, certain character combinations such as word prefixes are shared across multiple completion candidates which can speed up learning of the locations of completion candidates within sub-levels of the interactive search list 30 when multi-level searching (i.e. iterative searching) is available.
Each weight field 24 stores a weight value for ranking the corresponding completion candidate stored in the corresponding completion candidate field 22 with other completion candidates in the dictionary 20. The weight value stored in a weight field 24 may be based on one of many metrics. By way of example, each weight field 24 may contain a value representing a degree of common usage of the completion candidate stored in the corresponding completion candidate field 22 relative to the other completion candidates in the dictionary 20. As another example, the weight fields 24 may contain numeric values based on what words or phrase came before the completion candidate stored in the corresponding completion candidate from an analysis of a large corpus of text. Each weight field 24 may also be supplemented with one or more related fields.
Candidate Prediction System
Referring to FIG. 2, the candidate prediction system 32 is programmed to dynamically search the dictionary 20 for completion candidates that begin with the partial text entry entered by the user. The candidate prediction system 32 retrieves completion candidates from the dictionary 20 by determining which dictionary entries are more likely to be the ones that the user is attempting to type. In the first embodiment, the completion candidates are obtained from the dictionary 20 on the basis of frequency values stored in the weight field 24 for each entry. In the first embodiment, completion candidates having the highest weight values are retrieved. The frequency values represent the frequency of usage of the entries relative to each other. These frequency values may be predetermined on the basis of an analysis of a large corpus of text or may be dynamically generated or modified on the basis of the specific user's usage of words, phrases, and/or character sequences through the data entry system 26. Other statistical measures may also be employed to enhance the candidate prediction system 32, such as ranking information or identifying the frequency with which an entry in the dictionary 20 follows any of the other entries in the dictionary 20.
Preferably, the candidate prediction system 32 retrieves all possible completion candidates limited by one or more predetermined metrics. In the first embodiment, the total number of completion candidates retrieved is limited by a predetermined maximum number of displayable completion candidates. The maximum number of completion candidates retrieved by the candidate prediction system 32 is preferably a small but significant number sufficient enough to provide the user with as many potential candidates as possible without unduly saturating the user with an excessive number of candidates and without unduly delaying the user's ability to quickly review and select from the candidates selected. The maximum number of completion candidates is also preferably sufficiently large enough to provide a variety of completion candidates (if available) for the user to choose from so as to avoid an excessive amount of multi-level searching to complete a partial text entry. In the case of the first embodiment, the maximum number of displayable completion candidates is set to five. The data entry system 26 can be configured by the user to present a greater or lesser number of completion candidates. Preferably, the completion candidates are displayable in a configuration that can also be selected by the user. Making the number of completion candidates that can be displayed a user-configurable feature provides enhanced capabilities for computer-assisted text generation and entry with the data entry system 26.
Interactive Search List
The interactive search list 30 receives completion candidates retrieved from the dictionary 20 by the candidate prediction system 32 and presents the user with a list of these completion candidates. The interactive search list 30 can be displayed to the user in any of several different ways depending on which options the data entry system 26 has been programmed with and which of those options have been selected by the user, as will be discussed later in this specification. In the first embodiment, the interactive search list 30 is displayed on the touch-sensitive screen 14 as an interactive vertical list of completion candidates. As described further below, the interactive search list 30 is programmed to support multi-level searches so that the user can quickly use a completion candidate from one level of search results to drill deeper into the dictionary 20 for a narrower set of completion candidates.
In the first embodiment, the digital keyboard 28 and the interactive search list 30 are also interchangeably displayed on the graphical user interface 34, as further illustrated in FIG. 5. When the digital keyboard 28 and the interactive search list 30 are interchangeable, the user can easily and quickly swap between entering characters or otherwise modifying a partial text entry from the digital keyboard 28 and using the interactive search list 30 to rapidly and flexibly complete the entry of words, phrases, and/or character sequences. When the digital keyboard 28 and the interactive search list are interchangeable, the image of the digital keyboard 28 and the image of the interactive search list 30 share substantially the same display area on the graphical user interface 34. The data entry system 26 for the first embodiment is preferably programmed to automatically swap between the digital keyboard 28 and the interactive search list 30 depending upon the input provided by user from the pointing device. This latter feature minimizes disruption to the user's attention to the data entry process since the user's attention remains focussed on the same region of the graphical user interface 34 for both the digital keyboard 28 and the interactive search list 30. Interchanging the display of the digital keyboard 28 and the interactive search list 30 within the graphical user interface 34 provides for a space-efficient layout for the use of the data entry system 26 on the touch-sensitive screen 14. The interchangeability of the digital keyboard 28 and the interactive search list 30 can also minimize hand movement as the user switches between using the digital keyboard 28 and the interactive search list 30 without having to move the pointing device to another part of the touch-sensitive screen 14 (or, more generally, without having to move the pointing device to another part of the hardware input interface). This arrangement can be particularly useful for smaller personal computing devices such as PDAs or other hand-held devices, or where the amount of space on the graphical user interface 34 used by the data entry system 26 needs to be minimized.
The interactive search list 30 enables the user to more rapidly and flexibly complete the entry of words, phrases, and/or character sequences than would otherwise be required if the user were to simply type in each individual character of the desired entry.
Overview of Methodology
The basic data entry methodology of the present invention as applied to a personal computing device 10 of the first embodiment will now be described. In what follows, reference is made to FIGS. 1 to 5.
In the first embodiment, the user interfaces with the personal computing device 10 via the touch-sensitive screen 14 using a pen as a pointing device. However, it should be noted that the following methodology can be used with various other pointing devices, such as a stylus, finger, track ball, or mouse. If a mouse or an equivalent pointing device is used in place of a pen, stylus, or finger, then for the first embodiment the act of depressing a mouse button should be considered equivalent to touching the touch-sensitive screen 14 or touch pad with a stylus, pen, or finger, and similarly, releasing a depressed mouse button should be considered equivalent to lifting the stylus, pen, finger or other pointing device from the touch-sensitive screen 14 (or a touch pad or pressure sensitive pad).
With the data entry system 26 of the first embodiment two primary entry modes are available to the user: a keyboard mode and a search mode. In the keyboard mode, the user can enter text a character (or keystroke) at a time by simply pointing and selecting on keys on the digital keyboard 28 with the pointing device. With each selection of a key on the digital keyboard 28, the one or more characters that are associated with that key are forwarded to the application 27 for entry into text in, for example, a document or data entry field. In the search mode, the user can search for and select amongst completion candidate suggestions for the completion of a word, phrase, or character sequence as further described below.
In the first embodiment, when the user begins entering a word, phrase or character sequence using the digital keyboard 28, the candidate prediction system 32 automatically begins searching the dictionary 20 for candidate words, phrases, and/or character sequences that the user may be attempting to enter. The searching of the dictionary 20 performed by the candidate prediction system 20 begins automatically in response to the emerging character sequence entered by the user. This is done by searching the dictionary 20 for completion candidates that begin with the leading character or characters which the user has entered (i.e. the partial text entry). The leading characters manually entered by the user with the pointing device are stored temporarily as a search string. The search string is used by the candidate prediction system 32 to search the dictionary 20 for potential completion candidates. The candidate prediction system 32 retrieves completion candidates on the basis of which entries in the dictionary 20 are most likely to contain the completion candidate that the user is attempting to type based on the partial text entry currently entered as indicated by the weight fields in the dictionary.
The completion candidates retrieved from the dictionary 20 by the candidate prediction system 32 are provided to the interactive search list logic which causes the interactive search list 30 to be produced. As each new list of completion candidates is produced by the candidate prediction system 32 the interactive search list logic produces a revised list of completion candidates as the user adds or deletes characters to or from the partial text entry with the digital keyboard 28. Whether the interactive search list 30 is displayed on the graphical user interface 34, however, depends on whether or not the interactive search list 30 has been activated. In the first embodiment, the user may activate the interactive search list 30 by pausing for a predetermined delay period L1 with the pointing device in a selection made by touching down on a key on the digital keyboard 28 containing one or more characters. The amount of time that the user must pause with the key selected is a user-configurable option. Typically, the delay chosen will be less than one second, although the delay may be configured to any time period the user desires. As soon as the user pauses on a selected key for the delay period L1, the interactive search list 30 becomes active. When the interactive search list 30 is activated, the entry mode for the data entry system 26 changes from the keyboard mode to the search mode. In addition, when the interactive search list 30 is activated in the first embodiment, the image of the digital keyboard 28 is cleared from the graphical user interface 34 and replaced by an image of the interactive search list 30.
Although the interactive search list 30 is updated continuously in the first embodiment, in an alternative configuration the interactive search list 30 may be generated once the user activates the interactive search list 30 by pausing on a selected key on the digital keyboard 28. In this alternative configuration, the interactive search list 30 is revised each time the user calls up a new or modified interactive search list 30.
Once the user has activated the display of the interactive search list 30 containing a list of completion candidates, the data entry system 26 provides the user with the flexibility to proceed with one of several operations using the pointing device. With the interactive search list 30 displayed and active, the user can do any one of the following: (1) the user can deactivate the interactive search list 30 and return to modifying the current partial text entry and other text; or (2) the user can select one of the completion candidates in the interactive search list 30 and use the selected completion candidate to replace the partial text entry which the user is currently entering; or (3) the user can use one of the completion candidates in the interactive search list 30 to initiate a further automated search to dynamically obtain a more refined list of completion candidates from the dictionary 20; or (4) the user can scroll or cycle through the list of completion candidates displayed in the interactive search list 30, selecting and deselecting one completion candidate at a time as the user decides which of the completion candidates (if any) will be used to complete the current partial text entry or to initiate further searching; or (5) the user can gesture to or pause in a "dead zone" within the graphical user interface 34 for any length of time without triggering any further action by the data entry system 26 so as to pause to consider whether to continue using the displayed interactive search list 30 or to carry out one of the other operations listed above.
The user's action with the pointing device generates user input signals which are monitored and analyzed by the data entry system 26 to determine the type of user selection or action being made. Which operation is executed by the data entry system 26 once the interactive search list 30 is activated depends on what action the user takes with the pointing device.
If the user does not want to select any of the completion candidates presented in the interactive search list 30 and wishes to continue entering further characters or to otherwise modify the partial text entry from the digital keyboard 28, then with the interactive search list 30 active and displayed on the graphical user interface 34, the user may lift the pointing device without any significant movement or may lift the pointing device after dragging it to or returning it to a dead zone. Either of these actions causes the hardware input interface 17 to generate a user input signal which serves as an indication to the data entry system 26 that the user wishes to deactivate the interactive search list 30 and return to modifying the current partial text entry manually with the digital keyboard 28. In the first embodiment, when the interactive search list 30 is deactivated, the image of the interactive search list 30 is cleared from the graphical user interface 34 and replaced with the image of the digital keyboard 28 which is enabled for further use by the user. This allows the user to smoothly return to using the digital keyboard 28 in keyboard mode without having to relocate the pointing device. Once in keyboard mode, the user may reinitiate searching with the interactive search list 30 to obtain further completion candidates by pausing once again on a subsequently selected key (or character) on the digital keyboard 28 with the pointing device.
A completion candidate from the interactive search list 30 is selected by the user by generating a gesture with the pointing device. For the purposes of this specification the term "gesture" refers to a motion with the pointing device when the pointing device is in an active state. In general, motions making up gestures may be linear or in another computer-recognizable pattern. For the first embodiment shown in FIGS. 1 to 5 and for variations thereof, a gesture is a motion with the pointing device in a particular direction for at least a minimum distance when the pointing device is in an active state. The minimum distance is used by the data entry system 26 in the first embodiment to ignore small, insignificant gestures with the pointing device. This minimizes false selections arising from inadvertent movements with the pointing device.
In the first embodiment, the pointing device is in an active state when the pointing device is held to the touch-sensitive screen 14. In alternative variations of the graphical display device 15 and the hardware input interface 17, other conditions can be used to identify when the pointing device is in an active state and are considered equivalent. For example, with a pressure-sensitive pad, the pointing device is in an active state when the pointing device is either in contact with the pressure-sensitive pad or depressed on the pressure-sensitive pad with at least a measurable amount of pressure. Alternatively, when the pointing device is a mouse, it is in an active state when a button on the mouse is pressed.
The data entry system 26 monitors the gestures made with the pointing device to determine when and if a completion candidate in the interactive search list 30 has been selected. In the first embodiment the data entry system 26 does this by monitoring the current position coordinates of the pointing device on the touch-sensitive screen 14 relative to a point of origin generated when the pointing device first activates the interactive search list 30. The current position coordinates of the pointing device are monitored so long as the pointing device remains in contact with the touch-sensitive screen 14. As the current position coordinates are received, the data entry system 26 generates a vector using the current position coordinates of the pointing device and the point of origin. This vector is used by the data entry system 26 to determine if and when a particular completion candidate in the interactive search list 30 has been selected. The data entry system 26 is programmed to associate certain types of vectors with certain entries in the interactive search list 30. Each entry in the interactive search list 30 may have associated with it one or more predetermined vectors. When the vector formed by the user matches such a predetermined vector, the data entry system 26 recognizes that the user is selecting the completion candidate displayed in the corresponding entry of the interactive search list 30. If the data entry system 26 determines that the vector currently being generated by the user's gesture is associated with one of the completion candidates in the interactive search list 30, then the associated completion candidate is selected from the interactive search list 30. Preferably, when a completion candidate is selected from the interactive search list 30, the particular selection is indicated to the user by the data entry system 26 by highlighting the selected completion candidate in the interactive search list 30.
In the first embodiment, the data entry system 26 is programmed to require that the user gesture with the pointing device towards or onto a completion candidate in the interactive search list 30 in order to select that completion candidate. The gesture need only be a minimum distance. In an alternative arrangement, however, when a user makes selections from the interactive search list 30 using gestures with the pointing device, the movement performed by the user is relative. With this "relative" mode of gesture-based candidate selection, the user need only gesture in a direction associated by the data entry system 26 with a desired completion candidate without the pointing device necessarily moving towards or onto the portion of the graphical user interface 34 where the completion candidate is displayed. In this case, the predetermined vectors associated by the data entry system 26 with entries in the interactive search list 30 correspond to unique gestures with the pointing device but not necessarily gestures which are towards or onto a particular entry in the interactive search list 30. With this relative mode of selection, gestures with the pointing device are used to select amongst the completion candidates on the interactive search list 30 even if the user is not gesturing with the pointing device towards or onto a particular completion candidate within the interactive search list 30. In this case, the user need not move the pointing device toward or onto a particular fixed location within the interactive search list 30 in order to select a specific completion candidate. Associating a gesture with a particular completion candidate in the interactive search list 30 in the above manner minimizes the amount of movement required with the pointing device to make selections from the interactive search list 30 and provides the user with the flexibility of selecting completion candidates with gestures which do not necessarily need to be towards or onto the desired completion candidate.
Once the user has selected a completion candidate from the interactive search list 30, several options are available to the user. If the completion candidate represents the entry that the user wishes to add to the text, then the user can accept the selected completion candidate for insertion into the text by lifting the pointing device up from the touch-sensitive screen 14 in less than a predetermined time limit L2 while keeping the particular completion candidate selected. This latter event generates a user input signal which instructs the data entry system 26 to terminate all searching based on the partial text entry and to signal to the application 27 to use the selected completion candidate to permanently replace the partial text entry. When the selected completion candidate is used to permanently replace the partial text entry at this point, the interactive search list 30 is cleared from the graphical user interface 34 and the image of the digital keyboard 28 is re-enabled ready to receive the next keystroke from the pointing device. With the partial text entry thus completed, the data entry system 26 begins monitoring anew for text entries by the user.
If the user wishes to change selections, the user can move through the list of completion candidates displayed in the interactive search list 30 by gesturing with the pointing device to other completion candidates in the interactive search list 30. If the user is unsure of which completion candidate to use, or wishes to pause to consider whether to continue in search mode or to return to keyboard mode, the user can gesture to or pause in a "dead zone" within the graphical user interface 34 for any length of time without triggering any further action by the data entry system 26. In the first embodiment, when the interactive search list 30 is first activated by pausing on a key on the digital keyboard 28, the data entry system 26 begins monitoring the length of the vector that the user thereafter generates to determine when and if the user is selecting a particular completion candidate from the interactive search list 30 or when the user is carrying out another recognized operation such as moving the pointing device to a dead zone. If the data entry system 26 determines that the length of the vector generated from the current position coordinates and the point of origin is less than or equal to a predetermined length, then the pointing device is deemed by the data entry system 26 to be in a dead zone on the graphical user interface 34. When the pointing device is in a dead zone, the user has the freedom to pause without activating any further operation by the data entry system 26 and without clearing the interactive search list 30. This provides the user with the option to pause and consider what their next operation will be. Other dead zones may be programmed into the graphical user interface 34 within the data entry system 26 for the user to move to with the pointing device so as to further enhance the user's ability to pause in different parts of the graphical user interface 34. If the vector being generated through a gesture is found by the data entry system 26 to exceed a predetermined length, the data entry system 26 checks to determine whether or not the particular vector being generated is associated with any of the completion candidates in the interactive search list 30 or with any other operation on the screen such as alternative or additional dead zones.
If the completion candidate selected on the interactive search list 30 represents only part of the entry that the user wishes to add to the text, then the user can use the selected completion candidate to dynamically initiate a further search for a more refined list of completion candidates from the dictionary 20. The ability to dynamically search for a more refined list of completion candidates based on a selected completion candidate is also referred to in this specification as an iterative search. This iterative search tool is provided through the interactive search list 30 when the interactive search list 30 is active and displays a list of potential completion candidates for the user to select from. An iterative search is triggered in the first embodiment by the user continuing to keep a completion candidate in the interactive search list 30 selected for more than the predetermined time limit L2. The automated ability to use the interactive search list 30 to further search the dictionary 20 allows the user to make good use of search results that are only partially successful. When the user continues to keep a completion candidate selected in the interactive search list 30 for more than the predetermined time limit L2, the data entry system 26 determines that the user input corresponds to a user selection to initiate a new search using the selected completion candidate as the basis for the new search. When this happens, the candidate prediction system 32 dynamically obtains a refined list of completion candidates based on the selected completion candidate. As illustrated in FIG. 5A, the refined list of completion candidates provides a narrower list of completion candidates based on a more specific search string (ie. the selected completion candidate). Iterative searching enables one to perform multi-level searches with the interactive search list 30, so that the user can quickly use a completion candidate from one level of search results to drill deeper into the dictionary 30 for a narrower set of completion candidates.
In the first embodiment, iterative searching can be initiated when a completion candidate in the interactive search list 30 represents only a first part (i.e. a leading part) of the entry that the user wishes to add to the text.
Once the candidate prediction system 32 has obtained the refined list of completion candidates, the interactive search list 30 is redisplayed with the refined list of completion candidates. When the interactive search list 30 is redisplayed, the point of origin coordinates, used by the data entry system 26 to track vectors generated from gestures with the pointing device, are set to the position coordinates of the pointing device at the time the selected completion candidate is used to initiate the iterative search. With the interactive search list 30 redisplayed, the user can then choose one of the completion candidates in the refined list by lifting the pointing device up after selecting the particular completion candidate through a gesture, or the user can further repeat the iterative search process by selecting one of the completion candidates in the refined list and pausing with that particular completion candidate selected for the predetermined time limit L2. With the refined list of completion candidates displayed, the user may also lift the pointing device without selecting any completion candidates obtained from the iterative search. This latter action leaves the search string set to the last completion candidate selected by the user in the previous iteration of the interactive search list 30 and returns the data entry system 26 to keyboard mode. At this point, the interactive search list 30 is cleared from the graphical user interface 34, the digital keyboard 28 is redisplayed, and the data entry system 26 sends the completion candidate last selected by the user from the interactive search list 30 to the application 27 for entry into the text, thereby replacing the contents of the partial text entry currently under development by the user. New characters can then be added. As a variation, using the digital keyboard 28, the user can then, if desired, instruct the application 27 to cancel the entry of the modified partial text entry into the text by selecting a function button displayed on or associated with the digital keyboard 28.
Example
FIGS. 6 and 6A show an example 100 from the user's perspective of the operation and flexibility of the data entry system 26 for the first embodiment in FIGS. 1 to 5. For the example shown in FIGS. 6 and 6A, suppose the user wishes to enter in the word "endlessly" and begins by entering at block 102 the letter "e" on the digital keyboard 28. In order to activate the interactive search list 30, the user pauses on the letter "e" for at least the predetermined time limit L1 which automatically triggers at block 104 the candidate prediction system 32 to obtain a list of completion candidates that are then displayed to the user in the interactive search list 30. Depending upon the contents of the dictionary 20 and the ranking system used to rank completion candidates stored within the dictionary 20, the desired completion candidate "endlessly" may not be one of the choices displayed in the initial interactive search list 30. Suppose for the moment, however, that the word "end" is one of the completion candidates displayed in the initial interactive search list 30. The user can select by gesture at block 108 the completion candidate "end" and use it to automatically initiate a further search of the dictionary 20 in order to retrieve a list of prioritized completion candidates which all begin with the prefix "end". As discussed, this iterative searching technique is performed with the pointing device by simply pausing while selecting a completion candidate (in this case the word "end") in the interactive search list 30 for the predetermined time limit L2 at block 116 which thereby automatically initiates a new search using the selected completion candidate as the basis for such a search. If the desired completion candidate "endlessly" appears in the updated interactive search list 30, the user can then immediately add the desired completion word by selecting it (block 120) from the interactive search list 30 and lifting the pointing device up in less than the predetermined time limit L2.
If the interactive search list 30 displayed after block 104 does not include the prefix "end" or any other prefix that would lead the user to rapidly enter the desired word "endlessly" using the data entry system 26, then the user has the option of lifting the pointing device up from the touch-sensitive screen 14 at block 106 without selecting any of the completion candidates. This type of user input notifies the data entry system 26 to clear the interactive search list 30 from the screen and to re-enable and display the digital keyboard 28. It should be noted that in this latter operation the search string continues to contain the partial text entry which the user has generated with the digital keyboard 28 (in this case the search string contains only the letter "e").
As a further illustration of the flexibility of the present data entry system 26, if the partial completion candidate "end" is used to initiate a further search and the new list of completion candidates does not include the word "endlessly" in the list of completion candidates, then the user can choose to continue building upon the partial completion candidate "end" by lifting the pointing device up from the touch-sensitive screen 14 without any of the completion candidates in the new list selected and proceeding to continue entering in characters from the digital keyboard 28. In this example the partial completion candidate "end" permanently replaces the partial text entry that the user was generating with the digital keyboard 28 in the user's electronic text, the search string is cleared and any new user input is automatically treated as being part of a new partial text entry.
It will be appreciated from the above particular that the user can follow the final steps to completing a partial completion candidate by entering in the remaining letters at the end of the partial completion candidate. For instance, if the partial completion candidate "endless" was retrieved, then the user can simply tap on the digital keyboard 28 the letters "l" and "y" followed by a space (or the end-of-entry function button) in order to notify the data entry system 26 of the completion of the current text entry. Thus, even when a complete word, phrase, or sequence of characters is not found in the dictionary 20, the use of the data entry system 26 to retrieve a partial completion candidate can result in less time and effort being expended than if the user had simply typed in each letter of the desired word, phrase, or sequence of characters.
In an alternative embodiment, the data entry system 26 is programmed with the ability to re-initiate automated searching even once the application 27 is instructed by the data entry system 26 to permanently replace the partial text entry in the text with a partial completion candidate. In this alternative, the search string is set to the partial completion candidate when the partial completion candidate replaces the partial text entry in the user's electronic text. The user may then return at any time to the automated search facility of the data entry system 26 by pausing on a key on the digital keyboard 28 for the predetermined time limit L1. For instance, if the partial completion selection "end" has already been used to initiate a further search and the user then lifts up the pointing device without selecting any of the completion candidates in the new list of completion candidates, the user can then return to automated searching with the data entry system 26 by, for example, touching on the letter "l" on the digital keyboard 28 for a sufficient period of time to initiate a search on the basis of the prefix "endl". The partial text entry "endl" will then be used by the candidate prediction system 32 to obtain a list of completion candidates that are then displayed in the interactive search list 30. If the desired completion candidate "endlessly" appears in the new list of completion candidates, the user may then choose that candidate and add it to the text by selecting that candidate and lifting the pointing device before the time limit L2 is reached. If the desired completion candidate does not appear in the interactive search list 30, the user can simply lift the pointing device without selecting any of the completion candidates and continue building upon the partial text entry "endl" by entering further characters via the digital keyboard 28. Once the desired word is complete, in order to clear the search string and instruct the data entry system 26 to treat any new user input as being part of a new partial text entry in this alternative, the user selects a key or function from the digital keyboard 28 programmed to indicate that entry of the original partial text entry has ended, as discussed further in the section below.
http://tinyurl.com/39bb8a
enter 09631101
Look for the Tab "Published Documents"
FIELD OF THE INVENTION
The present invention relates generally to computer-assisted data entry and more particularly to a method, system, and apparatus for computer-assisted text generation and entry using a pointing device with a personal computing device, and to computer-readable media having executable instructions for supporting text generation and entry using a pointing device.
BACKGROUND OF THE INVENTION
The wide-spread adoption of miniaturized personal computing devices, such as hand-held devices and personal digital assistants (PDAs), has led to an increasing use of devices to send and receive text and data. One example of this trend is pen-based computing, wherein users enter text and commands into hand-held personal computers via a touch-sensitive screen. While such pen-based computing is popular, especially with the increasing power of miniature computing devices, it does present challenges to a user entering data in an application running on the hand-held device. For instance, many hand-held computers and personal digital assistants require that the user enter data according to a predetermined scripting style, such as with the PalmPilot.TM. series of PDAs. Other hand-held devices provide a handwriting recognition system which requires that the computer learn the user's handwriting style. While such data entry mechanisms are useful, they are relatively difficult to use and complex to learn and can be prone to error in the event the user deviates from the predetermined scripting style or the user's traditional handwriting style.
Many pen-based computing systems, both large and small, offer the user the option to enter text using an on-screen digital keyboard. On-screen digital keyboards are typically miniaturized replica of conventional full-sized physical keyboards, such as QWERTY keyboards. Many on-screen keyboards have shown themselves to be less than efficient for entering text. When using a pointing device such as a pen, a user is typically required to enter text one character at a time by tapping out individual character selections from the on-screen keyboard. This "hunt-and-peck" method of typing with a single pointing device is time-consuming, especially when a user is entering large amounts of data.
Another common challenge when entering data into a personal computing device with a single pointing device such as a pen or stylus, and in particular when entering text, is that each letter making up the word or phrase must be entered manually. The longer the word or phrase, the greater the amount of manual entry required.
Text completion systems have been developed in an effort to assist users with text entry. In general, these systems predict and suggest a complete word completion based on a partial text entry entered by a user. These systems allow a user to type in the partial text entry and then accept a predicted text completion for the partial text entry. This avoids the keystrokes that would otherwise be required to type the complete text desired by a user. While such text completion systems provide some basic assistance for users to more rapidly enter text than would be required if every character of the desired text had to be typed in independently, there remains a need in the art for a more flexible text completion system for use with a single pointing device. It would also be desirable for such a text completion system to employ a convenient selection technique which would reduce the amount of movement of the pointing device required to enter text into a computer. It would further be desirable if such a system were applicable to both large and small personal computing devices.
Another problem in the art is that soft, or digital, keyboards have tended to be continually displayed so as to permanently consume screen space or have needed to be manually invoked and dismissed by the user. It would be desirable if a digital keyboard could automatically appear and disappear as required.
SUMMARY OF THE INVENTION
The above and related desires are addressed in the present invention by providing a novel and non-obvious method, system and computer-readable instructions for computer-assisted text generation and entry using a pointing device with a personal computing device.
In one aspect of the present invention the user can rapidly enter and search for text using a data entry system through a combination of entering one or more characters on a digitally displayed keyboard with a pointing device and using a search list to obtain a list of completion candidates. The user can activate the search list to obtain a list of completion candidates at any time while entering a partial text entry with the data entry system. When the search list is active, a list of completion candidates is displayed on a graphical user interface for the user to select from and the user can perform one of several actions. The user can deactivate the search list and return to modifying the current partial text entry and other text. The user can select one of the completion candidates in the search list and use the selected completion candidate to replace the partial text entry which the user is currently entering. When the user deactivates the interactive search list, the user can immediately continue adding to or modifying the current partial text entry being entered, and may re-invoke the search list to further search for completion candidates based on the modified partial text entry. In the second case, the selected completion candidate is used to replace the partial text entry that the user is currently entering, and the data entry system begins monitoring for a new partial text entry from the user.
In one embodiment, when the search list is active the user may use one of the completion candidates in the search list to initiate a further automated search to obtain a more refined list of completion candidates. In this embodiment, multi-level search lists and searching are available to help accelerate completion of a partial text entry. As a result, the user can automatically initiate an iterative search wherein a completion candidate listed in the search list is used as the new partial text entry to dynamically obtain a new list of completion candidates, which is then displayed in the search list. The automated ability to use the search list to obtain a refined list of completion candidates allows the user to quickly make good use of search results that are only partially successful. When the search list is revised with a new list of completion candidates, the user can then choose one of the completion candidates in the new list, or the user can repeat the iterative search process once again by choosing one of the completion candidates in the new list and activating a further iterative search. In addition, the user may return to keyboard entry with the last completion candidate selected by the user in the previous iteration of the search list. This latter feature provides the user with the convenience of being able to automatically and seamlessly continue entering the desired word, phrase, or character sequence using the last completion candidate selected by the user in the previous iteration of the interactive search list. Thus, the user can quickly and easily replace the user's current partial text entry with a partially successful completion candidate and continue building upon or modifying this partial completion candidate using the keyboard while at the same time allowing the user the flexibility to re-enter the search list at any time.
In another aspect of the present invention, there is provided a method of processing text entered into a personal computing device with a pointing device. With this method, a partial text entry is received and used to obtain a dynamically generated list of completion candidates. The list of completion candidates is displayed in a search list within a graphical user interface. A user input signal associated with the pointing device is received. If the user input signal corresponds to a first type of user selection with the pointing device, then the search list is deactivated. If the user input signal corresponds to a second type of user selection with the pointing device, then the partial text entry is replaced with a completion candidate from the search list.
In one embodiment, if the user input signal corresponds to a third type of user selection with the pointing device, then a refined list of completion candidates is dynamically obtained based on one of the completion candidates from the search list. The refined list is displayed in the search list for further user selection.
In another aspect of the present invention, there is provided a method of processing an input string at least partially entered into a personal computing device with a pointing device. This aspect includes performing a search of a set of completion candidates to locate a plurality of possible completion candidates for completing the input string in response to a prior located possible completion candidate or a character selectable by a user. At least one of the plurality of possible completion candidates and characters selectable by the user are displayed.
In another aspect of the present invention, a method is provided of user-based text entry. With this aspect, a set of position coordinates for a pointing device is monitored relative to a user interface and a digital keyboard is displayed on the user interface at a last known set of coordinates for the pointing device whenever the digital keyboard is activated for user input.
In another aspect of the present invention, a method is provided for interchanging the display of a digital keyboard and a search list. The digital keyboard is displayed on a user interface when a user is entering text a keystroke at a time. As the digital keyboard is displayed, user input is monitored. If the user input corresponds to activating the search list, then the digital keyboard is replaced with the search list. If the user input corresponds to terminating use of the search list once activated, then the search list is replaced with the digital keyboard.
In another aspect of the present invention, a digital keyboard is configured to include a plurality of characters assigned to predetermined locations within a layout for the digital keyboard according to a predetermined frequency distribution associated with the plurality of characters. The plurality of characters includes less commonly used characters and more commonly used characters based on the predetermined frequency distribution. The digital keyboard is displayed on a graphical user interface with the less commonly used characters displayed substantially further from a center of the digital keyboard than the more commonly used characters.
In another aspect of the present invention there is provided a system for computer-assisted text generation and entry. The system includes an input interface, a processing unit and a computer-readable medium. The input interface receives user input signals based on actions with a pointing device. The computer-readable medium contains computer-readable instructions for directing the processing unit to assist with text generation and entry based on user input received via the input interface with the pointing device. The computer-readable medium includes instructions for receiving a partial text entry; for obtaining a dynamically generated list of completion candidates based on the partial text entry; for displaying the list of completion candidates in a search list on a display device; and for receiving a user input signal associated with the pointing device from the input interface. If the user input signal corresponds to a first type of user selection with the pointing device, the computer-readable instructions are programmed to deactivate the search list. If the user input signal corresponds to a second type of user selection with the pointing device, the computer-readable instructions are programmed to replace the partial text entry with a completion candidate from the search list.
Other aspects and features of the present invention will become apparent to those ordinarily skilled in the art upon review of the following description of specific embodiments of the invention in conjunction with the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
In the accompanying drawings which illustrate embodiments of the invention,
FIG. 1 is a schematic diagram of a personal computing device loaded with a data entry system, according to a first embodiment of the invention;
FIG. 2 is a schematic diagram of the data entry system of the first embodiment;
FIG. 3 is a schematic representation illustrating the display of a digital keyboard on a graphical user interface within the personal computing device of the first embodiment;
FIG. 4 is a schematic representation of a data structure for a dictionary according to the first embodiment;
FIG. 5 is a schematic representation illustrating the interchangeable display of the digital keyboard and an interactive search list according to the first embodiment;
FIG. 5A is a schematic representation illustrating potential completion candidates for retrieval and display in the interactive search list according to an example of the use of the first embodiment;
FIGS. 6 and 6A are flow diagrams illustrating, by way of example from the user's perspective, the use of the data entry system of the first embodiment;
FIGS. 7 to 9 are flow diagrams illustrating the flow of operation of the data entry system according to the first embodiment;
FIG. 10 is a schematic representation of an alternative embodiment of a digital keyboard layout according to the present invention;
FIG. 11 is a schematic representation of another alternative embodiment of a digital keyboard layout according to the present invention;
FIG. 12 is a schematic representation of another alternative embodiment of a digital keyboard layout according to the present invention;
FIG. 13 is a schematic representation of another configuration of a digital keyboard according to an embodiment of the present invention;
FIG. 14 is a schematic representation of the layout for the interactive search list according to the first embodiment;
FIGS. 15 to 18 are schematic representations of alternative layouts for the interactive search list according to alternate embodiments of the present invention;
FIG. 19 is a schematic representation of a configuration for the digital keyboard and the interactive search list according to an embodiment of the present invention;
FIG. 20 is a schematic representation of an alternative configuration for the digital keyboard according to an embodiment of the present invention;
FIGS. 21 to 22 are flow diagrams illustrating the flow of operation of the data entry system according to an alternative embodiment of the present invention;
FIGS. 23 to 24 are flow diagrams illustrating the flow of operation of the data entry system according to an alternative embodiment of the present invention;
FIGS. 25 to 26 are flow diagrams illustrating the flow of operation of the data entry system according to an alternative embodiment of the present invention;
FIG. 27 is a schematic representation of an alternative configuration for the digital keyboard according to an embodiment of the present invention.
It will be appreciated that for simplicity and clarity of illustration, elements illustrated in the accompanying drawings have not necessarily been drawn to scale. For example, the dimensions of some of the elements are exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals and labels have been repeated among the drawings to indicate corresponding or analogous elements and features.
DETAILED DESCRIPTION
Reference will now be made in detail to implementations and embodiments of the invention, examples of which are illustrated in the accompanying drawings.
Introduction
In one aspect of the present invention the user can rapidly enter and search for text using a data entry system through a combination of entering one or more characters on a digitally displayed keyboard with a pointing device and using an interactive search list to dynamically obtain a list of completion candidates. The user can activate the interactive search list to obtain a dynamically generated list of completion candidates at any time while entering a partial text entry with the data entry system. In this specification "partial text entry" means a sequence of one or more characters making up a leading portion of a word, phrase or character sequence. When the interactive search list is active, a dynamically generated list of completion candidates is displayed on a graphical user interface for the user to select from. The list of completion candidates is retrieved from at least one dictionary by a candidate prediction system which retrieves completion candidates that are most likely to contain the completion candidate desired by the user. Candidate prediction is based on statistical measures ranking the entries within the dictionary relative to each other. When the interactive search list is active, the user can perform one of several actions with the data entry system including any of the following: (1) the user can deactivate the interactive search list and return to modifying the current partial text entry and other text; or (2) the user can select one of the completion candidates in the interactive search list and use the selected completion candidate to replace the partial text entry which the user is currently entering.
In the first case, when the user deactivates the interactive search list, the user can immediately continue adding to or modifying the current partial text entry being entered, and may re-invoke the interactive search list to further search for completion candidates based on the modified partial text entry. In the second case, the selected completion candidate is used to replace the partial text entry that the user is currently entering, and the data entry system begins monitoring for a new partial text entry (of a word, phrase, or character sequence) from the user.
In certain embodiments of the present invention, as illustrated further below, a third action available to the user when the interactive search list is activated is to: (3) use one of the completion candidates in the interactive search list to initiate a further automated search to obtain a more refined dynamic list of completion candidates from the dictionary.
In the third case, multi-level search lists and searching are available to help accelerate completion of a partial text entry. In this latter case, the user can automatically initiate an iterative search wherein a completion candidate listed in the interactive search list is used as the new partial text entry to dynamically obtain a new list of completion candidates, which is then displayed in an updated interactive search list. The automated ability to use the interactive search list to dynamically obtain a refined list of completion candidates allows the user to quickly make good use of search results that are only partially successful. When the interactive search list is updated with a new list of completion candidates, the user can then choose one of the completion candidates in the new list, or the user can repeat the iterative search process once again by choosing one of the completion candidates in the new list and activating a further iterative search. In addition, the user may return to keyboard entry with the last completion candidate selected by the user in the previous iteration of the interactive search list. This latter feature provides the user with the convenience of being able to automatically and seamlessly continue entering the desired word, phrase, or character sequence using the last completion candidate selected by the user in the previous iteration of the interactive search list. Thus, the user can quickly and easily replace the user's current partial text entry with a partially successful completion candidate and continue building upon or modifying this partial completion candidate using the keyboard while at the same time allowing the user the flexibility to re-enter the interactive search list at any time.
As discussed further in this specification, other actions may also be carried out by the user when the interactive search list is active.
Operation Environment
FIG. 1 shows a schematic diagram of a personal computing device 10 for text entry according to a first embodiment of the invention. The personal computing device 10 shown in FIG. 1 contains at least one processing unit 12 (such as a CPU or a similar processor or multiprocessor) connected by a bus to a computer-readable medium 16. The computer-readable medium 16 provides a memory store for software and data residing within the personal computing device 10. The computer-readable medium 16 can include one or more types of computer-readable media including volatile memory such as Random Access Memory (RAM), and non-volatile memory, such as a hard disk or Read Only Memory (ROM). Preferably, the computer-readable medium 16 includes a combination of volatile and non-volatile memory. In the first embodiment, the computer-readable medium 16 contains an operating system, a data entry system 26 and an application 27 receptive to user-based text entry such as a word processor. The computer-readable medium 16 may also store alternative or other applications such as a browser or micro-browser, an e-mail application, and/or other end-user applications.
The operating system can be any of several well-known operating systems depending on the personal computing device used. For example, for hand-held devices, the operating system can be PalmOS.TM., Windows CE.TM., or an equivalent operating system. For larger systems, such as with work stations or desktop computers, a more robust operating system may be used such as, for example, Windows 95 .TM., Windows 98.TM., Windows NT.TM., Windows 2000.TM., MacOS.TM., UNIX, Linux or the like. For the purposes of the first embodiment, the operating system is PalmOS.TM..
The data entry system 26 is implemented as software that runs on the processing unit 12 to support computer-assisted text generation and entry for the user, although in other alternatives the data entry system 26 can be implemented as computer-readable instructions in firmware or embedded in hardware components. In the first embodiment, electronic text and documents are generated and maintained by the application 27 and the user authors and edits the electronic text and documents with the data entry system 26 which communicates with the application 27 through an application programming interface (API). This allows the data entry system 26 to be portable so that it can be used by one or more applications to accept text and data entry from the user. As an alternative, the data entry system 26 may be integrated into part of an application.
The personal computing device 10 includes a graphical display device 15 and a hardware input interface 17 receptive to user input from a pointing device. In this specification, the term "pointing device" means an input device that allows a user to select one choice amongst one or many choices (a user-based selection). Some pointing devices enable a user to make user-based selections by pointing to a desired choice and include, by way of example, a pen, stylus, or finger. More generally, pointing devices capable of supporting user-based selections include, by way of example, the pointing devices above capable of pointing, as well as other input devices such as a mouse, trackball or the like.
The graphical display device 15 is connected to and controlled by the processing unit 12 via a video display circuit 13. The graphical display device 15 may be a CRT, a liquid crystal display, or an equivalent computer display.
In the first embodiment, the personal computing device 10 is a personal digital assistant wherein the graphical display device 15 and the hardware input interface 17 are combined in the form of a touch-sensitive screen 14 that serves both as a graphical display 15 and as an input interface receptive to generating coordinate position signals in response to contact from a pointing device such as a pen or stylus. It will be appreciated by those skilled in the art that the personal computing device 10 is represented in the following discussion as a personal digital assistant for illustration purposes only, and that the invention may be practised with other personal computing devices including hand-held devices, personal computers and other microprocessor-based electronic devices, mobile telephones, internet appliances, and embedded devices, having a suitable graphical display and an input interface receptive to user input via a pen, stylus, finger, mouse, or an equivalent pointing device that allows the user to select one choice from many. Other types of equivalent personal computing devices to which the features and aspects of the present invention are applicable include, by way of example, an internet appliance controlled via a remote control (for instance, running an Internet service through a television via a set top box and a remote control). In other embodiments, the hardware input interface 17 may be a digitising tablet, a pressure-sensitive input surface or a proximity sensing input surface. It will also be appreciated that the personal computing device 10 may be powered by an internal (18) or external power source.
As shown in FIG. 2, the data entry system 26 includes computer-readable instructions for a digital keyboard 28, a candidate prediction system 32, a dictionary 20, and an interactive search list 30. The digital keyboard 28 provides an interface for the user to enter text and data into the personal computing device 10 (FIG. 1). As the user enters in characters via the digital keyboard 28 to construct a word, phrase, or character sequence, the characters entered by the user are stored in a search string as a partial text entry. The search string is used by the candidate prediction system 32 to search the dictionary 20 for completion candidates that begin with the current partial text entry being stored in the search string. The interactive search list 30 is used to display for user selection the list of completion candidates retrieved by the candidate prediction system 32. In the first embodiment, the data entry system 26 supports gesture-based user input for the selection of completion candidates from the interactive search list 30 as will be described in further detail below.
Digital Keyboard
As illustrated in FIG. 3, an image of the digital keyboard 28 is displayed on a graphical user interface 34 within the screen area of the touch-sensitive screen 14 once the data entry system 26 is initialised and ready to receive input from the user. The digital keyboard 28 contains a plurality of keys each of which is associated with at least one character from a set of characters. For example, when the English alphabet or a character set containing the English alphabet is used, each key on the digital keyboard 28 can contain one or more letters from the English alphabet. It should be noted that reference to the English alphabet is by way of example only, and that the digital keyboard 28 can be configured to contain and display any set of characters which the user may then select and use to enter text into the personal computing device 10. The terms "character set" and "set of characters" refer in this specification to a set containing a plurality of letters, numbers and/or other typographic symbols. Examples of character sets include, but are not limited to, one or more alphabets of a written language (e.g. English, French, German, Spanish, Chinese, or Japanese), and binary-coded character sets such as ASCII (American Standard Code for Information Interchange), EBCDIC (Extended Binary Coded Decimal Interexchange Code), and BCD (Binary Coded Decimal).
In the first embodiment, the digital keyboard 28 displays digital keys containing characters from the English alphabet along with special characters selected from the ASCII character set. Words, phrases, and character sequences can be typed into an electronic document or electronic text by simply using the pointing device to tap or select in sequence each key for the desired word, phrase, or character sequence. As will be discussed further below, the user can also use the digital keyboard 28 to initiate an automated search for completion candidates to more rapidly and flexibly enter words, phrases, and/or character sequences in an automated manner. As will also be discussed later in this specification, several enhancements to the digital keyboard 28 may be implemented to further enhance the user's ability to quickly, efficiently, and flexibly enter text using a pointing device.
Dictionary
FIG. 4 shows a sample data structure for the dictionary 20. Preferably, as in the first embodiment, the dictionary 20 contains completion candidates with weight values for ranking completion candidates relative to each other. The dictionary 20 contains a plurality of entries, with each entry having a completion candidate field 22 for storing a completion candidate and a weight field 24 for storing a numeric value associated with the completion candidate stored in a corresponding completion candidate field 22. Each completion candidate stored in the dictionary 20 represents a word, phrase, or character sequence according to a particular language. Character sequences may include, but are not limited to, word continuations. Word continuations represent a leading part of a word, but not an entire word. In the first embodiment where the dictionary is based on American english, word continuations may include, by way of example, such common word prefixes as "com", "con", "dis", "expl", "inter", "mis", "para", "pre", "syn", "tele", "trans" and "univers". Obtaining a list of completion candidates which can include common word prefixes reduces user effort in terms of the number of steps required to generate a desired text entry. Furthermore, certain character combinations such as word prefixes are shared across multiple completion candidates which can speed up learning of the locations of completion candidates within sub-levels of the interactive search list 30 when multi-level searching (i.e. iterative searching) is available.
Each weight field 24 stores a weight value for ranking the corresponding completion candidate stored in the corresponding completion candidate field 22 with other completion candidates in the dictionary 20. The weight value stored in a weight field 24 may be based on one of many metrics. By way of example, each weight field 24 may contain a value representing a degree of common usage of the completion candidate stored in the corresponding completion candidate field 22 relative to the other completion candidates in the dictionary 20. As another example, the weight fields 24 may contain numeric values based on what words or phrase came before the completion candidate stored in the corresponding completion candidate from an analysis of a large corpus of text. Each weight field 24 may also be supplemented with one or more related fields.
Candidate Prediction System
Referring to FIG. 2, the candidate prediction system 32 is programmed to dynamically search the dictionary 20 for completion candidates that begin with the partial text entry entered by the user. The candidate prediction system 32 retrieves completion candidates from the dictionary 20 by determining which dictionary entries are more likely to be the ones that the user is attempting to type. In the first embodiment, the completion candidates are obtained from the dictionary 20 on the basis of frequency values stored in the weight field 24 for each entry. In the first embodiment, completion candidates having the highest weight values are retrieved. The frequency values represent the frequency of usage of the entries relative to each other. These frequency values may be predetermined on the basis of an analysis of a large corpus of text or may be dynamically generated or modified on the basis of the specific user's usage of words, phrases, and/or character sequences through the data entry system 26. Other statistical measures may also be employed to enhance the candidate prediction system 32, such as ranking information or identifying the frequency with which an entry in the dictionary 20 follows any of the other entries in the dictionary 20.
Preferably, the candidate prediction system 32 retrieves all possible completion candidates limited by one or more predetermined metrics. In the first embodiment, the total number of completion candidates retrieved is limited by a predetermined maximum number of displayable completion candidates. The maximum number of completion candidates retrieved by the candidate prediction system 32 is preferably a small but significant number sufficient enough to provide the user with as many potential candidates as possible without unduly saturating the user with an excessive number of candidates and without unduly delaying the user's ability to quickly review and select from the candidates selected. The maximum number of completion candidates is also preferably sufficiently large enough to provide a variety of completion candidates (if available) for the user to choose from so as to avoid an excessive amount of multi-level searching to complete a partial text entry. In the case of the first embodiment, the maximum number of displayable completion candidates is set to five. The data entry system 26 can be configured by the user to present a greater or lesser number of completion candidates. Preferably, the completion candidates are displayable in a configuration that can also be selected by the user. Making the number of completion candidates that can be displayed a user-configurable feature provides enhanced capabilities for computer-assisted text generation and entry with the data entry system 26.
Interactive Search List
The interactive search list 30 receives completion candidates retrieved from the dictionary 20 by the candidate prediction system 32 and presents the user with a list of these completion candidates. The interactive search list 30 can be displayed to the user in any of several different ways depending on which options the data entry system 26 has been programmed with and which of those options have been selected by the user, as will be discussed later in this specification. In the first embodiment, the interactive search list 30 is displayed on the touch-sensitive screen 14 as an interactive vertical list of completion candidates. As described further below, the interactive search list 30 is programmed to support multi-level searches so that the user can quickly use a completion candidate from one level of search results to drill deeper into the dictionary 20 for a narrower set of completion candidates.
In the first embodiment, the digital keyboard 28 and the interactive search list 30 are also interchangeably displayed on the graphical user interface 34, as further illustrated in FIG. 5. When the digital keyboard 28 and the interactive search list 30 are interchangeable, the user can easily and quickly swap between entering characters or otherwise modifying a partial text entry from the digital keyboard 28 and using the interactive search list 30 to rapidly and flexibly complete the entry of words, phrases, and/or character sequences. When the digital keyboard 28 and the interactive search list are interchangeable, the image of the digital keyboard 28 and the image of the interactive search list 30 share substantially the same display area on the graphical user interface 34. The data entry system 26 for the first embodiment is preferably programmed to automatically swap between the digital keyboard 28 and the interactive search list 30 depending upon the input provided by user from the pointing device. This latter feature minimizes disruption to the user's attention to the data entry process since the user's attention remains focussed on the same region of the graphical user interface 34 for both the digital keyboard 28 and the interactive search list 30. Interchanging the display of the digital keyboard 28 and the interactive search list 30 within the graphical user interface 34 provides for a space-efficient layout for the use of the data entry system 26 on the touch-sensitive screen 14. The interchangeability of the digital keyboard 28 and the interactive search list 30 can also minimize hand movement as the user switches between using the digital keyboard 28 and the interactive search list 30 without having to move the pointing device to another part of the touch-sensitive screen 14 (or, more generally, without having to move the pointing device to another part of the hardware input interface). This arrangement can be particularly useful for smaller personal computing devices such as PDAs or other hand-held devices, or where the amount of space on the graphical user interface 34 used by the data entry system 26 needs to be minimized.
The interactive search list 30 enables the user to more rapidly and flexibly complete the entry of words, phrases, and/or character sequences than would otherwise be required if the user were to simply type in each individual character of the desired entry.
Overview of Methodology
The basic data entry methodology of the present invention as applied to a personal computing device 10 of the first embodiment will now be described. In what follows, reference is made to FIGS. 1 to 5.
In the first embodiment, the user interfaces with the personal computing device 10 via the touch-sensitive screen 14 using a pen as a pointing device. However, it should be noted that the following methodology can be used with various other pointing devices, such as a stylus, finger, track ball, or mouse. If a mouse or an equivalent pointing device is used in place of a pen, stylus, or finger, then for the first embodiment the act of depressing a mouse button should be considered equivalent to touching the touch-sensitive screen 14 or touch pad with a stylus, pen, or finger, and similarly, releasing a depressed mouse button should be considered equivalent to lifting the stylus, pen, finger or other pointing device from the touch-sensitive screen 14 (or a touch pad or pressure sensitive pad).
With the data entry system 26 of the first embodiment two primary entry modes are available to the user: a keyboard mode and a search mode. In the keyboard mode, the user can enter text a character (or keystroke) at a time by simply pointing and selecting on keys on the digital keyboard 28 with the pointing device. With each selection of a key on the digital keyboard 28, the one or more characters that are associated with that key are forwarded to the application 27 for entry into text in, for example, a document or data entry field. In the search mode, the user can search for and select amongst completion candidate suggestions for the completion of a word, phrase, or character sequence as further described below.
In the first embodiment, when the user begins entering a word, phrase or character sequence using the digital keyboard 28, the candidate prediction system 32 automatically begins searching the dictionary 20 for candidate words, phrases, and/or character sequences that the user may be attempting to enter. The searching of the dictionary 20 performed by the candidate prediction system 20 begins automatically in response to the emerging character sequence entered by the user. This is done by searching the dictionary 20 for completion candidates that begin with the leading character or characters which the user has entered (i.e. the partial text entry). The leading characters manually entered by the user with the pointing device are stored temporarily as a search string. The search string is used by the candidate prediction system 32 to search the dictionary 20 for potential completion candidates. The candidate prediction system 32 retrieves completion candidates on the basis of which entries in the dictionary 20 are most likely to contain the completion candidate that the user is attempting to type based on the partial text entry currently entered as indicated by the weight fields in the dictionary.
The completion candidates retrieved from the dictionary 20 by the candidate prediction system 32 are provided to the interactive search list logic which causes the interactive search list 30 to be produced. As each new list of completion candidates is produced by the candidate prediction system 32 the interactive search list logic produces a revised list of completion candidates as the user adds or deletes characters to or from the partial text entry with the digital keyboard 28. Whether the interactive search list 30 is displayed on the graphical user interface 34, however, depends on whether or not the interactive search list 30 has been activated. In the first embodiment, the user may activate the interactive search list 30 by pausing for a predetermined delay period L1 with the pointing device in a selection made by touching down on a key on the digital keyboard 28 containing one or more characters. The amount of time that the user must pause with the key selected is a user-configurable option. Typically, the delay chosen will be less than one second, although the delay may be configured to any time period the user desires. As soon as the user pauses on a selected key for the delay period L1, the interactive search list 30 becomes active. When the interactive search list 30 is activated, the entry mode for the data entry system 26 changes from the keyboard mode to the search mode. In addition, when the interactive search list 30 is activated in the first embodiment, the image of the digital keyboard 28 is cleared from the graphical user interface 34 and replaced by an image of the interactive search list 30.
Although the interactive search list 30 is updated continuously in the first embodiment, in an alternative configuration the interactive search list 30 may be generated once the user activates the interactive search list 30 by pausing on a selected key on the digital keyboard 28. In this alternative configuration, the interactive search list 30 is revised each time the user calls up a new or modified interactive search list 30.
Once the user has activated the display of the interactive search list 30 containing a list of completion candidates, the data entry system 26 provides the user with the flexibility to proceed with one of several operations using the pointing device. With the interactive search list 30 displayed and active, the user can do any one of the following: (1) the user can deactivate the interactive search list 30 and return to modifying the current partial text entry and other text; or (2) the user can select one of the completion candidates in the interactive search list 30 and use the selected completion candidate to replace the partial text entry which the user is currently entering; or (3) the user can use one of the completion candidates in the interactive search list 30 to initiate a further automated search to dynamically obtain a more refined list of completion candidates from the dictionary 20; or (4) the user can scroll or cycle through the list of completion candidates displayed in the interactive search list 30, selecting and deselecting one completion candidate at a time as the user decides which of the completion candidates (if any) will be used to complete the current partial text entry or to initiate further searching; or (5) the user can gesture to or pause in a "dead zone" within the graphical user interface 34 for any length of time without triggering any further action by the data entry system 26 so as to pause to consider whether to continue using the displayed interactive search list 30 or to carry out one of the other operations listed above.
The user's action with the pointing device generates user input signals which are monitored and analyzed by the data entry system 26 to determine the type of user selection or action being made. Which operation is executed by the data entry system 26 once the interactive search list 30 is activated depends on what action the user takes with the pointing device.
If the user does not want to select any of the completion candidates presented in the interactive search list 30 and wishes to continue entering further characters or to otherwise modify the partial text entry from the digital keyboard 28, then with the interactive search list 30 active and displayed on the graphical user interface 34, the user may lift the pointing device without any significant movement or may lift the pointing device after dragging it to or returning it to a dead zone. Either of these actions causes the hardware input interface 17 to generate a user input signal which serves as an indication to the data entry system 26 that the user wishes to deactivate the interactive search list 30 and return to modifying the current partial text entry manually with the digital keyboard 28. In the first embodiment, when the interactive search list 30 is deactivated, the image of the interactive search list 30 is cleared from the graphical user interface 34 and replaced with the image of the digital keyboard 28 which is enabled for further use by the user. This allows the user to smoothly return to using the digital keyboard 28 in keyboard mode without having to relocate the pointing device. Once in keyboard mode, the user may reinitiate searching with the interactive search list 30 to obtain further completion candidates by pausing once again on a subsequently selected key (or character) on the digital keyboard 28 with the pointing device.
A completion candidate from the interactive search list 30 is selected by the user by generating a gesture with the pointing device. For the purposes of this specification the term "gesture" refers to a motion with the pointing device when the pointing device is in an active state. In general, motions making up gestures may be linear or in another computer-recognizable pattern. For the first embodiment shown in FIGS. 1 to 5 and for variations thereof, a gesture is a motion with the pointing device in a particular direction for at least a minimum distance when the pointing device is in an active state. The minimum distance is used by the data entry system 26 in the first embodiment to ignore small, insignificant gestures with the pointing device. This minimizes false selections arising from inadvertent movements with the pointing device.
In the first embodiment, the pointing device is in an active state when the pointing device is held to the touch-sensitive screen 14. In alternative variations of the graphical display device 15 and the hardware input interface 17, other conditions can be used to identify when the pointing device is in an active state and are considered equivalent. For example, with a pressure-sensitive pad, the pointing device is in an active state when the pointing device is either in contact with the pressure-sensitive pad or depressed on the pressure-sensitive pad with at least a measurable amount of pressure. Alternatively, when the pointing device is a mouse, it is in an active state when a button on the mouse is pressed.
The data entry system 26 monitors the gestures made with the pointing device to determine when and if a completion candidate in the interactive search list 30 has been selected. In the first embodiment the data entry system 26 does this by monitoring the current position coordinates of the pointing device on the touch-sensitive screen 14 relative to a point of origin generated when the pointing device first activates the interactive search list 30. The current position coordinates of the pointing device are monitored so long as the pointing device remains in contact with the touch-sensitive screen 14. As the current position coordinates are received, the data entry system 26 generates a vector using the current position coordinates of the pointing device and the point of origin. This vector is used by the data entry system 26 to determine if and when a particular completion candidate in the interactive search list 30 has been selected. The data entry system 26 is programmed to associate certain types of vectors with certain entries in the interactive search list 30. Each entry in the interactive search list 30 may have associated with it one or more predetermined vectors. When the vector formed by the user matches such a predetermined vector, the data entry system 26 recognizes that the user is selecting the completion candidate displayed in the corresponding entry of the interactive search list 30. If the data entry system 26 determines that the vector currently being generated by the user's gesture is associated with one of the completion candidates in the interactive search list 30, then the associated completion candidate is selected from the interactive search list 30. Preferably, when a completion candidate is selected from the interactive search list 30, the particular selection is indicated to the user by the data entry system 26 by highlighting the selected completion candidate in the interactive search list 30.
In the first embodiment, the data entry system 26 is programmed to require that the user gesture with the pointing device towards or onto a completion candidate in the interactive search list 30 in order to select that completion candidate. The gesture need only be a minimum distance. In an alternative arrangement, however, when a user makes selections from the interactive search list 30 using gestures with the pointing device, the movement performed by the user is relative. With this "relative" mode of gesture-based candidate selection, the user need only gesture in a direction associated by the data entry system 26 with a desired completion candidate without the pointing device necessarily moving towards or onto the portion of the graphical user interface 34 where the completion candidate is displayed. In this case, the predetermined vectors associated by the data entry system 26 with entries in the interactive search list 30 correspond to unique gestures with the pointing device but not necessarily gestures which are towards or onto a particular entry in the interactive search list 30. With this relative mode of selection, gestures with the pointing device are used to select amongst the completion candidates on the interactive search list 30 even if the user is not gesturing with the pointing device towards or onto a particular completion candidate within the interactive search list 30. In this case, the user need not move the pointing device toward or onto a particular fixed location within the interactive search list 30 in order to select a specific completion candidate. Associating a gesture with a particular completion candidate in the interactive search list 30 in the above manner minimizes the amount of movement required with the pointing device to make selections from the interactive search list 30 and provides the user with the flexibility of selecting completion candidates with gestures which do not necessarily need to be towards or onto the desired completion candidate.
Once the user has selected a completion candidate from the interactive search list 30, several options are available to the user. If the completion candidate represents the entry that the user wishes to add to the text, then the user can accept the selected completion candidate for insertion into the text by lifting the pointing device up from the touch-sensitive screen 14 in less than a predetermined time limit L2 while keeping the particular completion candidate selected. This latter event generates a user input signal which instructs the data entry system 26 to terminate all searching based on the partial text entry and to signal to the application 27 to use the selected completion candidate to permanently replace the partial text entry. When the selected completion candidate is used to permanently replace the partial text entry at this point, the interactive search list 30 is cleared from the graphical user interface 34 and the image of the digital keyboard 28 is re-enabled ready to receive the next keystroke from the pointing device. With the partial text entry thus completed, the data entry system 26 begins monitoring anew for text entries by the user.
If the user wishes to change selections, the user can move through the list of completion candidates displayed in the interactive search list 30 by gesturing with the pointing device to other completion candidates in the interactive search list 30. If the user is unsure of which completion candidate to use, or wishes to pause to consider whether to continue in search mode or to return to keyboard mode, the user can gesture to or pause in a "dead zone" within the graphical user interface 34 for any length of time without triggering any further action by the data entry system 26. In the first embodiment, when the interactive search list 30 is first activated by pausing on a key on the digital keyboard 28, the data entry system 26 begins monitoring the length of the vector that the user thereafter generates to determine when and if the user is selecting a particular completion candidate from the interactive search list 30 or when the user is carrying out another recognized operation such as moving the pointing device to a dead zone. If the data entry system 26 determines that the length of the vector generated from the current position coordinates and the point of origin is less than or equal to a predetermined length, then the pointing device is deemed by the data entry system 26 to be in a dead zone on the graphical user interface 34. When the pointing device is in a dead zone, the user has the freedom to pause without activating any further operation by the data entry system 26 and without clearing the interactive search list 30. This provides the user with the option to pause and consider what their next operation will be. Other dead zones may be programmed into the graphical user interface 34 within the data entry system 26 for the user to move to with the pointing device so as to further enhance the user's ability to pause in different parts of the graphical user interface 34. If the vector being generated through a gesture is found by the data entry system 26 to exceed a predetermined length, the data entry system 26 checks to determine whether or not the particular vector being generated is associated with any of the completion candidates in the interactive search list 30 or with any other operation on the screen such as alternative or additional dead zones.
If the completion candidate selected on the interactive search list 30 represents only part of the entry that the user wishes to add to the text, then the user can use the selected completion candidate to dynamically initiate a further search for a more refined list of completion candidates from the dictionary 20. The ability to dynamically search for a more refined list of completion candidates based on a selected completion candidate is also referred to in this specification as an iterative search. This iterative search tool is provided through the interactive search list 30 when the interactive search list 30 is active and displays a list of potential completion candidates for the user to select from. An iterative search is triggered in the first embodiment by the user continuing to keep a completion candidate in the interactive search list 30 selected for more than the predetermined time limit L2. The automated ability to use the interactive search list 30 to further search the dictionary 20 allows the user to make good use of search results that are only partially successful. When the user continues to keep a completion candidate selected in the interactive search list 30 for more than the predetermined time limit L2, the data entry system 26 determines that the user input corresponds to a user selection to initiate a new search using the selected completion candidate as the basis for the new search. When this happens, the candidate prediction system 32 dynamically obtains a refined list of completion candidates based on the selected completion candidate. As illustrated in FIG. 5A, the refined list of completion candidates provides a narrower list of completion candidates based on a more specific search string (ie. the selected completion candidate). Iterative searching enables one to perform multi-level searches with the interactive search list 30, so that the user can quickly use a completion candidate from one level of search results to drill deeper into the dictionary 30 for a narrower set of completion candidates.
In the first embodiment, iterative searching can be initiated when a completion candidate in the interactive search list 30 represents only a first part (i.e. a leading part) of the entry that the user wishes to add to the text.
Once the candidate prediction system 32 has obtained the refined list of completion candidates, the interactive search list 30 is redisplayed with the refined list of completion candidates. When the interactive search list 30 is redisplayed, the point of origin coordinates, used by the data entry system 26 to track vectors generated from gestures with the pointing device, are set to the position coordinates of the pointing device at the time the selected completion candidate is used to initiate the iterative search. With the interactive search list 30 redisplayed, the user can then choose one of the completion candidates in the refined list by lifting the pointing device up after selecting the particular completion candidate through a gesture, or the user can further repeat the iterative search process by selecting one of the completion candidates in the refined list and pausing with that particular completion candidate selected for the predetermined time limit L2. With the refined list of completion candidates displayed, the user may also lift the pointing device without selecting any completion candidates obtained from the iterative search. This latter action leaves the search string set to the last completion candidate selected by the user in the previous iteration of the interactive search list 30 and returns the data entry system 26 to keyboard mode. At this point, the interactive search list 30 is cleared from the graphical user interface 34, the digital keyboard 28 is redisplayed, and the data entry system 26 sends the completion candidate last selected by the user from the interactive search list 30 to the application 27 for entry into the text, thereby replacing the contents of the partial text entry currently under development by the user. New characters can then be added. As a variation, using the digital keyboard 28, the user can then, if desired, instruct the application 27 to cancel the entry of the modified partial text entry into the text by selecting a function button displayed on or associated with the digital keyboard 28.
Example
FIGS. 6 and 6A show an example 100 from the user's perspective of the operation and flexibility of the data entry system 26 for the first embodiment in FIGS. 1 to 5. For the example shown in FIGS. 6 and 6A, suppose the user wishes to enter in the word "endlessly" and begins by entering at block 102 the letter "e" on the digital keyboard 28. In order to activate the interactive search list 30, the user pauses on the letter "e" for at least the predetermined time limit L1 which automatically triggers at block 104 the candidate prediction system 32 to obtain a list of completion candidates that are then displayed to the user in the interactive search list 30. Depending upon the contents of the dictionary 20 and the ranking system used to rank completion candidates stored within the dictionary 20, the desired completion candidate "endlessly" may not be one of the choices displayed in the initial interactive search list 30. Suppose for the moment, however, that the word "end" is one of the completion candidates displayed in the initial interactive search list 30. The user can select by gesture at block 108 the completion candidate "end" and use it to automatically initiate a further search of the dictionary 20 in order to retrieve a list of prioritized completion candidates which all begin with the prefix "end". As discussed, this iterative searching technique is performed with the pointing device by simply pausing while selecting a completion candidate (in this case the word "end") in the interactive search list 30 for the predetermined time limit L2 at block 116 which thereby automatically initiates a new search using the selected completion candidate as the basis for such a search. If the desired completion candidate "endlessly" appears in the updated interactive search list 30, the user can then immediately add the desired completion word by selecting it (block 120) from the interactive search list 30 and lifting the pointing device up in less than the predetermined time limit L2.
If the interactive search list 30 displayed after block 104 does not include the prefix "end" or any other prefix that would lead the user to rapidly enter the desired word "endlessly" using the data entry system 26, then the user has the option of lifting the pointing device up from the touch-sensitive screen 14 at block 106 without selecting any of the completion candidates. This type of user input notifies the data entry system 26 to clear the interactive search list 30 from the screen and to re-enable and display the digital keyboard 28. It should be noted that in this latter operation the search string continues to contain the partial text entry which the user has generated with the digital keyboard 28 (in this case the search string contains only the letter "e").
As a further illustration of the flexibility of the present data entry system 26, if the partial completion candidate "end" is used to initiate a further search and the new list of completion candidates does not include the word "endlessly" in the list of completion candidates, then the user can choose to continue building upon the partial completion candidate "end" by lifting the pointing device up from the touch-sensitive screen 14 without any of the completion candidates in the new list selected and proceeding to continue entering in characters from the digital keyboard 28. In this example the partial completion candidate "end" permanently replaces the partial text entry that the user was generating with the digital keyboard 28 in the user's electronic text, the search string is cleared and any new user input is automatically treated as being part of a new partial text entry.
It will be appreciated from the above particular that the user can follow the final steps to completing a partial completion candidate by entering in the remaining letters at the end of the partial completion candidate. For instance, if the partial completion candidate "endless" was retrieved, then the user can simply tap on the digital keyboard 28 the letters "l" and "y" followed by a space (or the end-of-entry function button) in order to notify the data entry system 26 of the completion of the current text entry. Thus, even when a complete word, phrase, or sequence of characters is not found in the dictionary 20, the use of the data entry system 26 to retrieve a partial completion candidate can result in less time and effort being expended than if the user had simply typed in each letter of the desired word, phrase, or sequence of characters.
In an alternative embodiment, the data entry system 26 is programmed with the ability to re-initiate automated searching even once the application 27 is instructed by the data entry system 26 to permanently replace the partial text entry in the text with a partial completion candidate. In this alternative, the search string is set to the partial completion candidate when the partial completion candidate replaces the partial text entry in the user's electronic text. The user may then return at any time to the automated search facility of the data entry system 26 by pausing on a key on the digital keyboard 28 for the predetermined time limit L1. For instance, if the partial completion selection "end" has already been used to initiate a further search and the user then lifts up the pointing device without selecting any of the completion candidates in the new list of completion candidates, the user can then return to automated searching with the data entry system 26 by, for example, touching on the letter "l" on the digital keyboard 28 for a sufficient period of time to initiate a search on the basis of the prefix "endl". The partial text entry "endl" will then be used by the candidate prediction system 32 to obtain a list of completion candidates that are then displayed in the interactive search list 30. If the desired completion candidate "endlessly" appears in the new list of completion candidates, the user may then choose that candidate and add it to the text by selecting that candidate and lifting the pointing device before the time limit L2 is reached. If the desired completion candidate does not appear in the interactive search list 30, the user can simply lift the pointing device without selecting any of the completion candidates and continue building upon the partial text entry "endl" by entering further characters via the digital keyboard 28. Once the desired word is complete, in order to clear the search string and instruct the data entry system 26 to treat any new user input as being part of a new partial text entry in this alternative, the user selects a key or function from the digital keyboard 28 programmed to indicate that entry of the original partial text entry has ended, as discussed further in the section below.
Join the InvestorsHub Community
Register for free to join our community of investors and share your ideas. You will also get access to streaming quotes, interactive charts, trades, portfolio, live options flow and more tools.
