ABSTRACT
This paper describes an alternative virtual keyboard for ALS patients that is controlled by a screen scanning mechanism triggered by eye-blink. The patients with ALS may lose the ability to communicate with others by speech or gestures. In most cases, the only remaining capability for those patients are the eye movements. Three different keyboard layouts were tested through simulations to account for typing efficiency. A good user experience in typing is essential to communicate with others or request help of the caretaker in cases of need and emergency. The proposed solution does not use any device attached to the patient. Instead, it uses an AI algorithm to detect eye-blink captured from a webcam to interact with the keyboard.
– An alternative keyboard for communication of patients with amyotrophic lateral sclerosis
André Herman Freire Bezerra1,2– andre.herman@lais.huol.ufrn.br
Ricardo Alexsandro de Medeiros Valentim2 – ricardo.valentim@lais.huol.ufrn.br
Daniele Montenegro da Silva Barros2 – daniele.barros@lais.huol.ufrn.br
John Victor Alves Luiz2– john.victor@lais.huol.ufrn.br
Hertz Wilton de Castro Lins2– hertz.lins@lais.huol.ufrn.br
Felipe Ricardo dos Santos Fernandes2– felipe.ricardo@lais.huol.ufrn.br
Danilo Alves Pinto Nagem2– danilo.nagem@lais.huol.ufrn.br
- Presenter.Ph.D. in Biosystems Engineering, currently Bachelor in Information Technology at UFRN and Resident Researcher at LAIS.
- Laboratory of Technological Innovation in Health (LAIS), UFRN, Natal, Brazil
An alternative keyboard for communication of patients with amyotrophic lateral sclerosis
Augmentative and alternative communication has been largely studied in the past decades [1, 2, 3] to help people with motor disabilities to communicate. Several studies test the efficiency of alternative tools in communication for those who are severely speech impaired and show some important factors to be taken into account. Overall, the main indicator of communication efficiency is the time taken by the user to write a sentence. This measured time is commonly reported as an average rate of words per minute or characters per minute. The lesser time to write, the closer to conventional speech. It improves patient autonomy and, therefore, its quality of life.
Our main target audience is amyotrophic lateral sclerosis (ALS) patients. ALS is a degenerative disease that causes the vast majority of the patient’s muscles to stop functioning due to degeneration of the motor neuron. In most cases, the cognitive aspect is preserved and muscles as of the eye region are still functional and can move and blink. These remaining moving muscles are usually explored as a mechanism for triggering some alternative communication apparatus.
Some solutions of assistive technology are classified as bio signal based technologies, in which they obtain signals from the patient, commonly through electroencephalogram (EEG), electrooculogram (EOG) or electromyogram (EMG), using assistive devices. Dhillon et al. [4] proposed a virtual keyboard solution that can be controlled by EOG and EMG signals. They proposed a method to obtain the eye gaze direction and eye gaze angle based on the EOG signals. EMG waves are obtained for eye blink actions. Alternatively, [5] proposed a keyboard that is controlled by EOG signalsonly.
About the solution
The proposed solution is an alternative virtual keyboard as an non-bio signal based assistive technology, meaning that there is no additional device attached to the patient to trigger events on the computer. All is needed is a laptop with a builtin webcam. It is a free solution with continuous support and constant update.
In the presented solution, the eye-blink is used as a way to select items sequentially illuminated on the screen, triggering events in the software. The keys are sequentially illuminated on the screen following either a linear or a line-column scan pattern and a timer. The timer is responsible to linger the scanner over the illuminated key for a certain period of time (default to 2 seconds) to give opportunity to the user to select it. An AI face recognition algorithm is responsible to detect user blinking. In short, the project is a free solution for those who cannot afford high priced communication equipment (both software and hardware). It is designed to operate in any laptop running Windows or Linux with a webcam.
The alternative keyboard was designed in order to answer the question: does this set of features makes the keyboard comfortable to ALS patients daily use?
Presenting the keyboard
According to [6], several factors are important to reduce writing time: scanning method, key order, grouping of keys. In addition, [7] shows that features such aseliminating keys momentarily impossible to use (key negation) and word prediction decrease writing time. The effect of those features in typing becomes more evident when using scanning key selection method instead of eye-tracking selection method as user waiting time to select the next desired letter diminishes. As a result of our literature review, the following keyboard features were observed to have the most significant effects on typing efficiency: keys ordered by usage frequency, key negation, list of predicted and most used words, methods of scanning (sequential and line-column), and text-to-speech (TTS), the latter, as a quality of life improvement.
Eyes detection is made by applying a convolutional neural network and a support vector machine to provide eye-coordinates, resulting in an open/closed classification. The flow of the detection process is presented in Figure 1. This binary classification serves to simulate a click action (closed eye) to trigger the event of selecting the current hovered key (Figure 1) on the screen.
The keyboard keys are ordered by most frequent characters used in the language (in our case, Portuguese) (Figure 2). A counting algorithm calculates each character frequency by scanning six public domain Portuguese books, in plain text format. Any accented vowel is transformed into their root to increase accuracy. Key negation uses an algorithm that checks for every following character to each letter of the alphabet, making possible to list all characters that do not come after a certain letter. Plain texts are also used to predict words. They are ordered by frequency of appearance in the text and, as the user types, words are suggested. Using this prediction method brings an advantage of training the algorithm to different user knowledge levels. The system can be trained with either simple or elaborate texts, depending on the user’s erudition. The TTS feature uses the open source engine eSpeak. All features are presented in Figure 3.
Testing the keyboard
A software that simulates the usage of the keyboard by a user was developed in order to run a synthetic test and check the keyboard typing efficiency. The simulations consisted in typing 97 sentences in the Portuguese language, taken from books of several subjects. The result is a typing speed average, measured in words per minute. In the simulations, the default value of 2 seconds for the timer was used. It means the software waits 2 seconds before selectingany key. In a real case scenario, the user selects a key before 2 seconds is elapsed.
We tested the current keyboard layout against the traditional qwerty layout and a new layout where it follows all characteristics of the current layout, but has a more square shape with 9 keys width and 6 keys in height. The layouts are shown in Figure 4.
The simulations results are shown on Table 1. The conventional qwerty layout performed worst of all 3 layouts, with an average of 0.9 words per minute, followed by the currently used layout, with 1.0 words per minute and the new layout with 1.1 words per minute. Unlike the qwerty layout, both the current and the new proposed layout benefit from the arrangement of the most used letters. Similarly, line-column scanning benefits from a more square layout, due to the shorter waiting time for letters that are further to the end of each keyboard line. As a comparison, [8] found an average time of 2.8 words per minute withthe qwerty layout with users using an eye-gaze device when typing.
Table 1. Performance of the different layouts
| Layout | Words per minute | Characters per minute |
| qwerty | 0.9 | 4.2 |
| current | 1.0 | 4.8 |
| new | 1.1 | 5.5 |
It is important to note that the mechanism of letter negation was not simulated, whichwould increase the count of word per minute by a significant amount. Also, with constant use, the user would get better with the keyboard usage, resulting in an increasingword per minute average. Additionally, by consideringthese factors, it becomes possible for avirtual keyboard that does not use eye-gaze technology to achieve similar results.
Of course, these are only synthetic results done by simulation. Other variables such asuser comfort with a new layout (different from the default qwerty) and user fatigue due to long time usage are not taken into account. Therefore, tests with real users are essential in order to either validate or refuse the results.
Final Remarks
Although the keyboard has been developed to be as comfortable to use as possible, some difficulties need to be overcome. The most obvious is the user’s initial difficulty in accepting a new keyboard layout. In addition, keyboards using scanning methods should always look for alternatives to reduce the waiting time. The new proposed layout is an example of it. Finally, for a better user experience, the system should be capable of detectingother languages within the text, in both prediction and speech features.
References
[1] Foulds, R. A., Soede, M., & van Balkom, H. (1987). Statistical disambiguation of multi-character keys applied to reduce motor requirements for augmentative and alternative communication. Augmentative and Alternative Communication, 3, 192–195.
[2] Levine, S. H., & Goodenough-Trepagnier, C. (1990). Customized text entry devices for motor-impaired users. Applied Ergonomics, 21, 55–62.
[3] Brumberg, J. S., Pitt, K. M., Mantie-Kozlowski, A., & Burnison, J. D. (2018). Brain–computer interfaces for augmentative and alternative communication: A tutorial. American journal of speech-language pathology, 27(1), 1-12.
[4] Dhillon, H. S., Singla, R., Rekhi, N. S., & Jha, R. (2009). EOG and EMG based virtual keyboard: A brain-computer interface. In 2009 2nd IEEE International Conference on Computer Science and Information Technology. 259-262. IEEE.
[5] Venkatagiri, H. (1999). Efficient keyboard layouts for sequential access in augmentative and alternative communication. Augmentative and Alternative Communication, 15(2), 126-134.
[6] Lingegowda, D. R., Amrutesh, K., & Ramanujam, S. (2017). Electrooculography based assistive technology for ALS patients. In 2017 IEEE International Conference on Consumer Electronics-Asia, 36-40. IEEE.
[7] Benligiray, B., Topal, C., & Akinlar, C. (2019). SliceType: fast gaze typing with a merging keyboard. Journal on Multimodal User Interfaces, 13(4), 321-334.
[8] Waddington, C. T., MacKenzie, I. S., Read, J. C., & Horton, M. (2017). Comparing a scanning ambiguous keyboard to the on-screen QWERTY keyboard. Electronic Visualisation and the Arts, 1-6.
Biography
Andre Herman Freire Bezerra, Ph.D. in Biosystems Engineering, currently Bachelor in Information Technology at UFRN and Resident Researcher at LAIS, andre.herman@lais.huol.ufrn.br
Andre Herman holds a degree in Agronomy from the Universidade Federal Rural do Semi-Árido (UFERSA) and a PhD in Agricultural Systems Engineering at University of São Paulo (USP), with a 1-year period working as visiting researcher at Wageningen University and Research Centre (WUR). He has experience in mathematical modeling of physical processes related to fluid dynamics, numerical methods for optimization problems and analysis of high frequency data. He is currently coursing Bachelor on Information Technology and is a research resident at LAIS, where he is part of the Autonomus development group.
– An alternative keyboard for communication of patients with amyotrophic lateral sclerosis
André Herman Freire Bezerra1,2– andre.herman@lais.huol.ufrn.br
Ricardo Alexsandro de Medeiros Valentim2 – ricardo.valentim@lais.huol.ufrn.br
Daniele Montenegro da Silva Barros2 – daniele.barros@lais.huol.ufrn.br
John Victor Alves Luiz2– john.victor@lais.huol.ufrn.br
Hertz Wilton de Castro Lins2– hertz.lins@lais.huol.ufrn.br
Felipe Ricardo dos Santos Fernandes2– felipe.ricardo@lais.huol.ufrn.br
Danilo Alves Pinto Nagem2– danilo.nagem@lais.huol.ufrn.br
- Presenter.Ph.D. in Biosystems Engineering, currently Bachelor in Information Technology at UFRN and Resident Researcher at LAIS.
- Laboratory of Technological Innovation in Health (LAIS), UFRN, Natal, Brazil
