ABSTRACT
Amyotrophic Lateral Sclerosis (ALS) is a degenerative disease, of unknown origin, which affects the nervous system in a limiting and irreversible way. With degeneration of the motor neuron, activation of patients’ muscle fibers are highly affected, causing movement speech losses, but not affecting, in most cases, cognitive aspects. Since gesticulation and communicative pathways are severely impaired due to these conditions, the ones who have social limitations, are affected by losing interaction with other people. It is not hard to identify communication as one of the greatest human skill, either through words or facial and body expressions.Thus, the use of assistive technologies focused in expanding communication skills and to develop alternative communication protocols for patients with ALS becomes essential. Traditionally, the main models of alternative communication are made of cards and communication boards in which actions are represented by contexts. However, the communicative complexity inherent to humans is not achieved with these tools. Therefore, the application of a human-machine interaction may help to mitigate some of these impediments.
Maria Luiza Pires de Souza1,2– maria.luiza@lais.huol.ufrn.br
Ricardo Alexsandro de Medeiros Valentim2 – ricardo.valentim@lais.huol.ufrn.br
Eulália Costa Ribeiro2– eulalia.ribeiro@lais.huol.ufrn.br
John Victor Alves Luiz2– john.victor@lais.huol.ufrn.br
Daniele Montenegro da Silva Barros2 – daniele.barros@lais.huol.ufrn.br
Gabriel Vinicius Sousa da Silva – gabriel.vinicius@lais.huol.ufrn.br
Hertz Wilton de Castro Lins2– hertz.lins@lais.huol.ufrn.br
Felipe Ricardo dos Santos Fernandes2– felipe.ricardo@lais.huol.ufrn.br
Danilo Alves Pinto Nagem2– danilo.nagem@lais.huol.ufrn.br
- Presenter: The author are Biomedical Engineering undergraduate students at UFRN and Researchers at LAIS.
- Laboratory of Technological Innovation in Health (LAIS), UFRN, Natal, Brazil.
Artificial Intelligence applied in a Human-Machine Interface for an alternative communication system to ALS patients
INTRODUCTION
Amyotrophic Lateral Sclerosis (ALS) is a degenerative disease of unknown origin, which affects the nervous system in a limiting and irreversible way (RICHARDS, 2020). With degeneration of the motor neuron, activation of patients’ muscle fibers is positively affected, causing movement speech losses, but not involving, in most cases, cognitive aspects. Since gesticulation and communicative pathways are severely impaired due to these conditions, those with social limitations are affected by losing interaction with other people. It is not hard to identify communication as one of the most excellent human skills, either through words or facial expressions.
Thus, the use of assistive technologies focused on expanding communication skills and developing alternative communication protocols for ALS patients becomes essential. Traditionally, the main models of alternative communication are made of cards and communication boards in which contexts represent actions. However, the communicative complexity inherent to humans is not achieved with these tools. Therefore, the application of a human-machine interaction may help to mitigate some of these impediments. Based on the presented problem, the proposed solution’s goal is to give, to some extent, autonomy back to ALS patients. Artificial Intelligence (AI) techniques are integrated into these protocols to offer the patient.’s best possible user experience (KROLAK and STRUMILLO, 2011).
Within the health area literature, several studies have shown different contexts to develop tools to aid patients’ communication. El-soussi et al. (2015) developed a tool to enable communication of intubated patients Fathi and Abadali-Mohammadi (2015) created a real-time eye blink detection method for communication. Following the same line, Krolak and Strumillo (2011) developed a human-machine interface using the eye-blink as an interface interaction method. Thus, the present article is relevant and has a promising research area.
MATERIALS AND METHODS
Figure 2 represents an overview of the proposed system. These are three subsystems interconnected through services developed with web technologies such as react, javascript and python. However, the electron was used to transform a desktop application. These technologies enable the system to be multiplatform. The user needs a personal computer with windows, mac, or linux to use this system.
The intelligence architecture has a general flow that uses several models of AI. The main objective is to make the patient communicate with the eye-blink. This module was focused on patients that cannot communicate and needs intense homecare – they do not speak and need an assistive tool to communicate. For the development of the intelligence module, the Single Shot MultiBox Detector (SSD) was used to perform face detection. Then, after an area of interest (ROI), it was cut out using computer vision techniques. The convolutional neural networks were used for the blink classification process. A plugin was developed to integrate the intelligence architecture with the alternative communication interface.
For the construction of the alternative communication interface, it was necessary to use Design Thinking techniques to solve problems, developing products and projects based on the designers’ thinking. The activities include research, brainstorms, selection of ideas and prototyping. The interface content elements play a crucial role in the development. Elements associated with text, colors, figures and shapes were used to give information to the user. Selected system sounds were used to make the system accessible. These are the main content elements of the system: text content associated with a figure (icon and/or picture); visual contrast (color/text-background) without using complementary colors (same amount of light); text clarity; simplified language; icons; use of universal icons for the default elements.
An essential part of this architecture is the interactive keyboard subsystem. The embedded keyboard has features such as key negation, word prediction and voice synthesization, essential to a better understanding and clarity of the system content.
RESULTS
An intelligent, modular and customizable structured system was developed as a software that uses tree architecture for navigation in its interface. The strategy is to use intelligent algorithms to recommend actions that take into account the context in which the patient is inserted, aiming to provide a medium that does not require a high cognitive demand in communication. These AI algorithms suggest items based on data filtering according to the user’s interest, i.e., what the user accesses the most or it is considered to be of greater importance. Each component of this module has its respective models and has been implemented with different approaches. The features are data acquisition, face detection, landmarks application, extraction of ROI, and classification.
The developed alternative communication interface is controlled by a non-invasive method using a webcam to detect eye-blink signals. The use of precise language, iconography, colors and user-based contexts is a fundamental element that marks the user interface in this system. A convolutional neural network is applied to catch the eyes, and a support vector machine is used to provide eye-coordinates, resulting in an open/closed classification. When the patient blinks, a click action is executed, and the screen item is selected.Therefore, this software facilitates ALS patients’ navigation, communication, and interaction, making communication more dynamic through the software on a device. The use of smart tools combined with vital signals supervision equipment adds quality to the caregiver’s work, making them more included in the needs and conditions of the patients.
CONCLUSION FUTURE WORK
ALS compromises people’s quality of life. Implementing assistive technologies and encouraging patient autonomy through intelligent systems proved to be a viable tool for improving communication. In this way, the communication process becomes more dynamic and facilitated, as it considers the context in which the individual is immersed. This work proposed developing a system that uses artificial intelligence techniques combined with alternative communication protocols to provide communication for patients with ALS.
As future work, the intelligence architecture will be tested new ways of detecting the blink, and a recommendation system will be developed for the interactive keyboard.
REFERENCES
- RICHARDS, D; MORREN, J A.; PIORO, E P. Time to diagnosis and factors affecting diagnostic delay in amyotrophic lateral sclerosis. Journal of the Neurological Sciences, p. 117054, 2020.
- MIAO, Y.; YIN, E.; ALLISON, B. Z. et al.An ERP-based BCI with peripheral stimuli: validation with ALS patients. Corn Neurodyn 14, 21–33 (2020). DOI: https://doi.org/10.1007/s11571-019-09541-0.
- BRUNIALTI, L.; PERES, S.; FREIRE, V.; LIMA, C. Aprendizado de Máquina em Sistemas de Recomendação Baseados em Conteúdo Textual: Uma Revisão Sistemática. In: SIMPÓSIO BRASILEIRO DE SISTEMAS DE INFORMAÇÃO (SBSI), 11., 2015, Goiânia. Anais do XI Simpósio Brasileiro de Sistemas de Informação. Porto Alegre: Sociedade Brasileira de Computação, may 2015 . p. 203-210.
- FATHI, A.; ABDALI-MOHAMMADI, F. Camera-based eye blinks pattern detection for intelligent mouse. SIViP9, 1907–1916 (2015). DOI: https://doi.org/10.1007/s11760-014-0680-1.
- KROLAK, A.; STRUMILLO, P. Eye-blink detection system for human–computer interaction. Institute of Electronics, Technical University of Lodz, Wolczanska, Poland. Springer, 2 October 2011.
- MAINSAH, B. O.; COLLINS, L. M.; COLWELL, K. A. et al. Increasing BCI communication rates with dynamic stopping towards more practical use: an ALS study. J Neural Eng. 2015;12(1):016013. DOI: https://doi.org/10.1088/1741-2560/12/1/016013.
- EL-SOUSSI, A. H. et al. Augmented alternative communication methods in intubated COPD patients: Does it make difference. Egyptian Journal of Chest Diseases and Tuberculosis, v. 64, n. 1, p. 21-28, 2015.
Biography
Maria Luiza Pires de Souza holds a degree in Science and Technology, is currently doing Bachelor in Biomedical Engineering at UFRN and working as a Researcher at LAIS. maria.luiza@lais.huol.ufrn.br
She has experience in educational programs, such as tutorials, and front-end development. At LAIS, she made part of the Autonomus development group. Currently she belongs to ALS National Register building group. She has interest in management and hospital technologies.
Maria Luiza Pires de Souza1,2– maria.luiza@lais.huol.ufrn.br
Ricardo Alexsandro de Medeiros Valentim2 – ricardo.valentim@lais.huol.ufrn.br
Eulália Costa Ribeiro2– eulalia.ribeiro@lais.huol.ufrn.br
John Victor Alves Luiz2– john.victor@lais.huol.ufrn.br
Daniele Montenegro da Silva Barros2 – daniele.barros@lais.huol.ufrn.br
Gabriel Vinicius Sousa da Silva – gabriel.vinicius@lais.huol.ufrn.br
Hertz Wilton de Castro Lins2– hertz.lins@lais.huol.ufrn.br
Felipe Ricardo dos Santos Fernandes2– felipe.ricardo@lais.huol.ufrn.br
Danilo Alves Pinto Nagem2– danilo.nagem@lais.huol.ufrn.br
- Presenter: The author are Biomedical Engineering undergraduate students at UFRN and Researchers at LAIS.
- Laboratory of Technological Innovation in Health (LAIS), UFRN, Natal, Brazil.
