Human Attention Assessment Using A Machine Learning Approach with GAN-based Data Augmentation Technique Trained Using a Custom Dataset

Sveva Pepe ¹, Simone Tedeschi ¹, Nicolo' Brandizzi ¹, Samuele Russo ^2,*, Luca Iocchi ¹, Christian Napoli ^1,*

1. Department of Computer, Automation and Management Engineering, Sapienza University of Rome, Via Ariosto 25, Roma 00185, RM, Italy

2. Department of Psychology, Sapienza University of Rome, Via dei Marsi 74, Roma 00185, RM, Italy

* Correspondences: Samuele Russo and Christian Napoli

Academic Editor: Raul Valverde

Special Issue: Neuroscience and Information Technology

Received: May 27, 2022 | Accepted: September 20, 2022 | Published: October 04, 2022

OBM Neurobiology 2022, Volume 6, Issue 4, doi:10.21926/obm.neurobiol.2204139

Recommended citation: Pepe S, Tedeschi S, Brandizzi N, Russo S, Iocchi L, Napoli C. Human Attention Assessment Using A Machine Learning Approach with GAN-based Data Augmentation Technique Trained Using a Custom Dataset. OBM Neurobiology 2022;6(4):17; doi:10.21926/obm.neurobiol.2204139.

© 2022 by the authors. This is an open access article distributed under the conditions of the Creative Commons by Attribution License, which permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is correctly cited.

1. Introduction

Human–robot interaction (HRI) [1], or more generally, human–computer interaction (HCI) [2], has received increased research interest in the last two decades thanks to the recent technological advances in the hardware and systems fields [3].

Extensive research has been performed to improve the quality of HRI by exploiting modern machine learning (ML) techniques [4,5] to allow voice and gesture commands. Such interactions are largely based on visual perception, and the system must be able to distinguish whether the user is attentive. However, few approaches relying upon eye gaze have been proposed to date [6,7].

Detecting attention is an essential stage in various applications. For example, when dealing with hyperactive children, knowing if the child is following the recommended action (e.g., an educational session) is necessary to decide whether the same activity must be presented again. Furthermore, in conventional HRI, if the user does not pay attention when the robot asks a question (e.g., “What do you want to order?” in a restaurant scenario), the robot can increase the volume and ask the question again to get the user’s attention. In a driver monitoring system, the robot can check if the driver is looking ahead and decide to notify him and refocus his attention.

Visual tracking plays an important role in the assessment of human attention. This task requires merging many computer vision applications, which include image and video processing, pattern recognition, information retrieval, automation, and control. While the correlation between eye movement and attention level is generally acknowledged, there are no standards to define a direct mapping between the two. However, eye tracking is an extensive field and is employed in many applications, such as mobile robotics, solar forecasting, particle tracking in microscopy images, biological applications, and surveillance [8,9,10,11]. Although there are numerous use cases of gaze tracking and attention detection, few datasets are freely available online. This shortage is due to the lack of a clear and established baseline for labeling image data. Moreover, constructing a dataset is time-consuming and expensive.

Massive amounts of data are required to train deep learning (DL) systems capable of solving such tasks in a supervised manner.

In this study, we focused on the attention detection task in its entirety, from addressing the issue of data scarcity, building a baseline system, and proposing unique ways to boost performance further to developing a web application. For the attention detection task, we constructed a new manually annotated dataset containing approximately 120,000 images from 18 users. Moreover, we developed a strong baseline system to achieve competitive detection results on the developed dataset. Furthermore, we enhanced the baseline system by adding an auxiliary face detection module, enhancing the performance. In addition, we proposed a novel GAN-based data augmentation technique to further enrich the collected data and achieve performance improvements. We extensively evaluated the benefits of using the constructed test set by performing a statistical analysis of the obtained results. Finally, we designed a web application to further analyze the system’s behavior and perform real-time testing. With this work, we aim to encourage further studies on the development of high-performance attention detection systems.

The paper is structured as follows. In Section 2, a brief overview of the related works for the topics covered in the introduction is presented, particularly attention assessment through different human signals and their use in real-world scenarios. In Section 3, the methodology employed for constructing the dataset and some statistical analysis are described. In Section 4, the proposed architecture for assessing human attention is presented, and the baseline system and the augmentation implementation by using GAN are discussed. The experimental setup and results are presented in Section V. The real-time application developed for evaluation is described in Section 6. Finally, the conclusion and ideas for future research are presented in Section 7.

2. Related Works

Extensive research has been conducted on human attention detection from many perspectives, and thanks to recent advances in sensor technologies and computer vision techniques, the research focus has shifted from manual to automated techniques [12,13,14].

Previously, studies mainly employed experiments on physiological factors such as fatigue [15], brain signal data [16], blood flow and heart rate [17], and galvanic skin conductance [18]. Recently, efforts have been made to accomplish driver attention prediction as a computer vision technique. In a previous study [19], semantics-induced scene variations were analyzed using a novel multi-branch deep architecture that integrates three sources of information: raw video, motion, and scene semantics. In another study [20], a semantic context-induced attentive fusion network was designed. Eye-tracking [21,22], gaze-tracking [23], and face-tracking algorithms [24] have also been proposed.

These methods are promising as they are inexpensive and unobtrusive [25,26,27,28]. In addition, the automatic recognition of facial expressions has resulted in applications that span several disciplines. For example, facial expressions are used to identify pain [29,30], diagnose syndromes in infants [31,32], detect driver’s drowsiness [33,34,35], recognize emotions [36,37,38,39,40], and detect engagement [41,42].

These systems have been employed for important tasks such as the detection of driver attention and autism-related diseases. There has been an increased interest in gaze detection information to determine a driver’s focus of attention. In a previous study [43], a model based on Sanger’s neural network [44] was proposed to monitor real-time driver attention through binary classifiers and iconic data reduction. In another study [45], the driver’s attention behavior and the road scene were parsed to predict potentially risky maneuvers. A DL-based gaze detection approach that can detect the driver’s head and eye movement by using a near-infrared (NIR) camera sensor has been proposed [46]. The driver’s gaze in a pre-attention environment has been investigated using intention prediction based solely on fixation maps [47]. However, these systems suffer from the problem of grabbing the attention of a driver and rely on a restricted number of predefined safety-inspired rules. In a previous study [48], the driver’s attention toward pedestrians and motorbikes at junctions was inspected, and object saliency was employed to avoid the looked-but failed-to-see effect. On the contrary, in the absence of eye-tracking systems and reliable gaze data, several studies [49,50,51,52,53] have focused on the driver’s head and the detection of facial expressions to predict head orientation. Such latter techniques are more robust to varying lighting conditions and occlusions; however, there is no certainty about the adherence of predictions to the true gaze during the driving task.

Most studies have focused on a specific aspect, such as the driver or the environment. To solve this problem, in a study [54], a novel driver attention estimation model that considers the environment’s saliency map and the driver’s gaze was proposed; both the gaze and the scenario image were used to estimate the driver’s attention area and establish a multiresolution network to model.

For children with autism spectrum disorders (ASDs), attention recognition plays a vital role in providing learning support. The unobtrusiveness of face-tracking techniques enables establishing automatic systems to detect and classify attentional behaviors. In a previous study [55], an attention detection model was established based on the kid’s behavior during engagement with the robot in HRI systems; the model was used in an adaptive interaction system where the robot detects the action depending on the child’s attention. However, constructing such systems is challenging because of the complexity of attentional behavior in ASDs. To overcome this problem, in a study [56], a face-based attention recognition model was presented based on geometric feature transformation and a support vector machine (SVM) classifier [57].

These semi-autonomous adaptive systems are complex and require high-performance hardware, such as GPU chips [58], to process real-time data and update interactions. Moreover, fully autonomous and complex robots and systems are not yet reliable outside controlled research setups. In a previous study [59], a simple autonomous assessment system based on attention cues was developed and deployed and then combined with an enhanced adaptive semi-autonomous interaction system. This technique can aid in ASD intervention to facilitate adaptive interactions with patients while involving minimal subjective biases.

In the present study, inspired by recent advancements, we developed a simple yet effective high-performance architecture to perform attention detection; the proposed system can be easily integrated into the aforementioned applications.

3. Data Collection Process

Thanks to the recent advances in neural networks [60], various tasks can be performed by training network models on large amounts of data. However, such data are scarcely available. Therefore, we created a new large dataset for the attention detection task.

The direction of the head is usually one of the determining aspects of whether the interlocutor is heedful, whether it’s between two humans or between a human and a machine [60,61,62,63,64].

Thus, we determined attention based on the direction of the interlocutor’s face.

In the proposed system, when an interlocutor faces their interaction partner, full attention is assumed. In contrast, if the interlocutor is looking in another direction (e.g., left, right, up, or down), the interlocutor is considered distracted. We used five classes—CENTER, LEFT, RIGHT, UP, and DOWN—rather than a binary label to better cluster different situations, letting the model distinguish between various kinds of inattention. The CENTER class is the only positive label, indicating that the interlocutor is heedful. We recorded 270 videos from 18 users, each video lasting approximately 20 s. Each user was asked to record 15 videos (each user recorded three series of five videos, where each of the five videos corresponded to one specific label), changing their location and/or their outfit—including glasses—every five videos to let the dataset be as general as possible. The videos were segmented manually and double-checked to ensure the validity of the annotations.

The created dataset was divided into the training, testing, and validation sets. The people involved in the training set were not included in the testing and validation sets, and vice versa. The average age of the people in the dataset is between 20 and 30 years. To test the proposed system’s ability to generalize across different ages, we added a person who was approximately 60 years old. Details regarding the dataset are presented in Table 1.

Table 1 Dataset statistics describing the number of samples for each class in the training, testing, and validation sets. The last line shows the total number of produced samples independently from the class and the split.

4. Methodology

In this section, we first describe the proposed baseline system in Section 4.1. Next, we propose two extensions to improve such systems: face detection (Section 4.2) and GAN-based data augmentation (Section 4.3).

4.1 Baseline System

Training ML models is a challenging task. The training algorithms may not work as intended, the training times are too long, and the training data may be problematic. Transfer learning (TL) [65] is one of the most effective ML techniques to facilitate training. TL involves storing the knowledge gained in solving one problem and applying it to a different but related problem. TL can be applied to many ML models, including DL models such as artificial neural networks and reinforcement learning models. In addition, TL offers several advantages in the development of ML models, such as resource-saving and improved efficiency in training new models. Furthermore, TL can be used to train models when only unlabeled datasets are available, as most of the model is already trained.

TL is being used in several areas of ML, such as strengthening natural language processing [66], machine vision [67], and other real-world applications [68].

We used the Visual Geometry Group 16 (VGG16) [69] pretrained model for the attention task. The proposed model’s architecture is illustrated in Figure 1 and consists of 13 convolutional layers (separated by five pooling layers) and three dense layers. We augmented the architecture with two additional dense layers and an output layer with the softmax activation function.

Figure 1 Architecture of the proposed model. Conv denotes convolutional layers, fc denotes fully connected layers, and gap is the global average pool.

The proposed system is designed to generalize as much as possible to achieve robustness to noise and unseen samples. Generally, the training data is augmented for this purpose [70]; however, not all data augmentation techniques are applicable in our case. For instance, horizontal flipping cannot be used because left-labeled images would become right-labeled and vice-versa, creating confusion in the training process. However, brightness and shifting can be used instead; brightness is beneficial if the test set contains samples whose brightness levels differ from those in the training data, whereas shifting is useful when the position of the interlocutor varies greatly between training and testing. In our case, shifting does not have any effect on the model’s performance, as we already have this feature in our training set. Further details are provided in Section 5.

4.2 Face Detection

Another fundamental task of computer vision is face detection [71], wherein the human face is identified by relying on the key points that characterize it (e.g., eyes, mouth, and nose). Face detection can be used as an additional step for a variety of tasks, such as understanding facial expressions [72], lip reading [73], user identification [74], and marketing applications [75].

In the proposed system, face detection is used to focus on the features that are relevant to our task. For instance, our attention detection system finds it challenging to classify low-contrast images where the background color is similar to the face color or in cases of overexposed images. To solve this problem, we employed an auxiliary face detection system (a pretrained OpenCV Caffe model) that outputs the four vertices of the bounding box surrounding the face and used them to crop the original image.

4.3 GAN-based Data Augmentation

As already mentioned in Section 4.1, data augmentation techniques are beneficial for several computer vision tasks and enable the development of highly robust systems. However, such techniques involve elementary mathematical operations such as shifting, rotating, and flipping images, which may not be sufficient in some cases.

The GAN framework estimates generative models by using an adversarial process in which two models are trained simultaneously: a generative model $G$ captures the data distribution $p_g$ over the data $x$, and a discriminative model $D$ estimates the probability that a sample came from the training data rather than $G$. The generator constructs a mapping from a prior noise distribution $p_z$ to a data space as $G(z; \theta _G)$, where $\theta _G$ denotes generator parameters. The training procedure for $G$ is to maximize the probability of $D$ making a mistake.

The original GAN framework poses this problem as a min-max game in which two players ($G$ and $D$) compete against each other, playing the following zero-sum min-max game:

\[\begin{smallmatrix}min max \\G \ \ D \end{smallmatrix} V(D, G)=E_{x \sim p_{ _ { \ d \ a \ t \ a \ }} \ (x)}[log D(x)]+E_{z \sim p_{z}(x)}\{log [1-D(G(z))]\}\tag{1} \]

Training GANs consists in finding a a point of equilibrium between the two competing concerns. Unfortunately, finding an equilibria is a very difficult problem. Both the generator model and the discriminator model are trained simultaneously in a game. This means that improvements to one model come at the expense of the other model. The proposed GAN architecture is illustrated in Figure 2.

Figure 2 GAN architecture.

We developed a novel GAN-based data augmentation technique to further improve the system’s performance. We used this technique to generate new samples, starting from the images of the users in the training set and making them a different age. For this purpose, we adopted SAM [76], a C-GAN for face aging, which takes as input an image of a person $P$ and a desired age $a_d$ and outputs the transformed image of $P$ with age $a_d$.

This strategy is fundamental in any computer vision application that deals with users of all ages, but the corresponding training set does not cover all the required ages. Only users between 20 and 30 years were included in our case; however, the proposed system is intended for use by both young and old people. To overcome this problem, in addition to the data augmentation techniques already described in Section 4.1, we applied this face-aging solution to make our system insensitive to age changes, as illustrated in Figure 3.

Figure 3 Example of face aging with target age $a_d$ = 60.

5. Experiments

In this section, first, the experimental setup is described (Section 5.1). Next, the results of the baseline system are presented (Section 5.2.1). Finally, the results of the baseline system + face detection system (Section 5.2.2) and the proposed baseline system + face detection system + GAN system are presented (Section 5.2.3).

5.1 Experimental Setup

We implemented the proposed baseline system and its extensions by using TensorFlow and the Keras framework to load and fine-tune the weights of the pretrained models. We trained every model by using the Adam optimizer [77] for a maximum of 30 epochs with a learning rate of 10^–5 and a cross-entropy loss criterion. The best models were determined according to their accuracy on the validation set at the end of each training epoch. The hyperparameter values are listed in Table 2. Furthermore, for obtaining the results shown in Sections 5.2.2 and 5.2.3, we repeated each training on five different seeds, fixed across experiments, and reported the mean and standard deviation of their accuracy scores. We compared the experimental results by using the Student’s t-test. All the experiments were performed on NVIDIA Tesla K80 with 12 GB of RAM.

Table 2 Hyperparameter values of the models used for the experiments.

5.2 Results

5.2.1 Baseline System

We aimed to develop a robust baseline system to perform attention detection. For this purpose, we performed model selection on the validation set by comparing different architectures. The results are presented in Table 3.

Table 3 Accuracy of the different models on the test set.

As can be seen from Table 3, the ResNet152V2, ResNet50 [78], and Xception [79] models exhibited low accuracy due to their high architectural complexity; they have 152, 50, and 36 layers, respectively, thus indicating that extremely deep architectures are not suitable. On the contrary, simpler architectures such as VGG16 and VGG19 [69] performed better. We selected VGG16 as the temporary baseline, on which we applied data augmentation techniques because it attains comparable performances with respect to the 19-layer version while being less complex.

To further improve the proposed system, we applied the data augmentation techniques discussed in Section 4.1. The results are presented in Table 4.

Table 4 Performances of the different data augmentation techniques when applied to the VGG16 model.

As can be seen from the results presented in Table 4, applying the shifting operation did not improve the system performance as some shifted data are already partially included in the dataset, as already mentioned in Section 4.1. In contrast, the brightness operation improved the model performance because the test set contained samples with highly variable levels of brightness. In the subsequent discussion, we refer to this final system, VGG16, with the brightness operation, as the baseline system.

5.2.2 Face Detection

As mentioned in Section 4.2, we started from the intuition that the background of the image, or the user’s outfit, does not provide any helpful information but can only introduce noise in the classification process. Therefore, we performed face detection as an auxiliary task to let the system focus only on the relevant features, yielding three advantages: it greatly improved the model performance, dramatically reduced the training times, and considerably decreased the amount of required disk space.

In terms of system performance, as can be seen from the second line of Table 5, the baseline system + face detection model yielded an average improvement of 12.85 accuracy points over the previously described baseline system. Moreover, an extreme statistical significance was observed in the results (p-value = 3.7e-07), thus, demonstrating how this technique is fundamental for our task. As can be seen in Figure 4, the validation accuracy of the two systems varied during epochs, and a consistent gap was observed between the two curves (pink and gray curves).

Table 5 Comparison of the performance of the improved models based on face detection and GAN-based data augmentation with the baseline system. ** denotes p - value < 0.001 , and * denotes p - value < 0.05. Both statistical significance and differences (Δs) are always expressed concerning the immediately above row, therefore the Δ of the third row refers to the difference of the results obtained between the model of the third and second row.

Figure 4 Validation performances of the baseline system, baseline system + face detection, and baseline system + face detection + GAN.

Considering training times instead, the baseline system required approximately 35 min for each epoch, whereas preprocessing the images by applying face detection and training the system directly on the cropped images increased the speed by up to 6×.

The final advantage is the reduced amount of required storage space. The standard dataset occupies approximately 25 GB of memory, whereas the processed dataset requires only approximately 4 GB of memory.

5.2.3 GAN-based Data Augmentation

As discussed in Section 3, our training set contains only images of users belonging to the age range of 20–30 years. Nevertheless, approximately 30% of our test set was constructed by including a 60-year-old user. While standard data augmentation techniques fail to let the system generalize well with users of different ages, the proposed GAN-based data augmentation technique (Section 4.3) helps overcome this problem. From Table 5, it can be seen that enhancing the baseline system by using GANs provides further improvements over the baseline for face detection system. The results were statistically significant.

6. Real-Time Application

After creating the dataset (Section 3), we developed the baseline system (Section 4.1) and applied the extensions—face detection (Section 4.2) and GAN-based data augmentation (Section 4.3)—to realize real-time applicability to further analyze and test the proposed system.

We developed an application consisting of a web interface that allows testing the proposed model by using the user’s webcam. The web application allows to test the proposed system and observe the predictions in real time, although the prediction also depends on the frame rate (fps) of the user’s computer. An example of positive and negative predictions is presented in Figure 5. We implemented the web application by using HTML, CSS, JavaScript, JQuery, and TensorFlowJS. The web application works on any browser and device, but it is not optimized for smartphones and tablets.

Figure 5 Example of positive and negative predictions in our web application.

7. Conclusions and Future Work

Although attention detection is a crucial step in various computer vision tasks and applications, particularly in the HRI field, few studies have been performed on this topic. In this study, we addressed the attention detection task in its entirety.

We created a manually annotated dataset consisting of approximately 120,000 images from 18 users with five labels (Section 3). Then, we used this dataset to train a baseline system and measure its performance, obtaining satisfactory results on the test set (Section 4.1).

Furthermore, as the first extension, we proposed an improvement over the baseline system by using an auxiliary face detection module to remove useless information (e.g., the background), obtaining consistent performance improvements over the baseline system (Section 4.2).

As the second extension, as our system is intended for users of all ages, we proposed a novel GAN-based data augmentation technique (Section 4.3) for face aging to make the proposed system robust to age changes. This extension yielded further performance improvements over the powerful baseline system. Moreover, we extensively evaluated the advantages of the proposed test set by performing statistical analysis and obtained statistically significant results (Section 5.2). Finally, to further analyze the behavior of the developed attention detection system and enable real-time testing, we designed a web application (Section 6).

Acknowledgments

This work has been supported by the project “HERMES WIRED” funded by Sapienza University of Rome within the Sapienza Research founding scheme 2020.

Author Contributions

All authors have equally contributed to this work.

Competing Interests

The authors have declared that no competing interests exist.