Fixation

The stabilization of the eye on part of a stimulus for a period of time (200–300 ms). The link between fixations and cognitive processes relies on two assumptions [3] the immediacy assumption, which states that, as soon as a participant sees the word, she tries to interpret it, and the eye-mind assumption, which states that a participant fixates her attention on the word until she comprehends it.

Saccade

The quick (and continuous) eye movements from one fixation to another. Saccadic eye-movements are extremely rapid (within 40–50 ms). Saccades are usually voluntary. Micro-saccades on the other hand are small jerky eye movements that are involuntary and occur during a long fixation to refresh the participant's visual memory.

Fig. 1.

Scanpath on a code snippet. Fixations are represented by circles, the size of a circle is proportional to the duration of the fixation. Saccades are lines between fixations. Numbers in the circles order the fixations.

Show All

Pupil Dilation

The widening of the pupil. which allows more light to get into the eye in low light Conditions. It also happens when a participant's mood or attitude changes or during complex cognitive tasks [11].

Scanpath

A series of fixations in chronological order that represents an participant's pattern of eye movements.

Researchers study eye-gaze data with respect to certain areas on the stimuli (e.g., diagrams or source code) called Areas of Interest (AOIs). An AOI can be relevant to the correct answer needed from the participant performing a task or can be irrelevant. For example, if we consider a class diagram as a stimulus, an irrelevant AOI can be any class or any notation while a relevant AOI could be the specific class that is relevant to the given task. For source code, it could be any source code element such as method call or identifier. See Figure 1 for an example of a gaze plot overlaid on a code snippet.

A. Model Stimulus

Several previous eye-tracking studies focused on the comprehension of UML class diagrams with respect to design patterns [6], [7], [12], the impact of layout [13], [14], and the impact of expertise [15]. Petrusel et al. [10] focused on the understanding of business process models (BPMN diagrams) while Cagiltay et al. [16] performed non-formal inspections of entity relationship diagrams (ERD). In addition, Sharafi et al. [17] investigated the efficiency of graphical representations vs. textual ones in modeling and presenting software requirements presented by TROPOS modeling technique.

B. Source Code and Text Stimulus

The majority of previous studies focused on the impact of expertise on comprehension and viewing strategies while comparing source-code reading with natural-language text reading [18]–​[21]. They reported that source-code reading is different from text reading [8] and that participants have higher fixation time and regression rate when reading source code compared to text [20]. They also reported that novices spend more visual attention on comments than experts while experts spend more time on relevant AOls [18], [19]. Busjahn et al. [22] also performed the first eye-tracking study that adapt several linearity metrics used to analyze natural-language reading to source-code reading. Kevic et al. [23] were able to conduct the first eye tracking study on large source code files in an open source system using the Eclipse plugin iTrace [24]. iTrace automatically maps eye movements to source code elements while maintaining context during scrolling.

Some previous studies analyzed the impact of software visualisation on code comprehension [25] and debugging [26], [27]. They reported that, by helping participants to find relevant AOls and their dependencies, visualizations guided participants to follow more systematic strategies [25]. Their results also showed that participants with higher performance mainly use graphical representations although they also used text from time to time [26]. Participants with lower performance performed attention switching very frequently [27].

Several previous studies focused on debugging and defect finding tasks [9], [26]–​[29]. Turner et al. [29] studied the impact of programming languages (C++ vs. Python) and reported that the programming language impacts the visual effort spent by novices compared to experts while working with buggy lines of code. The impact of identifier styles (camel case vs. underscore) on code comprehension has also been investigated [8], [30], [31]. No difference regarding accuracy, time, and effort for comprehension tasks were reported [30], [31] but expertise lessens the impact of identifier styles [30], [31]. Fritz et al. [32] combined pupil dilation, blink rates, electrodermal activity, and EEG (electroencephalogram) to characterise and predict task difficulty.

All these previous eye-tracking studies used different types of metrics to compute visual effort for specific tasks. No study summarizes and provides a list of all available metrics along with detailed definitions and suggestions on how to measure visual effort and use the metrics, the main contribution of this paper. In addition, some researchers use the same metric but name it differently. For example, while analyzing the distribution of visual attention on different code elements, all of the previous studies [18]–​[21] considered each code element as an AOI and calculated the sums of all fixation durations for each AOI but called this metric differently: fixation time [18]–​[20], [26], aggregated fixation time [20], total time of fixations [33], or total dwell time [21]. Conversely, the same name may refer to different metrics. Gaze time either refers to the sum of all fixation durations for an AOI for the whole study [34] or is defined as the sum of all fixation durations [21].

Such ambiguities make comparison and replication of previous studies difficult. This makes it even more important to clarify and standardize the names and definitions of eye-tracking metrics. This paper seeks to bring awareness to these ambiguities and pave the way towards future standardization.

A. Metrics Based on Fixations

Visual-effort metrics using fixations divide into two groups.

1) Metrics Based on the Number of Fixations

Table I presents a list of metrics used in previous studies measuring visual effort based on the number of fixations.

Fixation Count (FC) is the total number of fixations in each AOI. Several studies refer to this metric as the total number of fixations. Goldberg et al. [5] reports that a higher number of fixations devoted to a stimulus shows that the search for finding relevant information is not efficient. Previous eye-tracking studies in SE mainly use this metric to find AOIs that attract more visual attention or to report that more visual effort is required to perform a task. When working with text, the fixation count can be adjusted to the text length by dividing the number of fixations by the number of words in the text.

Fixation Rate (FR) [5] is calculated using Equation 1. The Area of Glance (AOG) can be either the whole stimulus, to calculate the ratio of the total number of fixations in one AOI to all fixations, or it can be another AOI, to show the ratio of fixations between two different AOIs. A smaller ratio shows lower efficiency in search tasks: participants spend more effort to find relevant areas [36]. For comprehension tasks, a higher ratio indicates that either the participant shows a great interest in an AOI or that this AOI is difficult to understand [37]. Other names for this metric are “ON-target ALL-target” [6], “Ratio of ON-target: All-target Fixation (ROAF)” [7], “Ratio of Fixation Count” [8], and “Relevant Fixation Count” [9].

FR=Total Number of Fixations in AOITotal Number of Fixations in AOG(1)

View Source\begin{equation*} FR= \frac{\text{Total Number of Fixations in AOI}}{\text{Total Number of Fixations in AOG}} {\tag{1}} \end{equation*}

Fixation Spatial Density (SD) [5] is equal to the number of cells containing at least one fixation, divided by the total number of cells, if we consider the stimulus as a grid. It is calculated using Equation 2 where n is the number of cells in the grid and ci is equal to 1 if the cell number i is visited, otherwise it is O. It represents the “coverage of an area” and measures the dispersion of the participant's fixations. Smaller spatial density shows less coverage.

SD=∑ni=1cin(2)

View Source\begin{equation*} SD= \frac{\sum\nolimits_{i=1}^{n}c_{i}}{n} {\tag{2}} \end{equation*}

Convex Hull Area [5] represents the area of the smallest convex set of fixations that contains all a participant's fixations to visualize the spatial distribution of fixations and show the preferred parts of a visual stimulus. It is very sensitive to outliers [4] and even one fixation that deviates from its true location can change the convex hull significantly. Thus, rigourous noise removal is necessary.

2) Metrics Based on the Duration of Fixations

The duration of fixations represents the required time to analyze a stimulus [5] and, thus, the depth of processing. Table II presents a list of metrics used by previous studies based on the duration of fixations. It is important to measure both fixation counts and their durations because it is possible to have a low fixation count but a high duration and vice versa.

Average Fixation Duration (AFD) [5] is the sum of the durations of all the fixations divided by the number of fixations, as in Equation 3, where ET(Fi) and ST(Fi) are the end time and start time for a fixation Fi and n is the total number of fixations in a given AOI. Another name for this metric is “Mean Fixation Duration” [20], [38].

AFD(AOI)=∑ni=1(ET(Fi)−ST(Fi))in AOin(3)

View Source\begin{equation*} AFD(AOI)= \frac{\sum\nolimits_{i=1}^{n}(ET(F_{i})-ST(F_{i}))\text{in AOi}}{n} {\tag{3}} \end{equation*}

Ratio of ON-target.·All-target Fixation Time (ROAFT) [5] is the sum of the durations of all fixations in an AOI, divided by the total duration of all fixations for the area of glance (AOG), as in Equation 4. An AOG can be the entire stimulus or a set of AOIs. A smaller ratio indicates lower efficiency while searching the stimulus [5]. Other names for this metric are “Proportional Fixation Time (PFT)” [26], “Ratio of Fixation Time” [8], “Relevant Fixation Duration” [9], and “Time in Region (TIR)” [10].

ROAFT=∑ni=1(ET(Fi)−ST(Fi))in AOI∑nj=1(ET(Fj)−ST(Fj))in AOG(4)

View Source\begin{equation*} ROAFT= \frac{\sum\nolimits_{i=1}^{n}(ET(F_{i})-ST(F_{i}))\text{in AOI}}{\sum\nolimits_{j=1}^{n}(ET(F_{j})-ST(F_{j}))\text{in AOG}} {\tag{4}} \end{equation*}

Fixation Time(FT) also known as gaze or fixation cluster, is the sum of the durations of all fixations in an AOI. Busjahn et al. [21] compute this metric for every AOI visit separately and called it “Dwell Time”. They define FT, which they call “Total Dwell Time”, as the sum of all dwell times on a AOI over an entire study [21]. Other names for this metric are “Aggregated Fixation Time” [20] and “Total Time of Fixations” [33].

Average Duration of Relevant Fixations (ADRF) is the total duration of the fixations for relevant AOIs as in Equation 5. A corresponding metric exists for non-relevant AOIs and is called “Average Duration of Non-Relevant Fixations (ADNRF)”.

ADRF=Fixations Duration of Relevant AOIsTotal Number of Relevant AOIs(5)

View Source\begin{equation*} ADRF= \frac{\text{Fixations Duration of Relevant AOIs}}{\text{Total Number of Relevant AOIs}} {\tag{5}} \end{equation*}

Normalised Rate of Relevant Fixations (NRRF) [12] is shown in Equation 6 and allows comparing two or more stimuli with each other. If a stimulus requires more relevant fixations than another, then it requires more visual effort [12]. To adjust for the size of stimulus, this metric must be normalized using the total number of AOIs in the stimulus [12].

NRRF=ADRF(Fixation Duration of All AOIsNumber of All AOIs)(6)

View Source\begin{equation*} NRRF= \frac{ADRF}{\left(\frac{\text{Fixation Duration of All AOIs}}{\text{Number of All AOIs}}\right)} {\tag{6}} \end{equation*}

B. Metrics Based on Saccades

Table III presents a list of saccade-based metrics used by previous studies to measure visual effort. Higher numbers of saccades indicate more searching [5], [36].

Number of Saccades and Saccade Duration are metrics whose definitions are identical to the corresponding fixations-based metrics, see the previous subsection. They also have similar interpretations in relation to the visual effort.

Regressions Rate indicates the percentage of backward saccades of any length [22]. Good readers are characterized by few regressions [22], thus higher regressions rates denote that the participants have difficulty reading and understanding a stimulus [5], [36].

Table I. Metrics for visual effort based on the number of fixations

Table II. Metrics for visual effort based on the duration of fixations

C. Metrics Based on Scanpaths

Table IV shows scanpath-based metrics for visual effort used by previous studies.

Attention Switching Frequency measures the dynamics of visual attention using the total number of switches between a set of AOIs per minute. A switch happens whenever the participant's focus of attention changes between any AOIs.

Transitional Matrix is a tabular representation of the frequencies of transitions between AOIs [39] computed using Equation 7 in which n is the number of fixations in a AOI (for one cell) and ci is equal to 1 if the AOI number i is visited and O otherwise. To compare two transition matrices, the density of a transition matrix is the number of non-zero matrix cells divided by total number of cells. Higher density indicates extensive search with inefficient scanning on a stimulus, while a sparse matrix is a proxy for an efficient and directed search [5]

View Source\begin{equation*} TM= \frac{\sum\nolimits_{i=1}^{n}\sum\nolimits_{j=1}^{n}c_{\iota,j}}{n^{2}} {\tag{7}} \end{equation*}

Edit Distance uses the Levenshtein algorithm, which computes the minimum editing cost to transform one string into another using three basic operations (insertion, deletion, and substitution). (For each operation, the cost of one is considered.) It only uses the location of fixations, not their duration. Mathematically, the Levenshtein distance between two strings a and b is Calculated recursively as in Equation 8.

View Source\begin{equation*} lev_{a,b}(i,\ j)=\begin{cases} max(i,\ j)&\text{if} \ \text{min}(\text{i,j})\ =0\\ min=\begin{cases} lev_{a,b}(i-1,j)+1 \\ lev_{a,b}(i,j-1)+1 \\ lev_{a,b}(i-1,j-1)+1_{a_{i}\neq b_{i}} \end{cases} &\text{Otherwise} \end{cases} {\tag{8}} \end{equation*}

Table III. Metrics for visual effort based on saccades.

Table IV. Metrics for visual effort based on scanpaths

Sequential PAttem Mining (SPAM) [40] uses the depth-first algorithm to mine and compare scanpaths by considering both fixation locations and durations.

ScanMatch [41] compares scanpaths based on the Needleman-Wunsch algorithm used in bioinformatics to compare sequences of DNA. After adjusting the length of the scan paths based on the fixations durations using a temporal binning, this metric calculates the similarity score to compare two scanpaths.

Linearity determines a participant's search strategy of a stimulus [36]. For source code, “linearity represents how closely readers follow a text's natural reading order” [22]. It uses eye-movements linearity (left-to-right and top-to bottom) to characterize how developers read source code. A set of local and global metrics exist to measure linearity. For example, local metrics are as follows:

• Vertical Next Text is the percentage of (forward)

• saccades that happen either on the same line or move only one line below.

• Vertical Later Text is the percentage of (forward)

• saccades that happen either on the same line or move any number of lines below.

• Horizontal Later Text is the percentage of (forward)

• saccades that happen on the same line.

• Regression Rate is defined in Section IV-B.

• Line Regression Rate is the percentage of backward saccades that happen on the same line.

• Saccade Length is defined as the average Euclidean distance between consecutive fixations.

• Element Coverage is the percentage of words that a participant puts visual attention on.

Global metrics use the order in which source code is read:

• Story Order is the extent to which the orders of fixations are similar (aligned) to the linear text reading order (left-to-right), using the ScanMatch metric [41].

• Execution Order is the extent to which the orders of fixations are similar (aligned) to the program control flow and also uses the ScanMatch metric [41].

D. Pupil Size and Blink Rate

Table V presents previous studies that measured visual effort using pupil sizes and blink rates. These two metrics are associated with cognitive workload. Lower blink rates indicate higher workload or attention [32] while higher rates are associated with fatigue [36]. Larger pupil sizes indicate more effort [36]. In addition, Beatty reported that the maximum amplitude of pupil sizes indicates memory and processing load that fluctuates with task difficulty [11].

A. Data Analysis

Previous eye tracking studies used a combination of the following data analysis approaches to investigate eye-gaze data: (1) start with a hypothesis or a theory and analyze the eye-tracking data to validate it and/or (2) work entirely based on observation without considering any theories in advance [37], [42]. Our review of previous works shows that fixation-based metrics have been mostly used for the first approach, whereby researchers are interested in calculating effort for specific AOIs. Scanpath-based metrics or saccade-based metrics have been mostly used for the second approach, e.g., for evaluating search and navigation strategies. Yet, some existing studies used both approaches, using different sets of metrics on different eye-movement data [15], [17], [31].

Most of the time, defining a set of AOIs is the main step towards analyzing eye-tracking data. Defining an AOI is a subjective task and is based on the researchers' assumptions and experimental goals and conditions. There are no detailed guidelines about defining AOIs [43], especially for SE tasks. Also, the majority of previous studies did not provide details about AOI definition and data extraction. Only, Goldberg et al. [43] proposed a set of general guidelines for defining AOIs. Thus, researchers must be careful when defining AOIs and gather and analyze data, especially if the AOIs overlap or are nested. In addition, we encourage researchers in the SE community to explain their choices of AOIs, data extraction, and data analysis approaches in detail. There is also a need for further studies investigating the impact of the type, the granularity, and the analysis of different AOIs for variety of software artifacts and tasks.

Table V. Metrics for visual effort based on the pupil size and blink rate.

B. Metric Popularity

All of the previous studies in SE used fixation-based and scanpath-based metrics. Only a handful of studies used metrics based on saccades because the general consensus in the eye-tracking research community is that cognitive processing and comprehension occur during fixations, while the processing (if any) happening during saccades is very limited. Even for eye-tracking research in usability studies, pupil size and blink rates have been scarcely used because these measures are very sensitive to ambient light level and will be contaminated easily [36]. Only one study used pupil size and blink rate, for measuring task difficulty [32]. It used an EEG device, filtering certain frequencies, to measure blink rates more accurately than using eye-trackers. In addition it applied noise removal and cleaning techniques on the data. The results were promising and show that the task difficulty can be classified using pupil size and other psycho-physiological factors.

C. Metric Representativeness

We provided a complete list of eye-tracking metrics used in SE research. However, there exists other metrics proposed and used in other domains, such as human-computer interaction (HCI) and usability [5], [36], [37]. These metrics have not been used in SE research yet. We analyze these metrics with respect to their adequacy and (possible) uses in SE research.

We categorize metrics in HCI and usability studies into two main groups [5]: (1) “Measure of processing”, which measures the amount of effort required to process, understand, and analyze a stimulus while performing a task and (2) “Measure of search”, which measures the amount of effort required to explore and navigate a stimulus when performing a task.

Metrics for the measure of processing are fixation-based metrics, including those presented in Section IV-A. They are mostly and frequently used in SE eye-tracking studies. They provide a single quantitative value to measure the amount of visual effort independently from the type of material that have been used because, to use these metrics, researchers only need to define the AOIs and their relevancy. Thus, these metrics can be used to assess source code or any models, as shown in Tables I and II. Yet, when reviewing the literature on HCI and usability, we found one metric that has not been used in SE.

Saccade-Fixation Ratio (SF Ratio) is a metric for processing that is calculated using Equation 9. It indicates the ratio of searching over processing. A higher value indicates more searching compared to processing. It can be used to compare different layouts (e.g., orthogonal vs. three-cluster vs. multicluster layouts for UML class diagrams [14]) or representations (e.g., graphical vs. textual [17]).

View Source\begin{equation*} SF Ratio = \frac{\text{Total Saccade Time}(\text{Search Time})}{\text{Total Fixations Time}(\text{Processing Time})} {\tag{9}} \end{equation*}

Metrics for search include scanpath-based and saccade-based metrics. Sections IV-B and IV-C present some of these metrics used in SE. There are several other metrics for search in the literature on HCI and usability that have not been used by SE researchers, including:

Average Saccade Amplitude indicates the angular distance that the eye travels and is computed by summing the distances between consecutive fixations, divided by the number of fixations minus one for the stimulus [5]. To filter out micro-saccades, a minimum amplitude is used. This metric can be used to compare two layouts [14] or representations [17] to determine their efficiency for a task. An efficient layout/representation must arrange elements such that participants' scanning to relevant AOIs is minimal [5], [36].

Scanpath Regularity is the ideal situation in which participants follow relevant cues until they reach the desired element. Any deviation from this regular path denotes less efficient search. For SE tasks, a researcher may define a set of relevant AOIs and then consider AOI-based scanpaths instead of ones based on fixations. The most efficient scanpath would be the one that visits all relevant AOIs in a specific order. Irregularity could be measured as either focusing on an irrelevant AOI or following a different path.

Metrics such as Average Saccade Amplitude must be used with care because they may not be adequate for SE tasks: they measure the distances between saccades or fixations in numbers of pixels. For SE materials (source code and models), distances between AOIs in numbers of pixels have no meaning because the amplitude is completely dependent on the size of the stimulus and of its elements. For example, two UML class diagrams that differ only by the sizes of the rectangles used to display classes will yield two different values of this metric.

A. Previous Eye-Tracking Studies

Several threats limit the validity of previous studies regarding effort measurement based on eye-movement data. We now discuss these potential threats and how previous studies tried to mitigate them. Reported eye-tracking studies use eye-movement data to measure visual effort. One threat to these studies is that there may be other factors that influence the amount of effort put forth by participants to complete their tasks, including stress (e.g., due to the Hawthorn effect), fatigue, or time of the day. Previous studies tried to mitigate this threat by performing the studies in quiet laboratories. They also avoided long studies to reduce fatigue and limited interactions between participants and researchers.

Like any other human psycho-physiological measure, eye movements may contain gaze data that are unintentional and unconscious. Any distracting events in the environment or in the visual stimulus may impact the eye movements and lead to noise in the collected data and consequently, possibly incorrect interpretations of the data. Previous studies used tasks and stimuli that were not distracting, i.e., without any interactive menus, and performed the experiments in a controlled environment to avoid any distractions.

Another major threat to validity of previous eye-tracking studies is that it is hard to compare results across studies, thereby limiting generalization. The main factor contributing to this lack of generalization was the confusing and redundant naming of visual effort metrics used as dependent variables. To mitigate this threat, we conducted the survey described in this paper. We hope that future studies using eye-trackers will take advantage of this one-stop cross reference for visual effort metrics, thereby eventually standardizing the terms and making it easier to compare results across different studies.

B. This Study

The main threat to validity of this study is whether or not we covered all visual effort metrics that have been proposed or used in previous eye-tracking studies. To reduce the possibility of missing relevant metrics, we performed a SLR [4] by following the process and advice given by Kitchenham [44]. Due to space limitations, we direct the reader to our previous study [4] for the protocol used in the SLR covering 1990–2014, which drives the comparisons done in this study. In this previous study, we did not compare metrics. The first author extracted data from previous eye-tracking studies to provide interpretation of the metrics. To reduce the likelihood of erroneous results, the three other authors checked and validated the extraction and interpretations.