DissLiteratur/storage/5MNP8ANX/.zotero-ft-cache

One-point Calibration Gaze Tracking Based on Eyeball Kinematics Using Stereo Cameras

Takashi Nagamatsu∗ Kobe University

Junzo Kamahara† Kobe University

Takumi Iko‡ Kobe University

Naoki Tanaka§ Kobe University

Abstract
This paper presents a one-point calibration gaze tracking method based on eyeball kinematics using stereo cameras. By using two cameras and two light sources, the optic axis of the eye can be estimated. One-point calibration is required to estimate the angle of the visual axis from the optic axis. The eyeball rotates with optic and visual axes based on the eyeball kinematics (Listing’s law). Therefore, we introduced eyeball kinematics to the one-point calibration process in order to properly estimate the visual axis. The prototype system was developed and it was found that the accuracy was under 1 ◦ around the center and bottom of the display.
CR Categories: H.5.2 [Information Interfaces and Presentation]: User Interfaces–Ergonomics; I.4.9 [Image Processing and Computer Vision]: Applications
Keywords: eye tracking, calibration, stereo camera, eyeball kinematics
1 Introduction
Gaze tracking technology is being used as a human-machine interface[Jacob 1991; Duchowski 2007; Nagamatsu et al. 2007]. However, most accurate gaze tracking systems require personal calibration process before the system can be used. Several works have attempted to reduce the calibration effort by using a model-based approach[Ohno and Mukawa 2004; Shih and Liu 2004; Ohno 2006; Guestrin and Eizenman 2006; Guestrin and Eizenman 2007; Villanueva and Cabeza 2007]. For example, Shih and Liu [2004], and Guestrin and Eizenman[2006; 2007] have presented methods to reconstruct the optic axis of the eye using stereo cameras without actually knowing the personal eye parameters. Since the optic axis is an approximation of the visual axis with an accuracy of approximately 5 ◦, after the reconstruction of the optic axis, the visual axis is estimated by at least a one-point calibration procedure in their work.
However, most previous works did not consider the eyeball kinematics when estimating the visual axis; in contrast, Villanueva and Cabeza[2007] pointed out the importance of eyeball kinematics. In this paper, we describe one-point calibration gaze tracking based on eyeball kinematics using stereo cameras.

2 Estimation of optic axis
In this section, we describe a method for the estimation of the optic axis. The cameras are modeled as pinhole cameras and the light sources are modeled as point sources. The position of the light sources must be measured and the intrinsic and extrinsic camera parameters must be determined in advance by the camera calibration.
2.1 Image processing of eye image
The eye images are captured by two cameras. From the images captured by camera 0, the center of the pupil, B00 , is detected by ellipse fitting, and the centers of the first Purkinje images (the reflection of light sources (L0 and L1) from the outer surface of the cornea), P000 and P001, are detected by seeking near the center of the pupil. Similarly, from the image captured by camera 1, B10 , P100, and P101 are detected, where B10 is the center of the pupil and P100 and P101 are the centers of the first Purkinje images of L0 and L1 captured by camera 1.
2.2 Center of corneal curvature
We estimate the position of the center of the corneal curvature, A. Figure 1 shows a cross section of the eyeball that includes A, light source 0, L0, the nodal point of camera 0, C0. L0 and C0 are known. A ray from L0 reflects at a point P00 on the corneal surface such that the reflected ray passes through C0 and intersects the camera image plane at a point P000. C0, P00, L0, P000, and A are coplanar, and the normal vector of the plane, π00, is (P000 − C0) × (L0 − C0). The plane can be expressed as

(P000 − C0) × (L0 − C0) · (x − C0) = 0

(1)

where x (= (x, y, z))is the point on the plane.
Similarly, with regard to the relation among the cameras, C0 and C1, the light sources, L0 and L1, and the Purkinje images, P001, P100, and P101, three other planes, π01, π10, and π11 can be expressed respectively as

(P001 − C0) × (L1 − C0) · (x − C0) = 0

(2)

(P100 − C1) × (L0 − C1) · (x − C1) = 0

(3)

(P101 − C1) × (L1 − C1) · (x − C1) = 0.

(4)

∗e-mail:nagamatu@kobe-u.ac.jp †e-mail:kamahara@maritime.kobe-u.ac.jp ‡e-mail:071w101w@stu.kobe-u.ac.jp §e-mail:ntanaka@maritime.kobe-u.ac.jp
Copyright © 2008 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail permissions@acm.org. ETRA 2008, Savannah, Georgia, March 26–28, 2008. © 2008 ACM 978-1-59593-982-1/08/0003 $5.00

All planes include A, and A can be known if three planes
are given. Two LEDs were installed to make an angle of 90 ◦ around a camera in order to improve its accuracy.

2.3 Center of pupil

Radius of cornea In order to estimate the radius of the

cornea, we will estimate the reflected point, P00, that is on

the middle line between L0A and C0A, as shown in figure

1. The line AP00 can be expressed in a parametric form as

µ

¶

x = A + t L0 − A + C0 − A

(5)

|L0 − A| |C0 − A|

95

Figure 1: Center of the Corneal Curvature.

where t is a parameter. On the other hand, the line from C0 to P00 can be expressed in parametric form as

x = C0 + t (C0 − P000) .

(6)

The position of P00 can be estimated as the intersection point of the above two lines. Therefore, the radius of the cornea, r, is determined as

r = |P00 − A|.

(7)

Since r can be determined using three other combinations (C0,L1,P001), (C1,L0,P100), and (C1,L1,P101), the average is used for improving the robustness.

Refraction As shown in figure 2, a ray that originates from the center of the pupil, B, refracts at the point B000, passes through the nodal point of camera 0, C0, and intersects the camera image plane at a point B00 . B000 can be determined to solve the equations given below.

x = C0 + t (C0 − B00 )

(8)

r = |x − A|

(9)

t is solved as follows:

√

t = −b − b2 − ac

(10)

a

where a = (C0x − B00 x)2 + (C0y − B00 y)2 + (C0z − B00 z)2, b = (C0x −A0x)(C0x −B00 x)+(C0y −A0y)(C0y −B00 y)+(C0z − A0z)(C0z − B00 z), and c = (C0x − A0x)2 + (C0y − A0y)2 + (C0z − A0z)2.
The refracted vector at B000, t0, can be calculated by using
Snell’s law as follows:

³

p

´

t0 = −ρn0 · v0 − 1 − ρ2 (1 − (n0 · v0)2) n0 + ρv0

(11) where the incident vector v0 = (C0 −B00 )/|C0 −B00 |, normal vector at the point of refraction n0 = (B000 − A)/|B000 − A|, and ρ = n1/n2 (n1: the index of air ≈ 1; n2 : the
effective refractive index ≈ 1.3375).

Center of pupil The center of the pupil, B, can be obtained from the intersection of two rays from the two cameras as follows:

x = B000 + tt0 x = B100 + tt1.

(12) (13)

where B100 and t1 are the refracted point and the refracted vector, respectively (using camera 1).

Figure 2: Refraction.

2.4 Optic Axis The optic axis can be written in parametric form as follows:

x = A + t(B − A).

(14)

3 One-Point Calibration
This section describes the calibration process that determines a unit direction vector of an optic axis, b, at the primary position, which is the position of the eye relative to the head when looking straight ahead at an object at eye level. In this paper, the unit direction vector of the visual axis at the primary position is (0, 0, −1)T (= a).
Figure 3 shows the eyeball model of rotation and translation. E0 and A0 are the rotation center of the eyeball and the center of the corneal curvature at the primary position, respectively. E, d, and c are the rotation center of the eyeball, a unit direction vector of the optic axis, and, a unit direction vector of the visual axis after the eye movement, respectively.

Figure 3: Model of an eyeball.

3.1 Movement of eyeball
The visual axis at the primary position can be expressed as x = A0 + ta, and the new visual axis is expressed as x = A + tc. If the eye moves, the visual axis rotates around E0 by the rotation matrix, R, and E0 is translated to E by the translation matrix, T . The relation can be written as

E + ((A − E) + tc) = T E0 + R((A0 − E0) + ta) (15)
Since E = T E0 and (A − E) = R(A0 − E0), the relation between a and c can be written as

c = Ra.

(16)

Therefore, in order to analyze the movement of the visual axis, we only have to analyze the movement of the direction vector of the visual axis.

96

3.2 Listing’s law
Listing’s law states that any eye position can be reached by a single rotation from the primary position and any rotation axis lies in a plane (Listing’s plane) [Zatsiorsky 1999]. In this paper, the rotation axis of the eyeball, l, is assumed to be parallel to the xy plane. We will calculate the eyeball rotation based on Listing’s law.
3.3 Visual axis
D is assumed to be a center on the display. When a user is gazing at D, the vector of the visual axis can be written as D −A. Figure 4 shows the relation among a, (D −A)/|D − A|, and l. The rotation from a to (D − A)/|D − A| by ψ can be written as a product of three rotation matrices as

D−A |D − A|

=

R(z, φ)R(x, ψ)R(z, −φ)a

(17)

= Ma

(18)

where R(z, φ) is the rotation matrix around the z-axis by φ, R(x, ψ) is the rotation matrix around the x-axis by ψ, R(z, −φ) is the rotation matrix around the z-axis by −φ, and M is a combined matrix of all the matrices.
The unit vector of the axis of rotation, l, and the angle, ψ, can be written as

l

=

a × (D − A) |a × (Dµ − A)|

¶

(19)

ψ

=

arccos

a · (D − A) |a||D − A|

.

(20)

Since φ is made by l and (1, 0, 0)T (= e), it can be obtained

as follows:

µ¶

φ = arccos

e·l |e||l|

.

(21)

=

(R(z, φ)R(x, ψ)R(z, −φ))−1

B |B

− −

A A|

=

R(z,

φ)R(x,

−ψ)R(z,

−φ)

B |B

− −

A A|

.

(24)

4 Estimation of visual axis
In this section, a method to estimate the visual axis using b, which was determined by calibration, is described.
4.1 Calculation of rotation axis and angle
d is a unit vector of the optic axis and it is given as d = (B −A)/|B −A|. l0 is a unit direction vector of the rotation axis of the eye, which is parallel to xy plane. Figure 5 shows the relation among b, d, and l0. l0 is included in the xy plane a · x = 0 in the figure. Since b rotates to d around l0, l0 is also included in the plane (d − b) · x = 0. Therefore the unit direction vector of the axis is given as

l0 = a × (d − b) .

(25)

|a × (d − b)|

The rotation axis can be expressed as x = tl0. The intersection point between x = tl0 and the plane perpendicular to x = tl0 including (d − b) must be determined in order to estimate ψ. t can be determined by the inner product of l0 and b, which is given as t = l0 · b. Therefore the center of rotation of b is (l0 · b)l0. The rotation angle of the eyeball is estimated as

µ

¶

ψ = arccos

((l0 · b)l0 − b) · ((l0 · b)l0 − d) |((l0 · b)l0 − b)||((l0 · b)l0 − d)|

.

(26)

Figure 4: Rotation about the visual axis.

3.4 Estimation of visual axis at primary position
If b is the optic axis at the primary position, then the relation between (B − A) and b can be written using M as follows:

B |B

− −

A A|

=

M b.

(22)

b can be obtained by using φ and ψ as

b = M −1 B − A

(23)

|B − A|

Figure 5: Rotation about the optic axis.

4.2 Calculation of visual axis
The visual axis can be calculated by rotating a around l0 by an angle ψ. A quaternion is convenient to express the rotation of a point around a certain axis. The elements of a and l0 are written as a = (ax, ay, az) and l0 = (lx0 , ly0 , lz0 ), respectively.
a is expressed in quaternion form as Pa = (0; ax, ay, az) and the rotation is expressed as Q = (cos(ψ/2); lx0 sin(ψ/2), ly0 sin(ψ/2), lz0 sin(ψ/2)). Then, the visual axis vector, Pc, in quaternion form can be obtained as follows:

Pc = (0; cx, cy, cz) = QPaQ−1.

(27)

Consequently we can obtain the vector of the visual axis, c = (cx, cy, cz).

97

4.3 Estimating gazing position on display The display can be expressed by four points: DT L, DT R, DBL, and DBR. The normal vector of the display, nd, is (DT L − DBR) × (DT R − DBL); threfore, the display can be expressed as nd · (x − DT L) = 0. Since the visual axis is written as x = A + tc, the intersection point is the point of gazing (P OG) on the display. After estimating the position of the intersection point, it is converted to graphical coordinates on the screen.
5 Implementation and experimental results
5.1 Implementation A prototype system was implemented, as shown in figure 6. This system uses two synchronized monochrome IEEE-1394 digital cameras using a 1/3” CMOS image sensor (Firefly MV, Point Grey Research Inc.) with 50-mm lens and an IR filter, two infrared light sources attached to a 17” LCD, and a Windows-platform-based PC. Software was developed using OpenCV[Intel ]. If a user clicks on the mouse, a fixation point is displayed. When the user gazing at the fixation point releases the mouse button, the calibration is performed.
Figure 6: System.
5.2 Experimental results The prototype system was evaluated experimentally with one adult subject. The subject was asked to fixate on 25 points on the display. The area within which the user could move was 3 cm laterally, 2.5 cm vertically, and 5 cm backward/forward, and was approximately 57.5 cm from the display. Figure 7 shows the result of the experiment. It was found that the accuracy was under 1 ◦, except that around the top left and top right, the measurement had a low accuracy or was impossible to determine. This may be because the penumbra of the cornea is not like a sphere or an LED that reflects on the sclera. This may be attributed to the limitation of the two-light-source and two-camera arrangement. The next step will be the improvement of the arrangement of light sources and cameras in order to observe Purkinje images on the corneal surface near the optic axis.
6 Conclusion
This paper presented a one-point calibration gaze tracking method based on eyeball kinematics using stereo cameras. A prototype system was developed based on this method and the accuracy was under 1 ◦ around the center and bottom of the display.
Acknowledgements
This research was partially supported by the Ministry of Education, Culture, Sports, Science and Technology, Grant-

Figure 7: Experimental results: the dotted points indicate the estimates of the POG, the 25 grid points represent the intended fixation points, and the numbers represent errors in view angle (degree).
in-Aid for Young Scientists (B), 18700645, 2006.
References
Duchowski, A. T. 2007. Eye Tracking Methodology: Theory and Practice, 2nd ed. Springer-Verlag.
Guestrin, E. D., and Eizenman, M. 2006. General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE Transactions on Biomedical Engineering 53, 6, 1124—1133.
Guestrin, E. D., and Eizenman, M. 2007. Remote pointof-gaze estimation with free head movements requiring a single-point calibration. In Proceedings of the 29th Annual International Conference of the IEEE EMBS, 4556—4560.
Intel. Open source computer vision library. http:// www.intel.com/technology/computing/opencv/index.htm.
Jacob, R. J. K. 1991. The use of eye movements in humancomputer interaction techniques: what you look at is what you get. ACM Transactions on Information Systems 9, 2, 152—169.
Nagamatsu, T., Kaieda, Y., Kamahara, J., and Shimada, H. 2007. Development of a skill acquisition support system using expert’s eye movement. In Proceedings of HCI International 2007, vol. 9, 430—439.
Ohno, T., and Mukawa, N. 2004. A free-head, simple calibration, gaze tracking system that enables gaze-based interaction. In Proceedings of the 2004 symposium on Eye tracking research & applications, 115—122.
Ohno, T. 2006. One-point calibration gaze tracking method. In Proceedings of the 2006 symposium on Eye tracking research & applications, 34—34.
Shih, S.-W., and Liu, J. 2004. A novel approach to 3-d gaze tracking using stereo cameras. IEEE Transactions on Systems, Man, and Cybernetics, Part B 34, 1, 234—245.
Villanueva, A., and Cabeza, R. 2007. Models for gaze tracking systems. EURASIP Journal on Image and Video Processing 2007, Article ID 23570.
Zatsiorsky, V. M. 1999. Kinematics of Human Motion (Japanese Edition). NAP Limited.

98