Eye-Tracking As a Tool in Process-Oriented Reading Test Validation

Oddny Judith SOLHEIM, Per Henning UPPSTAD


The present paper addresses the continuous need for methodological reflection on how to validate inferences made on the basis of test scores. Validation is a process that requires many lines of evidence. In this article we discuss the potential of eye tracking methodology in process-oriented reading test validation. Methodological considerations are highlighted and special significance is placed on the importance of studying the first reading of a text as well as reading while answering questions about it. This point of view expands the traditional scope of eye-tracking methodology in reading research. We conducted a small-scale study in which 18 12-year olds read and answered questions about a multimodal text. In this study comprehension scores were related to allocation of visual attention in two conditions: (i) reading a text passage for the first time; and (ii) rereading of the text passage while answering questions about it.


Reading Comprehension, Assessment, Validity, Eye-Tracking

Paper Details

Paper Details
Topic EU Education Programs
Pages 153 - 168
Issue IEJEE, Volume 4, Issue 1, Special Issue Reading Comprehension
Date of acceptance 01 October 2011
Read (times) 656
Downloaded (times) 236

Author(s) Details

Oddny Judith SOLHEIM

National Centre of Reading Education and Reading Research, University of Stavanger, Norway

Per Henning UPPSTAD

National Centre of Reading Education and Reading Research, University of Stavanger, Norway


Afflerbach, P. (2000). Verbal Reports and Protocol Analysis. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson & R. Barr (Eds.), Handbook of Reading Research (vol. 3, pp. 163-179). Mahwah, New Jersey, London: Lawrence Earlbaum Associates.

Afflerbach, P. & Johnston, P. (1984). Research methodology: On the use of verbal reports in reading research. Journal of Reading Behaviour 16 (307-322).

Alderson, J. C. (2000). Assessing Reading. Cambridge: Cambridge University Press.

Allan, A. I. C. G. (1992). EFL reading comprehension test validation: investigating aspects of process approaches. Unpublished PhD thesis, Lancaster University, Lancaster.

Andreassen, R. & Bråten, I. (2010). Examining the prediction of reading comprehension on different multiple-choice tests. Journal of Research in Reading 33 (263-283).

Campbell, J.R. (2005). Single Instruments, Multiple Measures: Considering the Use of Multiple Item Formats to Assess Reading Comprehension. In S. G. Paris & S. A. Stahl (Eds.), Children’s Reading Comprehension and Assessment (pp. 347–369). Mahwah, New Jersey: Lawrence Earlbaum Associates.

Cordon, L. A. & Day, J. D. (1996). Strategy use on standardized reading comprehension tests. Journal of Educational Psychology 88 (288–295).

Ericsson, K. & Simon, K. (1984). Protocol Analysis: Verbal reports as data. Cambridge, MA; MIT Press.

Farr, R., Pritchard, R. & Smitten, B. (1990). A Description of What Happens When an Examinee Takes a Multiple-Choice Reading Comprehension Test. Journal of Educational Measurement 27 (209–226).

Goetz, E.T., Schallert, D. L., Reynolds, R. E. & Radin, D. J. (1983). Reading in perspective: What real cops and pretend burglars look for in a story. Journal of Educational Psychology 75 (500–510).

Hannus, M. & Hyönä, J. (1999). Utilizations of Illustrations during Learning of Science Textbook Passages Among Low- and High-Ability Children. Contemporary Educational Psychology 2 (95–123).

Holmqvist, K., Holmberg, N., Holsanova, J., Tärning, J. & Engwall, B. (2006). Reading Information Graphics – Eyetracking Studies with Experimental Conditions. In J. Errea (Ed.), Malofiej Yearbook of Infographics (pp. 54–61). Society for News Design (SND-E), Navarra University, Pamplona, Spain.

Holsanova, J., Holmberg, N. & Holmqvist, K. (2005). Tracing Integration of Text and Pictures in Newspaper Reading. Lund University Cognitive Studies 125. Lund: Lund University.

Hyönä, J., Lorch, R. F. J., & Kaakinen, J. K. (2002). Individual Differences in Reading to SummarizeExpository Text: Evidence from Eye Fixation Patterns. Journal of Educational Psychology 94 (44–55).

Høien, T. & Tønnesen, G. (1998). Håndbok til Ordkjedetesten [Handbook for the word-chain test]. Stavanger: Stiftelsen Dysleksiforskning.

Johansen, E. B. & Steineger, E. (1999). Globus: Natur og miljøfag 7 [Globus: 7th year science and environment studies’]. Oslo: J.W. Cappelen Forlag.

Johnson, D. & Kress, G. (2003). Globalisation, Literacy and Society: redesigning pedagogy and assessment. Assessment in Education 10 (5-14).

Johnston, P. J. (1984). Assessment in Reading. In P.D. Pearson, R. Barr, M. Kamil and P. Mosenthal (Eds.),Handbook of Reading Research (2nd ed., pp. 147–182). New York: Longman.

Just, M. A. & Carpenter, P. A. (1980). A theory of reading. From eye fixations to comprehension. Psychological Review 87 (329-354).

Kaakinen, J.K. & Hyönä, J. (2005). Perspective Effects on Expository Text Comprehension: Evidence from Think-Aloud Protocols, Eyetracking and Recall. Discourse Process 40 (239–257).

Kamil, M. L. (2004). The current state of quantitative research. Reading Research Quarterly 39 (100-107).

Kress, G. & van Leuven, T. (2001). Multimodal discourse. The modes and media of contemporarycommunication. London: Arnold.

Land, M. & Tatler, B. (2009). Looking and Acting. Vision and Eye Movements in Natural Behaviour. Oxford, New York: Oxford University Press.

Langer, J. (1987). The construction of meaning and the assessment of comprehension: An analysis of reader performance on standardized test items. In R. O. Freedle & R. P. Duran (Eds.), Cognitive and linguistic analyses of text performance (pp. 225-244). Norwood, NJ: Ablex.

Li, W. (1992). What is a test testing? An investigation of the agreement between students´ test taking processes and test constructers´ presumptions. Unpublished MA thesis, Lancaster University, Lancaster.

Messick, S. (1988). The Once and Future Issues of Validity: Assessing the Meaning and Consequences of Measurement. In H. Wainer & H. I. Braun (Eds.), Test Validity (pp. 33–46). Hillsdale, New Jersey, Hove, London: Lawrence Earlbaum Associates.

Messick, S. (1995). Validity of Psychological Assessment: Validation of Inferences From Persons’ Responses and Performance as Scientific Inquiry Into Score Meaning. American Psychologist 50 (741–7499.

Mullis, I. V. S., Kennedy, A. M., Martin, M. O. & Sainsbury, M. (2006). PIRLS 2006 Assessment Framework and Specifications. (2nd ed.). Chestnut Hill, MA: Boston College.

Ozuru, Y., Best, R., Bell, C., Witherspoon, A. & McNamara, D. S. (2007). Influence of Question Format and Text Availability on the Assessment of Expository Text Comprehension. Cognition and Instruction 25 (399-438).

Ozuru, Y., Dempsey, K., McNamara, D. S. (2009). Prior knowledge, reading skill, and text cohesion in the comprehension of science texts. Learning and Instruction 19 (228-242).

Paulson, E.J. & Henry, J. (2002). Does the Degrees of Reading Power Assessment Reflect the Reading Process? An Eye-Movement Examination. Journal of Adolescent & Adult Literacy 46 (234–244).

Pearson, P. D. & Hamm, D. N. (2005). The Assessment of Reading Comprehension: A Review of Practices – Past, Present and Future. In S. G. Paris & S. A. Stahl (Eds.), Children’s Reading Comprehension and Assessment (pp. 13–69). Mahwah, New Jersey: Lawrence Erlbaum Associates.

Raven, J.C. (1958). Standard Progressive Matrices: Sets A, B, C, D & E. Oxford, UK: Oxford Psychologists Press Ltd.

Raven, J. C., Court, J. H. & Raven, J. (1988). Standard Progressive Matrices: 1988 Edition. London, UK: Oxford Psychologists Press Ltd.

Rayner, K. (1992). Eye Movements and visual cognition: Scene perception during reading. New York: Springer-Verlag.

Rayner, K. (1998) Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124 (372–422).

Rothkopf, E. Z. & Billington, M. J. (1979). Goal-guided learning from text: Inferring a descriptive processing model from inspection times and eye movements. Journal of Educational Psychology 71 (310–327).

Salmerón, L., Vidal-Abarca, E., Mana, A., Martínez, T., Gil, L. & Naumann, J. (submitted). Reading Strategies in task oriented reading: The case of PISA-like tasks. Manuscript submitted for publication.

Solheim, O. J. (2011). The Impact of Reading Self-Efficacy and Task Value on Reading Comprehension Scores in Different Item Formats. Reading Psychology 32 (1-27).

Solheim, O. J. & Skaftun, A. (2009). The problem of semantic openness and constructed response. Assessment in Education 16 (149-164).

Tai, R. H., Loehr, J. F. & Brigham, F. J. (2006). An exploration of the use of eye-gaze tracking to study problem-solving on standardized science assessments. International Journal of Research & Method in Education 29 (185–208).

Valencia, S. W. & Pearson, P. D. (1987). Reading Assessment: Time for a change. The Reading Teacher 40 (726–732).

Vidal-Abarca, E., Mãná, A. & Gil, L. (2010). Individual Differences for Self-Regulating Task-Oriented Reading Activities. Journal of Educational Psychology 102 (817-826).