Although mind wandering during reading is extremely common, researchers have only recently begun to study how it relates to reading behavior. In the present study, we used a word-by-word reading paradigm to investigate whether it could be possible to predict in real time whether a participant would report mind wandering when probed. By taking advantage of the finding that reaction times to individual words vary based on reports of mind wandering (with participants being less affected by length, number of syllables, and familiarity, and also showing an overall speed-up, during mindless reading), we were able to develop an algorithm that could successfully predict in real time whether a participant would report being on versus off task. In addition, for participants run without thought probes, there was a significant negative correlation between the number of predicted mind-wandering episodes and reading comprehension. Together, these findings offer a key advance toward the development of pedagogical tools for minimizing the negative impact of mindless reading, and they provide a new covert measure that could be used to study mind wandering without requiring participants to report on their mental states.