This study illustrates how computer agents in conversational trialogues (tutor agent, peer agent, student) can track the performance and engagement of adult readers with low literacy skills in a 4-month intervention to improve reading comprehension strategies.   One out of six adults in the United States possess low literacy skills so technology can potentially help these adults improve. AutoTutor trialogues were designed to teach comprehension strategies across different levels of discourse processing and capture their performance, namely the time and accuracy of answering questions in the conversation.   A study with 52 adult literacy students interacted with AutoTutor as part of an intervention with human instructors. Performance in AutoTutor was tracked  at four theoretical discourse levels (Words, Textbase, Situation Model, Rhetorical Structure) and engagement, with an objective measure of comprehension skill before and after the intervention.  The results showed how AutoTutor could provide nuanced performance and engagement measures that predicted comprehension improvements and can be used to guide formative assessment and enhanced AutoTutor adaptivity.