Evaluating Retrieval, Grounding, and Refusal in a Citation-Grounded Local RAG Teaching Assistant

Abstract

Learner-centered education requires students to access course knowledge independently while retaining the ability to verify, question, and trace the information provided by AI-supported learning tools. Citation-grounded retrieval-augmented generation (RAG) can support this goal by providing on-demand explanations grounded in instructor-approved course materials and by directing learners to the original evidence rather than presenting generated responses as authoritative answers. To examine whether such support can be delivered reliably, this study evaluates retrieval, grounding, citation correctness, and refusal behavior in a locally deployed, citation-grounded RAG teaching assistant for university course materials. The implementation combines OCR-aware ingestion, document- and page-level provenance, sentence-aware chunking, local semantic retrieval, evidence-constrained generation, deterministic refusal normalization, and instructor-facing observability. Two full textbooks, Deep Learning and Power Systems Analysis, were indexed as 2,293 chunks. A source-grounded benchmark contains 60 questions: 52 answerable questions with validated multi-page evidence and eight unsupported questions. With Top-5 retrieval, validated Chunk Hit@5 and Page Hit@5 reached 71.15% and 75.00%, with chunk and page MRR of 0.5135 and 0.5375. Manual evidence-grounded review of end-to-end responses yielded 81.82% normalized correctness among generated answers, 86.36% groundedness, and 86.36% citation correctness. The system answered 42.31% of answerable questions; importantly, 19 of 52 questions (36.54%) were refused even though validated evidence was already present in the retrieved Top-5. Conversely, all eight unsupported questions were correctly rejected with no false answers or citation leakage. The results expose a measurable safety–coverage trade-off: conservative refusal improves unsupported-question safety, but evidence-sufficiency calibration is necessary to avoid suppressing answerable questions.