Supporting Active Learning: Implementing a Socratic AI Tutor in a First-Year Economics Module

Context: First-year Economic Principles module; ~900 students; Spring Term 2025-26 academic year; Socratic AI tutor embedded in virtual learning environment (VLE); 291 baseline survey respondents; 65 post-implementation survey respondents.

Design approach: Curriculum-bounded knowledge base; Socratic dialogue; Adaptive scaffolding.

The challenge

By early 2025, the question facing economics educators was no longer whether students were using generative AI (GenAI) in their studies, but rather how they were applying these tools in their study practices. A baseline survey of 291 first-year students revealed that almost 91% were already using AI tools in their studies, with around 81% identifying ChatGPT as their preferred platform. Students reported using GenAI to summarise readings, define key concepts, and generate revision materials. Many of these GenAI uses are potentially beneficial, but the unguided use of AI by students can undermine learning even when students find it helpful (Bastani et al., 2025).

This presented us with an important pedagogical challenge. In economics, a solid understanding of content often depends on the process of knowledge scaffolding which requires working through concepts, building on assumptions, and testing causal relationships. This reinforced the case for deliberate pedagogical design. Our aim was not to provide students with another chatbot that simply provided direct answers to basic prompts. Instead, we wanted to explore whether an AI tool could be designed to support the kinds of questioning, explanation, and reflection that underpin the effective learning of economic content and whether such a tool could work at the scale of a 900-student cohort where individualised tutorial support is inherently limited (Bloom, 1984).

Why we designed a Socratic AI Tutor

We made three deliberate design choices, each responding to a specific risk we identified in unguided student AI use.

1. Curriculum Alignment Through a Module-Specific Bounded Knowledge Base

Students often express concerns about the reliability of AI-generated outputs. To address this concern, we designed the AI tutor to operate within a "walled garden", i.e. it had no internet access and could draw only on an approved module-specific knowledge base. These resources included lecture slides, PDF extracts of the textbook, tutorial worksheets and activities, assessment briefs, and previous examination materials. The intention was to ensure that tutor’s responses were aligned in terms of notation, graphical conventions, and simplifying assumptions with what students were expected to learn within the module.

2. Supporting Economic Reasoning Through Socratic Dialogue Approach

Instead of supplying immediate answers, we designed the tutor to ask guided questions and prompts to encourage students to think through problems step-by-step. This design was informed by Socratic approaches to teaching, which aim to promote active reasoning, reflection, and deeper understanding through structured questioning (Elder & Paul, 1998; Paul & Elder, 2007).

3. Adaptive Scaffolding for a Diverse Cohort

In a cohort of approximately 900 students, prior knowledge and confidence levels vary considerably. We, therefore, designed the tutor to provide graduated support, offering increasingly detailed prompts and explanations as and when needed. This aimed to deliver something closer to the personalised support that one-to-one tutoring provides (Bloom, 1984).

Figure 1: Linking Design Choices to Intended Learning Processes (click to expand)

These three principles reflect the broad goal of this study: to incorporate an AI tutor into our module to help students learn through Socratic questioning and knowledge scaffolding (See Figure 1 for our pedagogical model).

Tutor implementation

The tutor was built using the Learnwise platform and embedded directly within the module's Virtual Learning Environment (VLE), allowing students to access it alongside existing learning materials. It is important to note, however, that other available platforms supporting custom knowledge bases and Socratic prompting would be suitable for similar AI tutor integration.

Prior to the launch, we curated resources, uploaded supporting materials, and conducted preliminary testing of the tutor. During this process, two important limitations emerged. First, we found that the system was providing direct answers to questions and had not been set up to follow the ‘Socratic’ approach we had specified. Second, the tutor struggled with some forms of mathematical notation. These issues required some refinement before we released the tutor to students.

Due to procurement challenges, we were able to release the tutor to students only in Week 7 of an 11-week term (a delay that, as we discuss below, had significant consequences for adoption). We provided launch support through written guidance on the use of the AI tutor and regular email encouragement to incorporate the tutor into students’ revision sessions. Its release was also supported through bespoke activities such as in-class demonstrations and AI tutor prompting workshops. During these AI workshops, we provided students with the opportunity to build their skills in applying the AI Socratic reasoning technique to answering real-world economic problems. Students could begin to build their understanding of the subject content and scaffold this knowledge at their own pace.

It was noted during these workshops that the tutor could not provide graphical and diagrammatic representations of economic content. While initially unexpected, we communicated this to students explicitly, positioning the tutor as a tool for conceptual reasoning and revision support and was not a replacement for graphical or quantitative analysis.

Student interactions followed five stages: access the tutor; ask a question; the system retrieves information only from approved resources; receive a guided response; and continue through iterative dialogue. These interactions were private and staff deliberately decided to not have access to individual conversation histories. Students were informed of the university’s data governance policies and advised that any well-being queries should be directed to relevant university support services.

What students did and didn’t do

The most significant challenge we faced was not student satisfaction, but rather student adoption. Although 57% of the students surveyed reported accessing the tutor, 43% did not use it at all (n = 65 respondents).

Among those who did engage, however, engagement was largely sustained: around 66% of them used the tutor at least weekly, approximately 20% reported daily or multiple-daily use, 85% rated their overall experience positively, and most said they would recommend it to other students.

The more informative data came from those who did not adopt. The most frequently reported reasons were a lack of awareness of its existence (32%), a belief that they did not need additional support (25%), and continued reliance on existing AI tools, including a paid version of ChatGPT (25%). Surprisingly, dissatisfaction with the AI tutor itself was rarely the primary reason for non-adoption.

This shifted our thinking considerably. We had initially approached implementation as a technology adoption challenge. In practice, it proved equally to be a behavioural and pedagogical challenge. By Week 7, students had already developed study routines around commercial AI platforms, and introducing a new more constrained tool required changing established habits – something many students were evidently not willing to do (without a compelling reason to switch).

Lessons learnt

The success of the AI tutor depends at least as much on timing, communication, and curriculum integration as on the technology itself.

• The knowledge environment is the pedagogy: The "walled garden" only works if the content inside it is pedagogically sound, comprehensive, and tested against disciplinary needs. The quality of the curated knowledge base determined the quality of every student interaction, which meant that resource preparation was not a one-off task but an ongoing commitment.
• Test for discipline-specific needs: Economics presents distinct challenges for AI tools, particularly around graphs, diagrams, mathematical notations, and quantitative reasoning. While the tutor performed well for conceptual explanation and revision support, graphical analysis remained a clear limitation.
• Timing and visibility matters: Launching in Week 7 was too late. By that point students had already developed preferred ways of studying and established relationships with alternative GenAI tools.
• Position the tool educationally (and not technologically): Students naturally compared our AI tutor with commercial GenAI tools, for example, ChatGPT and Claude and on features alone, our AI tutor lost. Institutional tools are unlikely to compete in the context of tool capability and feature profiles. For example, our AI tutor was more constrained, could not produce diagrams, and sometimes offered questions where students just wanted answers. The case for adoption has to be pedagogical: this tool helps you understand, not just answer.
• Awareness cannot be assumed: Promote this through as many channels as you can and integrate the tutor in weekly learning activities.
• Keep human expertise visible: We found that the tutor worked best when positioned explicitly as a complement to teaching rather than a substitute for it, and when the boundaries between AI capability and human expertise were stated clearly. Questions requiring judgement, personalised feedback, quantitative reasoning, or nuanced discussion continued to benefit from human interaction. Students appeared most comfortable and most willing to persist with the tutor when those boundaries were explicit rather than discovered through frustrating interactions.
• The Staff Workload Paradox: Building the knowledge environment is the foundational investment and that the tutor's pedagogical credibility with students depends on the accuracy and specificity of the content it draws upon. As a result, the implementation of the AI tutor required considerable investment on staff’s part in resource preparation, system configuration, testing and workshop development. While we hoped that this would lessen our workload through personalised learning at scale, this was not the reality.

Recommendations

Final reflection

The most valuable lesson from this study was not about the technology. It was about the gap between building an AI tutor and integrating it into students’ learning practice. This was a gap that proved wider and more consequential than we had initially anticipated.

It is important to remember that this was a single implementation within one first-year economics module, and findings are based primarily on modest survey responses and student perceptions. The project was also not designed to measure causal effects on learning outcomes or examination performance. As a result, the evidence presented here is exploratory but provides a starting point for understanding student engagement with module-specific AI tools.

What we can report is that there is great potential in how the AI tutor supported conceptual explanations, provided formative questioning, and reinforced independent study between formal teaching sessions. These are key areas where AI tutor can complement teaching effectively, especially around economics education. Human expertise, however, remains essential for judgement, feedback, quantitative reasoning, and helping students make sense of complex economic ideas. The transferable elements for other economics modules are the three design principles of curriculum-bounded content, Socratic dialogue, and adaptive scaffolding. Perhaps and more importantly, the practical lessons about timing, communication, and the substantial staff investment that AI tutor integration demands.

For economics educators, the question remains: how best can we integrate AI into the curriculum in ways that strengthen rather than replace the reasoning processes through which students learn economics? This case study offers one practical (maybe imperfect) answer as well as a candid account of the implementation realities that any department considering an AI tutor should expect.

References

Bastani, H., O. Bastani, A. Sungu, H. Ge, O. Kabakci, and R. Mariman. 2025. “Generative AI without Guardrails Can Harm Learning: Evidence from High School Mathematics.” Proceedings of the National Academy of Sciences 122 (26): e2422633122. https://doi.org/10.1073/pnas.2422633122 .

Bloom, B. S. 1984. “The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring.” Educational Researcher 13 (6): 4–16. https://doi.org/10.3102/0013189X013006004

Elder, L., and R. Paul. 1998. “The Role of Socratic Questioning in Thinking, Teaching, and Learning.” The Clearing House 71 (5): 297–301. https://doi.org/10.1080/00098659809602729

Paul, R., and L. Elder. 2007. “Critical Thinking: The Art of Socratic Questioning.” Journal of Developmental Education 31 (1): 36.

Author notes