Digital Transformation in Higher Education: The Impact of LLM-Powered Chatbots on Student Support

CoE-EDP, VisionRICoE-EDP, VisionRI | Updated: 17-06-2024 12:12 IST | Created: 17-06-2024 12:12 IST
Digital Transformation in Higher Education: The Impact of LLM-Powered Chatbots on Student Support
Representative Images

Researchers Achraf Hsain and Hamza El Housni from Al Akhawayn University, Morrocco explore integrating GPT-3.5 and GPT-4 Turbo-powered chatbots into higher education to enhance internationalization and digital transformation. The study focuses on the design, implementation, and application of Large Language Models (LLMs) to improve student engagement, information access, and support. The research highlights the chatbot's development using Python 3, GPT API, LangChain, and Chroma Vector Store, emphasizing the creation of a high-quality dataset for chatbot testing. Findings show the chatbot's efficacy in providing comprehensive responses, its preference by users over traditional methods, and a low error rate. The study demonstrates the chatbot's potential to elevate accessibility, efficiency, and satisfaction in higher education by offering real-time engagement, memory capabilities, and critical data access.

Revolutionizing Student Support with Digital Technology

Digital technology is increasingly critical in supporting diverse student populations amid higher education's internationalization. LLM-based chatbots, leveraging GPT-3.5 and GPT-4, are seen as revolutionary tools for redefining student support and engagement. The chatbot's design includes a robust technological stack using Python 3 and various APIs and libraries for real-time data processing and retrieval. The literature review covers several dimensions, including the internationalization of higher education, the role of technology in this process, the implementation of chatbot technology, existing chatbot solutions for international student support, and gaps in current research. It highlights the transformative potential of AI technologies in education, focusing on the opportunities and challenges they present.

Innovative Methodology and Technological Stack

The methodology details the technological stack, including Python 3, LangChain, Chroma DB, and OpenAI embeddings. Two approaches for data collection were considered: scraping entire web pages and multiple PDFs or manual data scraping from the Al Akhawayn University (AUI) website. The final approach involved manual data scraping due to the website's relatively small size and the clustering of relevant data on specific pages. The chatbot's design ensures it can conduct context-aware conversations, drawing from a comprehensive database of relevant information. Data was stored in multiple text files and divided into embedded documents in the vector store, with each document having an average length of 1500 characters. The chatbot pipeline begins with receiving the user query and conducting a similarity search with all documents in the vector store, retrieving the five most relevant documents for the first prompt. The chatbot was trained to simulate an assistant at the OIP Department at AUI, tasked with addressing queries from international and Moroccan students.

Comprehensive Evaluation Metrics and Testing Procedures

Evaluation metrics used to measure the chatbot's performance include correctness, quality, relevance, formality, human-likeness, translation capabilities, and provocation proof. These metrics provide a comprehensive framework for assessing the chatbot's effectiveness in delivering accurate, engaging, and contextually appropriate responses. Testing procedures involve four distinct test sets: general capabilities, provocation and sensitivity, information retrieval, and multilingual capability. These tests evaluate the chatbot's proficiency in handling various queries, maintaining professionalism, retrieving relevant information, and providing accurate translations. The results reveal the chatbot's successful translation capabilities, resilience against provocative queries, and ability to retrieve relevant information. The survey evaluation among AUI students shows high ratings for the chatbot's quality, relevance, correctness, formality, and human likeness.

Promising Future for LLM-Based Chatbots in Education

Bootstrapped confidence intervals for these metrics indicate strong performance, with the chatbot scoring around 4 out of 5 across all metrics. The chatbot is capable of producing rich and engaging responses, successfully understanding user questions, avoiding hallucinations, maintaining professionalism, and emulating human-like conversational patterns. The chatbot's memory feature allows for context-aware conversations, enhancing its ability to provide meaningful and relevant responses. The discussion emphasizes the chatbot's potential to enhance student support services in higher education, particularly for international students. The positive reception by AUI students suggests that such technologies can be cost-effectively implemented by other universities to improve information retrieval and student support. Future work should focus on larger-scale testing, expert evaluation of translation capabilities, expanding the dataset, and refining evaluation metrics.

The study demonstrates that LLM-based chatbots, integrated with a thoughtful technological stack, significantly enhance student support services, particularly for international students. The chatbot's architecture, combining technologies like Python 3, GPT API, LangChain, and Chroma Vector Store, ensures rich, engaging, and accurate interactions. The study advocates for the broader adoption of such technologies to foster more inclusive and accessible learning environments in higher education. By utilizing advanced AI tools, educational institutions can better support international students, addressing their unique needs and challenges more effectively. This integration of LLM-powered chatbots represents a significant step towards modernizing student support and enhancing the overall educational experience for a global student body.

  • Devdiscourse
Give Feedback