We developed a course-specific chatbot called Phil to help in the Philosophy of Science course, based on the course’s syllabus and academic content. Throughout the course, students could communicate with Phil. Phil’s answers were only based on the course syllabus and could refer students to relevant sections in the syllabus.
Didactic challenges:
Teachers are unable to answer questions from all 550 students on the course.
Challenges of using conventional chatbots:
Hallucinations
Copyright
GDPR: No data processing agreement with chatbot providers
Refer to sources that don’t exist
LLMs like ChatGPT and CoPilot don’t have knowledge of the course-specific context
The Process | |||
Educator preparations | Building Phil the chatbot: Phil the chatbot combines RAG-technology (Retrieval-Augmented Generation) with a large language model such as ChatGPT (OpenAI). This integration makes it possible to retrieve relevant information from selected data sources, which is then combined with the language model to generate answers. Implementing this on the Azure platform requires a technical setup and programming to ensure students have stable and secure access to it.
| ||
Individually (In class) | Supporting students in class
| ||
Individuelt (out of class) | Supporting students during their independent study
| ||
RESOURCES FOR STUDENTS | SUPPORT FOR STUDENTS |
Access to Phil
| Teaching prompting techniques and good use of GAI tools. Examples of questions used to clarify technical terms and elaborate on articles could be:
|
Students interacted with the chatbot 20,000 times throughout the duration of the course. A follow-up survey was filled out by 77 students and found that:
91% said they wished other courses offered a similar tool.
76% preferred Phil over other chatbots (ChatGPT, Gemini or Copilot).
We coded 230 random answers from Phil and found that the answers were:
Not given, as there was no relevant material available 13.9%
Very useful 45.9%
Useful but not perfect 18.2%
Useful but inaccurate 10.4%
Incorrect and misleading 22%
Not properly answered 3.9%
Financial challenges: The costs associated with developing and operating Phil totalled DKK 4,500, including the subscription, test and setup.
Pedagogical challenges: You need to support and guide students in how to use chatbots correctly, including prompting methods and critical thinking.
Be aware that creating an RAG can be technically challenging and requires programming skills. If you don’t have the skills, ask for help from others in your department/school.
If you don’t have any programming experience, you can create your own chatbot app (GPT) in ChatGPT, or create a gem bot in Google Gemini. Here you can also upload material (for which you have copyright) and use a system prompt to guide the app on how to answer questions. Link to the chatbot can be shared with students. This allows you to shape the answers students get from the chatbot, although it’s not as delimited to the specific topic as Phil is.
Please contact the editors at AU Educate if you have any questions about the content of the platform or if you need consultation on your teaching from one of the many skilled professionals at the Centre for Educational Development.
