AU Educate

Aarhus University Seal
 Teaching cases Can we trust GAI? Students’ Critical approach to GAI in qualitative research analysis

Can we trust GAI? Students’ Critical approach to GAI in qualitative research analysis

Short description

In this course on research methods, students learn to code qualitative data using theory and explore how GAI can support their coding work. They test various coding tools and compare GAI results with their own codes and analyses to assess the validity and quality of the findings.

Motivation

I use GAI daily in my work and see how the technology can both support and challenge the analytical process. In my teaching. I therefore aim to create a space where students can test, discuss, and assess how GAI can support theory driven coding of qualitative data. The goal is to strengthen their critical thinking and reflection on the use of GAI in their academic work and prepare them to navigate their future careers with confidence.

Learning objectives

The purpose of the activity is for students to:

  • Gain an introduction to how GAI can support theory driven coding of qualitative data.
  • Practice and reflect on how to integrate GAI into qualitative analysis processes.
  • Develop insight into the differences between AI generated and human codes and how these differences affect the validity of the analysis.

Execution

THE PROCESS

Preperation 

Preparation of materials. For the activity, I select a set of simple, mutually exclusive codes, such as Mortimer and Scott, that suit theory driven coding.

  • I also select a transcript of a dialogue as a qualitative case that allows students to identify key patterns in the data. The case spans two to three pages and complies with current GDPR guidelines.
  • I draft the prompt that students will use to guide their chatbot during the activity.
  • I place the case and the codes in separate documents, along with step-by-step instructions and the prompt for the activity.

In class

Lecture

Introductory session (20 minutes). I begin the class by explaining and demonstrating theory driven deductive data analysis.

Plenary

Introduction to the activity (10 minutes). I introduce the students to the activity and its purpose, and hand out the case and the documents that contain the step-by-step instructions and the prompt for the activity.

In pairs

Theory driven analysis (15 minutes.) Based on the introductory session, students conduct their own theory driven analysis of the assigned case.

  • Working in pairs, they code the case manually and identify patterns.
  • As they work, they discuss how the concepts from the theory shape their interpretation.

In pairs

Theory driven analysis with GAI (10 minutes). After completing the manual coding, students repeat the exercise with support from GAI.

  • They upload the case to a chatbot and enter the assigned prompt:
    • Prompt: "Analyze the document using Mortimer and Scott’s communication model. For each teacher question, specify the communication type: interactive dialogic, interactive authoritative, non interactive dialogic, or non interactive authoritative."
  • Then they compare the chatbot’s coding with their own coding.

In pairs/groups

Follow up reflection (10 minutes). As a follow up, students reflect on their results, identify differences in interpretation, and discuss methodological and ethical challenges.

  • To what extent does the exercise help them understand theory driven coding?
  • How does GAI support their understanding and practice of theory driven analysis?
  • To what extent does GAI serve as a trustworthy aid, or does it do the opposite?
Plenary

Final wrap up (10 minutes). During the final session, the class discusses the results and reflects together on how GAI can support qualitative analysis in academic work.

  • The groups compare their manual coding with the chatbot’s coding and analysis, which often reveals clear differences in both code types and interpretations. These differences stand out in data where students draw meaning between the lines that the chatbot fails to detect.
  • The class also examines variations in the chatbot’s output when they reuse the same prompt, which sparks a critical discussion about the chatbot’s consistency and reliability.

RESSOURCES FOR STUDENTS

SUPPORT FOR STUDENTS
  • Clear instructions and activity descriptions available on Brightspace.
  • Introductory lecture with a review and demonstration of theory driven data analysis.

Reflections

Outcomes

I taught four classes with a total of 126 students. The students completed anonymous TaskCards about their experiences.

  • All groups highlighted the value of testing manual coding.
  • Several students noted GAI’s potential to validate their own findings.
  • The chatbot’s results matched the groups’ findings only to a limited extent.
  • Repeated prompts with the same wording produced different results.
  • The students view GAI as a strong tool for testing ideas, including which theory best suits a case for analysis.
  • They do not consider GAI highly valid or reliable, but regard it as a supplementary tool for inspiration.

Challenges

  • A case for analysis must comply with current GDPR regulations.
  • The selected GAI model should ensure data security and must not use the case for training purposes.

Advices for other educators

  • If students understand inter rater reliability, groups can calculate a kappa value to measure agreement between their own codes and the chatbot’s coding.
  • Alternatively, they can calculate a simple percentage agreement. This approach offers a clear understanding of how to assess and interpret reliability in qualitative analysis, both with and without GAI.

Basic information

Educator Niels Bonderup Dohn 
Faculty and department ARTS, DPU 
Degree programme Generel pædagogik
Level of study Master
Course/subject Research methods
Number of students 30
Teaching format Small class teaching
Implementation Autumn 2024

Niels Bonderup Dohn

Associate Professor
M
H 1483, 628
P +4593522537
P +4593522537

Contact

Please contact the editors at AU Educate if you have any questions about the content of the platform or if you need consultation on your teaching from one of the many skilled professionals at the Centre for Educational Development. 

Revised 17.02.2026
-
Martin Bruun Hansen