Methodology Vectors by Vecteezy
In this post I will outline the methodology I will implement for this project, the data types that will be collected, and the associated techniques to analyse the data. The Voice Learning Assistant (VLA) requires drawing from the domains of Education, Computer Science, Psychology and human-computer interaction. As such, the methodology must be selected to deal with this complexity whilst providing rigorous and accurate evaluation.
Chosen Methodology
The VLA will employ diagrams, dynamic prompts, mastery logging and user experience feedback which will naturally produce both qualitative and quantitative data. Since both use experience and learning progress are crucial for the success of this project, I have selected the Concurrent Triangulation Mixed Method for this use case as it collects both types of data simultaneously and treats them equally important. My aim is to analyse the two data sets independently and look for patterns and then identify if the patterns in one data are supported by the other. For example, if the quantitative data shows high student progress through a particular learning section and the qualitative data shows the students found that section to be intuitive and appropriately pitched, the corroboration between the two provides strong evidence for system efficacy (Hurst, 2023).
This methodology will work well with my development flow of prototype, alpha test, beta test and pilot as both forms of data will be collected concurrently and will offer a more complete picture of overall system performance.
Quantitative Data
The VLA will store numerical representations of user interactions. Currently, I have planned for the following data points:
-
- Progress scores based on student performance before and after interactions.
- Mastery tracking and progression.
- Task accuracy.
- Student response times.
- System error counts.
- Abandoned task counts.
- Number of prompts, hints or retries to produce a correct answer from the student.
- Timestamps and use duration.
Once collected, I plan to use descriptive statistics to gain insight into performance and engagement before applying inferential techniques. To deepen the evaluation of the system I will then use paired tests, ANOVA, correlation analysis and chi-square to explore progress, task difficulty and meaningful differences between users. Furthermore, I plan to use learning curves, distributions of response times and sequential interaction analysis to understand navigation and behaviour patterns.
Together, these analyses will provide a strong and reliable picture of both learning outcomes and student behaviour which can then be used with the qualitative data I will collect through feedback, observations and interviews (National University, 2023).
Qualitative Data
Alongside the quantitative data, I will be gathering user experience data from the people who actually use the system. This will include talking to users in interviews, observing interactions to identify positive and negative interaction, and collecting comments about how well the explanations and student prompts support learning. Furthermore, survey responses will add greater breadth to this picture. I plan to analyse these qualitative insights using Braun and Clarke’s Thematic Analysis to identify recurring patterns in how learners interpret the system, supported by content analysis to help to quantify common issues or themes. These findings will then be triangulated with the quantitative data to help explain unexpected results, connect specific behaviours in the system logs to user experiences, and identify behavioural explanations. Together, the complete data set will give a richer picture of system performance and learning progress (National University, 2023).
Data Integration
MY goal is to collect both data types concurrently to give real time feedback on the system performance to support iteration. By integrating the two forms of data, I aim to not only validate the system but to build a holistic picture. I expect to see qualitative responses that identify issues or successes that then show up in the quantitative data and quantitative trends that are explained by the qualitative responses. These finding will then be used to refine the system to improve both performance and user experience.
Problem-Orientated Interdisciplinary Research
As mentioned earlier, this project draws on four main fields of research which are integrated rather than parallel to one another. For example, in this project theory from the Education field such as pedagogy will have a direct impact on the prompts designed for the VLA's AI. The components of the project fall into the four categories as follows:
-
- Computer Science: AI, data structures, storage, dynamic prompting.
- Education: Pedagogy, student prompting (instruction), mastery framework, cognitive load (from Educational Psychology).
- Psychology: Student motivation, behaviour and cognition.
- Human-Computer Interaction: User experience, usability, accessibility and voice interaction.
The aim of this research project is to produce a functional system, that provides support to students in learning Key Stage 3 Science through the use of an adaptive AI VLA.. It does not aim to create a new theoretical framework or advance existing theory. As such, this project is best described as Problem-Orientated Interdisciplinary Research (Schmidt, 2011) (Woollen, 2020).
Final Thoughts
The Concurrent Triangulation methodology selected will offer strong data for evaluating the system performance as the project progresses and allows for the correct inclusion of performance data, behavioural logs, user feedback and observations. Identifying the project as Problem-Orientated Interdisciplinary Research correctly captures the project aim which is to bring together theory from different fields and integrate it to produce a solution that solves a real problem for many learners.
Thanks for reading and please check back for the next post.