Qualitative Analysis

This analysis is part of the design process - 01 Learn & 05 Refine - of the Case Study Design for Learning - AI Research Problem.

#Survey #ThematicAnalysis #QualitativeData
The Case Study Design for Learning - AI research Problem is based on the problem space of How do we shift from a concept of tracking student's progress to start tracking student's proficiency?

To understand how users perception variate for progress and proficiency according to certain types of graphic representations. I conduct a survey study.

01 Learn - My Approach & Survey Process

Contextual Inquiry
(empirical process)
Users & Business Goals
Data Gathering
raw work activity data
organized & structure work activity data
Needs & Requirements
(deductive analytic process)
Design Informing Models
(e.g., flow models, usage scenarios)
(integrative process)
Contextual Analysis
(inductive analytic process)
Data Interpretation & Consolidation
Hartson, Rex, and Pardha Pyla. 2012. The UX Book. San Francisco, United States: Elsevier Science & Technology.
For a more detailed understanding, feel free to download the full survey report here.
Contextual Inquiry / Data Gathering
Survey Process

1. I gathered 15 different graphic representations normally attribute to the concept of progress and/or proficiency.

2. I asked the users how they perceive the respective graphic individually.

3. I gather all the graphic representation on a grid and ask which of them they thought to better represent the notion of progress and the notion of proficiency


Test for Distinction Bias.
Contextual Analysis / Data Interpretation & Consolidation
The most voted graphic representation for Progress
20,69% of users voted with a significant expression for this representation as to the most recognizable for tracking progress.
The most voted graphic representation for Proficiency
30,43% of users voted with a significant expression for this representation as to the most recognizable for tracking proficiency.
Needs & Requirements / Extration
The two concepts are already complex to distinguish, so upfront, I exclude all the graphic representation that didn't have a significant result. I didn't want to fall in the mistake of creating an ambiguous representation. Accordantly, I immediately excluded the following representation:
Final Design Decision
Survey Study showed us that users associate proficiency more to starts representations and percentages and pie charts to the concept of progress. Accordantly, I followed the following graphic representation for the CPA exam review application. On the left for tracking proficiency and on the right for tracking progress.

05 Refine - My Approach & Survey Process

Thematic Analysis
To begging analysis the exploratory research, I conduct a qualitative analysis to uncover themes in the data.

I start it by doing a thematic analysis to explore similarities and relationships between the different chunks of the survey data. The goal was gather more visual means of analysis to present to the stakeholders in the final session review of the design process.
Important Note: Visualization helps my team to understand better the similarities and relationships in the data more clearly to discloser better design and business decision.
Thematic Analysis Process

1. I like to print the results transcripts and start to highlighting important keywords and sections that are relevant to the study.

2. Next, I start cutting the paper texts sections that have been highlighted and start organizing them on an Affinity Diagram and making some side notes to them (descriptive labels).
Important Note: Ideally this process shouldn't be done alone. But I am a team of one and my stakeholders are on the other side of the ocean. I had to remedy the situation and conduct the analysis myself and be subject to my own bias.
For a more detailed understanding, feel free to download the full outline survey here.
3. I organize the data into blocks of themes based on their relationships, and start to figure out some categories. At the end I found 4 major themes:
User Interface (UI)
Note: This was a time consuming and difficult task because I had to analyze the feedback from 300 students. It was an interesting challenge to reduce to 4 categories.
4. Once I found my 4 categories, I start thinking about how these all relate to each other.

5. Since we are a remote team, I used Miro platform to build a visualized representation of the categories relationships to present to my stakeholders.

6. With the Product Owners we explore the results from the 4 points of view - Behavior, Algorithm, User Interface and Feature. We analyse the positive and negative feedback and come up with solutions and strategies to improve the feature efficiency and experience.
Important Note: For privacy and competitor reasons, I am not allowed to share this dashboard real content.
The findings from this analysis where implement in the Design Review in the design process phase 05 Refine on the Case Study Design for Learning - AI Research Problem. Follow the case study below.

Becker CPA. Future of eLearning, bringing AI to eLearning

Introduction of Artificial intelligence engine to augment student studying skills to improve their learning process.
Made on