Agenda
Company
Becker Professional Education. A global professional education company that has been the market leader in accounting exam preparation for the past 60 years.
Becker Professional Education. A global professional education company that has been the market leader in accounting exam preparation for the past 60 years.
Product
CPA Exam Review - An online eLearning tool to prepare accounting professionals for the CPA exam to gain their certification.
CPA Exam Review - An online eLearning tool to prepare accounting professionals for the CPA exam to gain their certification.
Deliverable
New Feature
Timeline
8 months
New Feature
Timeline
8 months
Role
UX generalist who covers the entire design process
Main Tasks
User Research, Data Analysis, User Experience &
User Interface
UX generalist who covers the entire design process
Main Tasks
User Research, Data Analysis, User Experience &
User Interface
Design for Learning - Bringing AI to eLearning
Case Study Summary
Becker's Professional Education is moving from a fit-to-all learning path experience towards a personalized and adaptive experience that tailors the learning experience to the individual needs of each student.
To keep the company at the forefront and leverage technology to increase course completion and improve student engagement, Artificial Intelligence (AI) and Machine Learning (ML), emerged as the most natural solution - adaptation to real-time data and students' needs.
The main challenge of this feature was moving from tracking progress towards tracking proficiency. How do we measure proficiency?
To keep the company at the forefront and leverage technology to increase course completion and improve student engagement, Artificial Intelligence (AI) and Machine Learning (ML), emerged as the most natural solution - adaptation to real-time data and students' needs.
The main challenge of this feature was moving from tracking progress towards tracking proficiency. How do we measure proficiency?
Becker Professional Education
Becker offers educational resources for professionals in the areas of accounting, finance, and project management. It is the largest provider of training for candidates who are preparing to sit for the United States CPA Exam in order to become Certified Public Accountants.
Problem
In the past few years, Beckers has lost market share. To turnover this issue and leverage technology to increase course completion and improve student engagement, we change the CPA exam review strategy from
Moving from a fit-to-all learning path experience towards a personalized and adaptive learning experience for each individual needs.
GOAL
Provide a more efficient and personalized course experience.
Solution
We build a new feature called, Personalized Review Sessions (PRS).
It delivers the optimal next step of remediation for each student, in real-time.
Support by AI engine, it does:
It delivers the optimal next step of remediation for each student, in real-time.
Support by AI engine, it does:
- Identify the knowledge gaps specific to each learner
- Trace these back to their root cause
- Reads students' assessment tests and tailors a program of study personalized to them
- Maximize their study time and pain points
Project Scope
We partner up with SanaLabs, an AI company specialized in personalized education to build the next state of the art learning experience for professional education.
With SanaLabs technology within the course, Becker's CPA is now able to tracks every student interaction to generate a unique review experience for each student according to their individual weaknesses and learning needs.
SanaLabs achieves this by utilizing algorithms proven to be extremely accurate in predicting future mistakes. Leveraging spaced-repetition techniques, Sana's algorithms also ensure long-term knowledge retention in preparation for the final exam.
With these new technological possibilities, we were able to do the one step further towards our competitors and start using our users' behavior and real-time data to create highly contextual communication that is relevant to the user study proficiency.
With SanaLabs technology within the course, Becker's CPA is now able to tracks every student interaction to generate a unique review experience for each student according to their individual weaknesses and learning needs.
SanaLabs achieves this by utilizing algorithms proven to be extremely accurate in predicting future mistakes. Leveraging spaced-repetition techniques, Sana's algorithms also ensure long-term knowledge retention in preparation for the final exam.
With these new technological possibilities, we were able to do the one step further towards our competitors and start using our users' behavior and real-time data to create highly contextual communication that is relevant to the user study proficiency.
TEAM: Me and an Agile team comprised by 3 Developers, 2 Product Managers, 1 Software Developer Manager, and 4 Quality Assurance Engineers.
I played three roles
Management: I was responsible for determining the overall design direction; I define the design sprints; I wrote and managed Jira design stories and tasks. I was part of the design review sessions.
User Research: This project raises a lot of new technologies and design challenges. User research had to be constant in this project because a lot of decisions had to be made upon user research insights.
Product Designer: I collaborate with the team on the ideation process. I analyzed users' insights for both qualitative and quantitative (factor analysis) and made my design decision based on that. Developed the needed prototypes and then tested them with users.
User Research: This project raises a lot of new technologies and design challenges. User research had to be constant in this project because a lot of decisions had to be made upon user research insights.
Product Designer: I collaborate with the team on the ideation process. I analyzed users' insights for both qualitative and quantitative (factor analysis) and made my design decision based on that. Developed the needed prototypes and then tested them with users.
Key Problem
How to shift from a concept of tracking user's progress to start tracking user's proficiency?
Constraints
- Outsourcing the AI algorithm development to a third party.
- A design team of one.
- Only remote access to users.
- The feature had to run in an external application provided by AICPA. With a user interface, we cannot control or make changes.
Results
- 88% of the student-user Beta group felt that the Personalized Review Sessions improved their efficiency in studying for the exam.
- 92% of the students felt that they studied more effectively with personalized review sessions. Students also felt empowered by a newfound awareness of their knowledge gaps at a granular level.
My Approach
Contextual Inquiry
(empirical process)
(empirical process)
Users & Business Goals
Data Gathering
raw work activity data
organized & structure work activity data
Needs & Requirements
(deductive analytic process)
(deductive analytic process)
Extraction
Design Informing Models
(e.g., flow models, usage scenarios)
(e.g., flow models, usage scenarios)
Extraction
DESIGN
(integrative process)
(integrative process)
Contextual Analysis
(inductive analytic process)
(inductive analytic process)
Data Interpretation & Consolidation
Reference: The UX Book. By Hartson, Rex, & Pardha Pyla. 2012.
Process01 Learn. Gain knowledge of users, context, and technologies. Gather user data, and surveys, research competitive products, conduct stakeholder interviews. 02 Explore. Build user profiles on gathered data, produce materials that will aid the outlining of the project, site maps, content inventories, screen flows, navigation models, task flows, user journeys, scenarios. 03 Select. Evaluate, test, and select wireframe concepts for prototype development. 04 Develop. Create design specifications and evolve concept/wireframes into a full design solution. 05 Refine. Evaluate design with stakeholders to obtain feedback, conduct usability testing, and surveys. 06 Deliver. Complete design and produce deliverables. |
|
The most voted graphic representation for Progress
This graphic representation was the most voted (21%) by the users as the most recognizable to track progress.
The most voted graphic representation for Proficiency
This graphic representation was the most voted (31%) by the users as the most recognizable to track proficiency.
Interesting Fact: Our users are all accountants. So, I found out with this study that they have an intrinsic need for every graphic representation to be followed by numbers, results, or percentage description. They just love numbers!
|
Design Informing Models / Flow Models & Usage Scenarios
Becker's Content
CPA course has the following content structure:
As mentioned in our constraints,
we need to create a feature that could run in an external application with the following user interface.
I define the following user flow for the feature.
Important Note: The long-term goal is to apply the AI personalized experience to all levels of study content. But for the first phase, the feature was developed to run on a content Unit level.
|
Design / Wireframes & Prototyping - Interactive Process
Until we migrate the whole education system oriented to proficiency rather than progress, we need to have a hybrid experience between both. For Courses and Sections levels we still need to track progress. And we do it through the following graphic representations:
To not conflict with the current percentage progress tracking, with the support of the survey study results, a star representation system emerged as the most logical approach to represent proficient.
Problem: By exploring this representation I understand that a 5-star system was getting the UI too crowded.
Refine: I simplified it to a 3-star system with a half star achievement.
Problem: By exploring this representation I understand that a 5-star system was getting the UI too crowded.
Refine: I simplified it to a 3-star system with a half star achievement.
Badge System
Now we needed a more visually intuitive way to communicate the student level of proficiency. I explored several options and figure out that a badge approach was the most appropriate for the context. And we develop the following system.
Problem
I tested this concept and quickly learn that for a student earn a half-star he needs to work on several Multiple Questions (MCQs) or Simulations. And students replied that they needed more immediate feedback if they were progressing or not. So, I refine the design to this approach.
Refine
I included a kind of experience bar. Every time a student answered a question the bar would increase if the answer were right and decrease if the answer were wrong. Every time a student fill up a bar, he earns a half-star, until it reaches the 3 stars proficiency. Nonetheless, this kind of podium stage is not guaranteed. If the student answered wrongly to a question that he already accomplished 3 stats he can lose proficiency on it. Because we are evaluating knowledge not progress. That's why he may lose proficiency.
Proficiency Tracking
The review session uses badges containing progress bars and proficiency stars to track students proficiency within an unit. The badges report the module number or unit number and are color-coded based on the level of proficiency. Additionally, the progress bars within the badges fill as they answer questions correctly and empty and turn red as they answer questions incorrectly. The proficiency stars also fill by halves as you fill the progress bars.
The review session uses badges containing progress bars and proficiency stars to track students proficiency within an unit. The badges report the module number or unit number and are color-coded based on the level of proficiency. Additionally, the progress bars within the badges fill as they answer questions correctly and empty and turn red as they answer questions incorrectly. The proficiency stars also fill by halves as you fill the progress bars.
Important Note: To build an efficient AI personalized experience, first, the algorithm needs to collect data and learn from student-based knowledge. So, it would be reckless to encourage the student to jump in directly to the feature. Thus, we decide to place feature at the end of the Unit learning path.
|
With PRS
Students can now review their weaknesses.
Students can now review their weaknesses.
How does it work?
Until the student finishes a Unit learning path, the algorithm will be tracking their performance and knowledge.
Reaching the end of the path, the student can launch a study session with Multiple Questions and Simulations (the two types of exercises that appear in the AICPA exam) and work on his weakness.
The PRS came with a Proficiency Dashboard where they can see an overall picture of their proficiency by the Module knowledge level and by the overall Unit knowledge level. If the student didn't study any content for a Module, the badge will appear greyed down, since the algorithm didn't collect any information yet.
Until the student finishes a Unit learning path, the algorithm will be tracking their performance and knowledge.
Reaching the end of the path, the student can launch a study session with Multiple Questions and Simulations (the two types of exercises that appear in the AICPA exam) and work on his weakness.
The PRS came with a Proficiency Dashboard where they can see an overall picture of their proficiency by the Module knowledge level and by the overall Unit knowledge level. If the student didn't study any content for a Module, the badge will appear greyed down, since the algorithm didn't collect any information yet.
Important Note: We prepared on-boarding resources for students'. A pdf guide and a video tutorial on how the feature works.
Review Session
The PRS will accommodate their learning needs with targeted practice questions so that they can deepen in their understanding of the unit's key concepts.
Students can decide how many questions they would like to answer in each review session. Although students often asked for less than 20 questions during the testing sessions, we tested, and concluded that a pre set of 20 questions would be the least a student can do to concretely see any proficiency evolution. So, the default launch, is a session of 20 questions but the student can change it up until 100. Launching the PRS session the student will now see the AICPA external tool.
The feature will serve them a mixture of MCQs and TBSs that the AI technology has selected for them in light of they performance within the course.
The PRS will accommodate their learning needs with targeted practice questions so that they can deepen in their understanding of the unit's key concepts.
Students can decide how many questions they would like to answer in each review session. Although students often asked for less than 20 questions during the testing sessions, we tested, and concluded that a pre set of 20 questions would be the least a student can do to concretely see any proficiency evolution. So, the default launch, is a session of 20 questions but the student can change it up until 100. Launching the PRS session the student will now see the AICPA external tool.
The feature will serve them a mixture of MCQs and TBSs that the AI technology has selected for them in light of they performance within the course.
AICPA GUI
We adapted the UI tool to the following design.
As they answer each question, the course tracks whether their proficiency has increased or decreased. And with this information, the course continues to assemble the optimal mix of practice questions for the review session.
Each question within the review session indicates the CPA Exam skill level at which it is testing them. The question also comes with an explanation as to why the course provided that question to them.
Before a student answer each question in the review session, we decide to provide the possibility of watching the lectures discussing the topics tested by the practice questions so that they can ensure that they are mastering the information.
Then, once they have answered all of the questions in the review session, they can go through a detailed list of the questions that appeared within that module. From this dashboard, they can choose to return to particular questions so that they can read the in-depth explanations about why the correct answer was right and the incorrect answers were wrong. If they want, they can also work through additional review sessions to become even more proficient with the exam topics.
Each question within the review session indicates the CPA Exam skill level at which it is testing them. The question also comes with an explanation as to why the course provided that question to them.
Before a student answer each question in the review session, we decide to provide the possibility of watching the lectures discussing the topics tested by the practice questions so that they can ensure that they are mastering the information.
Then, once they have answered all of the questions in the review session, they can go through a detailed list of the questions that appeared within that module. From this dashboard, they can choose to return to particular questions so that they can read the in-depth explanations about why the correct answer was right and the incorrect answers were wrong. If they want, they can also work through additional review sessions to become even more proficient with the exam topics.
|
The Impact of My Work
After finishing this first MVP, we conduct a Beta A/B Test with ~3 000 students. The AI algorithm needed to learn from individual users but also from the collective usage. To give time to test the algorithm, we launch the A/B test for three months and the end surveyed the students.
I learned several aspects
1. Problem: The students felt that they didn't control the feature. The feature was controlling them with poor communication on why it was making certain actions on behalf of them.
Solution: I decide to incorporate in the question header, what kind/type of questions was that and why the AI decide to present it. With a tooltip explaining what the category means according to the AICPA parameters. E.g:
Solution: I decide to incorporate in the question header, what kind/type of questions was that and why the AI decide to present it. With a tooltip explaining what the category means according to the AICPA parameters. E.g:
Another aspect was "The PRS didn't seem to account for how many tries it took me to get MCQs right". So, we decide to include additional information.
2. Problem: Keeping the original AICPA timer came up problematic. A lot of students expressed that they were struggling with the exam clock. They didn't see the value and was distracting to them. We had this clock just because we were mimic the real exam. But this was harming the students. Students express that they wanted to pause the time well they look for the solution or explanation. And also, the clock was also conflicting with the question timer.
Solution: We decide to remove the AICPA exam timer and focus only on the question timer. The question timer is important because, for example, in the real exam the student has an average of less than 2 min to answer a multiple questions. So, I design the following solution, incorporating a tooltip to explain the feature and a pause time action.
Solution: We decide to remove the AICPA exam timer and focus only on the question timer. The question timer is important because, for example, in the real exam the student has an average of less than 2 min to answer a multiple questions. So, I design the following solution, incorporating a tooltip to explain the feature and a pause time action.
3. Problem: Several student's express things like "If the AI behind the program could more accurately figure out what I needed to study, this would be a great tool "
Solution: I redesign the Performance Dashboard at the end of the session in order to provide more detailed information to guide the student study. I want to give the flexibility to see a quick summary or a detailed report.
Solution: I redesign the Performance Dashboard at the end of the session in order to provide more detailed information to guide the student study. I want to give the flexibility to see a quick summary or a detailed report.
4. Problem: Some questions take the student's a lot of time to answer it. Because some involve calculations. Thus, a lot of students request a Skip Action when they had limited time to do a session but either way wanted to do some studying and didn't want to get blocked by a long question. Also, a lot of student also request the ability to mark questions so they can review or study it better later.
Solution: We include both suggestions, Skip Question button and Mark Question.
Solution: We include both suggestions, Skip Question button and Mark Question.
|
Becker's PRS UI
Final Design
