| AI-Assisted Anonymization for Legal Transparency Helping a Baltic court streamline the anonymization of legal documents with an AI-powered tool that balances efficiency, compliance, and human oversight. |
Company
Latvia Government
Product
AI-Assisted Document Anonymization System
Latvia Government
Product
AI-Assisted Document Anonymization System
Deliverable
UX strategy, high-fidelity prototype, interface design, user research analysis
Timeline
17 weeks (Discovery to MVP)
Discovery Phase (8-week duration)
3 x Delivery Sprints (2-week duration)
UX strategy, high-fidelity prototype, interface design, user research analysis
Timeline
17 weeks (Discovery to MVP)
Discovery Phase (8-week duration)
3 x Delivery Sprints (2-week duration)
Role
Lead UX Designer - Responsible for the end-to-end design process, including product vision, research, prototyping, usability testing, and final delivery.
Team
1 PM, 2 data scientists, 2 devs
Lead UX Designer - Responsible for the end-to-end design process, including product vision, research, prototyping, usability testing, and final delivery.
Team
1 PM, 2 data scientists, 2 devs
My Role
Leading the design efforts, from conception to delivery, collaborating with cross-functional teams, and ensuring that the automated solution serve the best interest of users.
Understand the user with semi-structured interview.
Design for explainable AI and iterating on the design.
Involve users early and often. Run a set of experiments.
Design for fairness and transparency through usability studies.
Understand user’s mental models regarding automation.
Impacts
-
Processing Time: 70% faster than manual -
Trust Calibration: Preferred precision-based model -
Human-in-the-loop validation introduced -
User Feedback: Increased satisfaction with HITL transparency
THE CHALLENGE
The Baltic court needs to provide public access to documents while protecting the personal data of participants in lawsuits.
The process for anonymizing documents is time-consuming, labor-intensive, and prone to errors, which can lead to non-compliance with data privacy regulations.
Each document could take up to 20 minutes to anonymize, creating delays and risking errors in sensitive data handling.
Document Life-cycle
Legar Transparency Under Pressure
PAIN POINTS
// High time and effort per case
// Error-prone process
// Growing backlog
// Lack of auditability
// User fears around automation replacing their jobs
ORG GOALS
Streamline document publication and ensure compliance with data privacy regulations by implementing an automated solution that reduces time and effort in the document anonymization process.
Enable court employees to focus on more important tasks, leading to faster document publication and improved access to justice for the public while protecting the personal data of participants in lawsuits.
USER GOALS
DESIGN GOALS
Create an intuitive and reliable system that automates repetitive tasks, with safeguards involving legal professionals through human-in-the-loop (HITL) control. It ensures transparency through confidence scoring and audit trails, building trust in machine-assisted decisions.
Design Process Summary01 Learn. Understand user needs, fears, and anonymization workflows. Conduct user interviews, review institutional constraints, and identify automation readiness. 02 Explore. Map current processes, build user profiles, outline task flows, and highlight automation opportunities. 03 Select. Identify key user journeys and prioritize opportunities for AI-assisted redaction, human validation, and auditability. 04 Develop. Prototype core interactions, including redaction flow, entity labeling, and confidence scoring with human-in-the-loop support. 05 Refine. Test early concepts with legal professionals, improve UI clarity, transparency, and alignment with legal procedures. 06 Deliver. Compiled findings, prepared full UX hand-off including annotated flows, UI specs, and interaction logic to support the development team in building the application. |
|
User recruitment: Direct recruitment with the institution. I conducted 7 moderated 60-minute interviews and created empathy maps to understand the user I was designing for their needs, motivations, and behavior.
Profiles: 1 System Admin, 2 Developers, and 4 Secretaries.
Profiles: 1 System Admin, 2 Developers, and 4 Secretaries.
|
User Groups
User research led us to redefine our user model. Initially, we had identified only two roles, but findings revealed the need to introduce a third: the Operator.
Our solution now supports three personas:
- Admin – Technically proficient but not an ML expert, responsible for managing the system and overseeing model integration.
- Tester – Supports the court’s internal systems and is tasked with training the model.
- Operator – Newly introduced based on findings. This role ensures a human-in-the-loop approach by reviewing and validating anonymized documents before publication.
This shift allowed us to align the solution more closely with real workflows and identify tailored value propositions for each role.
Design Guidelines from user journeys: Make system capabilities clear, communicate how well it performs, set expectations for adaptation, plan for trust calibration, ensure transparency, show cause-and-effect, and optimize for understanding.
User Flow
During the user-flow mapping, we faced a key challenge: determining if Classification, Anonymization, and Data Extraction are separate flows and functionalities.
Thanks to the persona work completed in the define phase, we understood that these were indeed three distinct flows, each associated with a specific user profile—admin, tester, and operator.
Reward function: To address failure scenarios, a confusion matrix was used to assess the risks of false positives and false negatives. This helped evaluate privacy implications, clarify which outcomes best support the user experience, and guide the trade-off between precision and recall.
|
|
A/B Testing
To validate which algorithmic behavior better supports user trust and system accountability, We ran a series of A/B test sessions. These sessions were informed by a prior analysis using a confusion matrix, which helped map the impact of false positives and false negatives on user experience and data privacy.
In each test, users interacted with two anonymization models:
- One tuned for precision, minimizing false positives to avoid over-redacting non-sensitive data;
- One tuned for recall, minimizing false negatives to ensure that all personal data is correctly anonymized.
Rather than focusing solely on performance metrics, these tests explored how each variant shaped user perception, trust calibration, and error recovery. I also observed how users responded to uncertainty and developed strategies to support meaningful oversight without undermining automation. These insights were critical to designing a system that is not only accurate, but trustworthy and transparent.
|
Goals
// Assess the trustworthiness of the NER (Named Entity Recognition) solution
// Examine the explainability and interpretability of the user interface and system feedback
// Verify the reliability and safety protocols of the solution
// Address any privacy and security concerns that may arise during the testing process
// Examine the explainability and interpretability of the user interface and system feedback
// Verify the reliability and safety protocols of the solution
// Address any privacy and security concerns that may arise during the testing process
Performance Metrics
We selected accuracy, efficiency, and satisfaction as key performance metrics to assess the success of our AI-powered anonymization system.
Accuracy. Measured precision (false positives) and recall (false negatives), then validated their impact through user interviews to assess how different errors influenced trust and usability.
Efficiency. Tracked anonymization time before and after implementation, showing a significant reduction in effort compared to manual workflows.
Satisfaction. Conducted interviews pre- and post-deployment to capture shifts in user trust, perceived value, and overall comfort with automation.
Accuracy. Measured precision (false positives) and recall (false negatives), then validated their impact through user interviews to assess how different errors influenced trust and usability.
Efficiency. Tracked anonymization time before and after implementation, showing a significant reduction in effort compared to manual workflows.
Satisfaction. Conducted interviews pre- and post-deployment to capture shifts in user trust, perceived value, and overall comfort with automation.
Lessons Learned
This project offered more than an opportunity to improve processes—it allowed me to help shape the UX foundations of an AI-driven solution from the ground up.
Working closely with data scientists, engineers, and product managers, I built trust across disciplines and led the UX process from discovery to delivery. I contributed to aligning our intelligent system with human-centered principles like transparency, explainability, and accountability.
Through this experience, I learned that UX for AI isn’t just about usability—it’s about building systems that people can trust, understand, and rely on.
Working closely with data scientists, engineers, and product managers, I built trust across disciplines and led the UX process from discovery to delivery. I contributed to aligning our intelligent system with human-centered principles like transparency, explainability, and accountability.
Through this experience, I learned that UX for AI isn’t just about usability—it’s about building systems that people can trust, understand, and rely on.
Key Takeaways
// Onboarding stakeholders and data scientists to the value of user-centered design is critical for AI products to succeed.
// Cross-functional collaboration is essential to align technical performance with human experience.
// Designing with care means addressing not only how well a system works, but also how clearly and ethically it operates.
// Cross-functional collaboration is essential to align technical performance with human experience.
// Designing with care means addressing not only how well a system works, but also how clearly and ethically it operates.
Human-Centered AI isn’t a final phase—it’s a foundation.
