Auditing an AI-based Cybersecurity Application
An independent audit turned a research prototype into a compliant, production-ready AI system to protect individuals and organizations from phishing attacks
Cybersecurity • EU • 2025
Situation
Eticas was commissioned to evaluate an AI-based cybersecurity application developed to reduce human vulnerability to phishing. The application combines behavioral analytics, psychometric profiling, contextual risk factors, and predictive modelling to assess exposure and recommend preventive actions.
The application analyses psychological data in workplace-like contexts. To be deployed in Europe, it is classed as a “high-risk AI system” under the EU AI Act.
Challenges
Before the audit, the application faced issues typical of early-stage high-risk AI systems:
- Its predictive engine was built on a small, incomplete dataset, raising concerns about bias and validity.
- It lacked a Data Protection Impact Assessment (DPIA), lawful basis documentation, and clear user transparency under the GDPR. Accountability roles were undefined, and it was unclear whether the application met the AI Act’s requirements for risk management, human oversight, and transparency.
- Its limited real-world data also hindered validation: some contextual variables unintentionally leaked outcome information, inflating accuracy.
- Combined with incomplete user consent and privacy notices, the application could not yet demonstrate full compliance or traceability.
Why solving these was hard
The application’s challenges lay at the intersection of law, data science, and ethics. Achieving compliance with both the AI Act and GDPR required collaboration across disciplines.
Technically, data scarcity threatened reliability and fairness. Ethically, assessing human “vulnerability” risked stigmatization, requiring careful framing and human-centered design.
Without remediation, the application developer risked fines under the AI Act (up to 3% of global turnover) and GDPR violations for processing psychometric data without explicit consent — and reputational damage: An unreliable model could have unfairly labeled users as “high-risk,” damaging trust and credibility.
Our approach
Eticas led a three-module audit addressing legal compliance, technical robustness, and ethical design.
Module 1 – Baseline audit:
Eticas reviewed classification, governance, and transparency under the AI Act and GDPR. Found mature technical architecture but incomplete regulatory coverage—missing DPIA, consent mechanisms, and transparency templates. Recommendations included a formal risk management plan, defined legal bases, and an internal compliance officer.
Module 2 – Deep-dive audit:
Eticas analyzed algorithms and data governance. Having identified data incompleteness, target leakage, and only moderate real-data accuracy, recommendations were made to improve data usage and model design.
An ethical review led to removing demographic data from dashboards and redesigning interfaces to highlight situational—not personal—risk. Transparency markers and confidence intervals were added to increase interpretability.
Module 3 – Production audit:
Eticas created a six-month monitoring framework with KPIs for technical and behavioral impact, including high-risk rate, positive predictive value, behavior-change rate and perceived stigmatization.
Deliverables
- Legal and technical audit reports confirming full compliance with GDPR and AI Act.
- Synthetic sample data validated for bias and representativeness.
- Six-month monitoring plan covering fairness, performance, and user trust.
- Updated transparency templates and privacy notices aligned with human-centred AI principles.
Outcome
After the audit, the application achieved full compliance with Articles 9–15 of the AI Act and GDPR. The system now runs with a DPIA, defined accountability roles, calibrated thresholds, and built-in monitoring for drift and fairness.
Predictive reliability and fairness improved significantly, while false positives dropped. Behavioral change metrics show that over 35% of alerted users now adopt protective actions. Model accuracy improved from 70% to 93%.
The application evolved from a research prototype to a fully compliant, trustworthy high-risk AI system—demonstrating measurable gains in user trust, fairness, and reduced economic risk.