Key Requirements for Trustworthy AI.docx

Key Requirements for Trustworthy AI The AI Act proposed by the Commission will lay down legally binding requirements for AI systems considered as "high-risk" in view of their important concerns, such as the risk of bias or error affecting educational outcomes: intended purpose⁴. This will include c...

Key Requirements for Trustworthy AI The AI Act proposed by the Commission will lay down legally binding requirements for AI systems considered as "high-risk" in view of their important concerns, such as the risk of bias or error affecting educational outcomes: intended purpose⁴. This will include certain AI systems used in the area of education and vocational training. When the AI Act becomes applicable, education institutions as users of AI systems will be able to rely on the trustworthiness of these "high-risk" AI systems based on the accompanying certification ensured by the provider, while having to comply with certain obligations. Irrespective of whether the AI systems will fall under the scope of the legal framework, companies developing and providing AI systems (system providers) are encouraged to implement and apply ethical requirements for trustworthy AI to their design and development processes. At the same time, it is important that schools and educators are aware of these and are able to formulate relevant questions in order to better reflect on them. The below requirements, which are based on the AI HLEG Ethics Guidelines for Trustworthy AI, are therefore recommendable for any AI system deployed and used in education. They address important concerns, such as the risk of bias or error affecting educational outcomes: - Human agency and oversight including fundamental rights, children's rights, human agency, and human oversight. - Transparency including traceability, explainability and communication. - Diversity, non-discrimination, and fairness including accessibility, universal design, the avoidance of unfair bias, and stakeholder participation, which allows use regardless of age, gender, abilities, or characteristics - with a particular focus for students with special needs. - Societal and environmental wellbeing including sustainability and environmental friendliness, social impact, society, and democracy. - Privacy and data governance including respect for privacy, quality and integrity of data, and access to data. - Technical robustness and safety including resilience to attack, security and general safety, accuracy, reliability, and reproducibility. - Accountability including auditability, minimisation and reporting of negative impact, trade-offs, and redress. The considerations and requirements can help educators, school leaders and technology providers to adequately assess the impact, address the potential risks, and realise the benefits of an AI system deployed and used in education. As such they guide the development, deployment and use of trustworthy AI systems.