Empathic technologies: children, toys and schools
It seems that children are forgettable. While critical attention to emotional AI and empathic technologies focuses on adult data, there is growing use with children. Our work by Prof. Andrew McStay and Dr. Gilad Rosner (founder of the IoT Privacy Forum) examines toys, wellbeing wearables, companions and schools.
Office of the Privacy Commissioner of Canada (OPC)
Most recent is the 2025 report for the Office of the Privacy Commissioner of Canada (OPC). Titled ‘Governance of Emotional AI Used with Canadian Children’, the report examines the governance of Emotional AI technologies used with children. Emotional AI, which interprets and responds to human emotions, is increasingly present in toys, education, and AI companions, raising significant privacy and ethical concerns. Children are particularly vulnerable due to their developing emotional skills, lack of awareness about data collection, and susceptibility to manipulation.
The study highlights gaps in Canada’s privacy law (PIPEDA), which lacks explicit protections for children’s biometric and emotion data. Compared with regulations like the EU’s AI Act and the UK’s Children’s Code, PIPEDA offers limited safeguards and enforcement powers. The report recommends reforms, including defining children’s emotion data as sensitive, strengthening regulatory oversight, and introducing explicit bans on harmful applications of Emotional AI in marketing and sales.
Additionally, the study underscores the need for a rights-based approach, ensuring Emotional AI aligns with the best interests of the child. It advocates for enhanced transparency, ethical AI design, and robust privacy impact assessments. To support governance, it proposes Canada-specific guidelines for fair and accountable Emotional AI deployment, ensuring child safety and data protection.
Rights of Childhood: Affective Computing and Data Protection
The toys work (Rights of Childhood: Affective Computing and Data Protection) funded by the HDI+Network and EPSRC (2019-20) identifies the socio-technical ethical terms by which affective child-inclusive Internet of Things (IoT), such as toys, but also home assistants, should function.
This is in order to: understand how they intersect and affect children’s information rights; identify which criteria are required to create consumer trust in emotion-sensitive products; and understand how children and parents can be empowered by technology design and rights.
Methods include UK national surveys of parents (n=1000), focus groups with parents (teaming with Dr. Kate Armstrong from the Institute of Imagination) and “elite” interviewing.
Broader work on education
In education, McStay explores how the application of emotional AI promises to assist with personalised learning, development of social and emotion learning, to understand if students are struggling with class material, and which students need to be challenged further by class content.
He finds a tension between schools’ pedagogical prerogatives (i.e. the wide latitude given to schools to set the terms of education) and the right for parents to have consent options in relation to data collection about their children.
Other issues are the empirics of face-based emotional AI employed in EdTech (does it work?), consequences of judgements about emotion and attention levels (for students and teachers alike), how profiling balances with the human rights of the child, and the impact of commercial imperatives. Is the public good being served?
Outputs
Academic Writing
McStay, A. (2023) Positive Education, in Automating Empathy: Decoding Technologies that Gauge Intimate Life. Oxford University Press.
McStay, A. (2022) Automated Empathy in Education. In L. Livingstone and K. Pothong (Eds) Education Data Futures: Critical Regulatory and Policy Futures. 5RightsFoundation. Online.
McStay, A. & Rosner, G. (2021) Emotional Artificial Intelligence in Children’s Toys and Devices: Ethics, Governance and Practical Remedies, Big Data & Society.
McStay, A. & Rosner, G. (2021) ““Emotoys”: Ethics, Emotions and Empathic Technologies” In A. Malinowska (Ed) Data Dating. Cambridge, MA: MIT Press.
McStay, A. (2019) Emotional AI and EdTech: Serving the Public Good, Learning Media & Technology.
Events
Computers, Privacy and Data Protection (CPDP 2020) panel. Slides and video.
Other
McStay, A. Rosner, G. Miyashita, H. and Urquhart, L. (2020) Comment on Children’s Rights In Relation To Emotional AI for UN Committee on the Rights of the Child.
McStay cited/discussed in UNICEF (2020) Policy guidance on AI for children (input from McStay on Emotional AI)
PROJECT REPORTs
Rosner, G. and McStay, A. (2025) The Machine-Readable Child: Governance of Emotional AI in Canada for Office of the Privacy Commissioner of Canada.
McStay, A. and Rosner, G. (2020) Emotional AI and Children: Parents, Ethics, Governance for HDI+Network.
