IGF 2026 WS #141 Raised with AI: Empowered or Dependent?

Description

AI companions and generative AI are no longer just tools for answering questions. For children and teenagers, they can become tutors, friends, role-play partners, and sources of emotional support and advice. As young people increasingly turn to them for companionship, they create a new child online safety and Internet governance challenge: how to protect children's agency, development, privacy, relationships, and digital resilience when AI systems are designed to simulate care, intimacy, trust, and constant availability. The session will be framed around three possible roles of AI companions in childhood. First, AI may empower children by supporting learning, creativity, communication, accessibility, and inclusion, helping them explore ideas, overcome barriers, and access personalized support. Second, AI may create dependence if children increasingly rely on AI companions for emotional reassurance, decision-making, learning, or social interaction, potentially weakening agency, resilience, and human relationships. Third, AI may exploit children through the commercialization of intimacy by collecting sensitive data, maximizing engagement, encouraging self-disclosure, manipulating attention, or providing harmful, inaccurate, or biased advice. The session will examine how governance can preserve AI's benefits while protecting children from dependency, manipulation, and unsafe design, and what limits should apply to systems designed to simulate friendship, intimacy, and trust. The discussion will move beyond AI in education to examine the governance of AI companions used by children, focusing on safety-by-design, age-appropriate design, data protection, transparency, platform accountability, crisis response, and meaningful youth participation. The session will bring together perspectives from the technical community and from experts in child protection, education, AI governance, public health, civil society, regulatory bodies, and industry. The expected outcome will be a practical set of “red lines and design principles” for child-centred AI companions.

Policy Question(s)

When does an AI companion empower a child, and when does it create emotional, social or cognitive dependence?
What safeguards should apply to AI systems designed to simulate friendship, intimacy, care, or emotional support for children?
How should responsibility be shared between parents, schools, platforms, AI providers, regulators, child protection actors and young users themselves?
Should AI companions for children be treated as ordinary digital services, educational tools, high-risk systems, or a separate category requiring child-specific rules?
What should be considered unacceptable exploitation of children’s attention, emotions, data or vulnerabilities in AI companion services?

Expected Outcomes

The session will aim to produce:

A short “Raised with AI: Red Lines and Design Principles” checklist for child-centred AI companions.
Practical recommendations for platforms, AI providers, policymakers, educators, parents, hotlines, helplines and youth communities.
A clearer distinction between AI as a lever for development, AI as a cognitive or emotional prosthesis, and AI as a potentially exploitative system.
A basis for follow-up cooperation through IGF youth, regional, intersessional or child online safety processes.

Reference Document

Format

Roundtable Duration (minutes): 60 Format description: The 60-minute interactive roundtable is designed to maximize meaningful participation rather than passive listening. Brief framing inputs establish a shared understanding, followed by a moderated discussion structured around the opportunities, risks and governance challenges of AI companions. The thematic format allows participants to explore different perspectives while maintaining a focused and dynamic exchange. Live polling, hybrid Q&A and equal opportunities for onsite and online interventions ensure continuous engagement, rather than limiting participation to the final minutes. Collaborative identification of red lines and design principles transforms the discussion into concrete, actionable outcomes, with remote participants contributing on equal terms through polling and shared collaborative tools. The room layout will facilitate dialogue among speakers and participants, encouraging interaction across stakeholder groups and regions. The session concludes with a rapporteur summary that synthesizes key insights and recommendations from both onsite and online discussions, providing a practical basis for future policy and governance conversations.

Hybrid Format

Hybrid Format: The session will be designed as a genuinely hybrid discussion. The online moderator will actively collect online questions and bring them into each segment of the discussion, rather than reserving online participation only for the final Q&A. The session will use:

live polling to ask participants whether they see AI companions mainly as support, risk or exploitation;
equal time for onsite and online questions;
collaborative collection of red lines and design principles;
a final rapporteur summary of points raised by both onsite and online participants. Possible tools include the IGF online platform, Slido/Mentimeter or equivalent polling tools, and a shared online document or form to collect recommendations from remote participants.

Organizers

Organizer 1: Julia Piechna, NASK - National Research Institute Organizer 2: Maciej Gron, 🔒

Speakers

Speaker 1: José Renato Laranjeira de Pereira, Civil Society, Latin American and Caribbean Group (GRULAC) Speaker 2: Julia Piechna, Technical Community, Eastern European Group Speaker 3: Pascal Bekono, Government, African Group Speaker 4: Svaradiva Devi, Civil Society, Asia-Pacific Group Speaker 5: Jannat Roshatey, Technical Community, Asia-Pacific Group

Moderator

Julia Piechna, Technical Community, Eastern European Group

Online Moderator

Maciej Gron, Technical Community, Eastern European Group

Rapporteur