A 2025 Pew survey found that 34% of US adults have used ChatGPT, a share that doubles among those under 30. Now, the consequences of that rapid uptake are becoming measurable. A separate poll of 2,000 Americans by Pearl.com reports that 19% lost more than $100 by acting on financial advice from an AI chatbot; among Gen Z investors, the figure rises to 27%. Simultaneously, research from Sentio University shows that 48.7% of users who report mental health difficulties use AI platforms for emotional support, with 73% turning to them for anxiety and 60% for depression. These parallel trends expose a common vulnerability: AI tools are being entrusted with high-stakes personal decisions for which they were never designed.

The mechanism behind the failures is not a lack of sophistication but a mismatch between fluency and accuracy. A chatbot can produce a confident, well-structured answer about Social Security claiming strategies or cognitive-behavioural therapy techniques, yet miss the contextual details—a spouse’s health status, a patient’s non-verbal cues—that a human professional would weigh. Researchers at Brown University simulated therapy sessions and identified 15 recurrent ethical violations, including artificial empathy and mishandling of crisis scenarios. In finance, the problem is compounded because the most consequential decisions—exercising stock options, navigating a divorce settlement—are rare events on which AI models have scant training data. As a finance professor notes, the dangerous failure is the one you cannot detect, leaving users to manage complex problems alone long past the point when they should seek help.

Viewed from Jakarta, the tension between AI’s promise and its perils is acute. The Indonesian fintech association AFTECH reports that 84% of its members have adopted AI, and 43% are now profitable. Yet the same digital acceleration has exposed consumers to AI-powered fraud: the national cyber agency recorded four billion anomalies in the first half of last year, with losses reaching Rp8 trillion. In Argentina, where 70% of young people say they lack financial knowledge and 60% rely on social media for money tips, Banco Galicia has launched a free digital literacy platform endorsed by the University of Buenos Aires, complete with an AI teaching assistant. Italian educators, meanwhile, are pushing beyond basic financial literacy to entrepreneurship training, arguing that students must learn to create work, not just find it, after OECD data showed 18% of Italian 15-year-olds fall below the baseline in financial competence.

The emerging response from policymakers and educators is to embed critical thinking into digital literacy. Indonesia’s Universitas Bakrie warns that AI and social media algorithms threaten local cultural resilience, urging a strategy that pairs technological adoption with cultural preservation. In the Vatican, Pope Leo XIV’s recent encyclical insists that AI must serve human dignity, not replace moral discernment. The next concrete step will come from the Indonesia Digital Bank Summit this week, where regulators and industry leaders are expected to draft governance standards for AI in consumer finance—a test of whether the institutions that enabled rapid digitisation can now build the guardrails that users, and their bank accounts, urgently need.