Why Blind Trust in AI Could Be as Risky as Rejecting It Completely

Generative AI is rapidly becoming more than a productivity tool. People increasingly use conversational systems for information, education, technical assistance, practical advice, entertainment and emotional support. As these systems become woven into everyday routines, the central policy challenge is changing. Users do not simply need to trust artificial intelligence more or less; they need to understand when, why and how much confidence a particular interaction deserves.

A new study, "Unpacking the Dynamics of Generative AI Use in Our Daily Lives: Towards an Integrative Trust Calibration Framework," published in AI & SOCIETY by Theo Araujo of the Amsterdam School of Communication Research at the University of Amsterdam, develops a conceptual framework for understanding how trust in conversational generative AI forms, changes and can become misaligned with what these systems are actually capable of delivering.

Excessive distrust can lead users to reject tools that may genuinely improve productivity, access to information or learning. Excessive confidence, however, can expose them to misinformation, manipulation, privacy risks or inappropriate dependence. The relevant policy objective is therefore not to maximize trust in AI, but to help users calibrate trust according to the task, the stakes and the system's real capabilities.

One User, One Chatbot, Many Completely Different Risks

Traditional research on trust in technology often assumes a relatively simple relationship: a person operates a specific system for a defined task, evaluates its performance and gradually learns whether it is reliable. Generative AI disrupts that model because the same conversational agent can serve radically different purposes within the same day. A user might ask it to draft a business email in the morning, explain a health issue at lunchtime, support a school assignment in the afternoon and provide emotional reassurance in the evening.

The author emphasizes that generative AI is now used for information seeking, productivity, learning, entertainment, social interaction, companionship and support. Those activities do not carry equal risk, and they should not demand the same form or level of trust. Accuracy and verifiability may be crucial when users seek factual information, while privacy, emotional safety and resistance to manipulation become more important when users discuss personal problems or seek companionship.

This difference means that the broad question "Do you trust AI?" is increasingly inadequate for policymakers and researchers. A far more useful question is whether users adjust their confidence appropriately as the context changes. A harmless mistake in a recreational conversation is fundamentally different from an incorrect recommendation involving health, finance, education or another consequential decision. Trust therefore has to be understood in relation to vulnerability, not simply usage.

The implication for digital literacy is significant. Generic warnings that AI systems "can make mistakes" may not be enough. Users need to recognize which situations require independent verification, when personal data should be withheld, when advice should be treated with caution and when qualified human expertise should take precedence. This challenge may be especially acute in countries where AI access spreads faster than media literacy, institutional safeguards or consumer protection.

The Hardest Part of Trusting AI Is Knowing What 'Trustworthy' Means

Trust calibration appears straightforward in theory: users should place confidence in a system in proportion to its actual trustworthiness. Generative AI makes that benchmark unusually difficult to define because its performance depends heavily on purpose and context. Accuracy, safety, fairness, transparency, privacy and auditability may all contribute to trustworthiness, but their importance changes depending on what the user is trying to accomplish.

The paper describes this difficulty as arising partly from the technology's extraordinary flexibility. Conversational AI can perform so many different functions that determining what it should reliably do becomes difficult. A model may perform well in summarization but poorly in specialized factual reasoning; it may provide useful general guidance while being inappropriate for a high-stakes medical judgment. Trustworthiness therefore cannot always be treated as a single stable property of an entire chatbot or model.

Outputs also vary according to prompts, languages, topics and system updates. Providers continuously release new models, add capabilities and alter personalization or memory features. That raises a difficult research question: should trustworthiness be assessed at the level of the AI service, the particular use case or even an individual answer? The more granular the standard becomes, the harder it is for regulators, researchers and users to evaluate consistently.

This creates a governance challenge for technology companies and public authorities. Universal claims about "trustworthy AI" may be too broad to guide real decisions. A system that is perfectly acceptable for brainstorming ideas may not satisfy the standards required for health guidance, political information or financial decisions. The paper therefore points toward context-specific expectations of trustworthiness, rather than assuming that one assurance mechanism can cover every use.

Transparency is important, but it is not a simple solution. Explanations may sometimes improve users' judgment, yet they can also increase confidence without necessarily improving understanding. A polished explanation can make an uncertain answer sound more credible than it deserves. Effective disclosure should therefore help users evaluate uncertainty and limitations rather than merely make systems appear transparent.

Users May Be Trusting a Personality When They Should Be Evaluating a System

Generative AI also complicates trust because users may not know exactly who or what they are trusting. Confidence could be directed toward the chatbot's answer, the underlying model, the company providing the service, the sources presented in the response or the wider technical infrastructure supporting it. Araujo describes this as a complex network of potential trustees, making responsibility and accountability harder to interpret.

The problem becomes more important as conversational AI adopts increasingly human-like characteristics. First-person language, friendly personalities, voice interfaces, persistent memory and personalized responses can make systems feel less like software and more like interaction partners. Some users may still perceive them as tools, while others may treat them as social counterparts capable of understanding, remembering or caring.

The psychological shift can distort trust. An AI assistant may sound empathetic, confident and attentive without being more accurate or safer. Personalization may create familiarity, and familiarity can be mistaken for reliability. A system that remembers previous conversations may feel more trustworthy simply because the interaction resembles a continuing relationship, even though memory says little about whether its next recommendation is correct.

This is where the paper's discussion of synthetic relationships becomes particularly important. Repeated AI interactions can influence users' thoughts, emotions and behavior, especially when systems are used for companionship or emotional support. From a regulatory perspective, the issue is therefore not only whether an AI produces factually correct information, but whether interface design, anthropomorphic cues and personalization encourage unjustified confidence or dependence.

Consumer-protection policy may eventually need to treat trust-shaping design as an important category of AI risk. Just as regulators examine misleading advertising or manipulative interface design, they may increasingly need to consider whether certain AI features encourage users to attribute competence, authority or emotional understanding that the technology does not possess.

AI Trust Is Becoming a Relationship, Not a One-Time Decision

Perhaps the paper's most important insight is that trust calibration unfolds over time. Users learn from past interactions, remember helpful answers and failures, improve their prompting techniques and gradually build assumptions about what an AI system can do. Those assumptions can strengthen confidence, reduce it or become difficult to revise after repeated positive experiences.

The system is evolving at the same time. Longer-term memory and personalization allow some AI agents to adapt responses based on earlier conversations. The paper argues that this can create feedback loops in which users alter their behavior in response to the system while the system simultaneously adjusts to information learned about them. Trust therefore becomes an evolving relationship between user expectations, system behavior and context rather than a one-time judgment about reliability.

The integrative framework developed in the paper connects user characteristics, learned experiences, system design, personalization, situational risks and broader institutional conditions. Its central message is that responsible AI use cannot be secured through a single disclosure or literacy intervention. Education systems may need to teach AI literacy as a continuing judgment skill, developers may need to encourage verification when stakes rise, and regulators may need to examine how memory, personalization and social cues affect long-term reliance.

The paper is conceptual rather than empirical, and its propositions still require testing through longitudinal and cross-cultural research. Future studies will need to examine how trust develops across months of everyday use, how individuals from different socioeconomic and educational backgrounds respond to AI, and how actual trustworthiness can be measured at the fine-grained level required by highly flexible systems.

Nevertheless, the framework identifies a strategic issue that governments, educators and technology companies cannot afford to ignore. The next phase of responsible AI will not be defined simply by persuading people that artificial intelligence is trustworthy, nor by encouraging blanket skepticism. The more important objective is to build societies in which people can distinguish between confidence that is justified, confidence that is misplaced and skepticism that genuinely protects them from harm.