ChatGPT’s rise as a communication proxy exposes gaps in AI governance

ChatGPT is beginning to stand in for human communicators in tasks that involve advice, explanation, emotional response, authorship and judgment, according to a new study published in AI & Society that warns the risk is not limited to wrong answers. The larger issue is that users may not always know when a machine has moved from helping communication to replacing it.

The study, titled Communicative surrogates and the ethics of LLM-mediated communication, claims that large language model interfaces such as ChatGPT are becoming communicative surrogates, or nonhuman systems that take over communicative labor while unsettling the usual markers of responsibility, authority, care and human presence.

ChatGPT's smooth interface makes substitution easy to miss

ChatGPT can draft messages, explain concepts, translate text, prepare teaching material, summarize information and respond to personal concerns. However, it also allows the system to enter spaces where communication usually depends on human accountability. Users are not only deciding whether ChatGPT is correct or helpful, they are also trying to work out what the system is in a given moment. It may feel like a tool when it edits a sentence, like an adviser when it gives guidance, like a teacher when it explains a topic, or like a companion when it responds to distress. The same interface can shift roles without clearly showing that a shift has happened.

The authors point out that ethical judgment in communication depends on knowing who or what is participating. When a human gives advice, writes a message or offers care, responsibility can be traced to a person, profession, institution or relationship. When ChatGPT produces similar language, the same expectations become harder to place.

The system's ease of use intensifies this problem. ChatGPT asks little of the user: type a prompt, receive a fluent response, continue the exchange. The smoothness removes the pause that might otherwise make people ask whether a task should be delegated at all. A user may begin with harmless wording help and gradually rely on the system to frame arguments, answer others, interpret information or simulate concern.

The researchers describe this as communicative surrogacy. Earlier tools such as autocomplete or basic text prediction supported communication at the margins. Large language models can occupy the communicative space itself. They generate context-sensitive responses that appear socially usable, responsive and meaningful. Practically, they can sound as if someone is present, even when no human speaker carries the responsibility usually attached to the words.

The paper is based on semi-structured interviews with 18 academic knowledge workers in human-computer interaction and related fields during the early public uptake of ChatGPT. It wasn't aimed at measuring how often people use the system, but to examine how users described the system's role, where they felt uncertain and how ethical discomfort surfaced when ChatGPT entered communicative tasks.

Participants did not simply reject ChatGPT - many saw clear value in it. A system that can be a writing assistant, tutor, translator, search-like tool and emotional responder becomes harder to classify. Its flexibility makes it attractive, but it also weakens the social cues that usually tell people what kind of communication they are engaged in.

The authors call this process ontological drift - a term describing the loss of stable footing about what kind of entity is involved in an exchange. If the user cannot tell whether ChatGPT is acting as a tool, proxy, adviser or participant, then the user also struggles to know what kind of responsibility, trust or caution should apply. The confusion often appears first as hesitation, ambivalence or the sense that something about the interaction feels wrong before the user can explain why.

Four AI roles show where ethical risks take shape

The study identifies four recurring forms of communicative surrogacy: the frictionless chatbot, the emotional companion, the epistemic actor and the laboring assistant. These are not separate products or fixed user categories; they are ways ChatGPT can appear as it moves through different communicative settings.

• The frictionless chatbot reflects the role created by ease of access. Participants described ChatGPT as simple, quick and likely to spread widely because it requires little technical knowledge. The ethical risk is that familiar design can make consequential delegation feel routine. When a system feels as easy as search, users may hand over tasks that involve judgment, care or responsibility before shared norms have caught up.
• The emotional companion emerges when ChatGPT is used or imagined as a source of comfort, reflection or emotional support. The issue is not that users necessarily believe the system is human. It is that the system's conversational style, responsiveness and memory-like behavior can trigger expectations normally tied to human care. A person may share distress with a chatbot, then feel unsettled when the exchange appears to continue or be remembered.
• The epistemic actor appears when ChatGPT gives explanations or advice with the tone of authority. Participants were concerned that the system can sound confident while offering limited visibility into sources, reasoning or accountability. This is not only a misinformation problem. It is a problem of authority without clear responsibility. In areas such as health, education, research or public decision-making, fluent language can make uncertain output appear more reliable than it is.
• The laboring assistant appears when ChatGPT takes over practical communication work, including translation, drafting, summarizing, preparing materials and supporting language learning. Participants recognized the value of these uses, especially when people face time pressure or limited resources. But usefulness did not erase concern. Several worried that routine delegation could weaken skills, hide labor, shift accountability or make machine substitution seem normal in work that still requires human judgment.

The authors observed the same pattern across all four roles. Ethical concern usually follows uncertainty about what role ChatGPT has taken. For example, a user may welcome drafting help while worrying about authorship. A teacher may appreciate saved time while worrying about deskilling. A person may find emotional expression easier with a chatbot but recoil when the system feels too present. The discomfort is not random, it signals that a communicative boundary has become unclear.

The study treats such discomfort as meaningful evidence. Unease, hesitation and ambivalence are not framed as irrational resistance to new technology. They are early signs that communication norms are under pressure. People may sense the ethical problem before they can fully name it. That matters because LLMs are spreading faster than institutions, policies and social rules can define appropriate use.

The authors also point out that users should not carry the full burden of judgment. ChatGPT's interface gives limited information about how outputs are produced, what data may shape them, where claims come from or what forms of hidden labor sustain the system. When the system hides those conditions behind a clean chat box, users are left to make ethical decisions with too little context.

Why AI-mediated communication needs policy guardrails

The key policy message is that designers and institutions must make AI substitution visible. The problem is not only that people may misuse ChatGPT, but that current systems often make very different kinds of communication feel the same. Asking for a grammar fix, drafting a condolence message, seeking medical-style guidance and relying on the system for emotional support may all happen through the same simple interface, even though the ethical stakes are not comparable.

Friction, as the authors suggest, can be useful in such moments. In technology design, friction is often treated as something to remove. But when AI enters communication involving care, authority, trust or professional judgment, a pause may be necessary. A well-designed interruption can help users recognize when assistance is becoming substitution.

Systems that draft sensitive messages, respond to emotional distress, support students, provide health-related information or generate professional communication may need clearer role signals. They may also need stronger source cues, better memory controls, warnings about limits and prompts that distinguish drafting support from delegated communication.

The study also raises concerns for schools, universities and workplaces. ChatGPT may be adopted as a labor-saving tool before organizations decide which forms of communicative work should remain human-led. Faculty may use it to prepare course material. Students may use it to write assignments. Employees may use it to draft messages. Institutions may use it to manage service or support communication. Each use carries a different question: who is speaking, who is responsible and what kind of human judgment is being replaced or hidden.

The paper warns that communicative surrogacy could become part of the background infrastructure of daily life. If that happens, machine-generated communication may become routine in settings where people still expect human presence or professional accountability. The risk is not only deception, but a subtle shift in what people accept as communication.

For governance, the authors suggest that rules focused only on accuracy, disclosure or misuse will not be enough. LLMs also change how people understand communication itself. Policy must therefore address when AI-mediated communication is appropriate, when it should be disclosed, when users should be able to interrupt or refuse it, and who remains accountable when a system speaks on behalf of a person or organization.

Overall, the study contends that society needs ways to identify the moment when AI stops being a background aid and starts acting as a stand-in for a human communicator. Achieving this will require coordinated action from designers, educators, employers, regulators and users.