Chatbots Versus SALT: Can AI Accurately Analyse Children’s Complex Language?

Artificial intelligence could reduce the time speech-language pathologists spend analysing children's language samples. New evidence suggests that today's chatbots remain far more dependable at counting words than interpreting complex sentence structures.

The original study, 'Convergence between AI chatbot-calculated microstructure measures and SALT analysis of child language samples,' by Richy Lewis Hayes and Pui Fong Kan, was published in Frontiers in Language Sciences. The researchers compared calculations produced by ChatGPT, Microsoft Copilot, MagicSchool AI and Google Gemini with results from Systematic Analysis of Language Transcripts, widely known as SALT.

The findings reveal strong correspondence for several common language measures, especially those based on words and morphemes. The results do not show that chatbots can replace SALT, trained clinicians or careful human review, particularly when a task requires identifying relationships between clauses.

Why faster language-sample analysis could matter for children

Language sample analysis allows speech-language pathologists to examine how children communicate in natural situations, capturing vocabulary, grammar, sentence construction, fluency and narrative ability in ways that standardised tests may miss. It can be valuable when assessing multilingual children, whose abilities may not be represented fairly by conventional assessment tools.

Clinical use remains limited because recording, transcribing, coding and calculating a sample can take considerable time, creating a difficult workload for professionals already managing large caseloads. User-friendly chatbots offer an appealing possibility: a clinician could enter a transcript, request several measurements and receive results almost immediately.

Large language models generate responses by predicting probable sequences of text rather than independently confirming that every calculation is correct, meaning a confident-looking answer can still contain counting errors, invented information or incorrect linguistic interpretations. Child language data also raises privacy concerns because speech patterns can be highly personal, even after obvious identifying details have been removed.

The researchers treated chatbot analysis as an exploratory method that required validation, not as an established clinical shortcut.

Four chatbots were tested on 84 children's English narratives

The study included 84 typically developing children between two and nine years old, including 35 girls. Thirty participants were Japanese-English bilinguals, and 54 were English monolinguals, with only their English-language narratives used in the analysis.

Children listened to a story from the Multilingual Assessment Instrument for Narratives and retold it in their own words using the parallel Dog and Cat stories. Recordings were transcribed and coded according to SALT conventions by the principal investigator and a trained research assistant, whose coding reached 94.9% item-by-item agreement across 20 independently examined transcripts.

Six language features were calculated:

• Mean length of utterance in words, or MLUw

• Mean length of utterance in morphemes, or MLUm

• Number of total words, or NTW

• Number of different words, or NDW

• Subordination index, or SI

• Mazes, including repetitions, filled pauses and false starts

Each transcript was divided into communication units, with one independent clause and its attached subordinate clauses placed on a separate line. Mazes were marked in parentheses, giving the chatbots more structured material than an unprepared conversational transcript.

The researchers used free standard versions of ChatGPT, Copilot, MagicSchool and Gemini during data collection from September to November 2025. Every transcript was submitted first with a general instruction and later with a more detailed prompt explaining the calculation rules for MLUm and SI.

On a few occasions, Copilot, MagicSchool and Gemini initially calculated SI using the wrong formula. The researchers corrected them by explaining that SI divides the total number of clauses by the total number of utterances, and the revised answers were used.

Word-based measurements aligned closely, but clause analysis remained difficult

Every chatbot-generated measure was associated with its SALT equivalent, with all results reaching a statistical significance level below 0.001. The strength of that relationship differed substantially by measurement.

Mean length of utterance in words produced standardised coefficients ranging from 0.90 to 0.97, and total word counts ranged from 0.85 to 0.98. Mean length in morphemes also performed strongly, with coefficients between 0.89 and 0.93, followed by maze counts at 0.89 to 0.94 and vocabulary diversity at 0.74 to 0.97.

The subordination index was the clear weak point, producing coefficients between 0.43 and 0.70. Calculating SI requires a system to recognise clause boundaries and distinguish main clauses from subordinate ones, demanding deeper linguistic interpretation than simply counting words or marked pauses.

Agreement scores followed a similar pattern: MagicSchool recorded high agreement with SALT for MLUw, MLUm, NTW and NDW, with intraclass correlation coefficients between 0.93 and 0.97. ChatGPT produced the strongest association for maze counting. These differences were descriptive because the study was not designed to test whether one chatbot statistically outperformed another.

Strong association does not mean identical results: Two systems can rise and fall together across samples while producing values that differ enough to affect an individual assessment, making agreement and decision-level accuracy essential considerations before clinical use.

Children's backgrounds influenced a limited number of results: Age altered only Copilot's NDW relationship with SALT, which was stronger among school-age children than preschoolers. Language background affected ChatGPT and Copilot NDW, MagicSchool SI and mazes, and Gemini MLUw, MLUm and NTW. Most of these relationships were stronger for bilingual children, with MagicSchool's maze calculation as the exception.

The authors cautioned that this pattern may reflect unequal sample sizes, shorter bilingual narratives or differences in score distributions rather than better chatbot accuracy for bilingual speakers.

More detailed prompts failed to deliver more accurate calculations

The detailed prompt did not strengthen the relationship with SALT for either MLUm or SI on any chatbot. Copilot's MLUm and SI associations became weaker, as did MagicSchool's MLUm association, challenging the popular assumption that adding instructions automatically produces a better answer.

Longer prompts may increase processing demands or introduce instructions that a model applies inconsistently. Preparing transcripts with useful structural markings might prove more effective than filling a prompt with technical definitions, though extensive manual coding would also reduce the time-saving value of chatbot assistance.

Several limitations keep these findings from supporting immediate clinical adoption. The sample included only typically developing children, the age range was narrow, and the average narrative contained 14.88 utterances, far below 50 utterances often recommended for language assessment. The researchers did not test diagnostic decisions, compare results directly with practising speech-language pathologists, measure workflow efficiency or investigate every source of chatbot error.

The models are continually updated, making exact replication difficult, and manually separating transcripts into communication units created unusually controlled conditions that may not reflect routine practice. Small numerical differences could still change how a clinician interprets a child's abilities.