The Clues Inside Fake AI Voices: What Enhanced FusionNet Reveals About Detection

Researchers are investigating which details in recorded speech can help computers separate a real person’s voice from a synthetic imitation.

A familiar voice on a call can feel reassuring, especially when it sounds like someone you know and trust. Artificial intelligence can imitate the qualities that make a voice recognisable, creating opportunities for impersonation and fraud across services that depend on spoken communication. Researchers are investigating which details in recorded speech can help computers separate a real person's voice from a synthetic imitation.

Bhargavi Anumula and Anil V. Turukmane of VIT-AP University explored this challenge in "Enhanced FusionNet: a deep learning structure for multimodal input-based emotion-aware deepfake speech identification," published in Frontiers in Artificial Intelligence. Their system combines several ways of analysing speech and examines whether information about emotion helps identify fake recordings. The research combines strong detection results and a revealing finding about the limits of adding emotional clues.

Enhanced FusionNet achieved 98.67% accuracy on an independent test set drawn from the Fake or Real dataset. Adding a separately learned emotion representation reduced accuracy slightly, showing that extra information does not automatically make a detector better. These findings describe performance under specific research conditions, leaving important questions about unfamiliar voices, new synthesis tools and everyday recordings.

How the system searches for clues in speech

The researchers used 8,358 recordings from the Fake or Real dataset, comprising 4,167 genuine samples and 4,191 synthetic samples after quality checks and duplicate removal. They reserved 5,963 recordings for training, 1,196 for validation during development and 1,199 for independent testing. A separate collection called RAVDESS supplied 1,439 recordings from 24 actors expressing eight emotions, with different speakers assigned to training, validation and testing.

Audio was standardised to a sampling rate of 16 kHz and converted into numerical descriptions of sound, including Log-Mel features that represent energy across frequency bands. The system combined these acoustic measurements with information extracted by a pretrained Wav2Vec 2.0 model, producing a representation containing 1,016 values per recording. The paper's use of "multimodal" refers here to combining speech representations, with emotion information learned separately and added in a controlled experiment.

Inside FusionNet, a MultiScaleCNN searches for patterns at different scales, and a bidirectional sequence network called BiLSTM processes relationships across the input. Cross-attention connects information from these branches, residual fusion preserves earlier information, and adaptive gates learn which features deserve more weight. Attention pooling gathers useful details for the final real-or-fake decision, creating a system designed to examine several aspects of a recording together.

Strong detection results with identifiable mistakes

On the independent test set, FusionNet correctly classified 1,183 of 1,199 recordings, identifying 595 of 600 fake samples and 588 of 599 genuine samples. Five synthetic recordings passed as real, and 11 genuine recordings were incorrectly flagged as fake. Precision reached 98.18%, recall reached 99.17%, and the F1-score, which balances those two measures, reached 98.67%.

The model's ROC-AUC score reached 99.88%, indicating strong separation between genuine and synthetic recordings across different decision thresholds. Five-fold cross-validation, which repeated evaluation across different portions of the dataset, produced an average accuracy of 99.08%, with a standard deviation of 0.35 percentage points. Across those evaluations, the system made 77 mistakes among 8,358 recordings, including 52 genuine recordings labelled fake and 25 fake recordings labelled real.

The combined cross-validation predictions produced 99.40% sensitivity, 98.75% specificity and an equal error rate of about 0.91%, describing different aspects of the detector's mistakes. Using 10,000 resamples of those predictions, the researchers estimated a 95% confidence interval of 98.863% to 99.282% for accuracy. The model outperformed the four implemented baselines on reported hold-out accuracy, with matched fold-level comparisons unavailable to establish stronger claims of superiority.

Emotion added information without improving overall accuracy

The emotion encoder achieved 61.51% accuracy and a macro-F1 score of 59.81% when tested on speakers it had not encountered during training. Recognition varied across emotions, with anger and surprise receiving the highest F1-scores and happiness and neutrality proving more difficult. Confusion between calm and neutral speech illustrates how similar emotional expressions can be challenging to separate across different voices.

Adding 128 emotion-related values to the detector increased its input from 1,016 to 1,144 values and reduced accuracy from 98.67% to 98.42%. Precision and specificity improved slightly, accompanied by declines in sensitivity, F1-score and ROC-AUC, and an increase in equal error rate from 0.83% to 1.50%. The authors describe emotion information as complementary, with no overall detection benefit demonstrated by this experiment.

A separate component-removal analysis found the largest accuracy drop when MultiScaleCNN was removed, falling from 99.16% to 85.82%. Removing the sequence network, cross-attention, residual fusion, adaptive gating or attention pooling produced smaller declines, supporting further investigation of their individual roles. That analysis used an implementation differing from the main architecture, and visual plots showing real and fake samples clustering apart offered illustrative evidence rather than independent proof of reliability.

What needs testing before everyday use

Reliable speaker identities were unavailable throughout the Fake or Real dataset, preventing the researchers from guaranteeing that training and testing contained entirely different speakers. An audit found no cross-partition content overlap under its procedure, and the study did not test an external dataset containing previously unseen speech-generation methods. Recording conditions, channel characteristics and synthesis-specific traces could influence the results, limiting what these scores tell us about performance outside the benchmark.

Enhanced FusionNet contains 976,779 trainable parameters, substantially more than the smaller comparison models, and its processing speed, memory requirements and throughput need measurement. The researchers identify multilingual speech, noisy recordings, unfamiliar synthesis methods and comparisons with contemporary anti-spoofing systems as priorities for further research. Its current task is classifying recordings as genuine or synthetic, leaving complete speaker verification and voice authentication outside the demonstrated system.

The study used public datasets and recruited no new participants, with the researchers reporting no research funding or commercial conflicts of interest. Emotion analysis raises privacy concerns, making authorised use and appropriate handling of voice data part of responsible deployment. Enhanced FusionNet offers encouraging benchmark results and a useful lesson for future detectors: emotional information needs to earn its place through measured improvements.