The promise of silent, thought-driven communication has long anchored the vision for non-invasive brain-computer interfaces. If a system could reliably parse covert speech—the internal monologue of a user silently pronouncing words without moving their mouth or vocal cords—it would unlock an exceptionally intuitive control mechanism for assistive devices, software, and consumer electronics. A user would simply imagine speaking a command, and signal processing algorithms operating on non-invasive electroencephalography recordings would translate those mental events into digital actions. Over the past decade, dozens of studies have published encouraging results, claiming that computational decoders can successfully categorise specific words or phrases from continuous brain signals.
However, a fundamental assumption underpins this field: that silent speech generates consistent, distinct, and repeatable spectral patterns across the human scalp that modern machine learning algorithms can reliably classify. When signal decoding relies on complex, high-dimensional neural data, distinguishing genuine neurophysiological activity from random noise, muscle artifacts, or processing quirks requires rigorous methodology. Without strict validation protocols and full algorithmic transparency, artificial intelligence models can easily fit to subtle artifacts or temporal quirks in a specific dataset, presenting high classification performance that disappears when applied to new data or tested under identical conditions by independent reviewers.
Re-evaluating published speech decoding pipelines
To determine whether reported successes in speech imagery decoding reflect genuine neural tracking or methodological missteps, researchers conducted an extensive audit focused on both reproducibility—the ability to duplicate published results using original data and procedures—and replicability—the ability to achieve consistent outcomes across different datasets using standardised methods.
The evaluation began with two of the most widely cited, open-access datasets dedicated to speech imagery. Across these datasets, four prominent published decoding pipelines were selected for complete technical reproduction. The audit required building each software pipeline step-by-step according to the published literature, noting every point where methodological instructions were ambiguous, incomplete, or absent.
The audit uncovered widespread reporting deficiencies. Every single speech imagery study examined omitted critical implementation details necessary to execute the pipeline as originally designed. Furthermore, several published studies failed to incorporate essential cross-validation protocols, a fundamental oversight that allows machine learning models to evaluate their own performance on data they have already seen during training, artificially inflating reported accuracy.
When these four pipelines were reconstructed and tested under rigorous evaluation frameworks, the reported performance collapsed. Across all attempted reproductions, the classification accuracies achieved were substantially lower than the figures published in the original papers. The discrepancies between reported claims and audited realities were severe, with accuracy drops ranging from 2 to 39 percentage points. Pipelines that had been presented as high-performing solutions performed barely better than chance once missing steps were filled in and standardised cross-validation was enforced.
Covert speech versus motor imagery
To test whether speech imagery produces stable neural signatures that can generalise across different settings, the investigation broadened into a large-scale replicability study. Standard decoding algorithms were applied across a wide variety of time-frequency feature configurations. This testing encompassed three public speech imagery datasets as well as a newly collected speech imagery dataset.
For direct comparison and baseline validation, the exact same signal processing procedures were applied to four widely used, publicly available datasets for motor imagery—the well-established brain-computer interface paradigm where users imagine moving physical body parts, such as a hand or foot.
The contrast between the two control paradigms was stark. Motor imagery data demonstrated robust, consistent neurophysiological patterns across time and frequency bands. When evaluated against strict statistical significance thresholds, 91 percent of participants in the motor imagery datasets yielded decoding accuracies significantly above random chance, confirming that motor imagery produces dependable neural markers across diverse subjects and testing conditions.
Speech imagery data, by contrast, failed to exhibit any consistent time-frequency patterns across datasets. Spectral signatures varied unpredictably between individuals and across different testing runs. Crucially, across all evaluated speech imagery datasets, only 36 percent of participants achieved classification performance that exceeded statistical significance thresholds. For the remaining nearly two-thirds of subjects, the decoders performed no better than guessing, indicating that the algorithms were attempting to classify noise or non-specific background signals rather than structured speech-related neural activity.
Methodological fragility in non-invasive neural interfaces
These findings point to systemic vulnerabilities in how covert speech research has been conducted and reported. The failure to find consistent spectral markers across multiple speech imagery datasets suggests that non-invasive surface electroencephalography may lack the spatial and functional resolution required to capture the subtle, distributed cortical processes underlying internal speech. Unlike motor imagery, which generates prominent, localised sensorimotor rhythms across the cortex, covert speech involves intricate, low-amplitude networks that are easily masked by background neural noise, eye movements, or subtle muscular tension.
When signal-to-noise ratios are low, machine learning models are prone to exploiting incidental features within small datasets. If evaluation procedures do not enforce strict trial-level cross-validation, or if signal processing parameters are tuned after inspecting the full dataset, models appear to perform with impressive precision. When independent auditors attempt to apply those same parameters to new data or enforce standard validation splits, the apparent predictive power vanishes.
This analysis does not mean that internal speech leaves no trace in brain activity, but it strongly indicates that current non-invasive signal processing pipelines cannot reliably extract those traces for practical brain-computer interface applications. The widespread reliance on incomplete reporting and flawed evaluation procedures has created an inflated sense of technical maturity around speech imagery, misdirecting attention and investment toward paradigms that remain fundamentally unproven at a practical level.
Sources
- Decoding speech imagery or just noise?: a symptom of the replicability crisis, Tates A et al., Journal of neural engineering, 2026-09-15
- Publisher record (DOI)
The R&D takeaway
Standardised benchmarking protocols and fully open, complete code repositories must become mandatory requirements when evaluating non-invasive brain-computer interface paradigms. Engineering teams developing neural interface products should treat non-invasive speech imagery as an unproven, high-risk research concept, prioritising robust paradigms like motor imagery for near-term product roadmaps. Auditing literature pipelines for hidden cross-validation errors and missing implementation parameters will protect early-stage research capital from being committed to unrepeatable signal processing architectures.
The R&D Innovate desk