REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing
Researchers found that autoregressive speech recognition models drift their timestamps during long silences, making transcripts misaligned with audio. A new replay-based method corrects this without catastrophic forgetting.
- Autoregressive ASR systems suffer from timestamp drift during long silences, causing misalignment between transcript and audio.
- Researchers created gap and long-gap benchmarks to evaluate 15 timestamp-producing ASR and audio-language systems.
- Naive fine-tuning to correct timestamps can degrade non-target performance, but replay-based distribution editing avoids this issue.
- The proposed method maintains alignment without catastrophic forgetting, preserving model capabilities.
Autoregressive automatic speech recognition (ASR) systems often generate timestamps alongside transcriptions by emitting tokens that represent time. While this approach avoids the need for frame-level aligners or post-processing, researchers have discovered a critical flaw: timestamps can drift significantly during extended periods of silence in the audio.
The team built two benchmarks, gap and long-gap, to test 15 timestamp-producing ASR and audio-language systems. Their experiments showed that while the transcript itself remains plausible, the decoded time axis progressively drifts away from the actual audio timeline. This misalignment becomes especially problematic in applications requiring precise synchronization, such as captioning or transcription of meetings.
To address this issue, the researchers propose a replay-based distribution editing method. Unlike naive fine-tuning approaches that can severely degrade performance on non-target tasks, their solution corrects timestamp drift while preserving the model's broader capabilities. The method leverages replay buffers to maintain knowledge of previously learned distributions, preventing catastrophic forgetting during correction.
Source: REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing. Read the full piece at the source.
ASR developers need to address timestamp drift in long silences to ensure accurate transcriptions and audio alignment.
Companies relying on ASR for captioning, transcription, or voice interfaces must account for this drift to maintain product reliability.
Investments in ASR technology may benefit from solutions addressing this critical alignment issue.
Accurate speech recognition with proper timestamping is essential for accessibility and real-world applications.
- ASR
- Automatic Speech Recognition, technology that converts spoken language into written text.
- timestamp drift
- A phenomenon where the time markers in a transcript progressively misalign with the actual audio timeline.
- autoregressive
- A model that generates output sequentially, using previous outputs as inputs for subsequent steps.
- catastrophic forgetting
- The loss of previously learned knowledge when a model is updated or fine-tuned on new data.
Mantle cell lymphoma artificial intelligence prognostic index using hematoxylin and eosin histology - Nature
MIRA: Multiplayer Interactive World Models trained on Rocket League [R]
Claude's 'J-Space' Reveals AI's Internal Reasoning Before Output - 조선일보
AI Researchnvidia/Nemotron-Labs-Audex-30B-A3B · Hugging Face
AI Behaved Well Until Scientists Made It Think Nobody Was Watching - NDTV
BusinessOpenAI and Anthropic are giving away millions in computing power to attract startups
OpenAI and Anthropic are offering up to $3 million in free computing credits to startups, with Y Combinator startups alone eligible for $800 million annually. The move aims to lock startups into their ecosystems ahead of potential IPOs.
How OpenAI Plans To Win Over Doctors, Patients And Hospitals - Forbes
OpenAI is rolling out AI tools designed to integrate with healthcare systems, aiming to assist doctors and improve patient outcomes.
BusinessApollo economist warns AI profit gains outside tech could take "well beyond" what Wall Street expects
Apollo’s chief economist warns that AI-driven productivity gains in non-tech sectors like healthcare and banking could take years longer than expected due to regulatory hurdles.
EXCLUSIVE: Beijing is looking at curbing overseas access to China's top AI models, sources say - Reuters
China is reportedly exploring measures to limit overseas access to its most advanced AI models, citing national security risks.
The New Playbook for Enterprise AI Contracts - Emerj Artificial Intelligence Research
Emerj Research outlines a new framework for structuring enterprise AI contracts, focusing on risk management, compliance, and vendor accountability.
Police use of artificial intelligence grows as rules lag behind - Macomb Daily
Law enforcement agencies are increasingly deploying AI tools despite a lack of comprehensive regulations to govern their use.