StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHow Can I Deal with C# .NET TimeSpan Progressive Rounding Error while Recording Video Frame-by-Frame?
text
Body
copied!<p>This is a neat issue, and not a "tell me what code works", but rather a "how do I logically handle this situation" question.</p> <p>I have, in short, video + audio coming in a from an IP camera via RTSP.</p> <p>The video and audio are being decoded and recorded frame-by-frame to a single mp4 container, by separate threads (shown below).</p> <p>The problem is video and audio becoming progressively more out of sync over time, due to lack of precision with the TimeSpan end time and start times for each video frame.</p> <p>It should be a duration of 1 / framerate = 0.0333667000333667 for each video frame, but it's using (even with the FromTicks() method), start time = 0.0 and end time of 0.0333667 for the first frame.</p> <p>I can adjust the video decoder framerate value from 29.97 (it's pulling that in from the camera's settings declared framerate), resulting in either video that precedes the audio, or lags after the audio--this is simply making each video mediaBuffer.StartTime and mediaBuffer.EndTime either too early or too late, in comparison to the audio.</p> <p>Over time, the miniscule decimal truncation ends up making the video and audio out of sync--the longer the recording, the more out of sync the two tracks get.</p> <p>I don't really understand why this is happening, because, rounding error shouldn't logically matter.</p> <p>Even if I only had a precision of 1 second, I'd only write a video frame each second, and it's placement in the timeline would be roughly where it should be +- 1 second, and that should make every progressive frame the same +- 1 second to where it should be, not adding up progressively more misplacement. I'm imagining this would look like for each frame:</p> <p>[<-------- -1 second --------> exact frame time expected <-------- +1s -------->] ---------------------------------------------------- recorded frame time --------</p> <p>Am I missing something here?</p> <p>I'm not doing "new frame start time = last frame end time, new frame end time = new frame start time + 1 / framerate"--I'm actually doing "new frame start time = frame index - 1 / framerate, new frame end time = frame index / framerate".</p> <p>That is, I'm calculating the frame start and end times based on the expected time they should have (frame time = frame position / framerate).</p> <p>What my code is doing is this:</p> <p>time expected ---------- time expected ---------- time expected frame time frame time frame time </p> <p>I understand the issue mathematically, I just don't understand why decimal truncation is proving such a problem, or logically know what the best solution is to fix it.</p> <p>If I implement something that says "every x frames, use "(1 / framerate) + some amount" to make up for all the missing time, will that be possible to have frames match where they should be, or just result in messy video?</p> <pre><code> public void AudioDecoderThreadProc() { TimeSpan current = TimeSpan.FromSeconds(0.0); while (IsRunning) { RTPFrame nextFrame = jitter.FindCompleteFrame(); if (nextFrame == null) { System.Threading.Thread.Sleep(20); continue; } while (nextFrame.PacketCount > 0 && IsRunning) { RTPPacket p = nextFrame.GetNextPacket(); if (sub.ti.MediaCapability.Codec == Codec.G711A || sub.ti.MediaCapability.Codec == Codec.G711U) { MediaBuffer<byte> mediaBuffer = new MediaBuffer<byte>(p.DataPointer, 0, (int)p.DataSize); mediaBuffer.StartTime = current; mediaBuffer.EndTime = current.Add(TimeSpan.FromSeconds((p.DataSize) / (double)audioDecoder.SampleRate)); current = mediaBuffer.EndTime; if (SaveToFile == true) { WriteMp4Data(mediaBuffer); } } } } } public void VideoDecoderThreadProc() { byte[] totalFrame = null; TimeSpan current = TimeSpan.FromSeconds(0.0); TimeSpan videoFrame = TimeSpan.FromTicks(3336670); long frameIndex = 1; while (IsRunning) { if (completedFrames.Count > 50) { System.Threading.Thread.Sleep(20); continue; } RTPFrame nextFrame = jitter.FindCompleteFrame(); if (nextFrame == null) { System.Threading.Thread.Sleep(20); continue; } if (nextFrame.HasSequenceGaps == true) { continue; } totalFrame = new byte[nextFrame.TotalPayloadSize * 2]; int offset = 0; while (nextFrame.PacketCount > 0) { byte[] fragFrame = nextFrame.GetAssembledFrame(); if (fragFrame != null) { fragFrame.CopyTo(totalFrame, offset); offset += fragFrame.Length; } } MediaBuffer<byte> mediaBuffer = new MediaBuffer<byte>( totalFrame, 0, offset, TimeSpan.FromTicks(Convert.ToInt64((frameIndex - 1) / mp4TrackInfo.Video.Framerate * 10000000)), TimeSpan.FromTicks(Convert.ToInt64(frameIndex / mp4TrackInfo.Video.Framerate * 10000000))); if (SaveToFile == true) { WriteMp4Data(mediaBuffer); } lock (completedFrames) { completedFrames.Add(mediaBuffer); } frameIndex++; } } </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload