Databases
StackOverflow2013
Postsdbo
POPausing AVAssetWriter when using appendSampleBuffer for audio and video results in desynchronization. Is there a way to pause recording?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPausing AVAssetWriter when using appendSampleBuffer for audio and video results in desynchronization. Is there a way to pause recording?
    primarykey
    data
    text
    <p>I have two methods (one for audio, and one for video) that append their respective samples to an AVAssetWriter via respective AVAssetWriterInput's. Everything works well for the simple start-and-stop case. Some background here is that the audio stream is coming from coreAudio, where I've applied echo suppression, while the video feed is coming via AVCaptureVideoDataOutputSampleBufferDelegate fed from an AVCaptureVideoDataOutput via setSampleBufferDelegate.</p> <p>The problem is when I want to pause recording, and later resume recoding, different things happen for the audio and video portion of the feed. Thus far, I have been unable to keep the audio and video synchronized after a resume-from-pause.</p> <p>It appears that when appending audio sample data via appendSampleBuffera, it does not seem to 'skip over' time periods for which there are no samples, even though the CMSampleBufferRef has accurate time stamps. With that I mean, when audio samples are appended, it doesn't really care what the timestamp is on the CMSampleBuffer... it just keeps piling the samples back-to-back, even if there was (for example) a 120 second gap in the audio data, it doesn't make an 'empty space' for it.</p> <p>I frankly wouldn't care, as long as the behaviour for Video samples would be the same (but it is not). For video samples resulting video encoding does skip over the samples though it makes a prolonged empty video frame for the period for which there is no data.</p> <p>So from a naive implementation, during the pause, you get a blank video section for the duration of the pause, but for the audio section, you get the samples of the next unpaused section back-to-back. This means that pausing and unpausing (by supressing adding buffer data to the avasset writers), causes a complete desynchronization between audio and video. </p> <p>Remedies I have tried:</p> <ol> <li><p>using CMSetAttachment() set kCMSampleBufferAttachmentKey_FillDiscontinuitiesWithSilence -- for the audio channel, hoping it would do similarly what the video stream is doing. This had no effect at all. I may have done it wrong, but there is no real documentation on this, so shooting in the dark on this one.</p></li> <li><p>add my own empty samples for the affected duration of 'pause' (via appendLapsedAudiotime). Issue is that I don't think I got the buffers correct, and it's unclear to me if it is possible to add an audio buffer with just 0 or 1 sample, and stretch that out over a very long period of time. It may be possible if I also change the audio buffer format to do 1 sample for the entire pause period, but it's not likeley that downstream low-level API's will appreciate wonky sample rate settings). </p></li> <li><p>using CMSampleBufferCreateCopyWithNewTiming() change the presentation timing of the video, to coincide with the behavior of the audio. This proved both unstable, and didn't work. I iterated several times on this, including adjust both timestamps for audio and video. It's probably I made mistakes there, but given the next-to-no documentaiton, I'm flying blind on this approach as well.</p></li> <li><p>usse multiple startSessionAtSourceTime / endSessionAtSourceTime along with suppressing writing stamples during the pause-state, but that didn't work, and in fact a note in Apples header file alerts you that this isn't supported (in contrast to their online documentation): NOTE: Multiple sample-writing sessions are currently not supported. It is an error to call startSessionAtSourceTime: a second time after calling endSessionAtSourceTime:</p></li> </ol> <p>So my next approach is to create multiple AVAssets. When there is a 'pause start', I'll finalize the recording, and when there is a 'pause stop', I'll start a whole new recording, and when all is said and done, I'll use an AVAssetComposition to string it all back together again. I'm a bit nervous about this approach because I have found that rapid av capture stops and starts wind up with inexplicable failures in various places when setting up a new av capture. I may have to force the 'unpause' to wait some period of time before I can try to start a new av capture.</p> <p>So the question is... IS THERE A BETTER WAY?</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USBernt Habermeier
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. POPausing AVAssetWriter when using appendSampleBuffer for audio and video results in desynchronization. Is there a way to pause recording?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POPausing AVAssetWriter when using appendSampleBuffer for audio and video results in desynchronization. Is there a way to pause recording?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POPausing AVAssetWriter when using appendSampleBuffer for audio and video results in desynchronization. Is there a way to pause recording?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI am also trying to create a CMSampleBuffer to feed an AVAssetWriter for audio data. My audio data are in memory and I can't figure out how to convert them to CMSampleBuffers. Could you share some code or point me to the right direction?
      singulars
      1. POPausing AVAssetWriter when using appendSampleBuffer for audio and video results in desynchronization. Is there a way to pause recording?
      1. USPetrakeas
    2. COSure -- do you have a question posted already? I might as well answer your question there so others might benefit.
      singulars
      1. POPausing AVAssetWriter when using appendSampleBuffer for audio and video results in desynchronization. Is there a way to pause recording?
      1. USBernt Habermeier
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload