Databases
StackOverflow2013
Postsdbo
POHow do I achieve better granularity when performing streaming operations?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow do I achieve better granularity when performing streaming operations?
    primarykey
    data
    text
    <p>Okay so I'm working on my file transfer service, and I can transfer the files fine with WCF streaming. I get good speeds, and I'll eventually be able to have good resume support because I chunk my files into small bits before streaming.</p> <p>However, I'm running into issues with both the server side transfer and the client side receiving when it comes to measuring a detailed transfer speed as the messages are streamed and written.</p> <p>Here's the code where the file is chunked, which is called by the service every time it needs to send another chunk to the client.</p> <pre><code> public byte[] NextChunk() { if (MoreChunks) // If there are more chunks, procede with the next chunking operation, otherwise throw an exception. { byte[] buffer; using (BinaryReader reader = new BinaryReader(File.OpenRead(FilePath))) { reader.BaseStream.Position = currentPosition; buffer = reader.ReadBytes((int)MaximumChunkSize); } currentPosition += buffer.LongLength; // Sets the stream position to be used for the next call. return buffer; } else throw new InvalidOperationException("The last chunk of the file has already been returned."); </code></pre> <p>In the above, I basically write to the buffer based on the chunk size I am using(in this case it's 2mb which I found to have the best transfer speeds compared to larger or smaller chunk sizes). I then do a little work to remember where I left off, and return the buffer.</p> <p>The following code is the server side work.</p> <pre><code>public FileMessage ReceiveFile() { if (!transferSpeedTimer.Enabled) transferSpeedTimer.Start(); byte[] buffer = chunkedFile.NextChunk(); FileMessage message = new FileMessage(); message.FileMetaData = new FileMetaData(chunkedFile.MoreChunks, buffer.LongLength); message.ChunkData = new MemoryStream(buffer); if (!chunkedFile.MoreChunks) { OnTransferComplete(this, EventArgs.Empty); Timer timer = new Timer(20000f); timer.Elapsed += (sender, e) =&gt; { StopSession(); timer.Stop(); }; timer.Start(); } //This needs to be more granular. This method is called too infrequently for a fast and accurate enough progress of the file transfer to be determined. TotalBytesTransferred += buffer.LongLength; return message; } </code></pre> <p>In this method, which is called by the client in a WCF call, I get information for the next chunk, create my message, do a little bit with timers to stop my session once the transfer is complete and update the transfer speeds. Shortly before I return the message I increment my <code>TotalBytesTransferred</code> with the length of the buffer, which is used to help me calculate transfer speed.</p> <p>The problem with this, is it takes a while to stream the file to the client, and so the speeds I'm getting are false. What I'm trying to aim for here is a more granular modification to the <code>TotalBytesTransferred</code> variable so I have a better representation of how much data is being sent to the client at any given time.</p> <p>Now, for the client side code, which uses an entirely different way of calculating transfer speed.</p> <pre><code>if (Role == FileTransferItem.FileTransferRole.Receiver) { hostChannel = channelFactory.CreateChannel(); ((IContextChannel)hostChannel).OperationTimeout = new TimeSpan(3, 0, 0); bool moreChunks = true; long bytesPreviousPosition = 0; using (BinaryWriter writer = new BinaryWriter(File.OpenWrite(fileWritePath))) { writer.BaseStream.SetLength(0); transferSpeedTimer.Elapsed += ((sender, e) =&gt; { transferSpeed = writer.BaseStream.Position - bytesPreviousPosition; bytesPreviousPosition = writer.BaseStream.Position; }); transferSpeedTimer.Start(); while (moreChunks) { FileMessage message = hostChannel.ReceiveFile(); moreChunks = message.FileMetaData.MoreChunks; writer.BaseStream.Position = filePosition; // This is golden, but I need to extrapolate it out and do the stream copy myself so I can do calculations on a per byte basis. message.ChunkData.CopyTo(writer.BaseStream); filePosition += message.FileMetaData.ChunkLength; // TODO This needs to be more granular TotalBytesTransferred += message.FileMetaData.ChunkLength; } OnTransferComplete(this, EventArgs.Empty); } } else { transferSpeedTimer.Elapsed += ((sender, e) =&gt; { totalElapsedSeconds += (int)transferSpeedTimer.Interval; transferSpeed = TotalBytesTransferred / totalElapsedSeconds; }); transferSpeedTimer.Start(); host.Open(); } </code></pre> <p>Here, my <code>TotalBytesTransferred</code> is also based on the length of the chunk coming in. I know I can get a more granular calculation if I do the stream writing myself instead of using the <code>CopyTo</code> for the stream, but I'm not exactly sure how to best go about this.</p> <p>Can anybody help me out here? Outside of this class I have another class polling the property of <code>TransferSpeed</code> as it's updated internally.</p> <p>I apologize if I posted too much code, but I wasn't sure what to post and what not.</p> <p><strong>EDIT:</strong> I realize at least with the Server side implementation, the way I can get a more granular reading on how many bytes have been transferred, is by reading the position of the return message value of the stream. However, I don't know a way to do this to ensure absolute integrity on my count. I thought about maybe using a timer and polling the position as the stream was being transferred, but then the next call might be made and I would quickly become out of sync.</p> <p>How can I poll data from the returning stream and know immediately when the stream finishes so I can quickly add up the remainder of what was left of the stream into my byte count?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCowman
    1. USCowman
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow do I achieve better granularity when performing streaming operations?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow do I achieve better granularity when performing streaming operations?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POHow do I achieve better granularity when performing streaming operations?
      1. USSchautDollar
      1. VTFavorite
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload