Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>I would personally look at using Linq to Xml utilizing the streaming techniques outlined in the Microsoft help file: <a href="http://msdn.microsoft.com/en-us/library/system.xml.linq.xstreamingelement.aspx#Y1392" rel="noreferrer">http://msdn.microsoft.com/en-us/library/system.xml.linq.xstreamingelement.aspx#Y1392</a></p> <p>Here's a quick benchmark test reading from a 200mb xml file with a simple filter:</p> <pre class="lang-cs prettyprint-override"><code>var xmlFilename = "test.xml"; //create test xml file var initMemoryUsage = GC.GetTotalMemory(true); var timer = System.Diagnostics.Stopwatch.StartNew(); var rand = new Random(); var testDoc = new XStreamingElement("root", //in order to stream xml output XStreamingElement needs to be used for all parent elements of collection so no XDocument Enumerable.Range(1, 10000000).Select(idx =&gt; new XElement("child", new XAttribute("id", rand.Next(0, 1000)))) ); testDoc.Save(xmlFilename); var outStat = String.Format("{0:f2} sec {1:n0} kb //linq to xml ouput streamed", timer.Elapsed.TotalSeconds, (GC.GetTotalMemory(false) - initMemoryUsage) / 1024); //linq to xml not streamed initMemoryUsage = GC.GetTotalMemory(true); timer.Restart(); var col1 = XDocument.Load(xmlFilename).Root.Elements("child").Where(e =&gt; (int)e.Attribute("id") &lt; 10).Select(e =&gt; (int)e.Attribute("id")).ToArray(); var stat1 = String.Format("{0:f2} sec {1:n0} kb //linq to xml input not streamed", timer.Elapsed.TotalSeconds, (GC.GetTotalMemory(false) - initMemoryUsage) / 1024); //xmlreader initMemoryUsage = GC.GetTotalMemory(true); timer.Restart(); var col2 = new List&lt;int&gt;(); using (var reader = new XmlTextReader(xmlFilename)) { while (reader.ReadToFollowing("child")) { reader.MoveToAttribute("id"); int value = Convert.ToInt32(reader.Value); if (value &lt; 10) res2.Add(value); } } var stat2 = String.Format("{0:f2} sec {1:n0} kb //xmlreader", timer.Elapsed.TotalSeconds, (GC.GetTotalMemory(false) - initMemoryUsage) / 1024); //linq to xml streamed initMemoryUsage = GC.GetTotalMemory(true); timer.Restart(); var col3 = StreamElements(xmlFilename, "child").Where(e =&gt; (int)e.Attribute("id") &lt; 10).Select(e =&gt; (int)e.Attribute("id")).ToArray(); var stat3 = String.Format("{0:f2} sec {1:n0} kb //linq to xml input streamed", timer.Elapsed.TotalSeconds, (GC.GetTotalMemory(false) - initMemoryUsage) / 1024); //util method public static IEnumerable&lt;XElement&gt; StreamElements(string filename, string elementName) { using (var reader = XmlTextReader.Create(filename)) { while (reader.Name == elementName || reader.ReadToFollowing(elementName)) yield return (XElement)XElement.ReadFrom(reader); } } </code></pre> <p>And here's the processing time and memory usage on my machine:</p> <pre><code>11.49 sec 225 kb // linq to xml ouput streamed 17.36 sec 782,312 kb // linq to xml input not streamed 6.52 sec 1,825 kb // xmlreader 11.74 sec 2,238 kb // linq to xml input streamed </code></pre>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. This table or related slice is empty.
    2. VO
      singulars
      1. This table or related slice is empty.
    3. VO
      singulars
      1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload