Databases
StackOverflow2013
Postsdbo
POOut Of Memory Exception - unmanaged memory
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POOut Of Memory Exception - unmanaged memory
    primarykey
    data
    text
    <p>I am working on a web scraper, and it gernally works quite well. It will go through thousands of pages on most sites and complete sucessfully with no issues.</p> <p>On a few sites, I am repeatedly seeing the same issue.</p> <pre><code>Insufficient memory to continue the execution of the program. </code></pre> <hr> <p><strong>Edit:</strong> I used perfmon to determine that the leak is happening in unmanaged memory. I know because "private bytes" keeps increasing as the program runs, while bytes in all heaps stays stable.</p> <p>(actually, it goes up and down, but gradually climbs. it usually runs out of memory in the code section i listed above, but i dont think that section is the cause, but rather a likely first victim because it uses a lot of memory... i think it releases it afterwards though)</p> <hr> <p><strong>Edit 2:</strong></p> <p>I followed the directions on this site: <a href="http://www.codeproject.com/Articles/42721/Best-Practices-No-5-Detecting-NET-application-memo" rel="nofollow">http://www.codeproject.com/Articles/42721/Best-Practices-No-5-Detecting-NET-application-memo</a></p> <p>and i used debugDiag to inspect the program.</p> <p>After analyzing the data, debug diag told me what was responsible for the leak:</p> <pre><code>jscript.dll is responsible for 1.10 GBytes worth of outstanding allocations. The following are the top 2 memory consuming functions: jscript!Parser::GenerateCode+167: 498.19 MBytes worth of outstanding allocations. jscript!NoRelAlloc::PvAlloc+96: 292.99 MBytes worth of outstanding allocations. </code></pre> <p>I am not referencing jscript.dll in my application, it must be being used by the web browser controls which I am using.</p> <pre><code>System.Windows.Forms.WebBrowser </code></pre> <p>Thats my guess, at least.</p> <p>I am also getting a message box that pops up with the title "Message From webpage" that says something to the effect of "out of memory at line X."</p> <p>So, i figured that i could just dispose of the webbrowser objects and get my memory back - so i added a button with the following code:</p> <pre><code>Me.wbMain.Dispose() 'dispose all of thwe web-browsers frmDebugger.wbDebugMain.Dispose() Me.WBNewWin.Dispose() GC.Collect() 'just for the heck of it </code></pre> <p>So, after running it for awhile, i stopped scraping and clicked my new button... it didnt make any difference at all. I was watching the total "Private Bytes" in perfmon, and it didn't even move.</p> <p>Any ideas, anyone?</p> <hr> <p><strong>Edit 3:</strong></p> <p>I have tried a bunch of the recommended solutions, none of them seem to be working.</p> <p>Someone suggested that it may be due to images not being cleared from the cache, but i disabled images from loading, so i know that is not the problem.</p> <p>I also heard that IE7 had an issue, and that upgrading to IE8 would resolve it. I have IE8 and it still leaks memory.</p> <p>Someone suggested that minimizing the form with the webbrowser control would release some memory. I tried, and it does not make a difference.</p> <p>I have also been told that i should not expect the memory use to just drop, as i will have to wait for the garbage collector. It is not a leak in managed code, so GC.Collect() wont do anything. It is in unmanaged memory. Apparently the javascript functionality uses different memory, and theres no manual way to force a collection. But its getting to the point where it crashes, so obviously there is a problem.</p> <p>I am adding a bounty of 50 to this question, and i will award it to anyone who helps me solve the leak. I wanted to try this solution: <a href="http://www.codeproject.com/Questions/322884/WPF-WebBrowser-control-vs-Internet-Explorer-browse" rel="nofollow">http://www.codeproject.com/Questions/322884/WPF-WebBrowser-control-vs-Internet-Explorer-browse</a> but i am unable to figure out what the vb.net equivalent would be. I have tried online converters, and they error when converting this code (though they work fine for other code i have converted in the past)</p> <p>If i am unable to solve the leak, i will award it to anyone who converts the page i mentioned above from c# to vb.net.</p> <p>My fallback plan is to create a separate application that only contains the webbrowser, and communicate with that process, until it runs low on memory, at which point i will restart it (memory is releasd when i clsoe my application completely). This is far from ideal for my application, as the webbrowser is woven pretty tightly into my project.</p> <hr> <p><strong>Edit 4</strong></p> <p>I tried to implement the javascript injection suggested - here is my code:</p> <p>(I fire it just before navigating to a new page)</p> <pre><code>Public Shared Sub Clean_JS(ByRef wb As System.Windows.Forms.WebBrowser) Dim args As Object() = {"document.body"} Dim head As HtmlElement = wb.Document.GetElementsByTagName("head")(0) Dim scriptEl0 As HtmlElement = wb.Document.CreateElement("script") Dim element0 As mshtml.IHTMLScriptElement = DirectCast(scriptEl0.DomElement, mshtml.IHTMLScriptElement) element0.text = "function ReleaseHandler() {" + vbCrLf + " var EvtMgr = (function() {" + vbCrLf + " var listenerMap = {};" + vbCrLf + " " + vbCrLf + " // Public interface" + vbCrLf + " return {" + vbCrLf + " addListener: function(evtName, node, handler) {" + vbCrLf + " node[""on"" + evtName] = handler;" + vbCrLf + " var eventList = listenerMap[evtName];" + vbCrLf + " if (!eventList) {" + vbCrLf + " eventList = listenerMap[evtName] = [];" + vbCrLf + " }" + vbCrLf + " eventList.push(node);" + vbCrLf + " }," + vbCrLf + " " + vbCrLf + " removeAllListeners: function() {" + vbCrLf + " for (var evtName in listenerMap) {" + vbCrLf + " var nodeList = listenerMap[evtName];" + vbCrLf + " for (var i = 0, node; node = nodeList[i]; i++) {" + vbCrLf + " node[""on"" + evtName] = null;" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " })();" + vbCrLf + " }" head.AppendChild(scriptEl0) Dim scriptEl1 As HtmlElement = wb.Document.CreateElement("script") Dim element1 As mshtml.IHTMLScriptElement = DirectCast(scriptEl1.DomElement, mshtml.IHTMLScriptElement) element1.text = "function ReleaseHandler() {" + vbCrLf + " var EvtMgr = (function() {" + vbCrLf + " var listenerMap = {};" + vbCrLf + " " + vbCrLf + " // Public interface" + vbCrLf + " return {" + vbCrLf + " addListener: function(evtName, node, handler) {" + vbCrLf + " node[""on"" + evtName] = handler;" + vbCrLf + " var eventList = listenerMap[evtName];" + vbCrLf + " if (!eventList) {" + vbCrLf + " eventList = listenerMap[evtName] = [];" + vbCrLf + " }" + vbCrLf + " eventList.push(node);" + vbCrLf + " }," + vbCrLf + " " + vbCrLf + " removeAllListeners: function() {" + vbCrLf + " for (var evtName in listenerMap) {" + vbCrLf + " var nodeList = listenerMap[evtName];" + vbCrLf + " for (var i = 0, node; node = nodeList[i]; i++) {" + vbCrLf + " node[""on"" + evtName] = null;" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " }" + vbCrLf + " })();" + vbCrLf + " }" head.AppendChild(scriptEl1) wb.Document.InvokeScript("ReleaseHandler") wb.Document.InvokeScript("purge", args) End Sub </code></pre> <p>unfortunately, i am still seeing privaty bytes increasing in perfmon.</p> <p>can anyone see any flaws in my logic? I am trying to implement this fix: <a href="http://www.codeproject.com/Questions/322884/WPF-WebBrowser-control-vs-Internet-Explorer-browse" rel="nofollow">http://www.codeproject.com/Questions/322884/WPF-WebBrowser-control-vs-Internet-Explorer-browse</a></p> <p>btw - i tested it using simple code such as this:</p> <pre><code>object[] args = {"my important message"}; webBrowser1.Document.InvokeScript("alert",args); </code></pre> <p>and this:</p> <pre><code>Dim head As HtmlElement = wb.Document.GetElementsByTagName("head")(0) Dim scriptEl As HtmlElement = wb.Document.CreateElement("script") Dim element As mshtml.IHTMLScriptElement = DirectCast(scriptEl.DomElement, mshtml.IHTMLScriptElement) element.text = "function sayHello() { alert('hello') }" head.AppendChild(scriptEl) wb.Document.InvokeScript("sayHello") </code></pre> <p>and it showed the message in both test cases.</p> <p>Curiously, when i tried to test the script injection by doing this:</p> <pre><code> Dim head As HtmlElement = wbMain.Document.GetElementsByTagName("head")(0) Dim scriptEl As HtmlElement = wbMain.Document.CreateElement("script") Dim element As mshtml.IHTMLScriptElement = DirectCast(scriptEl.DomElement, mshtml.IHTMLScriptElement) element.text = "function sayHello() { alert('hello') }" head.AppendChild(scriptEl) wbMain.Document.InvokeScript("sayHello") RTB_RawHTML.Text = "TEST" + vbCrLf + wbMain.DocumentText </code></pre> <p>I didnt see the injected code reflected in the text box - the only change i saw was the word "test" appearing (i run the code RTB_RawHTML.Text = wbMain.DocumentText when the pages finish loading from the documentCompleted event...)</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USAllen
    1. USAllen
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POOut Of Memory Exception - unmanaged memory
      1. USAllen
      1. VTBountyStart
    2. VO
      singulars
      1. POOut Of Memory Exception - unmanaged memory
      1. This table or related slice is empty.
      1. VTUpMod
    1. COHave you tried debugging it (press F5) and seeing what the sizes of the entities invloved are on the site that causes the problem? Maybe a judicious use of StringBuilders instead of Strings for variables which are re-used a lot would help (seems unlikely if it works for most sites).
      singulars
      1. POOut Of Memory Exception - unmanaged memory
      1. USAndrew Morton
    2. COi seems i have an unmanaged memory leak somewhere. I think that the section i was originally looking at may be a victim and not the cause of the issue... it would usually crash there because it does in fact use a decent amount of memory... however, i recently noticed that the program was leaking unmanaged memory gradually - but i do not know where from. I updated my post above.
      singulars
      1. POOut Of Memory Exception - unmanaged memory
      1. USAllen
    3. COInstead of using MSHTML, could you get the HTML with a WebClient and parse it with something like Html Agility Pack ( http://htmlagilitypack.codeplex.com/ )?
      singulars
      1. POOut Of Memory Exception - unmanaged memory
      1. USAndrew Morton
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload