Databases
StackOverflow2013
Postsdbo
POOptimizing a set 20 webrequests with threads
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POOptimizing a set 20 webrequests with threads
    primarykey
    data
    text
    <p>This is for ASP.NET. I want to improve the time it takes run my function, today it takes around 20-30 seconds, more towards 30secs than 20secs though. That's running on one thread making 20 webrequests.</p> <p>I'm thinking threads that do all the 20 webreqeusts, in order to quickly find the result or just go through the data (IE do all the 20 requests not finding anything).</p> <p><br> <strong>Here's how it works.</strong><br> 1. I'm using html agility pack to fetch htmldocuments.<br> 2. Then I parse them for information<br> 3. Lastly I add that information to a dictionary <strong>OR</strong> I move on to the next webrequest until I reach 20 requests made. </p> <p>I make at most 20 webRequests, at minimum 1. I have set the function to end when the info I'm searching for is found. Sometimes the info isn't there hence the 20 webrequests(it goes through all the data).</p> <p>Every webrequest adds between 5-20 entries to the dictionary. This is then compared with the information I sent to it, if it's in the list I get the Key back, otherwise it returns 201. If found it gets added to the database.</p> <p><strong>QUESTIONS</strong></p> <p>*A:*If I want to do this with threads, how many should I create? 20 One for each request and let them all loose to do the job? Or should i create like 4 of them making at most 5 requests each?<br><br><em>B:</em> What if two threads are finished at the same time and wants to add info to the directory, can it lock the whole site(I'm using ASP.NET), or will it try to add one from thread A and then one result from Thread B? I have a check already today that checks if the key exists before adding it.</p> <p><strong>C:What would be the fastest way to this?</strong></p> <p>This is my code, depicting the loop which just shows that 20 requests are being made?</p> <pre><code>public void FetchAndParseAllPages() { int _maxSearchDepth = 200; int _searchIncrement = 10; PageFetcher fetcher = new PageFetcher(); for (int i = 0; i &lt; _maxSearchDepth; i += _searchIncrement) { string keywordNsearch = _keyword + i; ParseHtmldocuments(fetcher.GetWebpage(keywordNsearch)); if (GetPostion() != 201) { //ADD DATA TO DATABASE InsertRankingData(DocParser.GetSearchResults(), _theSearchedKeyword); return; } } } </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. US8bitcat
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload