Databases
StackOverflow2013
Postsdbo
PONodeJS - Force users to wait until global event is completed
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PONodeJS - Force users to wait until global event is completed
    primarykey
    data
    text
    <p>I have a Node server that does the following:</p> <p>I have a list of URLs in an external server, call it URLServer. When a user hits my NODE server, my node server makes a request to the URLServer and gets a list of say 20 URLs. As soon as we get those 20 URLs, I want my node server to go and get the title for each of these URLs, which means that I will fetch the URLs and create a DOM and then extract the title, I also get other data, so this is the way it has to get done. Once I have done that, I want the title of the URLs and the URLs to be saved in internal memory and/or database. So I have a URL-cache and a title-cache (I don't want to fetch the URLs all the time).</p> <p>I have something like this: if(URL-cache is empty) get URLS from URLServer and cache these URLs</p> <p>I then want to check each of those URLs to see if their titles are in my cache, so I do: for each URL if title-cache[URL], good else fetch site, create DOM, extract title + other data and cache</p> <p>This works great for one user, but I when I try a heavy load in the server, the server will hang. I have concluded the server hangs for the following reason:</p> <p>User 1 Request - Empty Caches - Fetch URLs and when done fetch Content for URLs User 2 Request - The caches still look empty to this user because the request for user 1 has not yet completed!!! Therefore, User 2 forces once again a fetch of the URLs and their respective content. User 3 Request - User 1 and User 2 requests are not yet completed so the same issue...</p> <p>So, assuming I have 10 URLs I need to fetch, instead of opening 10 connections, one per URL and then caching the data, if I have 20 users hitting the server at the exact same time, I will be opening 200 connections (each user opens 10 connections).</p> <p>How can I block User X (where X>1) from causing these events? I basically want the server to close a gate and ask every user to wait until it has populated the caches, then opening the gates once these are populated, is there any way to do this?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USEl Moreno
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PONodeJS - Force users to wait until global event is completed
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload