Databases
StackOverflow2013
Postsdbo
POweb application receiving millions of requests and leads to generating millions of row inserts per 30 seconds in SQL Server 2008
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POweb application receiving millions of requests and leads to generating millions of row inserts per 30 seconds in SQL Server 2008
    primarykey
    data
    text
    <p>I am currently addressing a situation where our web application receives at least a Million requests per 30 seconds. So these requests will lead to generating 3-5 Million row inserts between 5 tables. This is pretty heavy load to handle. Currently we are using multi threading to handle this situation (which is a bit faster but unable to get a better CPU throughput). However the load will definitely increase in future and we will have to account for that too. After 6 months from now we are looking at double the load size we are currently receiving and I am currently looking at a possible new solution that is scalable and should be easy enough to accommodate any further increase to this load. Currently with multi threading we are making the whole debugging scenario quite complicated and sometimes we are having problem with tracing issues.</p> <p>FYI we are already utilizing the SQL Builk Insert/Copy that is mentioned in this previous post</p> <p><a href="https://stackoverflow.com/questions/3974275/sql-server-2008-performance-tuning-features-for-insert-large-amount-of-data/3974606">Sql server 2008 - performance tuning features for insert large amount of data</a> </p> <p>However I am looking for a more capable solution (which I think there should be one) that will address this situation. </p> <p><strong>Note:</strong> I am not looking for any code snippets or code examples. I am just looking for a big picture of a concept that I could possibly use and I am sure that I can take that further to an elegant solution :)</p> <p>Also the solution should have a better utilization of the threads and processes. And I do not want my threads/processes to even wait to execute something because of some other resource.</p> <p>Any suggestions will be deeply appreciated.</p> <p><strong>Update</strong>: Not every request will lead to an insert...however most of them will lead to some sql operation. The appliciation performs different types of transactions and these will lead to a lot of bulk sql operations. I am more concerned towards inserts and updates. and these operations need not be real time there can be a bit lag...however processing them real time will be much helpful.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USuser153410
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POweb application receiving millions of requests and leads to generating millions of row inserts per 30 seconds in SQL Server 2008
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POweb application receiving millions of requests and leads to generating millions of row inserts per 30 seconds in SQL Server 2008
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POweb application receiving millions of requests and leads to generating millions of row inserts per 30 seconds in SQL Server 2008
      1. USNiK
      1. VTFavorite
    1. COI would add more info to your question. Why do hits to the webpage result in rows in the database? Do these rows have to be written in real time?
      singulars
      1. POweb application receiving millions of requests and leads to generating millions of row inserts per 30 seconds in SQL Server 2008
      1. USPhil Sandler
    2. COHave you ever considered using a NOSql db? I've made good experience with MongoDB in a high write/read environment.
      singulars
      1. POweb application receiving millions of requests and leads to generating millions of row inserts per 30 seconds in SQL Server 2008
      1. USNick Weaver
    3. CO@Phil...please see the update section...
      singulars
      1. POweb application receiving millions of requests and leads to generating millions of row inserts per 30 seconds in SQL Server 2008
      1. USuser153410
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload