Databases
StackOverflow2013
Postsdbo
POSuggestions for a scalable architecture solution to large data problem
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSuggestions for a scalable architecture solution to large data problem
    primarykey
    data
    text
    <p>I am in the process of building/architecting a business social network web application that has a component that I think will lead to major scalability issues and I'd like to get some feedback/thoughts on the best way forward.</p> <p>The application has a User object. The idea is, that every time a new user joins the system he ranks everyone else's "usefulness" to him based on a set of factors. Similarly, every other user on the system ranks him/her.</p> <p>However, I'm worried about the scalability implications of this approach. For example, if 10,000 users join the system we are talking about 10,000^2 calculations to be stored to the database. That is 100 million records so that clearly becomes problematic both in terms of time taken to calculate these rankings but also in terms of storing this in a database.</p> <p>Thus, I'm looking for help/inspiration :)</p> <p>My background is in java and I've been looking at hadoop/map-reduce as a possible way to implement the calculations in a parallel manner, however I really am not sure whether this problem is applicable to Map Reduce or as to what is the best approach in general. </p> <p>So, I suppose there are two specific parts to my query..</p> <p>1) To do the actual calculations, should I do these in a parallel manner, ie..is Map Reduce a good approach for this problem</p> <p>2) To store the rankings, what should I be using...is a standard relational database a bad idea, ie...this won't be a good fit for MySQL...should I look at something like Cassandra, HBase or some other NoSQL solution?</p> <p>Any help/ideas is greatly appreciated.</p> <p>cheers, Brian</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USBrian
    1. USBrian
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COIf you're just basically funneling all combinations of users through one ranking function then yes, map/reduce is your friend. If you're updating a bunch of records try to group them to decrease lock contention (updating several records in the same DB page can be done with a page lock, updating several on different pages may cause lock escalation to a table lock).
      singulars
      1. POSuggestions for a scalable architecture solution to large data problem
      1. USTMN
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload