Databases
StackOverflow2013
Postsdbo
POScalable way of logging page request data from a PHP application?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POScalable way of logging page request data from a PHP application?
    text
    copied!<p>A web application I am developing (in PHP) requires the ability to log each page request.</p> <p>Just like a normal access_log, it will store details like url requested, source ip address, date/time but I also need it to store the User ID of the logged in user (which is stored in a php session variable).</p> <p>This data will then be queried to create site-wide or per user analytics reports as required at a later date - things such as total number of visits/unique visits, page views in a certain time period, geo-locating the ip addresses and looking at locations, most active times of day, most active members etc.</p> <p>The obvious thing to do would be to have a mysql insert statement on each page but if the application is receiving thousands of req/sec, this is going to be a hugh bottleneck on the database so I am looking at alternative, scalable ways of doing this without big infrastructure requirements.</p> <p>A few of the ideas i've had are:</p> <p>1) Work on a way for Nginx to be able to log the user_id from the session/application in the normal web server access_log, which can be parsed and loaded into a database periodically (nightly). This feels like a bit of a hack and will need doing on each web server as the system scales out.</p> <p>2) Log each page request into Redis which has high write speeds - the problem with this is the lack of ability to query the date at a later date.</p> <p>3) Log each page request into either Memcache/Redis acting as a cache (or a message queue) and from there it would be regularly extracted, inserted into MySQL and removed.</p> <p>4) Would something like MongoDB which has more query capability be suitable?</p> <p>I'm interested in how you would approach this and if anyone has any experience of a similar application (or has come across anything online).</p> <p>I'm also interested on thoughts on how the data could be suitably structured to be stored in memcache/redis.</p> <p>Thanks</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload