StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POAnalysing alogithm possibly based on regular intervals to check for bots and spiders
primarykey
Id
5102988
data
AcceptedAnswerId
0
AnswerCount
1
ClosedDate
CommentCount
1
CommunityOwnedDate
CreationDate
2011-02-24T09:55:10.510
FavoriteCount
1
LastActivityDate
2011-02-24T11:14:11.800
LastEditDate
LastEditorUserId
0
OwnerUserId
512238
ParentId
0
PostTypeId
1
Score
0
ViewCount
103
LastEditorDisplayName
text
Body
I'm trying to build a script which shows me a list of IP's that are bots/spiders. I wrote a script which imports the access log of Apache to a mysql db so I can try to manage it with php and mysql. I've noticed a lot of bots have regular intervals, they send out a request every 2 or 3 seconds. Is there an easy way of showing these patterns with a query or php script? Or, even harder I think, is there an algorithm that can recognise these bots / spiders. DB: <pre><code>CREATE TABLE IF NOT EXISTS `access_log` ( `IP` varchar(16) NOT NULL, `datetime` datetime NOT NULL, `method` varchar(255) NOT NULL, `status` varchar(255) NOT NULL, `referrer` varchar(255) NOT NULL, `agent` varchar(255) NOT NULL, `site` smallint(6) NOT NULL ); </code></pre>
Tags
<php><apache><analytics><web-crawler><bots>
Title
Analysing alogithm possibly based on regular intervals to check for bots and spiders
singulars
PostAcceptedAnswerId
1. This table or related slice is empty.
PostParentId
1. This table or related slice is empty.
PostTypePostTypeId
1. PTQuestion
UserLastEditorUserId
1. This table or related slice is empty.
UserOwnerUserId
1. USPvdL
plurals
PostLinksPostIdRelatedPostId
1. PL
 singulars
 LinkTypeLinkTypeId
 LTLinked
PostLinksRelatedPostIdPostId
1. This table or related slice is empty.
PostsAcceptedAnswerId
1. This table or related slice is empty.
PostsParentIdCreationDate
1. PO
 singulars
 PostTypePostTypeId
 PTAnswer
VotesPostIdCreationDate
1. VO
 singulars
 PostPostId
 POAnalysing alogithm possibly based on regular intervals to check for bots and spiders
 UserUserId
 USDon
 VoteTypeVoteTypeId
 VTFavorite
CommentsPostId
1. COSee [tell bots apart from human visitors for stats?](http://stackoverflow.com/questions/1717049/tell-bots-apart-from-human-visitors-for-stats) it might answer your question already
 singulars
 PostPostId
 POAnalysing alogithm possibly based on regular intervals to check for bots and spiders
 UserUserId
 USPekka 웃

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.