StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POBest data store w/full text search for lots of small documents? (e.g. a Splunk-like system)
primarykey
Id
5100871
data
AcceptedAnswerId
5109079
AnswerCount
5
ClosedDate
CommentCount
2
CommunityOwnedDate
CreationDate
2011-02-24T05:49:53.217
FavoriteCount
1
LastActivityDate
2016-11-28T19:03:59.203
LastEditDate
LastEditorUserId
0
OwnerUserId
147637
ParentId
0
PostTypeId
1
Score
1
ViewCount
1793
LastEditorDisplayName
text
Body
We are specing out a system that will index and store zillions of Syslog messages. These are text messages, with a few attributes (system name, date/time, message type, message body), that are typically 100 to 1500 bytes each. We generate 2 to 10 gb of these messages per day, and need to retain at least 30 days of them. The splunk system has a really great indexing and document compression system. What to use? I thought of mongodb, but it seems inappropriate for documents of this small size. SQL Server is a possibility, but seems perhaps not super efficient for this purpose. Text files with lucene? -- The windows file system doesn't always like dirs with zillions of files Suggestions ? Thanks!
Tags
<mongodb><full-text-indexing><datastore>
Title
Best data store w/full text search for lots of small documents? (e.g. a Splunk-like system)
singulars
PostAcceptedAnswerId
1. PO
 singulars
 PostTypePostTypeId
 PTAnswer
PostParentId
1. This table or related slice is empty.
PostTypePostTypeId
1. PTQuestion
UserLastEditorUserId
1. This table or related slice is empty.
UserOwnerUserId
1. USJonesome
plurals
PostLinksPostIdRelatedPostId
1. This table or related slice is empty.
PostLinksRelatedPostIdPostId
1. This table or related slice is empty.
PostsAcceptedAnswerId
1. This table or related slice is empty.
PostsParentIdCreationDate
1. PO
 singulars
 PostTypePostTypeId
 PTAnswer
2. PO
 singulars
 PostTypePostTypeId
 PTAnswer
3. PO
 singulars
 PostTypePostTypeId
 PTAnswer
VotesPostIdCreationDate
1. VO
 singulars
 PostPostId
 POBest data store w/full text search for lots of small documents? (e.g. a Splunk-like system)
 UserUserId
 USjrojo
 VoteTypeVoteTypeId
 VTFavorite
2. VO
 singulars
 PostPostId
 POBest data store w/full text search for lots of small documents? (e.g. a Splunk-like system)
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
CommentsPostId
1. CO"""I thought of mongodb, but it seems inappropriate for documents of this small size.""" - why this?
 singulars
 PostPostId
 POBest data store w/full text search for lots of small documents? (e.g. a Splunk-like system)
 UserUserId
 USAndreas Jung
2. COGood question. One of the devs here is doing a mongodb project, and he said he thought mongodb was a bad fit for 300 million documents of 100-1500 bytes each with full text search. Perhaps he is mistaken?
 singulars
 PostPostId
 POBest data store w/full text search for lots of small documents? (e.g. a Splunk-like system)
 UserUserId
 USJonesome

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.