Databases
StackOverflow2013
Postsdbo
POElasticSearch types and indexing performance
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POElasticSearch types and indexing performance
    primarykey
    data
    text
    <p>I would like to understand the performance impact of indexing documents of multiple types to a single index where there is an imbalance in the number of items of each type (one type has millions, where another type has just thousands of documents). I have spotted issues on some of my indexes, and ruling out whether types are indexed separately within a single index (or not) would help me. Can I assume that types are indexed separately along the lines of a relational database where each table is effectively separate?</p> <p>If the answer to the above is no and that types are effectively all lumped together, then I'll lay out the rest of what I'm doing to try and get some more detailed input.</p> <p>The use case for this example is capturing tweets for Twitter users (call it owner for clarity). I have multi-tenant environment with one index per twitter owner. That said, focusing on a single owner: </p> <ul> <li>I capture the tweets from each timeline (mentions, direct messages, my tweets, and the full 'home' timeline) into a single index, with each timeline type having a different mapping in ElasticSearch</li> <li>Each tweet refers to a parent type, the user who authored the tweet (which may or may not be the owner), with a parent mapping. There is only a single 'user' type for all the timeline types</li> <li>I search and facet only ever on one owner in a single query, so I don't have to concern myself searching across multiple indexes</li> <li>The home timeline may capture millions of tweets, where the owner's own tweets may result in hundreds or thousands</li> <li>The user documents are routinely updated with information outside of the Twitter timelines, therefore I would like to avoid (if possible) the situation where I have to keep multiple copies of the same user object in sync across multiple indexes</li> </ul> <p>I have noticed a much slower response querying on the indexes with millions of documents, even when excluding the 'home timeline' type with millions of documents indexed, leaving just the types with a few thousand entries. I don't want to have to split the types into separate indexes (unless I have to), due to the parent-child relationship between a tweet and a user. </p> <p>Is there a way I can understand if the issue is with the total number of documents in a specific index, something to do with the operation of 'has_child' filtered queries, some other poor design of queries or facets, or something else?</p> <p>Any input would be appreciated.</p> <p><strong>EDIT</strong></p> <p>To clarify the statement that tweets are stored per timeline. This means that there is an ElasticSearch type defined for home_timeline, my_tweets_timeline, mentions_timeline, direct_messages_timeline, etc, which correspond to what you see in the standard twitter.com UI. So there is a natural split between the sets of tweets, although with some overlap too.</p> <p>I have gone back in to check out the has_child queries, and this is a definite red-herring at this point. Basic queries on the larger indexes are much slower, even when querying a type with just a few thousand rows (my_tweets_timeline).</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USPhil
    1. USPhil
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POElasticSearch types and indexing performance
      1. This table or related slice is empty.
      1. VTUpMod
    1. COMy answer feels incomplete, but so does your question: please provide the `has_child` query you're using, as well as examples of the different documents with their relationships. In particular I wasn't sure what you meant by "excluding the 'home timeline' type" - I only got a sense of the tweet and user types, so that confused me.
      singulars
      1. POElasticSearch types and indexing performance
      1. USPaul Bellora
    2. COPaul, I edited the question a little to clarify the timelines. Also, going back to look at the queries, has_child is not any more of a performance issue than regular queries.
      singulars
      1. POElasticSearch types and indexing performance
      1. USPhil
    3. COHmm, okay. Seems like it's a general scalability issue then. Hopefully someone else can chime in. +1
      singulars
      1. POElasticSearch types and indexing performance
      1. USPaul Bellora
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload