Databases
StackOverflow2013
Postsdbo
POSolr Custom RequestHandler - optimizing results
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSolr Custom RequestHandler - optimizing results
    primarykey
    data
    text
    <p>Yet another potentially embarrassing question. Please feel free to point any obvious solution that may have been overlooked - I have searched for solutions previously and found nothing, but sometimes it's a matter of choosing the wrong keywords to search for.<br> Here's the situation: coded my own RequestHandler a few months ago for an enterprise-y system, in order to inject a few necessary security parameters as an extra filter in all queries made to the solr core. Everything runs smoothly until the part where the docs resulting from a query to the index are collected and then returned to the user.</p> <p>Basically after the filter is created and the query is executed we get a set of document ids (and scores), but then we have to iterate through the ids in order to build the result set, one hit at a time - which is a good 10x slower that querying the standard requesthandler, and only bound to get worse as the number of results increase. Even worse, since our schema heavily relies on dynamic fields for flexibility, there is no way (that I know of) of previously retrieving the list of fields to retrieve per document, other than testing all possible combinations per doc. </p> <p>The code below is a simplified version of the one running in production, for querying the SolrIndexSearcher and building the response. </p> <p>Without further ado, my questions are:</p> <ul> <li>is there any way of retrieving all results at once, instead of building a response document by document?</li> <li>is there any possibility of getting the list of fields on each result, instead of testing all possible combinations?</li> <li>any particular WTFs in this code that I should be aware of? Feel free to kick me!</li> </ul> <pre><code>//function that queries index and handles results private void searchCore(SolrIndexSearcher searcher, Query query, Filter filter, int num, SolrDocumentList results) { //Executes the query TopDocs col = searcher.search(query,filter, num); //results ScoreDoc[] docs = col.scoreDocs; //iterate & build documents for (ScoreDoc hit : docs) { Document doc = reader.document(hit.doc); SolrDocument sdoc = new SolrDocument(); for(Object f : doc.getFields()) { Field fd = ((Field) f); //strings if (fd.isStored() && (fd.stringValue() != null)) sdoc.addField(fd.name(), fd.stringValue()); else if(fd.isStored()) { //Dynamic Longs if (fd.name().matches(".*_l") ) { ByteBuffer a = ByteBuffer.wrap(fd.getBinaryValue(), fd.getBinaryOffset(), fd.getBinaryLength()); long testLong = a.getLong(0); sdoc.addField(fd.name(), testLong ); } //Dynamic Dates else if(fd.name().matches(".*_dt")) { ByteBuffer a = ByteBuffer.wrap(fd.getBinaryValue(), fd.getBinaryOffset(), fd.getBinaryLength()); Date dt = new Date(a.getLong()); sdoc.addField(fd.name(), dt ); } //... } } results.add(sdoc); } } </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USjavanna
    1. US12N
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COFor clarification, you created this custom RequestHandler for the sole purpose of adding an additional Filter to every single request. Is that correct?
      singulars
      1. POSolr Custom RequestHandler - optimizing results
      1. USrfeak
    2. CO@rfeak - Short answer, yes. Is it relevant? Any different approach you'd suggest?
      singulars
      1. POSolr Custom RequestHandler - optimizing results
      1. US12N
    3. COYes. If I recall correctly (it's been 2 years) there are ways to add default filters in the XML files that are added to each query. Then the default filters are cached in the FilterCache like any other. Their aggregation with other Filters should be mighty fast and you wouldn't have to fetch documents to apply the filter. Not sure if you considered that approach.
      singulars
      1. POSolr Custom RequestHandler - optimizing results
      1. USrfeak
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload