Databases
StackOverflow2013
Postsdbo
POImporting 10000 rows to Solr fetches hundreds of thousands of rows
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POImporting 10000 rows to Solr fetches hundreds of thousands of rows
    primarykey
    data
    text
    <p>I'm trying to import five or so tables from MS SQL Server 2005 to Solr 4.3.0 on Ubuntu. Ultimately I want to import all the rows in each table, but for now I'm using "top (10000)" to limit my import time, until I can get my configuration and queries debugged.</p> <p>It's mostly working now, but I have the problem that when I check the status with:</p> <pre><code>http://192.168.1.3:8983/solr/dataimport </code></pre> <p>it tells me it's been running for forty-eight minutes, and fetched over 800,000 rows. Clearly something is wrong in my configuration but I don't have a clue.</p> <pre><code>&lt;str name="status"&gt;busy&lt;/str&gt; &lt;str name="importResponse"&gt;A command is still running...&lt;/str&gt; &lt;lst name="statusMessages"&gt; &lt;str name="Time Elapsed"&gt;0:48:35.0&lt;/str&gt; &lt;str name="Total Requests made to DataSource"&gt;821&lt;/str&gt; &lt;str name="Total Rows Fetched"&gt;8165164&lt;/str&gt; &lt;str name="Total Documents Processed"&gt;0&lt;/str&gt; &lt;str name="Total Documents Skipped"&gt;0&lt;/str&gt; &lt;str name="Full Dump Started"&gt;2013-05-21 18:14:15&lt;/str&gt; &lt;/lst&gt; </code></pre> <p>My queries all look like:</p> <pre><code>select top (10000) foo, bar from [DBName].dbo.TableName </code></pre> <p>Possibly my problem is that I'm using nested entities rather than un-nested ones. I've seen examples of both when studying how to configure data-config.xml, and am unclear what the difference is.</p> <p>This:</p> <pre><code>&lt;entity&gt; &lt;/entity&gt; &lt;entity&gt; &lt;/entity&gt; </code></pre> <p>Or this:</p> <pre><code>&lt;entity&gt; &lt;entity&gt; &lt;/entity&gt; &lt;/entity&gt; </code></pre> <p>I'm using the TemplateTransformer to append the primary keys from my tables into a single uniqueKey called id:</p> <pre><code>&lt;field column="id" template="${GetAds.ADS_WPID},${Foo.MI_WPID},${Bar.MERCHANTID},${Baz.R4SE_WPID},${Boo.CR_WPID}" /&gt; </code></pre> <p>I've been beating on Solr for over two weeks, so my new boss is impatient for me to have it done. I'm so close I can taste it, but don't know how to proceed from here.</p> <p>Thanks for any advice you can give me.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USMike Crawford
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. COIf you have access to the sql server, run a trace to see the actual sql that solr is sending to the database to execute.
      singulars
      1. POImporting 10000 rows to Solr fetches hundreds of thousands of rows
      1. USbrian
    2. COThanks brian. I've gone home for the night, but I'll give your suggestion a try first thing in the morning.
      singulars
      1. POImporting 10000 rows to Solr fetches hundreds of thousands of rows
      1. USMike Crawford
    3. COIf I don't nest the entities, but make them vertically adjacent, I get `org.apache.solr.common.SolrException: Document is missing mandatory uniqueKey field: id`, I expect because my use of the TemplateTransformer to concatenate my primary keys into a single id fails to resolve the now out-of-scope symbols.
      singulars
      1. POImporting 10000 rows to Solr fetches hundreds of thousands of rows
      1. USMike Crawford
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload