Databases
StackOverflow2013
Postsdbo
POHow to efficiently feed FAST ESP with gigabytes of data with .NET
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to efficiently feed FAST ESP with gigabytes of data with .NET
    primarykey
    data
    text
    <p>This will be a tricky question but I will try anyway: our task is to feed Microsoft FAST ESP with gigabytes of data. The final amount of indexed data is somewhere in the neighborhood of 50-60GB.</p> <p>FAST has a .NET API but core components are written in Python (processing pipelines to index documents). The challenge is to reliably communicate with the system while feeding it gigabytes of data for indexing.</p> <p><strong>The problems that arise with FAST here are:</strong></p> <blockquote> <ol> <li><p>the system is quirky when it is fed too much data at once as it wants to reindex its data during which the system remains unreachable for hours. Unacceptable.</p></li> <li><p>it is not an option to queue up all data and serially feed one item at a time since this will take too long (several days).</p></li> <li><p>when an item cannot be be indexed by FAST the client has to re-feed the item. For this to work, the system is supposed to call a callback method to inform the client about the failure. However, whenever the system times out the feeding client is unable to react to the timeout because that callback is never called. Hence the client is starving. Data is in the queue but cannot be passed along to the system. The queue collapses. Data is lost. You get the idea.</p></li> </ol> </blockquote> <p><strong>Notes:</strong></p> <blockquote> <ol> <li>feeding an item can take seconds for a small item and up to 5-8 hours for a single large item.</li> <li>the items being indexed are both binary and text based.</li> <li>the goal is for the full indexing to take "only" 48-72h, i.e. it must happen over the weekend.</li> <li><strong>The FAST document processing pipelines (Python code) here have around 30 stages each. There are a total of 27 pipelines as of this writing.</strong></li> </ol> </blockquote> <p><strong>In summary:</strong></p> <blockquote> <p>The major challenge is to feed the system with items, big and small, at just the right speed (not too fast because it might collapse or run into memory issues; not too slow because this will take too long), simultaneously, in a parallel manner like asynchronously running threads. In my opinion there has to be an algorithm that decides when to feed what items and how many at once. Parallel programming comes to mind.</p> <p>There could also be multiple "queues" where each queue (process) is dedicated to certain-sized items which are loaded in a queue and then fed one by one (in worker threads). </p> </blockquote> <p>I am curious if anyone has ever done anything like this, or how how you would go about a problem like this.</p> <blockquote> <p><strong>EDIT: Again, I am not looking to "fix" FAST ESP or improve its inner workings. The challenge is to effectively use it!</strong></p> </blockquote>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USZero Piraeus
    1. USJohn
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow to efficiently feed FAST ESP with gigabytes of data with .NET
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow to efficiently feed FAST ESP with gigabytes of data with .NET
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POHow to efficiently feed FAST ESP with gigabytes of data with .NET
      1. This table or related slice is empty.
      1. VTUpMod
    1. COHow many "large items" do you have? Seems like all other issues aside that could prevent you from hitting your 72-hour mark.
      singulars
      1. POHow to efficiently feed FAST ESP with gigabytes of data with .NET
      1. USYuck
    2. COYou are going to have to provide something for us to work with. At this point it would be extremely hard to answer your question since we have no context. You need to provide more information be as detailed as you can be.
      singulars
      1. POHow to efficiently feed FAST ESP with gigabytes of data with .NET
      1. USSecurity Hound
    3. COCurrently, it's hard to tell, whether your problems come from bugs in the 3rd party product or from something else. Data loss etc. sound like bugs in this product. Maybe a non-technical approach would be to request those bugs to be fixed.
      singulars
      1. POHow to efficiently feed FAST ESP with gigabytes of data with .NET
      1. USDaniel Hilgarth
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload