Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>The easiest way to add gaps (= PositionIncrement > 1) is to provide a custom TokenStream. You do not need to change your Analyzer for that. However, HTML parsing should be done upstream (i.e., you should segment and clean your input text accordingly before feeding it to Lucene).</p> <p>Here is a full, working example (imports omitted):</p> <pre><code>public class GapTest { public static void main(String[] args) throws Exception { final Directory dir = new RAMDirectory(); final IndexWriterConfig iwConfig = new IndexWriterConfig(Version.LUCENE_4_10_1, new SimpleAnalyzer()); final IndexWriter iw = new IndexWriter(dir, iwConfig); Document doc = new Document(); doc.add(new TextField("body", "A B C", Store.YES)); doc.add(new TextField("body", new PositionIncrementTokenStream(10))); doc.add(new TextField("body", "D E F", Store.YES)); System.out.println(doc); iw.addDocument(doc); iw.close(); final IndexReader ir = DirectoryReader.open(dir); IndexSearcher is = new IndexSearcher(ir); QueryParser qp = new QueryParser("body", new SimpleAnalyzer()); for (String q : new String[] { "\"A B C\"", "\"A B C D\"", "\"A B C D\"", "\"A B C D\"~10", "\"A B C D E F\"~10", "\"A B C D F E\"~10", "\"A B C D F E\"~11" }) { Query query = qp.parse(q); TopDocs docs = is.search(query, 10); System.out.println(docs.totalHits + "\t" + q); } ir.close(); } /** * A gaps-only TokenStream (uses {@link PositionIncrementAttribute} * * @author Christian Kohlschuetter */ private static final class PositionIncrementTokenStream extends TokenStream { private boolean first = true; private PositionIncrementAttribute attribute; private final int positionIncrement; public PositionIncrementTokenStream(final int positionIncrement) { super(); this.positionIncrement = positionIncrement; attribute = addAttribute(PositionIncrementAttribute.class); } @Override public boolean incrementToken() throws IOException { if (first) { first = false; attribute.setPositionIncrement(positionIncrement); return true; } else { return false; } } @Override public void reset() throws IOException { super.reset(); first = true; } } </code></pre> <p>}</p>
    singulars
    1. This table or related slice is empty.
    1. POHow to increase position offsets in a lucene index to correspond to <p> tags?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USChristian Kohlschütter
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POHow to increase position offsets in a lucene index to correspond to <p> tags?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COThanks @Christian. This is interesting. I hadn't thought of this approach of segmenting the text first, and then introducing the gaps. This could certainly be used to build up a more principled document parser. I had taken a more brute-force approach and created a variant of the `StandardTokenizer` based on a modified jflex file, into which I inserted a rule to detect paragraphs: `MARKUP = ("<" {ALPHANUM}+ {ANY}* ">" | "</" {ALPHANUM}+ {ANY}* ">") ` and `ANY = ({ALPHANUM} | {WHITESPACE} | [\"='&;])`. Of course my approach is more brittle, but it seems to work well on my TREC documents.
      singulars
      1. PO
      1. USGene Golovchinsky
    2. CO@Gene. Yup, tokenizing at HTML level might work at a first glance but opens up a bunch of problems that will be revealed only when it's too late ;-)
      singulars
      1. PO
      1. USChristian Kohlschütter
    3. CO@Gene. What might be useful here is to use [boilerpipe](http://code.google.com/p/boilerpipe/) with a KeepEverythingExtractor. This already provides you some segmentation that can be used as input for the suggested solution above.
      singulars
      1. PO
      1. USChristian Kohlschütter
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload