Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>I'll cover the basics of textual document matching...</p> <p>Most document similarity measures work on a word basis, rather than sentence structure. The first step is usually <a href="http://en.wikipedia.org/wiki/Stemming" rel="noreferrer">stemming</a>. Words are reduced to their root form, so that different forms of similar words, e.g. "swimming" and "swims" match.</p> <p>Additionally, you may wish to filter the words you match to avoid noise. In particular, you may wish to ignore occurances of "the" and "a". In fact, there's a lot of conjunctions and pronouns that you may wish to omit, so usually you will have a long list of such words - this is called "<a href="http://en.wikipedia.org/wiki/Stop_words" rel="noreferrer">stop list</a>".</p> <p>Furthermore, there may be bad words you wish to avoid matching, such as swear words or racial slur words. So you may have another exclusion list with such words in it, a "bad list".</p> <p>So now you can count similar words in documents. The question becomes how to measure total document similarity. You need to create a score function that takes as input the similar words and gives a value of "similarity". Such a function should give a high value if the same word appears multiple times in both documents. Additionally, such matches are weighted by the total word frequency so that when uncommon words match, they are given more statistical weight.</p> <p><a href="http://lucene.apache.org/java/docs/index.html" rel="noreferrer">Apache Lucene</a> is an open-source search engine written in Java that provides practical detail about these steps. For example, here is the information about how they weight query similarity:</p> <p><a href="http://lucene.apache.org/java/2_9_0/api/all/org/apache/lucene/search/Similarity.html" rel="noreferrer">http://lucene.apache.org/java/2_9_0/api/all/org/apache/lucene/search/Similarity.html</a></p> <blockquote> <p>Lucene combines Boolean model (BM) of Information Retrieval with Vector Space Model (VSM) of Information Retrieval - documents "approved" by BM are scored by VSM.</p> </blockquote> <p>All of this is really just about matching words in documents. You did specify matching sentences. For most people's purposes, matching words is more useful as you can have a huge variety of sentence structures that really mean the same thing. The most useful information of similarity is just in the words. I've talked about document matching, but for your purposes, a sentence is just a very small document.</p> <p>Now, as an aside, if you don't care about the actual nouns and verbs in the sentence and only care about grammar composition, you need a different approach...</p> <p>First you need a <a href="http://en.wikipedia.org/wiki/Link_grammar" rel="noreferrer">link grammar parser</a> to interpret the language and build a data structure (usually a tree) that represents the sentence. Then you have to perform inexact graph matching. This is a hard problem, but there are algorithms to do this on trees in polynomial time. </p>
    singulars
    1. This table or related slice is empty.
    1. POBest way to rank sentences based on similarity from a set of Documents
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USTim Gee
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POBest way to rank sentences based on similarity from a set of Documents
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThank you very much for your answer. Now I got a clear idea about how to proceed.
      singulars
      1. PO
      1. USAnantha Krishnan
    2. COvery interesting post
      singulars
      1. PO
      1. USAlex
    3. COGreat answer Tim. +1 of course.
      singulars
      1. PO
      1. USMichał Šrajer
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload