Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Based on your comments, it sounds like the feature in your documents that's highly predictive of the class is a near-tautology: "paper accepted on" correlates with accepted papers because at least some of the papers in your database were scraped from already-accepted papers and have been annotated by the authors as such.</p> <p>To me, this sounds like a useless feature for trying to predict whether a paper will be accepted, because (I'd imagine) you're trying to predict paper acceptance <em>before</em> the actual acceptance has been issued ! In such a case, none of the papers you'd like to test your algorithm with will be annotated with "paper accepted on." So, I'd remove it.</p> <p>You also asked about how to determine whether a feature correlates strongly with one class. There are three things that come to mind for this problem.</p> <p>First, you could just compute a basic frequency count for each feature in your dataset and compare those values across classes. This is probably not super informative, but it's easy.</p> <p>Second, since you're using a log-linear model, you can train your model on your training dataset, and then rank each feature in your model by its weight in the logistic regression parameter vector. Features with high positive weight are indicative of one class, while features with large negative weight are strongly indicative of the other.</p> <p>Finally, just for the sake of completeness, I'll point out that you might also want to look into <a href="http://en.wikipedia.org/wiki/Feature_selection" rel="nofollow"><strong>feature selection</strong></a>. There are many ways of selecting relevant features for a machine learning algorithm, but I think one of the most intuitive from your perspective might be <strong>greedy feature elimination</strong>. In such an approach, you train a classifier using all N features in your model, and measure the accuracy on some held-out validation set. Then, train N new models, each with N-1 features, such that each model eliminates one of the N features, and measure the resulting drop in accuracy. The feature with the biggest drop was probably strongly predictive of the class, while features that have no measurable difference can probably be omitted from your final model. As larsmans points out correctly in the comments below, this doesn't scale well at all, but it can be a useful method sometimes.</p>
    singulars
    1. This table or related slice is empty.
    1. POPredictive features with high presence in one class
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USlmjohns3
    1. USlmjohns3
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COYour proposed algorithm is going to scale poorly in text classification, where the number of features is typically on the order of thousands to millions. Also, feature selection would typically *keep* the "paper accepted on" feature, so I don't see how it relates to the OP's problem.
      singulars
      1. PO
      1. USFred Foo
    2. CO@larsmans I agree with both your points and also the tautology which lmjohns3 indicated. Other than removing such tautology by hand, I don't seem to find any algorithmic method to do this.
      singulars
      1. PO
      1. UStan
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload