Databases
StackOverflow2013
Postsdbo
POMatLab Missing data handling in categorical data
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POMatLab Missing data handling in categorical data
    primarykey
    data
    text
    <p>I am trying to put my dataset into the MATLAB <code>[ranked,weights] = relieff(X,Ylogical,10, 'categoricalx', 'on')</code> function to rank the importance of my predictor features. The <code>dataset&lt;double n*m&gt;</code> has <code>n</code> observations and <code>m</code> discrete (i.e. categorical) features. It happens that each observation (row) in my dataset has at least one NaN value. These NaNs represent unobserved, i.e. missing or null, predictor values in the dataset. (There is no corruption in the dataset, it is just incomplete.)</p> <p><strong>relieff()</strong> uses this function below to remove any rows that contain a NaN:</p> <pre><code>function [X,Y] = removeNaNs(X,Y) % Remove observations with missing data NaNidx = bsxfun(@or,isnan(Y),any(isnan(X),2)); X(NaNidx,:) = []; Y(NaNidx,:) = []; </code></pre> <p>This is not ideal, especially for my case, since it leaves me with <code>X=[]</code> and <code>Y=[]</code> (i.e. no observations!)</p> <p>In this case:</p> <p><strong>1)</strong> Would replacing all NaN's with a random value, e.g. 99999, help? By doing this, I am introducing a new feature state for all the predictor features so I guess it is not ideal. </p> <p><strong>2)</strong> or is replacing NaNs with the mode of the corresponding feature column vector (as below) statistically more sound? (I am not vectorising for clarity's sake)</p> <pre><code>function [matrixdata] = replaceNaNswithModes(matrixdata) for i=1: size(matrixdata,2) cv= matrixdata(:,i); modevalue= mode(cv); cv(find(isnan(cv))) = modevalue; matrixdata(:,i) = cv; end </code></pre> <p><strong>3)</strong> Or any other sensible way that would make sense for "categorical" data? </p> <p>P.S: <a href="http://www.dtreg.com/MissingValues.htm" rel="nofollow">This link</a> gives possible ways to handle missing data.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USRhubarb
    1. USRhubarb
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POMatLab Missing data handling in categorical data
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POMatLab Missing data handling in categorical data
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POMatLab Missing data handling in categorical data
      1. USteng
      1. VTFavorite
    1. COMy first question would be why are `NaN`s appearing in your data? Is this a corruption of the data set, or is this an explainable phenomenon?
      singulars
      1. POMatLab Missing data handling in categorical data
      1. USmacduff
    2. COIt is a manually entered dataset and NaNs are due to omission by the personnel who enter the data. There is no corruption in the dataset; it is, however, sparse.
      singulars
      1. POMatLab Missing data handling in categorical data
      1. USRhubarb
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload