Databases
StackOverflow2013
Postsdbo
POprocess logger files in R
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POprocess logger files in R
    primarykey
    data
    text
    <p>I’m using R version 2.15.3 (2013-03-01) with RStudio 0.97.312 on Ubuntu 12.10. I’m trying to create some histograms of logger data in R. However, some sensors weren’t always working, so I got some tables with <code>#N/A</code> and <code>O/C</code> in it. Here’s an excerpt of the log:</p> <pre><code>Date Time Type control.value (V) light.barrier (V) T hotplate ('C) T mesh ('C) T exhaust ('C) T camera ('C) Ref. Junction 1 ('C) 30.03.2012 13:47:50 Interval 0.001 23.556 411.0 O/C 30.5 35.1 23.14 30.03.2012 13:47:51 Interval 0.001 23.556 411.1 O/C 30.3 35.2 23.14 30.03.2012 13:47:52 Interval 0.001 23.556 411.1 O/C 30.2 35.5 23.14 30.03.2012 13:47:53 Interval 0.001 23.556 410.9 O/C 29.8 35.5 23.14 30.03.2012 13:47:54 Interval 0.001 23.556 410.9 O/C 30.1 35.3 23.14 30.03.2012 13:47:55 Interval 0.001 23.556 411.1 O/C 30.2 35.4 23.14 30.03.2012 13:47:56 Interval 0.001 23.556 410.8 O/C 29.8 35.4 23.14 30.03.2012 13:47:57 Interval 0.001 23.556 410.2 O/C 29.4 35.3 23.14 30.03.2012 13:47:58 Interval 0.001 23.556 409.5 O/C 29.1 35.0 23.14 30.03.2012 13:47:59 Interval 0.000 23.556 408.9 O/C 29.3 34.6 23.14 30.03.2012 13:48:00 Interval 0.000 23.556 408.7 O/C #N/A #N/A 23.14 </code></pre> <p>Output of <code>dput (head(logs), file = "dput.txt")</code>: <a href="http://pastebin.de/34176" rel="nofollow noreferrer">http://pastebin.de/34176</a></p> <p>R refuses to process the columns with <code>#N/A</code> and <code>O/C</code>. I can’t reformat it by hand, the file has 185 000 lines. </p> <p>When I load the log and try to create a histogram:</p> <pre><code>&gt; logs &lt;- read.delim("../data/logger/logs/logs.txt", header=TRUE) &gt; hist (logs$mesh) </code></pre> <p>I get this error message:</p> <pre><code>Fehler in hist.default(logs$mesh) : 'x' muss nummerisch sein </code></pre> <p>Rough translation (see: <a href="https://stackoverflow.com/q/16347731/2333821">How to change the locale of R in RStudio?</a>):</p> <pre><code>Error in hist.default(logs$mesh) : 'x' must be numeric </code></pre> <p>The only columns I can create histograms from are the numerical ones listed by sapply. So I thought I have to remove these invalid values to get numeric ones.</p> <p>How can I remove the invalid rows? I’m also open to other ways than processing them with R, e.g Perl or Python if that’s more suitable for this task.</p> <p>This is the output of sapply after loading the log:</p> <pre><code>&gt; sapply (logs, is.numeric) date time type control.value light.barrier hotplate mesh exhaust FALSE FALSE FALSE TRUE FALSE TRUE FALSE FALSE camera reference FALSE TRUE </code></pre> <p>After replacing the <code>#N/A</code> and <code>O/C</code> with <code>NA</code> (<a href="https://stackoverflow.com/a/16350443/2333821">https://stackoverflow.com/a/16350443/2333821</a>)</p> <pre><code> logs.clean &lt;- data.frame (check.rows = TRUE, apply(logs, 2, sub, pattern = "O/C|#N/A", replacement = NA)) </code></pre> <p>I get this:</p> <pre><code>&gt; sapply (logs.clean, is.numeric) date time type control.value light.barrier hotplate mesh exhaust FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE camera reference FALSE FALSE </code></pre>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USTobias Schula
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POprocess logger files in R
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POprocess logger files in R
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POprocess logger files in R
      1. This table or related slice is empty.
      1. VTDownMod
    1. COWhat do you mean by 'processing'? can you run `dput(head(DF))` where `DF` is your data.frame and post here the result?
      singulars
      1. POprocess logger files in R
      1. USFerdinand.kraft
    2. COIf you want to drop those rows, look at `?subset` and construct a logical argument to exclude those rows, e.g., `myvar != "O/C & myothervar != "#N/A"`.
      singulars
      1. POprocess logger files in R
      1. USFrank
    3. COvoted down for no code, no error message, and no details on what "R refuses to process these columns" means. Sheesh.
      singulars
      1. POprocess logger files in R
      1. USSpacedman
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload