Databases
StackOverflow2013
Postsdbo
POR: Improving workflow and keeping track of output
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POR: Improving workflow and keeping track of output
    primarykey
    data
    text
    <p>I have what I think is a common enough issue, on optimising workflow in R. Specifically, how can I avoid the common issue of having a folder full of output (plots, RData files, csv, etc.), without, after some time, having a clue where they came from or how they were produced? In part, it surely involves trying to be intelligent about folder structure. I have been looking around, but I'm unsure of what the best strategy is. So far, I have tackled it in a rather unsophisticated (overkill) way: I created a function <code>metainfo</code> (see below) that writes a text file with metadata, with a given file name. The idea is that if a plot is produced, this command is issued to produce a text file with exactly the same file name as the plot (except, of course, the extension), with information on the system, session, packages loaded, R version, function and file the metadata function was called from, etc. The questions are:</p> <p>(i) How do people approach this general problem? Are there obvious ways to avoid the issue I mentioned?</p> <p>(ii) If not, does anyone have any tips on improving this function? At the moment it's perhaps clunky and not ideal. Particularly, getting the file name from which the plot is produced doesn't necessarily work (the solution I use is one provided by @hadley in <a href="https://stackoverflow.com/questions/1815606/rscript-determine-path-of-the-executing-script">1</a>). Any ideas would be welcome!</p> <p>The function assumes git, so please ignore the probable warning produced. This is the main function, stored in a file <code>metainfo.R</code>:</p> <pre><code>MetaInfo &lt;- function(message=NULL, filename) { # message - character string - Any message to be written into the information # file (e.g., data used). # filename - character string - the name of the txt file (including relative # path). Should be the same as the output file it describes (RData, # csv, pdf). # if (is.null(filename)) { stop('Provide an output filename - parameter filename.') } filename &lt;- paste(filename, '.txt', sep='') # Try to get as close as possible to getting the file name from which the # function is called. source.file &lt;- lapply(sys.frames(), function(x) x$ofile) source.file &lt;- Filter(Negate(is.null), source.file) t.sf &lt;- try(source.file &lt;- basename(source.file[[length(source.file)]]), silent=TRUE) if (class(t.sf) == 'try-error') { source.file &lt;- NULL } func &lt;- deparse(sys.call(-1)) # MetaInfo isn't always called from within another function, so func could # return as NULL or as general environment. if (any(grepl('eval', func, ignore.case=TRUE))) { func &lt;- NULL } time &lt;- strftime(Sys.time(), "%Y/%m/%d %H:%M:%S") git.h &lt;- system('git log --pretty=format:"%h" -n 1', intern=TRUE) meta &lt;- list(Message=message, Source=paste(source.file, ' on ', time, sep=''), Functions=func, System=Sys.info(), Session=sessionInfo(), Git.hash=git.h) sink(file=filename) print(meta) sink(file=NULL) } </code></pre> <p>which can then be called in another function, stored in another file, e.g.:</p> <pre><code>source('metainfo.R') RandomPlot &lt;- function(x, y) { fn &lt;- 'random_plot' pdf(file=paste(fn, '.pdf', sep='')) plot(x, y) MetaInfo(message=NULL, filename=fn) dev.off() } x &lt;- 1:10 y &lt;- runif(10) RandomPlot(x, y) </code></pre> <p>This way, a text file with the same file name as the plot is produced, with information that could hopefully help figure out how and where the plot was produced.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. UStcam
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POR: Improving workflow and keeping track of output
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POR: Improving workflow and keeping track of output
      1. USAnthony
      1. VTFavorite
    1. COI should also add that there is a good chance that Makefiles would be part of the solution. However, they are not always possible, because some analyses might require some 'manual' steps, such that you can't run R files in sequence.
      singulars
      1. POR: Improving workflow and keeping track of output
      1. UStcam
    2. COSince you are considering Make, I added reference to Drake to my answer.
      singulars
      1. POR: Improving workflow and keeping track of output
      1. USAlex Vorobiev
    3. COThanks for the suggestion! Drake looks interesting. Still trying to get my head around it. I'm not particularly used to Make either except in its most rudimentary forms. The part I struggle with is that Make makes sense once you have a clear cut analysis and you have a Makefile to run the whole analysis from beginning to end. The problem is before getting to that stage, during exploratory analysis, Make seems less well suited. But it perhaps it induces good practice by forcing you to make the analysis as reproducible as possible.
      singulars
      1. POR: Improving workflow and keeping track of output
      1. UStcam
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload