Databases
StackOverflow2013
Postsdbo
PORead list of file names from web into R
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PORead list of file names from web into R
    primarykey
    data
    text
    <p>I am trying to read a lot of csv files into R from a website. Threa are multiple years of daily (business days only) files. All of the files have the same data structure. I can sucessfully read <em>one</em> file using the following logic:</p> <pre><code># enter user credentials user &lt;- "JohnDoe" password &lt;- "SecretPassword" credentials &lt;- paste(user,":",password,"@",sep="") web.site &lt;- "downloads.theice.com/Settlement_Reports_CSV/Power/" # construct path to data path &lt;- paste("https://", credentials, web.site, sep="") # read data for 4/10/2013 file &lt;- "icecleared_power_2013_04_10" fname &lt;- paste(path,file,".dat",sep="") df &lt;- read.csv(fname,header=TRUE, sep="|",as.is=TRUE) </code></pre> <p>However, Im looking for tips on how to read all the files in the directory at once. I suppose I could generate a sequence of dates an construct the file name above in a loop and use rbind to append each file but that seems cumbersome. Plus there will be issues when attempting to read weekends and holidays where there is no files.</p> <p>The impages below show what the list of files look like in the web browser:</p> <p><img src="https://i.stack.imgur.com/OXMcB.png" alt="file list in browser part 1"></p> <p>... ... ...</p> <p><img src="https://i.stack.imgur.com/U9nOg.png" alt="file list in browser part 2"></p> <p>Is there a way to scan the path (from above) to get a list of all the file names in the directory first that meet a certin crieteia (i.e. start with "icecleared_power_" as there are also some files in that location that have a different starting name that I do not want to read in) then loop the read.csv through that list and use rbind to append?</p> <p>Any guidance would be greatly appreciated?</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USMikeTP
    1. USMikeTP
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PORead list of file names from web into R
      1. This table or related slice is empty.
      1. VTUpMod
    1. COWhat does the actual webpage look like in a browser? Is there an index page of some sort that you can easily screen/web-scrape to get the list relevant files to construct your download commands?
      singulars
      1. PORead list of file names from web into R
      1. USA5C1D2H2I1M1N2O1R2T1
    2. COAnanda: please see edits with web browser screen shots.
      singulars
      1. PORead list of file names from web into R
      1. USMikeTP
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload