Databases
StackOverflow2013
Postsdbo
POPandas ParserError EOF character when reading multiple csv files to HDF5
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPandas ParserError EOF character when reading multiple csv files to HDF5
    primarykey
    data
    text
    <p>Using Python3, Pandas 0.12</p> <p>I'm trying to write multiple csv files (total size is 7.9 GB) to a HDF5 store to process later onwards. The csv files contain around a million of rows each, 15 columns and data types are mostly strings, but some floats. However when I'm trying to read the csv files I get the following error:</p> <pre><code>Traceback (most recent call last): File "filter-1.py", line 38, in &lt;module&gt; to_hdf() File "filter-1.py", line 31, in to_hdf for chunk in reader: File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 578, in __iter__ yield self.read(self.chunksize) File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 608, in read ret = self._engine.read(nrows) File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 1028, in read data = self._reader.read(nrows) File "parser.pyx", line 706, in pandas.parser.TextReader.read (pandas\parser.c:6745) File "parser.pyx", line 740, in pandas.parser.TextReader._read_low_memory (pandas\parser.c:7146) File "parser.pyx", line 781, in pandas.parser.TextReader._read_rows (pandas\parser.c:7568) File "parser.pyx", line 768, in pandas.parser.TextReader._tokenize_rows (pandas\parser.c:7451) File "parser.pyx", line 1661, in pandas.parser.raise_parser_error (pandas\parser.c:18744) pandas.parser.CParserError: Error tokenizing data. C error: EOF inside string starting at line 754991 Closing remaining open files: ta_store.h5... done </code></pre> <p><strong>Edit</strong>:</p> <p>I managed to find a file that produced this problem. I think it's reading an EOF character. However I have no clue to overcome this problem. Given the large size of the combined files I think it's too cumbersome to check each single character in each string. (Even then I would still not be sure what to do.) As far as I checked, there are no strange characters in the csv files that could raise the error. I also tried passing <code>error_bad_lines=False</code> to <code>pd.read_csv()</code>, but the error persists.</p> <p>My code is the following:</p> <pre><code># -*- coding: utf-8 -*- import pandas as pd import os from glob import glob def list_files(path=os.getcwd()): ''' List all files in specified path ''' list_of_files = [f for f in glob('2013-06*.csv')] return list_of_files def to_hdf(): """ Function that reads multiple csv files to HDF5 Store """ # Defining path name path = 'ta_store.h5' # If path exists delete it such that a new instance can be created if os.path.exists(path): os.remove(path) # Creating HDF5 Store store = pd.HDFStore(path) # Reading csv files from list_files function for f in list_files(): # Creating reader in chunks -- reduces memory load reader = pd.read_csv(f, chunksize=50000) # Looping over chunks and storing them in store file, node name 'ta_data' for chunk in reader: chunk.to_hdf(store, 'ta_data', mode='w', table=True) # Return store return store.select('ta_data') return 'Finished reading to HDF5 Store, continuing processing data.' to_hdf() </code></pre> <p><strong>Edit</strong></p> <p>If I go into the CSV file that raises the CParserError EOF... and manually delete all rows after the line that is causing the problem, the csv file is read properly. However all I'm deleting are blank rows anyway. The weird thing is that when I manually correct the erroneous csv files, they are loaded fine into the store individually. But when I again use a list of multiple files the 'false' files still return me errors.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USMatthijs
    1. USMatthijs
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPandas ParserError EOF character when reading multiple csv files to HDF5
      1. USunutbu
      1. VTFavorite
    2. VO
      singulars
      1. POPandas ParserError EOF character when reading multiple csv files to HDF5
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POPandas ParserError EOF character when reading multiple csv files to HDF5
      1. USAndy Hayden
      1. VTFavorite
    1. COdon't pass the ``mode='w'``; you are truncating the hdf file on each iteration
      singulars
      1. POPandas ParserError EOF character when reading multiple csv files to HDF5
      1. USJeff
    2. COyou can try catching the CParserError and just skip that file (until you fix it)
      singulars
      1. POPandas ParserError EOF character when reading multiple csv files to HDF5
      1. USJeff
    3. COHi Jeff, how do you suggest I catch the CParserError. It's way too cumbersome to check each of the individual files.
      singulars
      1. POPandas ParserError EOF character when reading multiple csv files to HDF5
      1. USMatthijs
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload