Databases
StackOverflow2013
Postsdbo
POSplitting 25mb .txt file into smaller files using text delimiter
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSplitting 25mb .txt file into smaller files using text delimiter
    primarykey
    data
    text
    <p>Regards, SO </p> <p>I am new to python and Perl. I have been trying to solve a simple problem and getting tied in knots with syntax. I hope someone has the time and patience to help. I have a 25mb file in ".txt" format which contains news-wire articles going back to 1970. Each news story is concatenated to the next, with only the "Copyright" statement to delimit. Each news story starts with "Item XX of XXX DOCUMENTS". There are certain metadata that are repeated throughout, I will use these for tagging later on.</p> <p>I wish to split this 25mb file into separate .txt files, each containing one news story (i.e. the text between "DOCUMENTS" and "Copyright", saving each with a different name (obviously).</p> <p>I am trying to 1 ) open the file... 2) iterate over lines in the file checking for the eof delimiter, and if it is not present writing the line to a list 3)write that list to a seperate small file.</p> <p>I'm having big problems with changing filenames using the counter, and how do I make Python start from where I left off, is the "seek" function appropriate?</p> <p>so far I have been trying this approach, completely unsuccessfully:</p> <pre><code>myfile = open ("myfile.txt", 'r') filenumber = 0 for line in myfile.readline(): filenumber += 1 w=0 while myfile.readline() != '\s+DOCUMENTS\s*\n' ### read my line into a list mysmallfile()['w'] = [myfile.readline()] w += 1 output = open('C:\\Users\\dunner7\\Documents\###how do I change the filename each iteration???', 'w') output.writelines(mysmallfile) ###go back to start. </code></pre> <p>Thank you for your time and patience.</p> <p>RD</p> <p>Here is a sample of the text file:</p> <pre><code> 1 of 575 DOCUMENTS The Washington Daybook January 28, 2011 </code></pre> <p>Health and Human Services Department (HHS); Food and Drug Administration (FDA) (F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the Medical Devices Advisory Committee to discuss and make recommendations regarding the possible reclassification of devices indicated for use in electroconvulsive therapy, January 27-28.</p> <p>TIME: 8 a.m.</p> <p>LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway, Gaithersburg, Md.</p> <p>CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when calling for information.]</p> <p>LOAD-DATE: November 28, 2010</p> <p>LANGUAGE: ENGLISH</p> <p>TYPE: Meeting</p> <pre><code> Copyright 2011 Federal Information and News Dispatch, Inc. 2 of 575 DOCUMENTS The Washington Daybook January 27, 2011 </code></pre> <p>Health and Human Services Department (HHS); Food and Drug Administration (FDA) (F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the Medical Devices Advisory Committee to discuss and make recommendations regarding the possible reclassification of devices indicated for use in electroconvulsive therapy, January 27-28.</p> <p>TIME: 8 a.m.</p> <p>LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway, Gaithersburg, Md.</p> <p>CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when calling for information.]</p> <p>LOAD-DATE: November 28, 2010</p> <p>LANGUAGE: ENGLISH</p> <p>TYPE: Meeting</p> <pre><code> Copyright 2011 Federal Information and News Dispatch, Inc. 3 of 575 DOCUMENTS FNS DAYBOOK January 12, 2011 Wednesday FUTURE EVENTS </code></pre> <p>EVENT: MEETING - HEALTH AND HUMAN SERVICES DEPARTMENT (HHS); FOOD AND DRUG ADMINISTRATION (FDA) (F.R. PAGE 72832); LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway, Gaithersburg, Md. -- January 27, 2011 8:00 am</p> <p>SECTION: FEDERAL AGENCIES AND DEPARTMENTS - FUTURES</p> <p>LENGTH: 72 words</p> <p>SUBJECT: Health and Human Services Department (HHS); Food and Drug Administration (FDA) (F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the Medical Devices Advisory Committee to discuss and make recommendations regarding the possible reclassification of devices indicated for use in electroconvulsive therapy, January 27-28.</p> <p>CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when calling for information.]</p> <p>LOAD-DATE: January 10, 2011</p> <p>LANGUAGE: ENGLISH</p> <p>PUBLICATION-TYPE: Event Schedules</p> <pre><code> Copyright 2011 Federal News Service All Rights Reserved </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USrosser
    1. USrosser
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COcan you provide a snippet of the file?
      singulars
      1. POSplitting 25mb .txt file into smaller files using text delimiter
      1. USnosklo
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload