Databases
StackOverflow2013
Postsdbo
POProcessing large files in PHP
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POProcessing large files in PHP
    primarykey
    data
    text
    <p>What is the best way of processing very large files in php. This is my current the scenario:</p> <ol> <li>I extract a raw file from a Network Mgmt System (NMS) about all parameters of all network elements (NMS is running in a UNIX box).</li> <li>I FTP the raw file in my PC box using PHP.</li> <li>I process the raw file line by line using the PHP's fget() function.</li> <li>Every line i use string matching and regexp matching to extract necessary data until i am able to compose 1 line of necessary data separated by commas (",").</li> <li>I repeat step 4 until EOF and have the full CSV file.</li> <li>I then throw this data to my database using sql's "LOAD DATA INFILE"</li> </ol> <p>My problem now is, I have 1 raw file that reaches more/less 200MB, and has more/less 180 columns of data and because of these my php script cannot finish proccessing the whole file because upon processing it is exhausting all the 1024MB memory I allocate on my php.ini file.</p> <p>Hoping to have recomendations on the best workaround of this problem. Thanks!</p> <p>code of the processing part below:</p> <pre><code>while( !feof($fh) ){ set_time_limit(0); $l_buffer = fgets( $fh, $fsize ); $l_LineStream = explode( ' ', trim( $l_buffer ) ); $l_FilteredLineStream = array_filter( $l_LineStream, array( $this, 'RemoveEmptyElement' ) ); $l_GrepMatchArray = preg_grep( '/^BSC.*_.*$/', $l_FilteredLineStream ); if( count( $l_GrepMatchArray ) &gt; 0 ){ foreach( $l_GrepMatchArray as $l_BSCFound ){ $l_BSCFound = explode( '_', $l_BSCFound ); $l_BSCHoming = $l_BSCFound[1]; } } $l_GrepMatchArray = preg_grep( '/^BTS-[0-9]*$/', $l_FilteredLineStream ); if( count( $l_GrepMatchArray ) &gt; 0 ){ foreach( $l_GrepMatchArray as $l_BTSFound ){ $l_CurrBTS = $l_BTSFound; } } /**/ if( $l_PrevBTS != $l_CurrBTS &amp;&amp; isset( $l_BTSArray ) &amp;&amp; count( $l_BTSArray ) &gt; 0 ){ #$this-&gt;BTS_Array[] = $l_BTSArray; if( $l_FirstLoop == true ){ $this-&gt;WriteDataToCSVFile( $l_BTSArray, $l_FilePath, true ); $l_FirstLoop = false; }else{ $this-&gt;WriteDataToCSVFile( $l_BTSArray, $l_FilePath ); } } /**/ if( count( $l_GrepMatchArray ) &gt; 0 ){ #var_dump( $l_FilteredLineStream ); $l_BTSArray = $this-&gt;InstantiateEmptyBTSArray(); #$l_BTSArray['CI'] = ''; $l_BTSArray['BSC'] = $l_BSCHoming; $l_BTSArray['BCF'] = $l_FilteredLineStream[0]; $l_BTSArray['BTS'] = $l_FilteredLineStream[3]; $l_BTSArray['CELL NAME'] = $l_FilteredLineStream[6]; } if( $l_GetPLMNNextLineData == true &amp;&amp; isset( $l_BTSArray['PLMN'] ) ){ $l_BTSArray['PLMN'] .= trim( $l_buffer ); $l_GetPLMNNextLineData = false; } $l_GrepMatchArray = preg_match( '/\.\(.*$/', $l_buffer, $reg_match ); if( count( $reg_match ) &gt; 0 ){ $l_KeyName = substr( $reg_match[0], 2, strpos( $reg_match[0], ')' ) - 2 ); preg_match( '/[[:space:]].*|[-].*/', $reg_match[0], $param_value ); $l_BTSArray[$l_KeyName] = trim( $param_value[0] ); if( $l_KeyName == 'PLMN' ){ $l_GetPLMNNextLineData = true; } } $l_PrevBTS = $l_CurrBTS; } </code></pre>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. UShakre
    1. USJCm
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COHow do i do that? Im sorry if it sound so dumb, but im not very much familiar with how do things work around here.. Thanks for ur consideration.
      singulars
      1. POProcessing large files in PHP
      1. USJCm
    2. COyou choose an answer when you get a good one...
      singulars
      1. POProcessing large files in PHP
      1. USCosminO
    3. COSee FAQ or : http://cdn.sstatic.net/img/faq/faq-accept-answer.png
      singulars
      1. POProcessing large files in PHP
      1. USadrien
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload