Board index   FAQ   Search  
Register  Login
Board index php forum :: php coding PHP coding => General

Fatal error: Allowed memory size of 134217728 bytes exhauste

Ask about general coding issues or problems here.

Moderators: macek, egami, gesf

Fatal error: Allowed memory size of 134217728 bytes exhauste

Postby Kerryh » Wed Mar 27, 2013 9:06 pm

:help: I am new to PHP. I have written a program to compare 2 csv files. It reads in the first file, puts it into an array, and reads in the 2nd file and puts it into an another array, then compares 2 arrays.
I got some very big files today and run them with my program and I got "Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes)".
My colleague told me to update the program so that it can split the file into smaller files (which I am not sure how to achieve this with PHP?)? Or are there any other ways to solve this problem?

Below is my code -

Code: Select all
<?php


if (($handle = fopen($_POST["file1"], "r")) !== FALSE) {
    # Set the parent multidimensional array key to 0.
    $nn1 = 0;
    while (($data = fgetcsv($handle, 0, "¦")) !== FALSE) {   
        # Count the total keys in the row.
        $c1 = count($data);
        # Populate the multidimensional array.
        for ($x=0;$x<$c1;$x++)
        {
            $csvarray1[$nn1][$x] = $data[$x];
         //echo "$csvarray1($nn1)($x) = ". $csvarray1[$nn1][$x] . "<br />\n";
         //echo "<br />\n";
        }
        $nn1++;
      //echo "<br />\n";
    }
    # Close the File.
    fclose($handle);
}

if (($handle = fopen($_POST["file2"], "r")) !== FALSE) {
    # Set the parent multidimensional array key to 0.
    $nn2 = 0;
    while (($data = fgetcsv($handle, 0, "¦")) !== FALSE) {
        # Count the total keys in the row.
        $c2 = count($data);
        # Populate the multidimensional array.
        for ($x=0;$x<$c2;$x++)
        {
            $csvarray2[$nn2][$x] = $data[$x];
         //echo "$csvarray2($nn2)($x) = ". $csvarray2[$nn2][$x] . "<br />\n";
         //echo "<br />\n";
        }
        $nn2++;
      //echo "<br />\n";
    }
    # Close the File.
    fclose($handle);
}

 
function array_diff_no_cast(&$ar1, &$ar2) {
    $diff = Array();
          foreach ($ar1 as $key => $val1) {
                    if (array_search($val1, $ar2) === false) {
                       $diff[$key] = $val1;
                    }
                 }
                 return $diff;
               }


$result1 = array_diff_no_cast($csvarray1,$csvarray2);
$result2 = array_diff_no_cast($csvarray2,$csvarray1);



//output the 2 results array in a table
Kerryh
New php-forum User
New php-forum User
 
Posts: 5
Joined: Wed Mar 27, 2013 8:53 pm
Location: australia

Re: Fatal error: Allowed memory size of 134217728 bytes exha

Postby seandisanti » Thu Mar 28, 2013 10:35 am

you shouldn't need to break the file down, just don't load everything into memory. You could even use mysql to do your comparisons and speed up your execution while reducing memory usage. Instead of reading everything into associative arrays, throw the files into 2 database tables, then run a query that joins them on the field that you're comparing, and then you get all of the same information without having to step through each iteration or load more than a line or two at a time into memory. When you're all done, you can truncate or drop the tables to dump the data or the whole table if you like.
seandisanti
php-forum Fan User
php-forum Fan User
 
Posts: 838
Joined: Mon Oct 01, 2012 12:32 pm

Re: Fatal error: Allowed memory size of 134217728 bytes exha

Postby Kerryh » Thu Mar 28, 2013 2:18 pm

Thank you for your reply. Much appreciated.
Kerryh
New php-forum User
New php-forum User
 
Posts: 5
Joined: Wed Mar 27, 2013 8:53 pm
Location: australia

Re: Fatal error: Allowed memory size of 134217728 bytes exha

Postby seandisanti » Thu Mar 28, 2013 10:01 pm

no problem, let me know if you run into any issues rewriting and I'll be happy to help.
seandisanti
php-forum Fan User
php-forum Fan User
 
Posts: 838
Joined: Mon Oct 01, 2012 12:32 pm

Re: Fatal error: Allowed memory size of 134217728 bytes exha

Postby Kerryh » Sat Mar 30, 2013 10:22 pm

Hi, thanks for offering to help. I have been trying to rewrite the program and trying to learn PHP MySQL at the same time and have been struggling. I think I have got the uploading file to database tables ok, but not sure how to compare 2 tables? You mentioned to use join command? The goal here is to compare 2 tables and display differences.
Can you help me by getting me started with some code that might help me please? Or direct me to other tutorial websites etc... I haven't done coding for a while and am a little rusty. Thank you.
Kerryh
New php-forum User
New php-forum User
 
Posts: 5
Joined: Wed Mar 27, 2013 8:53 pm
Location: australia

Re: Fatal error: Allowed memory size of 134217728 bytes exha

Postby seandisanti » Sun Mar 31, 2013 7:52 pm

no problem, what fields do you have in your tables?
seandisanti
php-forum Fan User
php-forum Fan User
 
Posts: 838
Joined: Mon Oct 01, 2012 12:32 pm

Re: Fatal error: Allowed memory size of 134217728 bytes exha

Postby Kerryh » Mon Apr 01, 2013 2:04 pm

Thank you!

I have 5 types of files (each type from 2 sources) to compare. I will just give you an example of one type of file.

Address file -
CDN (client number)
type
street Address
other designation (addr line 2)
city
state
post code
country

Sample Data in the file
CD8516766¦M¦PMITEST Mail Addr 1¦Mail Addr 2¦Sydney¦NSW¦2000¦1101
CD8516766¦R¦PMITEST Res Addr 1¦Res Addr 2¦Brisbane¦QLD¦4000¦1101
CD8516766¦T¦PMITEST Temp Addr 1¦Temp Addr 2¦Melbourne¦VIC¦3000¦1101


Thanks again!
Kerryh
New php-forum User
New php-forum User
 
Posts: 5
Joined: Wed Mar 27, 2013 8:53 pm
Location: australia

Re: Fatal error: Allowed memory size of 134217728 bytes exha

Postby Kerryh » Mon Apr 01, 2013 2:11 pm

Sorry, you asked for fields in the table.

CDN
type
street Address
other designation (addr line 2)
city
state
post code
country

In my code, I created a table in the database with fields above, and loaded the file (from source A) to the table. Then created another table with a different table name with exactly the same fields above and loaded file (from source B).
Then I will compare 2 tables, then display the differences, then drop those tables.

Thanks.
Kerryh
New php-forum User
New php-forum User
 
Posts: 5
Joined: Wed Mar 27, 2013 8:53 pm
Location: australia

Re: Fatal error: Allowed memory size of 134217728 bytes exha

Postby seandisanti » Mon Apr 01, 2013 2:51 pm

cool, so you have a common field, CDN. so you can do something like:
Code: Select all

$db 
Database::getInstance();
$sql "SELECT 
            t1.cdn as cdn,
            concat_ws(',',t1.field1,t1.field2,t1.field3,t1.field4)as t1.value,
            concat_ws(',',t2.field1,t2.field2,t2.field3,t2.field4) as t2.value 
        FROM 
            table1 t1 INNER JOIN table2 t2 ON t1.cdn = t2.cdn
        WHERE 
            t1.value<>t2.value"
;
$result $db->query($sql);
if (!
$result ||$result->rowCount()==0){
    return 
FALSE;
}
$output 'These cdns have updates';
while (
$row $result->fetch(PDO::FETCH_ASSOC)){
    
$output .= $row['cdn'] . ', ';
}
$output rtrim($output,', ');
return 
$output;
 

my example makes use of PDO to interact with the database, by way of the Database.php file i've posted many times already. check my recent posts that have database in the title and you should be able to find it and an explanation for using it fairly quickly. (or just take the logic above and apply it to your current database interaction method)
seandisanti
php-forum Fan User
php-forum Fan User
 
Posts: 838
Joined: Mon Oct 01, 2012 12:32 pm

Re: Fatal error: Allowed memory size of 134217728 bytes exha

Postby seandisanti » Mon Apr 01, 2013 2:54 pm

You could even just update table1 values to match table2 values if the cdn is in both, and let triggers log the changes. but as with any issue you can think of there are several viable solutions.
seandisanti
php-forum Fan User
php-forum Fan User
 
Posts: 838
Joined: Mon Oct 01, 2012 12:32 pm


Return to PHP coding => General

Who is online

Users browsing this forum: Google [Bot] and 4 guests

Sponsored by Sitebuilder Web hosting and Traduzioni Italiano Rumeno and antispam for cPanel.