Long time execution -exit(idle timeout), terminated by calling exit(), return code: 0

[kilburn]

Quote:many people around the world use fcgi for conversion and i don't see why it shouldn't work longer than few minutes

I don't think so. Both the fastcgi protocol and the fcgid implementation are designed to run web scripts, this is, fast responding scripts. You can subvert them to try to run long data-crunching tasks, but this is only a hack. Additionally, there are multiple reasons why this is undesirable:

- When running under fcgid, apache workers are left locked until the script responds. Therefore, you are locking apache workers with your long-running tasks. This can even derive into a DoS condition.
- Fcgid-spawned processes should be able to terminate within a given time range. Otherwise, when apache is restarted, these processes will have to be killed by fcgid after a timeout, causing both long delays to the apache restart and the running script to terminate improperly.
- If the client cancels the request (using the "stop" button from their browser) or looses connection, the script will be finished without letting the long operation finish his task.

A much better solution is to delegate such long tasks to background, asynchronous processes. Something like the following (this is rought code, but the idea is clear):

PHP Code:

function convert_video($id, $infile, $outfile) {
    function shutdown() {
        posix_kill(posix_getpid(), SIGHUP);
    }

    // Go to background
    $pid = pcntl_fork(); // fork
    if ($pid < 0) return false; // Unable to fork
    else if ($pid) return true;
    else {
         // child, but not detached from the main process yet. Cleanup things
        ob_end_clean(); // Discard the output buffer and close
        fclose(STDIN);  // Close all of the standard
        fclose(STDOUT); // file descriptors as we
        fclose(STDERR); // will be running in background.
        register_shutdown_function('shutdown');
        // Try to become master
        if (posix_setsid() < 0) exit(1);
        // And do the final fork out of apache/fastcgi control
        if (pcntl_fork()) exit(0);
        
        // Perform your long running task. You *must* reopen any db connection,
        // as it will have been closed by the first child.
        mysql_connect($host, $user, $pass);
        mysql_select_db($db);
        $cmd = 'ffpmeg -i ' . escapeshellarg($infile) . " -o " . escapeshellarg($outfile);
        $output = exec($cmd, $output = array(), $ret = 0);
        $ret = $ret == 0 ? 'ok' : 'fail';
          $output = implode("\n", $output);
        $qry = "INSERT INTO processed_videos (id, file, result, log) VALUES('" .
            mysql_real_escape_string($id) . "', '"
            mysql_real_escape_string($outfile) . "', '"
            mysql_real_escape_string($ret) . "', '"
            mysql_real_escape_string($output) . "')";
        mysql_query($qry);
        
        // Exit cleanly when the task has been completed.
        exit(0);
    }
} 


An even better solution would be simply submitting the tasks to some backend (which might very well be a cron running every X minutes) that makes sure to process at most X conversions at the same time to avoid resource exhaustion at the server.

[circut_breaker]

Hmm, I think it might be possible.

I got the point but i can't get it to work.

I have already installed php5-cli package what should(?) contain pcntl extension.

So i enable it using php.ini of user website.

Quote:extension=php_pcntl.dll

But when i want run following code.

Code:

<?
function convert_video() {
    function shutdown() {
        posix_kill(posix_getpid(), SIGHUP);
    }

    // Go to background
    $pid = pcntl_fork(); // fork
    if ($pid < 0) return false; // Unable to fork
    else if ($pid) return true;
    else {
         // child, but not detached from the main process yet. Cleanup things
        ob_end_clean(); // Discard the output buffer and close
        fclose(STDIN);  // Close all of the standard
        fclose(STDOUT); // file descriptors as we
        fclose(STDERR); // will be running in background.
        register_shutdown_function('shutdown');
        // Try to become master
        if (posix_setsid() < 0) exit(1);
        // And do the final fork out of apache/fastcgi control
        if (pcntl_fork()) exit(0);

        mail('admin@xxx.xxx', 'start', 'start');
        sleep(600);
        mail('admin@xxx.xxx', 'stop', 'stop');

        // Exit cleanly when the task has been completed.
        exit(0);
    }
}
convert_video();
?>

And of course it says it don't have it

Code:

Fatal error: Call to undefined function pcntl_fork()

Any idea... except recompile php with pcntl - enabled ?

[kilburn]

Two things:

1. Websites run the php5-cgi version, which (as you correctly spotted) is *not* compiled with pcntl support enabled in debian.
2. Your idea about installing it as a module for this client is correct. Nevertheless, you have two errors there: (1) *.dll files are windows modules, linux ones are *.so; and (2) as php5-cli is compiled with pcntl support builtin (not as module), the pcntl.so module is simply not available in your system.

Thus, you have to compile and install the module first (and you'll have to do it each time the php version is updated by debian). These steps should get you going:

Code:

apt-get install php5-dev dpkg-dev
mkdir -p /usr/local/src
cd /usr/local/src
apt-get source php5
cd php5-{VERSION}/ext/pcntl
phpize
./configure
make install

Don't forget to replace "extension=php_pcntl.dll" line with "extension=pcntl.so" in your client's php.ini file and restart apache.

[circut_breaker]

I made it almost like you but i don't use make install, just copy compiled file

Quote:cp modules/pcntl.so /usr/lib/php5/20060613+lfs

And it's working, i can throw this process into background.

Of course i run my application from cron, it's not accessible via browser.

.dll - I still think in 2k3 server


I have trouble with getting output from exec...

That's why i use shell_exec instead exec... but shell_exec don't seem working ass good as exec. I ran few tests and shell_exec was stopped... all tests with just exec completed successful.


The trouble is i'm getting only "last" row of output...

I had try:

Code:

$output = exec($cmd, $output, $return);

foreach ($output as $line) {
              mail('admin@xxx.xxx', "test$line", "test$line tes");
}

I don't get any e-mails...

But $output contain last row... i was thinking it's an array that's why i use foreach... it's console so for test purpose i wan't debug it for my e-mail.... Later i will use print_r to ob_buffer and e-mail to see what's going on....


Except getting status looks working

Tell me is there any possible for fetching that output in same look as in real console ?
I need that output for debugging.

If it's more complex maybe faster will be throw output to file

Code:

comand >random_file_name.txt

And then using file_get_contents(random_file_name.txt) take output
Maybe it will be faster
What do You think about it ?

Regards

[kilburn]

Quote:I made it almost like you but i don't use make install, just copy compiled file

Fine, any special reason to avoid make install though?

About the output thing, notice that you are using "$output" two times in your call. Change it to:

Quote:exec($cmd, $output=array(), $return);

And you're done. See that you were using "$output" at the left side of the assignation. The php manual for "exec()" clearly states that the return value is the last line of the command's output, so this was overwriting the values assigned inside the exec function (this is, $output was first getting an array with all the output as lineas, and then overwriten by only the last line, which is what you were getting).

Oh, and no, I don't think that redirecting stdout to a file would improve performance, quite the opposite in fact. Without redirecting, the output will be stored in memory only, whereas if you redirect it will go to the disk. Disk is slower than memory.... so at the end you would get worse performance.

[circut_breaker]

Welcome,

Sorry for late reply but i was sick.

Finally i use shell_exec, that error about what i talk previous was created by print($output); command... I just deleted printing because i don't need it...


So i will start from beginning.

After installing pcntl, putting application into threads everything was working great about half month 24/24. Application was starting by cron, every minute using following code.

Code:

* * * * *  /usr/bin/wget --delete-after "http://xxx.xxx/xxx.php" >/dev/null 2>&1

* * * * *  /usr/bin/wget --delete-after "http://xxx.xxx/xxxy.php" >/dev/null 2>&1

As i said, everything work smooth half month...

About 4 days back, i got emergency call.... server goes down...
They take me out from bed and put in front of server...

I was trying connect via ssh but she didn't respond me... i can't access apache, webmin... anything...

I had cut power off.

After restart, everything work great...
I run some tests, check disks, look at memory, make all updates, everything look ok so i left server alone.

After 24 hours i get same call...

Force reboot, and from 2 days i'm looking at logs, everything and found where error is...

Application started by pcntl process don't close correctly... and create zombie process

Code:

    1646    vu2xxx    10:07    [php5-cgi] <defunct>
               1656    vu2xxx    10:08    [php5-cgi] <defunct>
               1676    vu2xxx    10:09    [php5-cgi] <defunct>
               1689    vu2xxx    10:10    [php5-cgi] <defunct>
               1699    vu2xxx    10:11    [php5-cgi] <defunct>
               1710    vu2xxx    10:12    [php5-cgi] <defunct>
               1720    vu2xxx    10:13    [php5-cgi] <defunct>
               1730    vu2xxx    10:14    [php5-cgi] <defunct>
               1744    vu2xxx    10:15    [php5-cgi] <defunct>
               1757    vu2xxx    10:16    [php5-cgi] <defunct>
               1767    vu2xxx    10:17    [php5-cgi] <defunct>
               1777    vu2xxx    10:18    [php5-cgi] <defunct>
               1788    vu2xxx    10:19    [php5-cgi] <defunct>
               1803    vu2xxx    10:20    [php5-cgi] <defunct>
               1821    vu2xxx    10:21    [php5-cgi] <defunct>
               1832    vu2xxx    10:22    [php5-cgi] <defunct>
               1842    vu2xxx    10:23    [php5-cgi] <defunct>
               1856    vu2xxx    10:24    [php5-cgi] <defunct>
               1867    vu2xxx    10:25    [php5-cgi] <defunct>
               1877    vu2xxx    10:26    [php5-cgi] <defunct>

vu2xxx represent user who use pcntl threads.

After 12 hours it create 800 process.

I check date of sources, and no modifications was made in that half month... i don't know why it stops working.
Right now i have every 12hours reboot server !



This is code of application:
( i remove code and put comments )

Code:

<?php
/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/


function download_xxx() {
    function shutdown() {
        posix_kill(posix_getpid(), SIGHUP);
    }

    // Go to background
    $pid = pcntl_fork(); // fork
    if ($pid < 0) return false; // Unable to fork
    else if ($pid) return true;
    else {
         // child, but not detached from the main process yet. Cleanup things
        ob_end_clean(); // Discard the output buffer and close
        fclose(STDIN);  // Close all of the standard
        fclose(STDOUT); // file descriptors as we
        fclose(STDERR); // will be running in background.
        register_shutdown_function('shutdown');
        // Try to become master
        if (posix_setsid() < 0) exit(1);
        // And do the final fork out of apache/fastcgi control
        if (pcntl_fork()) exit(0);

        // Perform your long running task. You *must* reopen any db connection,
        // as it will have been closed by the first child.

      
//start mysql code
//here i START mysl connection


//Here i call mysql select


//Here  i download file...

//Here i save file

//Here i call mysql and put information about downloaded file


//Here i disconnect mysql
  

        // Exit cleanly when the task has been completed.
        exit();
    }
}
download_xxx();
exit();
?>

I really don't know what's going wrong.
It's really working half month... every single minute starting 2 process.... right now i't don't close thread...


Please help

[kilburn]

Who is the parent owner of these zombie processes?

[circut_breaker]

Welcome,

I attach screenshot of process.

http://www.freeimagehosting.net/image.ph...031529.png
(i recommend look at tree)

Parent is "/usr/bin/php5-cgi"

   

[circut_breaker]

I found temporary solution.

1) Set up new server using Ubuntu 8.04
2) On webserver (ispcp) allow access to mysql using local ip adress (192.x.x.x.x)
3)On Ubuntu server install headless php, using php_cli
4)Copy scripts, add little changes... and it's ready

Working really great... and much faster

That's my solution, it help

Regards