Long-running PHP processes

Giorgio Sironi

If you are looking at these slides on your pc, there are speaker notes in the HTML

Giorgio Sironi (@giorgiosironi)

Software Engineer in Test (automates stuff for a living)
What do I do
- Distributed systems
- Automated complex tests, integrating many different projects
- Continuous Delivery
- Pasta and risotto

(Main) PHP SAPIs

The type of PHP SAPIs and how the CLI one is used by all long-running processes

Differentiation

CLI category	Running time	STDIN	STDOUT
Script	seconds, minutes	sometimes	always
Cron	seconds (best), minutes	no	no
Long-running process	minutes, hours, days	no	no

Quick examples

RabbitMQ/ZeroMQ/SQS consumer
Gearman worker
Database projection generator

Essential characteristics

runs for hours, and it's supposed to
instantiated multiple times for parallelism
child of a daemon instead of a web server

PHP?!

Not exactly built for this (non-exhaustive list)

multiprocess is not memory efficient
no standard supervision system or framework
not much fault tolerance (Fatal Error)

In general, PHP provides parallel, independent execution by spawning multiple processes. This is not memory efficient as each process, unless you go to the touble of forking your own processes, which ties all your copies together and I prefer to avoid. Unlike with Erlang or other platform build for high concurrency and fault-tolerance, there is no standard architecture to put processes into a framework like you would do for classes with PSR standards instead. And when you encounter a (non-Catchable) Fatal Error, there's no way to recover inside the same process. There are many more problems coming up, such as memory leaks, which I'll list later. Just wanted to let you know that I am aware of the limitations of PHP for this job.

PHP!

your application is already written with it
you use libraries written in PHP (open source and private)
deployment tooling is already in place
you know the language
ability to move existing functionality into the background

Minimal structure


#!/usr/bin/env php
require_once 'bootstrap.php';
while (!$shouldStop()) {
    $item = $blockingCall($timeout = 20);
    $doWork($item);
    $commit($item);
}

Once the PHP CLI script has finished bootstrapping, it goes into a loop for processing transactions (not necessarily a database transaction: using the term in a more conceptual . This loop will only exit if some conditions are reached in $shouldStop, for example a signal being received or some very bad, unrecoverable exception being caught before. Inside the loop, a single transaction consists of stopping on a blocking call, like receiving a message from a queue or a task from a job server. This may be a TCP connection or HTTP long polling, and should always be configured with a timeout, resulting in a null $item and an empty iteration that can bring us to re-evaluate the loop condition periodically. Once we have $item, we have to process it: calculate a quote, resize an image, send an email, and so on. To implement at-least-once-delivery, the only sensible option for processes like these which are often integrating many different systems, many queues/job systems require a commit action to be performed as the last step. Committing the item will remove it permanently from the queue, while if $doWork crashed the item will eventually reappear to retry processing. Alternatively, they require to specify a callback and the successful execution of that callback without exceptions will be interpreted as committing.

Problems: supervising (1)

Is the process still alive?
If it's crashed, we should restart it
There's a Class 'X' not found, we should stop restarting it

Even if we strive for a process being long-running, it will sooner or later stop due to external unforeseen conditions (uncaught exceptions, Fatal Error, VM problems, full disks...) Therefore we would like the process to be restarted if it's a sensible thing to do, to continue to process new transactions, possibly pursuing a let-it-crash philosophy. Howewer, if a process continues to crash because there is some underlying problem like a bad deploy, restarting it is only going to saturate the CPU with its bootstrap. You may have noticed we are reinventing the first level of an Erlang supervision tree. To get good ideas from other platforms without getting their battle-tested implementations is the price we pay for the advantages of using PHP to solve this problem.

Problems: supervising (2)

Supervisord (everywhere)
Upstart (e.g. Ubuntu 14.04)
Systemd (e.g. Ubuntu 16.04)

Problems: stopping


$flag = false;
pcntl_signal('SIGTERM', function () use (&$flag) {
    $flag = true;
});
$shouldStop = function() use (&$flag) {
    pcntl_signal_dispatch();
    return $flag;
}

Problems: memory


# php.ini memory_limit = -1
$shouldStop = function() use ($maximum) {
    return memory_get_usage(true) > $maximum;
}

As a PHP process works for hours or days, it may progressively leak memory that remains allocated. These leaks may be beyond your control, coming from the insides of a library or a PHP extension; even if they are almost unnoticeable, they accumulate and can make your process crash or, worse, make your machine swap heavily impacting performance greatly. Web-facing processes do not suffer this problem as most (but not all, e.g. database persistent connections) of the memory is not persistent and is wiped at the end of every HTTP request. PHP is not the JVM so I decided not to try to optimize garbage collection or to track down every possible leakage. A more resilient approach is to replicate the process and have each instance monitor its memory usage as a stopping condition. If the memory usage is too high, the process exits gracefully and a new copy is respawned by its supervisor, with fresh and empty RAM segments. memory_get_usage(true) is a good approximation of what top reports in the RES(ident) column, total memory allocated to the script (in my experience top always reports some more, probably due to a fixed overhead of the interpreter. memory_limit can usually be set to no limit for CLI processes to avoid for outliers to complete without crashing, as they will be followed by a restart. The overhead of a restart may be paid just once every several thousand iterations, depending on how much memory is being leaked.

Problems: global state


                      global $db;
                      CustomerSingleton::instance();
                      $entityManager->close();

A subtle problem is that of unwanted interaction between the transactions being performed by a long-running process, due to global state being carried over from one to the next. Not all global state is bad, as long as it is immutable or seen as immutable from the point of view of the application code (e.g. a buffering Logger). In fact, we plan on shared resources such as the PHP interpreter memory and the database connections being carried over between transactions; it is a foundation for the efficiency of long-running processes. However, global variables, singletons or Kernel-Application level objects may require tuning to correctly work in this environment. Some objects may need an explicit reset, like singletons pointing to specific customers or users (I hope you are not using those). Other more generic objects like Doctrine's EntityManager may need to be recreated or reopened if possible. This is usually not a problem if you have a clean architecture. When I started doing this talk I didn't even include it, but in the wild global state may bite you hard so it's useful to have a reminder.

Problems: bottlenecks

(in order of appearance)

Single threading ...create more than one instance
Memory ...horizontal scaling with more servers
CPU ...horizontal scaling too
Database you're reading/writing ...the usual profiling
External resources (e.g. rate limiting) ...congratulations

Usually these bottlenecks in throughput come up roughly in this order as you expand the number of transactions your long-running processes are daeling with. First that comes up is usually the single thread approach of PHP: every process can only work for a small interval of time before it's blocked by latency, waiting on an external API call or a database query. It only uses several unit percentage of CPU utilization. Optimizing latency is also a matter of algorithms and overall architecture, but as for throughput you should replicate your process leveraging parallelism. When you start replicating processes, available RAM (or sometimes CPU) become the next bottlenecks, depending on whether the work is data-intensive or computationally intensive. Fortunately, you can scale both by paying for more servers/containers to be added to your application, and this is almost linear scaling. Next up is the bottleneck of the database you are reading or writing from. Usually queues and job servers are optimized for this not to be a problem, but the work you are performing is relying on the databases backing up the state of your application to perform well. The usual profiling&optimization routines apply here. If you really have optimized everything (optimized enough), you may end up in a situation where it's an external API that rate limits you. I can only say congratulations if you have reached this point, unless you have overlooked some optimizations there your problem may now be at the social or business level rather than technical.

Tools

```
ps afuxww
```
New Relic APM and Server
Symfony Console and your own libraries

`ps` is a wonderful Unix utility. The `afuxww` options try to give you a comprehensive listing of all the processes running on a server, plus some metadata about them like their state and user. New Relic can monitor both single instrumented processes (through the `newrelic` PHP extension) and full servers with CPU, RAM and I/O measurements. Especially if you have many processes or you spread them through different services, it's important to be notified of error rates going up on one category without having to check them manually. The PHP standard library is a thin veil over the POSIX standards. Symfony Console is a useful addition to build any command line process in a OO way, you won't use interactivity of course. You should also extract your own libraries to share between services once your team agrees on some design decisions.

Resources

Conclusions

PHP long-running processes work well if you carefully work on the operations, not just development
Know your limits: performance and efficiency
Cohesion and simplicity of your application win over technological terror

Thanks!

@giorgiosironi @eLife

g.sironi@elifesciences.org

https://joind.in/talk/b89dd