Avoiding OOM in Symfony with Phunkie Streams

Every night, the fleet system of a car rental company produces the day's returns: a line per car that came back, with the rental it closes, the time, the mileage and whether the car needs an inspection. The rental desk service has to turn each line into a status change. On a quiet night that file has a few thousand lines. On the Monday after a bank holiday it has a million, which is 96 MB of NDJSON.

{"rental_id":"R-0000042","returned_at":"2026-10-13T06:04:38+00:00","mileage":11554,"condition":"ok"}
{"rental_id":"R-0000043","returned_at":"2026-10-13T07:21:11+00:00","mileage":11591,"condition":"damaged"}

The two services talk over HTTP, and the desk runs where PHP usually runs: behind nginx and PHP-FPM, with memory_limit = 128M. This post builds the exchange with Phunkie Streams, then measures what happens, because the first two attempts at streaming did nothing at all, and the reason is worth knowing.

The version everybody writes first

Here is the consumer most of us would write, and have written:

#[Route('/returns/naive', methods: ['POST'])]
public function returns(Request $request, ReturnRecordParser $parser, RentalDesk $desk): JsonResponse
{
    $lines = explode("\n", $request->getContent());
    foreach ($lines as $line) {
        if ('' === trim($line)) {
            continue;
        }
        $desk->receive($parser->parse($line));
    }
 
    return new JsonResponse(['records' => \count($lines)]);
}

It is correct, and it is fine for the quiet nights. getContent() reads the whole body into a string; explode() makes a second copy as an array. With 100,000 records (9.6 MB of body) the request peaks at 25.7 MB. With the full million:

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted
(tried to allocate 81788960 bytes)

The waste is not the memory. It is the shape of the work: the desk does nothing for as long as the upload takes, then does everything at once. Records that arrived in the first second are closed minutes later, and the whole run fails or succeeds as one piece.

Streams, on both sides

A stream here is a pull-based pipeline: the sink asks for the next element, the source produces one, it flows through every stage, and only then is the next one pulled. Nothing accumulates between stages unless a stage decides to accumulate. Phunkie Streams gives that pipeline a Phunkie shape, with map, filter, flatMap and friends, and IO from phunkie/effect for the stages that touch the world. The code below needs phunkie/streams 1.6, and the two services are on GitHub, with the script that produces every number in this post.

The producer sends the file as it reads it

The fleet service never holds the file in memory either. readAll gives a Stream of 64 KB chunks: the file is opened on the first pull and closed when the stream ends. toGenerator() compiles the pipeline into a generator that runs as it is pulled, and Symfony's HTTP client takes a generator as a body and pulls from it as the socket drains, so the file goes out one chunk at a time. If the client gives up halfway, the generator's finally closes the file:

use function Phunkie\Streams\Functions\file\readAll;
 
final class ReturnsPusher
{
    public function __construct(private readonly HttpClientInterface $http)
    {
    }
 
    public function push(string $file, string $url, int $chunkBytes = 65536): ResponseInterface
    {
        return $this->http->request('POST', $url, [
            'headers' => [
                'Content-Type: application/x-ndjson',
                'Content-Length: '.filesize($file),
            ],
            'body' => readAll(Path($file), $chunkBytes)->compile()->toGenerator(),
        ]);
    }
}

The Content-Length header is not optional. Without it the client falls back to chunked transfer encoding, and PHP-FPM only reads a request body up to the CONTENT_LENGTH it was given: the desk saw an empty body and reported zero records. If the size is genuinely unknown ahead of time, an application server that speaks HTTP itself (FrankenPHP, RoadRunner) does not have this limitation. For a file, the size is one filesize() away.

The consumer closes rentals as the lines arrive

On the desk, the controller does one thing: it turns the request body into a stream, hands it to the domain, and runs the program it gets back.

use function Phunkie\Effect\Functions\io\io;
use function Phunkie\Streams\Functions\io\readHandle;
 
final class ReturnsController
{
    private const int CHUNK_BYTES = 65536;
 
    #[Route('/returns', methods: ['POST'])]
    public function returns(Request $request, ReturnsIngestion $ingestion): JsonResponse
    {
        $body = readHandle(io(fn () => $request->getContent(true)), self::CHUNK_BYTES);
 
        $records = $ingestion->ingest($body)->unsafeRun();
 
        return new JsonResponse(['records' => $records]);
    }
}

getContent(true) is Symfony handing over the php://input handle instead of the string. readHandle takes the opening as an IO rather than as a handle, so the stream owns it: nothing is opened until the first pull, and the handle is closed when the stream ends, whether the body ran out, a stage cut it short, or something threw.

The ingestion is the pipeline. Chunks come in at arbitrary byte boundaries, so the first stage reassembles lines; lines() keeps the partial tail of one chunk and completes it with the next, and emits the last line when the input ends. Then each line becomes a record, records are grouped a thousand at a time, and each group is closed in one database transaction:

final class ReturnsIngestion
{
    public function __construct(
        private readonly ReturnRecordParser $parser,
        private readonly RentalDesk $desk,
        private readonly int $batchSize = 1000,
    ) {
    }
 
    /**
     * @return IO<int> the number of records closed
     */
    public function ingest(Stream $chunks): IO
    {
        return $chunks
            ->lines()
            ->filter(fn (string $line): bool => '' !== trim($line))
            ->map(fn (string $line): ReturnRecord => $this->parser->parse($line))
            ->chunk($this->batchSize)
            ->evalTap(fn (array $batch): IO => io(fn () => $this->desk->receiveBatch($batch)))
            ->foldLeft(0)(fn (int $closed, array $batch): int => $closed + \count($batch));
    }
}

Two details matter. evalTap is where the effect lives: it runs an IO for each batch and passes the batch through, so the database write is part of the pipeline rather than a loop around it. And nothing runs here. foldLeft() turns the pipeline into an IO<int> that counts the records as the batches go by, and the controller is where it runs; the pipeline is a value until the edge of the application.

Memory holds one chunk of text, the tail of a line, and one batch of records. The 64 KB chunk size and the batch size are the only two numbers that decide the footprint, and neither depends on the size of the file.

Backpressure comes free. If the database is slow, evalTap blocks, so the pull blocks, so fread() on php://input is not called, so the socket buffer fills, so the producer's pull waits. The file is read at the pace the desk can close rentals, and nothing in between queues it up.

Then nothing happened

That was the code. Here is what the first push produced:

HTTP/1.1 413 Request Entity Too Large

Three things sit between the producer and the pipeline, and each one of them is a buffer by default.

Layer 1: nginx refuses, then spools

nginx caps request bodies at 1 MB (client_max_body_size). Everybody hits this, and everybody applies the same fix:

client_max_body_size 0;

Now the push goes through, and the numbers look like this (the client is throttled to about 12 MB/s so the timeline is readable; times are measured from the start of the push):

upload finished            +8.71s
desk: request started      +9.98s
desk: first record closed  +10.07s
desk: last record closed   +19.83s

PHP started ten seconds into the push, more than a second after the last byte arrived. That is fastcgi_request_buffering, on by default: nginx reads the whole body first, into client_body_buffer_size (16 KB) and then a temp file under client_body_temp_path, and only then opens the FastCGI connection and replays it. The pipeline streamed beautifully from a file on nginx's disk, after the upload was over. Proxying to an application server has the same switch under a different name, proxy_request_buffering.

Layer 2: PHP reads the body before your script runs

So, the obvious fix:

fastcgi_request_buffering off;

upload finished            +8.15s
desk: request started      +8.15s
desk: first record closed  +8.26s
desk: last record closed   +17.07s

Nothing changed. The body now reaches PHP-FPM as it arrives, and PHP's request startup reads all of it before the first line of the script runs. With enable_post_data_reading on (the default), PHP's default POST reader handles any content type it has no parser for by consuming the body into an internal temp stream, so that php://input can serve it later. For application/x-ndjson that means: wait for 96 MB, then start. The script's peak memory was still low because the temp stream spills to disk, which is exactly why this layer is invisible until you look at the timestamps.

The same setting is what enforces post_max_size, 8 MB by default. Before I raised it, the desk logged a warning and carried on:

PHP Request Startup: POST Content-Length of 10116125 bytes exceeds the
limit of 8388608 bytes

With display_errors on, that warning is output, which sends the response headers, which breaks the JSON response the controller returns afterwards. In production it is a quiet line in the error log and an empty $_POST.

Layer 3: the fix, scoped to one location

Both layers are switched off per endpoint. The streaming route gets its own nginx location, and the ordinary one states its own PHP_VALUE too:

server {
    listen 80;
    root /app/public;
 
    client_max_body_size 0;
    gzip off;
 
    location ~ ^/returns$ {
        fastcgi_pass desk-php:9000;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $realpath_root/index.php;
        fastcgi_param SCRIPT_NAME /index.php;
 
        # Request side: send the body to PHP as it arrives.
        fastcgi_request_buffering off;
        # Response side: send PHP's output as it is flushed.
        fastcgi_buffering off;
        # A long push must not hit the 60s default.
        fastcgi_read_timeout 600;
 
        # PHP's side: do not read the body at request startup. This also
        # makes PHP ignore post_max_size.
        fastcgi_param PHP_VALUE "enable_post_data_reading=0";
    }
 
    location / {
        try_files $uri /index.php$is_args$args;
    }
 
    location ~ ^/index\.php(/|$) {
        fastcgi_pass desk-php:9000;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        # Every PHP location states its own value, see below.
        fastcgi_param PHP_VALUE "enable_post_data_reading=1";
        internal;
    }
}

PHP_VALUE reaches PHP-FPM as a FastCGI parameter, and FPM applies it before the request starts, which is early enough for a startup setting. Be careful about setting the PHP streaming configuration only on the streaming nginx location. Because PHP-FPM reuses workers, that setting can remain on the worker and affect a later non-streaming request: the next form posted to that worker arrives with an empty $_POST. I am, in this post, explicitly setting it per location, but have a look at FPM pools: you could create an FPM pool for streaming as a more polished alternative. Routing the endpoint straight to index.php instead of through try_files keeps the body out of any internal redirect.

desk: request started      +0.02s
desk: first record closed  +0.04s
upload finished            +8.28s
desk: last record closed   +9.02s

The desk starts closing rentals within a twentieth of a second of the push and finishes three quarters of a second after the last byte. The run took 9.1 seconds instead of 17.2, because the upload and the work overlapped, which is the whole point of streaming, and the peak memory of the PHP process was 2 MB.

If nginx proxies to an application server instead of PHP-FPM, the knobs are proxy_request_buffering off, proxy_buffering off and proxy_http_version 1.1, and the PHP setting is irrelevant, since the server reads the socket itself.

The numbers

Measured on the same two containers, 128 MB memory_limit, one million records, 96.5 MB of body. Peak memory is memory_get_peak_usage(true) of the desk's PHP process.

The last two rows are the reason for the code. The streaming configuration is necessary, and it is not sufficient: give the naive controller a perfectly streamed body and getContent() will still materialise it. A pipeline that pulls, a server that passes the stream through, and a producer that does not hold the file either; all three, or you get the nightly fatal on the busy Monday.

The checklist

For a PHP endpoint that has to accept a body of unknown size:

• client_max_body_size 0 (or the real ceiling) in nginx.
• fastcgi_request_buffering off, or proxy_request_buffering off for an application server, in the location of that endpoint only.
• fastcgi_param PHP_VALUE "enable_post_data_reading=0" in the same location, for PHP-FPM.
• fastcgi_param PHP_VALUE "enable_post_data_reading=1" in every other PHP location, because PHP-FPM keeps the value on the worker after the request. Or a separate FPM pool for the streaming endpoint.
• fastcgi_read_timeout above the longest push you expect.
• A producer that declares Content-Length.
• A consumer that reads the body as a stream (getContent(true) hands over the php://input handle) and never asks for it as a string.
• Measure REQUEST_TIME_FLOAT against the time the push started. If PHP starts when the upload ends, something in between is still buffering.