PhpSpec 9 is light speed fast!

PhpSpec 9 is a rewrite. The DSL changed, the doubles are built in, the Gherkin runner is built in, there is a browser client, a parallel runner and coverage from PCOV or Xdebug. I wanted to know what all of that costs at the one moment that matters every day: when you press enter and wait for the suite. So I built one code base, wrote the same checks four times, once per tool, and measured.

Short version: phpspec 9.0.0-rc.7 starts in 26 ms where PHPUnit takes 68 ms and Pest 58 ms, runs a thousand examples in 124 ms against 232 ms and 438 ms, runs the same five hundred Gherkin scenarios in 147 ms against Behat's 195 ms, and with code coverage on it is three to fourteen times faster than PHPUnit and Pest, depending on the engine. It also uses the least memory in every case. The rest of this post is how that was measured, the tables, and why the numbers come out the way they do.

The project is at github.com/MarcelloDuarte/phpspec-benchmark. Everything below can be reproduced with two commands in a container.

The same checks, four times

The code under test is a hundred small shops, Bench\Shop1 to Bench\Shop100, each the same nine files: an Order with its Lines, a Checkout that prices a basket through a Catalogue interface and hands the order to an Orders interface, two in-memory implementations, and three exceptions. Nine hundred files, generated from one template, so that every shop is identical and the suites are big enough for the framework to be the thing measured.

Each tool then gets the same work:

Every example and its test differ only in syntax. This is one of the ten examples each shop gets, as a phpspec example and as a PHPUnit test:

it('saves the order it placed', function (Catalogue $catalogue, Orders $orders) {
    allow($catalogue->priceOf('apple'))->toReturn(100);
    expect($orders->save(anInstanceOf(Order::class)))->toBeCalled()->once();
 
    (new Checkout($catalogue, $orders))->place(['apple' => 1]);
});

#[Test]
public function savesTheOrderItPlaced(): void
{
    $catalogue = $this->createStub(Catalogue::class);
    $catalogue->method('priceOf')->willReturn(100);
    $orders = $this->createMock(Orders::class);
    $orders->expects($this->once())->method('save')->with($this->isInstanceOf(Order::class));
 
    (new Checkout($catalogue, $orders))->place(['apple' => 1]);
}

Five of the ten examples per shop use doubles, two expect an exception, the rest are plain value checks. The Pest version uses describe(), it() and expect() with the same PHPUnit doubles underneath.

The features are literally the same files for Behat and phpspec, a Background, two scenarios and a Scenario Outline with three rows per shop:

Feature: Checking out at shop 7
 
  Background:
    Given shop 7 sells "apple" at 100
    And shop 7 sells "bread" at 250
 
  Scenario: Pricing a basket
    When I check out 2 "apple" and 1 "bread"
    Then the order total should be 450
    And the order should be saved

phpspec implements the eight step definitions in one steps.php; Behat in one context class with #[Given], #[When] and #[Then] attributes. Behat ships no assertion library, so its steps throw a plain exception when a value is wrong, the way its documentation shows, rather than pulling PHPUnit in.

The HTTP suites hit a small JSON application served by php -S: list a shop's catalogue, price a basket, refuse an empty one, look up a product, get a 404 for one the shop does not sell. phpspec has get() and post() built in. PHPUnit and Pest get a forty-line client written the same way phpspec's is, file_get_contents over a stream context, so nobody pays for Guzzle.

How it was measured

Everything runs in one container: PHP 8.4.26 CLI with OPcache off, PCOV 1.0.12 and Xdebug 3.5.3 loaded only for the coverage cases. phpspec 9.0.0-rc.7, PHPUnit 13.4.6, Pest 5.3.5, Behat 4.0.0 with the Behat Parallel Extension 1.3.0, ParaTest 7.27.5. Eight cores.

Each case is run once to check it passes with the expected counts, then timed by hyperfine with two warm-up runs and ten measured runs, output to /dev/null, and the tool's default formatter. Peak memory is the median maximum resident set size over five runs under GNU time. Every tool sees the same files, the same autoloader settings (optimize-autoloader) and the same PHP.

One thing I got wrong the first time round, and which is worth knowing if you ever time anything in Docker Desktop: the project must live on the container's own filesystem. A directory shared from macOS is served through a file-sharing layer that makes every stat() and read() several times slower, and a test run is mostly files being read. On the share, the thousand phpspec examples took 615 ms and PHPUnit's 733 ms; on a copy inside the container they take 124 ms and 232 ms. The share did not just slow everything down, it flattened the differences, because it charged every tool the same price per file. The figures below are from the copy, which is the footing a Linux machine or CI gives.

Starting up

One file, five examples. This is what you wait for every time you run a single spec from the editor.

The last column is count(get_included_files()) at shutdown. PHPUnit loads 1509 PHP files to run five tests. phpspec loads 193, and that includes the spec, the nine shop classes and Symfony Console.

Specs against tests

A thousand examples, two hundred files, half of them with doubles.

Take the start-up out and the examples themselves cost phpspec a tenth of a millisecond each, PHPUnit 0.16 ms per test and Pest 0.38 ms. Pest pays its own start-up plus a layer on top of PHPUnit, and holds on to the most memory.

Features against features

The same hundred .feature files, five hundred scenarios, 2200 steps.

phpspec parses Gherkin with the official cucumber/gherkin library, as Cucumber itself does, and runs the pickles. Behat has its own parser. The gap is modest and mostly start-up again: 31 ms against 68 ms for a single feature.

Browser tests against HTTP tests

Five hundred requests to the shop application, with a response matcher per example.

The server is the same php -S for all three, and the client code does the same thing on both sides. phpspec spends 74 ms of CPU in those 102 ms and waits on the socket for the rest; PHPUnit spends 160 ms and Pest 248 ms around the same five hundred requests.

Code coverage

This is where the tools part ways. Coverage needs an engine, PCOV or Xdebug, and the engine is the same for everyone. What differs is what each tool does with the lines the engine reports.

PHPUnit and Pest hand them to php-code-coverage, which parses every source file with php-parser to decide which lines are executable, includes the files no test loaded so they appear at 0%, and then renders the report. phpspec counts the lines the engine reports for the files the examples loaded, and renders the report. That is less work, and it also means the two print different percentages for the same run: phpspec reports 100% of the 700 files the specs loaded, PHPUnit 68% of all 900. Neither is wrong; they count different denominators.

To keep the comparison like for like I ran PHPUnit and Pest twice: with their defaults, and with includeUncoveredFiles="false", which stops php-code-coverage parsing the 300 files no test touched.

Under PCOV, the thousand examples:

Under Xdebug:

Switching coverage on costs phpspec 100 ms under PCOV and 300 ms under Xdebug on top of the plain run. It costs PHPUnit 0.46 s and 2.4 s, and Pest 1.5 s and 5.3 s. Leaving the untouched files out of php-code-coverage's analysis turns out to change little, 17 ms for PHPUnit and 80 ms for Pest: the cost is in what it does with the files that were loaded, not in the ones that were not. A coverage run with phpspec is something you can leave on. The five hundred scenarios with coverage take phpspec 255 ms under PCOV and 541 ms under Xdebug; Behat has no coverage of its own, so there is nothing to put beside them.

Parallel

Eight workers. phpspec with --parallel=8, PHPUnit through ParaTest, Pest with --parallel, and Behat through the Behat Parallel Extension, which works with Behat 4, in each of its three modes. The CPU column counts every process, workers included; the memory column is the parent's only.

The thousand examples:

The five hundred scenarios:

Examples this small have nothing to gain from parallelism, and the tables say so honestly: phpspec's eight workers finish in the 121 ms its single process takes, having burnt five times the CPU starting up; ParaTest is slower than PHPUnit on its own; Pest gains 60 ms. The point of the tables is the shape of the cost. phpspec's workers start once and are handed the next file over a socket as they finish, so a parallel run is eight start-ups. The Behat extension starts a fresh behat process for every job: three hundred start-ups in its default mode, which is why 195 ms of serial work becomes 5.45 s of wall time and 41 s of CPU, a hundred in feature mode, and only when twenty scenarios share a job does it get close to its own serial time.

Why

Speed was not a goal of the rewrite and nothing in it was tuned. Between the proof of concept and rc.7 there is no performance work to speak of: a lazy Story BDD registry, PCOV support, the stop conditions handed to the workers. What I set out to do was make phpspec a smaller thing, with less between an example and the code it describes: closures instead of a class hierarchy, its own doubles, its own Gherkin runner, as few dependencies as it could carry. The speed is what a smaller thing does.

Specs are closures, not classes. A phpspec 8 spec was a class extending ObjectBehavior; every $this->method() went through __call, a wrapper, an unwrapper and a Prophecy-backed subject, and the runner found examples by reflection over the class. I made a spec a PHP file that calls describe() and it(): loading it with require_once is what registers the examples, and running an example is calling a closure. There is nothing to reflect over and nothing between the example and the code.

The doubles are built in. I wrote phpspec's doubles into phpspec. mock() generates one class per doubled type, with eval(), caches it, and instantiates it per example. Each method records the call and answers from the stubs declared for it. Prophecy, with its prophets, object prophecies and method prophecies per call, is gone, and with it doctrine/instantiator.

Ten runtime dependencies became three. phpspec 8 required Prophecy, php-diff, sebastian/exporter, symfony/console, event-dispatcher, process, finder, yaml and doctrine/instantiator. phpspec 9 requires symfony/console, symfony/yaml and cucumber/gherkin. Installed, that is 14 packages and 776 PHP files in vendor/, against PHPUnit's 40 and 2203, Pest's 58 and 3255, Behat's 27 and 1384. Fewer files on disk is fewer files to autoload, which is the start-up table.

No container to assemble. phpspec 8 built a service container of a few hundred services on every start, in an 835-line ContainerAssembler, then let extensions rewire it. I dropped it: phpspec 9 constructs the handful of objects a run needs and passes them along.

Coverage reads the engine. A PerExampleCollector starts and stops the engine around each example and scenario, and the report counts what it recorded, scoped to the source directories. I did not want a second pass over the source tree, so there is none.

Workers are handed one file at a time. The parallel runner starts the workers once, keeps a socket to each, and feeds it the next file as soon as it reports the last one, features first and largest first. The parent only ever lists files; each worker loads what it runs.

What this does and does not say

These are small examples on purpose, so that the framework is what is being measured. A real suite that talks to a database spends its time in the database, and the gaps in the plain runs shrink towards the start-up difference. The coverage gap does not shrink: it is paid per file and per line the engine reports, so it grows with the code. The HTTP numbers depend on the client as much as the framework, which is why the clients are the same. Behat's steps throw exceptions rather than call an assertion library, which is the cheapest thing a context can do. Pest's memory includes Collision and Termwind, which is where its output comes from. The numbers are from one machine, one container, one evening; the repository has the script, so the same tables can be produced on yours.

What it does say is that a rewrite that removed the machinery between an example and the code, and that reads coverage straight from the engine, is the fastest PHP test runner of the four on every one of these tables, by a little when the suite is running, and by a lot when it is starting up or measuring coverage.

composer require --dev phpspec/phpspec:^9.0@RC