Small history about QA

Despite I'm mainly a developer, I now use most of my time on doing QA on PHP projects.

Here is, around release of versions7.2.17RC1 and 7.3.4RC1 a report which should help to understand this activity.

1. Presentation

Usually, tests are done by PHP developers, particularly thanks to travis and then by users who will install the RC version available 2 weeks before a GA version.

The PHP project follow a release process (cf README.RELEASE_PROCESS) which gives 2 days between the preparation of a version, the Tuesday on git, and the Thursday its announcement in the mailing lists. These 2 days are especially designed to allow the build of binary packages (mostly by Microsoft and often by me for my repository) and to allow a last QA check which mays allow to discover some late issue.

When the new versions were available (on Tuesday afternoon) I start building the packages for my repostiory, givinf more coverage than the current travis configuration:

• Fedora 27 to 31
• RHEL 6, 7 and 8-Beta
• i386 and x86_64
• NTS and ZTS
• various compiler versions  (GCC 4 to 9) and system library versions

I also run the build of the 7.3.4RC1 package in Fedora rawhide to trigger the re-build of all the PHP stack in Koschei, one of the CI tools of the Fedora project.

Notice : time to build all the packages for all the targets is about 3h for each version !  (I really need a faster builder).

2. Discoverd issues

2.1. Failed tests with pcre2 version 10.33RC1

Already available in rawhide, this version introduce a change in some error message, making 2 tests to fail.

Minor issue, fixed in PHP 7.3+: commit c421d9a.

2.2. Failed tests on 32-bit

In fix of bug #76117 the output of var_export have changed, make 2 tests to fail on 32-bit.

After confirmation by the autor of the change, tests have been fixed in PHP 7.2+ : commits a467a89 and 5c8d69b.

2.3. Regression

Koschei allow to discover very quickly a important regression in the run of the "make test" command. After digging, this regression was introduced in the fix of bug #77609, read the comments on the commit 3ead672.

After discussion between the Release managers, it have been choosen to:

• revert this change to get back to a sane situation
• to re-run the release process (new tag onr git)

The version which wil be announced shortly will not be affected byt this regression.

3. Conclusion

To ensure of the quality of PHP, of no regression is a complex, long and serious work. Thanks to all the actors, developers, QA team and users, this works pretty well.

So, if you use PHP in a development environment, it is essential to install the RC versions to detect and report us quickly any problem, so we can react before the finale version.

For users of my repository, the RC versions of PHP and various extensions are nearly always available in the testing repositories.