In this issue

To follow up from our article last month which looked at security flaws introduced by vendors shipping Apache, this month we decided to take a look at what custom patches vendors add. The Apache Software Foundation would rather that vendors of Apache didn't add any third-party modifications to Apache at all - it adds to brand confusion. It's rather like going to buy Coke, complaining to the Coca-Cola company that the drink was too sour, then finding that your supermarket was adding lemon.

There are hundreds of distributions and hundreds of vendors so in order to make this manageable we started out by looking at just Linux vendors that have publicised security updates for Apache in the last few months to the bugtraq mailing list. Where a vendor has multiple versions of products we tried to look at the most recent version of Apache 1.3 (since most vendors do not yet ship Apache 2).

Our survey consisted of Connectiva, Debian, EnGarde, Gentoo, Mandrake, OpenPKG, Red Hat, SCO, SuSE, and Trustix.

At the time of the survey, not all the Linux vendors were shipping Apache 1.3.27. Several shipped older versions for which they had backported security fixes. Mandrake, Debian, and Connectiva included Apache 1.3.26 with backported patches for CAN-2002-0839, CAN-2002-0840, and CAN-2002-0843. SuSE included Apache 1.3.23 with backported security fixes for only CVE-2002-0392 and CAN-2002-0839. SuSE also add a backported patch for mod_proxy (PR#8234)

All the vendors shipped with EAPI, the interface that links Apache to mod_ssl, and most bundled some selection of extra modules.

All the vendors shipped a custom httpd.conf file or made patches to the default file. Examining the configuration file changes was outside the scope of this survey since these are things that can be easily changed by the user.

All the vendors except OpenPKG and SuSE pointed the magic mime types file at the system /etc/mime.types file, with many adding additional types using AddType directives in httpd.conf.

SysV init is a standard process used by Linux distributions to control which software the init command launches or shuts off on a given runlevel. These sometime get confused with the apachectl command which provides similar functionality. All the vendors except OpenPKG included custom init scripts or patches with their Apache packages.

All the vendors provided patches to help build Apache on their particular Linux distribution and to customise it to their environment. Connectiva, Gentoo, and Mandrake added a serverroot configuration option and then used that to help build Apache. Most vendors patched apxs and changed file and directory locations.

Debian, Gentoo, Mandrake, Red Hat, and SuSE added dbm patches to ensure that the files created for dbm-based authentication from Perl tools like dbmmanage are in a format that Apache can understand.

Connectiva, Debian, EnGarde, Gentoo, Mandrake, Red Hat, and SCO all included a patch for CAN-2001-0131, a vulnerability in htpasswd and htdigest that could allow local users to overwrite arbitrary files via a symlink attack. This vulnerability is not yet fixed in Apache, as it's tricky to get right cross-platform. The vendors patching this themselves only have to worry about the Linux architecture so can add a specific fix.

Altering the server version string can help users determine that they are running a vendor-modified version of Apache. It can also help the vendor track market share through surveys like those from Netcraft. Four of the distributions had patches to make sure that they added a customised string to the server version string. These distributions were quite well behaved and did not add their customised string if the ServerTokens directive is set to 'product only' or 'minimum'.

• Debian GNU/arch
• (Gentoo/Linux)
• (Red-Hat/Linux)
• (Trustix Secure Linux/Linux)

Connectiva and SCO were a little more invasive, with Connectiva adding (Connectiva/Linux) to the server version string no matter what the ServerTokens directive was set to. SCO did a similar thing, with their extra string giving the version of an acceleration patch they add.

Finally, Mandrake changed the base product name altogether, renaming from Apache to Apache-AdvancedExtranetServer.

In Apache 1.3, a compile-time constant defines the maximum possible number of server processes, defaulting to 256. Only three vendors changed this default: Debian set it to 512 processes via a build-time define, EnGarde patch it to 1024, and SuSE set it to 2048 via a define.

Debian, Mandrake, SuSE, and SCO build Apache with Large File support, so that on 32-bit systems Apache can use files larger than 2 gigabytes - this is particularly useful for log files. Enabling LFS does slightly change the Apache 1.3 binary module ABI, which can cause problems if using binary modules built against a different version of Apache.

After taking account of all the patches and modifications above, we're left with only four vendors that add additional patches.

SuSE added:

• A patch to change the ap_set_content_length API function to accept a length of type off_t instead of long, to improve the support for Large Files mentioned above.

Gentoo added:

• A patch to make the regexp library work with Large File Support on 32-bit systems. This is a modification the affects the ABI.
• A patch to fix a segmentation fault when using a custom response in a module, (PR#8334)
• A patch to fix a problem when using server-parsed HTML with suexec where an <--#exec tag with a cmd attribute contains more than one word. (Debian bug 47951)
• A patch to allow SSL environment variables to be accessible when using mod_ssl and suExec. (similar to PR#9163)
• A patch to cause Apache to not run if user or group directives are found within a VirtualHost but suExec is not configured correctly. (Debian bug 21525)

Debian added the same patches as Gentoo and additionally:

• A fix for a htdigest buffer overflow if arguments passed to it are too long. This is only a security issue if htdigest is used setuid
• Changes to ApacheBench to support round-robin DNS

SCO added:

• A patch to mod_proxy needed for mod_backhand
• A patch to add a new API function, ap_call_execute, needed by the old mod-frontpage-VR module
• the "Accelerating Apache" performance patches from SGI.

The "Accelerating Apache" performance patches were first submitted to the Apache Group by SGI in 1999. We reported that they were designed to improve the performance of Apache when measured specifically by the SPECweb96 benchmark. The patches were named after the ten fold increase in speed they gave over regular Apache on a dual processor SGI IRIX machine. Some of the patches were folded in to Apache in 2000, but other parts were rejected by the Apache developers. The Accelerating Apache project was dropped by SGI in February 2001.

What we found in our survey was that no two of the ten vendors were alike; some vendors like OpenPKG made only the expected build and configuration changes, whilst others made fairly substantial changes including affecting the ABI. ABI changes mean that you can't reliably take a module precompiled for one distribution and start using it on another. Third party modifications to Apache have been known to cause bugs and security issues. This is often frustrating for the Apache Software Foundation who end up receiving all the bug reports for issues that don't even exist in the official Apache releases. This is one of the reasons why the Apache Software Foundation insists that when vendors make modifications to Apache that they change the name of their version so it is not confused with official Apache releases.

One thing that impressed us was how easy it was to identify the changes that the vendors had made. In almost all cases the vendor's source package contained a pristine copy of Apache along with one or more patch files for the various changes. Working out what those changes did and where they came from was another issue though, vendors could do a much better job of labelling the origin of, and reason for, each of the patches they make.

In this section we highlight some of the articles on the web that are of interest to Apache users.

If you act as a Certificate Authority (CA), you will need to install your own CA certificate into browsers who are accessing websites with certificates signed by your CA in order for them to trust you. Rob Flickenger shows you how to do this on an Apache web server and also advises you to review your pre-installed CA to ensure that you know who you trust by default in "Distributing Your CA to Client Browsers".

Here's another article on how to block spambots by using mod_rewrite if you are using an Apache web server. It provides you with a sample of the rewrite rules used and supplies you with a good further reading list to help you fight your own personal battle with the bad robots.

Meanwhile, in "Creating an Apache Site with Public and Secure Access", Peter Laurie runs through a httpd.conf file for Apache 2 with a brief explanation of how it works. Just a bit to jog the memory if you are already an expert.