In this issue
To follow up from our article
last month which looked at security flaws introduced by vendors
shipping Apache, this month we decided to take a look at what custom
patches vendors add. The Apache Software Foundation would rather that
vendors of Apache didn't add any third-party modifications to Apache
at all - it adds to brand confusion. It's rather like going to buy
Coke, complaining to the Coca-Cola company that the drink was too
sour, then finding that your supermarket was adding lemon.
There are hundreds of distributions and hundreds of vendors so in
order to make this manageable we started out by looking at just Linux
vendors that have publicised security updates for Apache in the last
few months to the bugtraq mailing list. Where a vendor has multiple
versions of products we tried to look at the most recent version of
Apache 1.3 (since most vendors do not yet ship Apache 2).
Our survey consisted of Connectiva, Debian, EnGarde, Gentoo,
Mandrake, OpenPKG, Red Hat, SCO, SuSE, and Trustix.
At the time of the survey, not all the Linux vendors were shipping
Apache 1.3.27. Several shipped older versions for which they had
backported security fixes. Mandrake, Debian, and Connectiva included
Apache 1.3.26 with backported patches for
SuSE included Apache 1.3.23 with
backported security fixes for only
CVE-2002-0392 and CAN-2002-0839.
SuSE also add a backported patch for mod_proxy (PR#8234)
All the vendors shipped with EAPI, the interface that links Apache to
mod_ssl, and most bundled some selection of extra modules.
All the vendors shipped a custom httpd.conf file or
made patches to the default file. Examining the configuration file
changes was outside the scope of this survey since these are things
that can be easily changed by the user.
All the vendors except OpenPKG and SuSE pointed the magic
mime types file at the system /etc/mime.types file, with
many adding additional types using AddType
directives in httpd.conf.
SysV init is a standard process used by Linux distributions to
control which software the init command launches or shuts off on a
given runlevel. These sometime get confused with the
apachectl command which provides similar functionality.
All the vendors except OpenPKG included custom init scripts or patches
with their Apache packages.
All the vendors provided patches to help build Apache on their
particular Linux distribution and to customise it to their
environment. Connectiva, Gentoo, and Mandrake added a
serverroot configuration option and then used that to
help build Apache. Most vendors patched apxs and changed
file and directory locations.
Debian, Gentoo, Mandrake, Red Hat, and SuSE added dbm patches to ensure
that the files created for dbm-based authentication from Perl tools like
dbmmanage are in a format that Apache can understand.
Additional security fixes
Connectiva, Debian, EnGarde, Gentoo, Mandrake, Red Hat, and SCO all
included a patch for CAN-2001-0131, a vulnerability in
htpasswd and htdigest that could allow local users to overwrite
arbitrary files via a symlink attack. This vulnerability is not yet
fixed in Apache, as it's tricky to get right cross-platform. The
vendors patching this themselves only have to worry about the Linux
architecture so can add a specific fix.
Server version strings
Altering the server version string can help users determine that
they are running a vendor-modified version of Apache. It can also
help the vendor track market share through surveys like those from
Netcraft. Four of the distributions had patches to make sure that
they added a customised string to the server version string. These
distributions were quite well behaved and did not add their customised
string if the ServerTokens directive is set to
'product only' or 'minimum'.
- Debian GNU/arch
- (Trustix Secure Linux/Linux)
Connectiva and SCO were a little more invasive, with Connectiva
adding (Connectiva/Linux) to the server version string no
matter what the ServerTokens directive was set
to. SCO did a similar thing, with their extra string giving the
version of an acceleration patch they add.
Finally, Mandrake changed the base product name altogether, renaming
from Apache to Apache-AdvancedExtranetServer.
In Apache 1.3, a compile-time constant defines the maximum possible
number of server processes, defaulting to 256. Only three vendors
changed this default: Debian set it to 512 processes via a build-time
define, EnGarde patch it to 1024, and SuSE set it to 2048 via a
Large File Support
Debian, Mandrake, SuSE, and SCO build Apache with Large File
support, so that on 32-bit systems Apache can use files larger than 2
gigabytes - this is particularly useful for log files. Enabling LFS
does slightly change the Apache 1.3 binary module ABI, which can cause
problems if using binary modules built against a different version of
After taking account of all the patches and modifications above,
we're left with only four vendors that add additional patches.
- A patch to change the ap_set_content_length API function to accept
a length of type off_t instead of long,
to improve the support for Large Files mentioned above.
- A patch to make the regexp library work with Large File Support on
32-bit systems. This is a modification the affects the ABI.
- A patch to fix a segmentation fault when
using a custom response in a module, (PR#8334)
- A patch to fix a problem when using server-parsed HTML with suexec where
an <--#exec tag with a cmd attribute contains more than one word.
(Debian bug 47951)
- A patch to allow SSL environment variables to be accessible when using
mod_ssl and suExec. (similar to PR#9163)
- A patch to cause Apache to not run if user or
group directives are found within a
VirtualHost but suExec is not configured correctly.
(Debian bug 21525)
Debian added the same patches as Gentoo and additionally:
- A fix for a htdigest buffer overflow if arguments passed to it are too long.
This is only a security issue if htdigest is used setuid
- Changes to ApacheBench to support round-robin DNS
- A patch to mod_proxy needed for
- A patch to add a new API function, ap_call_execute, needed
by the old mod-frontpage-VR module
- the "Accelerating Apache" performance patches from SGI.
The "Accelerating Apache" performance patches were first submitted
to the Apache Group by SGI in 1999. We
reported that they were designed to improve the performance of
Apache when measured specifically by the SPECweb96 benchmark. The
patches were named after the ten fold increase in speed they gave over
regular Apache on a dual processor SGI IRIX machine. Some of the
patches were folded in to Apache in 2000, but other parts were
rejected by the Apache developers. The Accelerating Apache project
was dropped by
SGI in February 2001.
What we found in our survey was that no two of the ten vendors were
alike; some vendors like OpenPKG made only the expected build and
configuration changes, whilst others made fairly substantial changes
including affecting the ABI. ABI changes mean that you can't reliably
take a module precompiled for one distribution and start using it on
Third party modifications to Apache have
been known to cause bugs and security issues. This is often
frustrating for the Apache Software Foundation who end up receiving
all the bug reports for issues that don't even exist in the official
Apache releases. This is one of the reasons why the Apache Software
Foundation insists that when vendors make modifications to Apache that
they change the name of their version so it is not confused with
official Apache releases.
One thing that impressed us was how easy it was to identify the changes
that the vendors had made. In almost all cases the vendor's source package
contained a pristine copy of Apache along with one or more patch files for
the various changes. Working out what those changes did and where they came
from was another issue though, vendors could do a much better job of
labelling the origin of, and reason for, each of the patches they make.
In this section we highlight some of the articles on the web
that are of interest to Apache users.
If you act as a Certificate Authority (CA), you will need to install
your own CA certificate into browsers who are accessing websites
with certificates signed by your CA in order for them to trust you.
Rob Flickenger shows you how to do this
on an Apache web server and also advises you to review your
pre-installed CA to ensure that you know who you trust by default in
"Distributing Your CA to Client Browsers".
article on how to block spambots
by using mod_rewrite if you are using an
Apache web server. It provides you with a sample of the rewrite
rules used and supplies you with a good further reading list to
help you fight your own personal battle with the bad robots.
"Creating an Apache Site with Public and Secure Access",
Peter Laurie runs through a httpd.conf file for Apache 2
with a brief explanation of how it works. Just a bit to jog the memory if
you are already an expert.