First published: 3rd March 2003
We decided to take a look at what custom patches vendors add to the versions
of Apache 1.3 they ship. The Apache Software Foundation would rather that
vendors of Apache didn't add any third-party modifications to Apache
at all - it adds to brand confusion. You might think you are getting a
copy of the Apache web server but you're actually getting something
that is based on the Apache web server.
There are hundreds of distributions and hundreds of vendors so in
order to make this manageable we started out by looking at just Linux
vendors that have publicised security updates for Apache in the first
few months of 2003 to the bugtraq mailing list. Where a vendor has multiple
versions of products we tried to look at the most recent version of
Apache 1.3 (since most vendors do not yet ship Apache 2).
Our survey consisted of Conectiva, Debian, EnGarde, Gentoo,
Mandrake, OpenPKG, Red Hat, SCO, SuSE, and Trustix.
Versions
At the time of the survey, not all the Linux vendors were shipping
Apache 1.3.27. Several shipped older versions for which they had
backported security fixes. Mandrake, Debian, and Conectiva included
Apache 1.3.26 with backported patches for
CAN-2002-0839,
CAN-2002-0840,
and CAN-2002-0843.
SuSE included Apache 1.3.23 with
backported security fixes for only
CAN-2002-0392 and CAN-2002-0839.
SuSE also add a backported patch for mod_proxy (PR#8234)
All the vendors shipped with EAPI, the interface that links Apache to
mod_ssl, and most bundled some selection of extra modules.
Configuration files
All the vendors shipped a custom httpd.conf file or
made patches to the default file. Examining the configuration file
changes was outside the scope of this survey since these are things
that can be easily changed by the user.
All the vendors except OpenPKG and SuSE pointed the magic
mime types file at the system /etc/mime.types file, with
many adding additional types using AddType
directives in httpd.conf.
SysV init is a standard process used by Linux distributions to
control which software the init command launches or shuts off on a
given runlevel. These sometime get confused with the
apachectl command which provides similar functionality.
All the vendors except OpenPKG included custom init scripts or patches
with their Apache packages.
Build Patches
All the vendors provided patches to help build Apache on their
particular Linux distribution and to customise it to their
environment. Conectiva, Gentoo, and Mandrake added a
serverroot configuration option and then used that to
help build Apache. Most vendors patched apxs and changed
file and directory locations.
Debian, Gentoo, Mandrake, Red Hat, and SuSE added dbm patches to ensure
that the files created for dbm-based authentication from Perl tools like
dbmmanage are in a format that Apache can understand.
Additional security fixes
Conectiva, Debian, EnGarde, Gentoo, Mandrake, Red Hat, and SCO all
included a patch for CAN-2001-0131, a vulnerability in
htpasswd and htdigest that could allow local users to overwrite
arbitrary files via a symlink attack. This vulnerability is not yet
fixed in Apache, as it's tricky to get right cross-platform. The
vendors patching this themselves only have to worry about the Linux
architecture so can add a specific fix.
Server version strings
Altering the server version string can help users determine that
they are running a vendor-modified version of Apache. It can also
help the vendor track market share through surveys like those from
Netcraft. Four of the distributions had patches to make sure that
they added a customised string to the server version string. These
distributions were quite well behaved and did not add their customised
string if the ServerTokens directive is set to
'product only' or 'minimum'.
- Debian GNU/arch
- (Gentoo/Linux)
- (Red-Hat/Linux)
- (Trustix Secure Linux/Linux)
Conectiva and SCO were a little more invasive, with Conectiva
adding (Conectiva/Linux) to the server version string no
matter what the ServerTokens directive was set
to. SCO did a similar thing, with their extra string giving the
version of an acceleration patch they add.
Finally, Mandrake changed the base product name altogether, renaming
from Apache to Apache-AdvancedExtranetServer.
Hard Limits
In Apache 1.3, a compile-time constant defines the maximum possible
number of server processes, defaulting to 256. Only three vendors
changed this default: Debian set it to 512 processes via a build-time
define, EnGarde patch it to 1024, and SuSE set it to 2048 via a
define.
Large File Support
Debian, Mandrake, SuSE, and SCO build Apache with Large File
support, so that on 32-bit systems Apache can use files larger than 2
gigabytes - this is particularly useful for log files. Enabling LFS
does slightly change the Apache 1.3 binary module ABI, which can cause
problems if using binary modules built against a different version of
Apache.
The remainder
After taking account of all the patches and modifications above,
we're left with only four vendors that add additional patches.
SuSE added:
- A patch to change the ap_set_content_length API function to accept
a length of type off_t instead of long,
to improve the support for Large Files mentioned above.
Gentoo added:
- A patch to make the regexp library work with Large File Support on
32-bit systems. This is a modification the affects the ABI.
- A patch to fix a segmentation fault when
using a custom response in a module, (PR#8334)
- A patch to fix a problem when using server-parsed HTML with suexec where
an <--#exec tag with a cmd attribute contains more than one word.
(Debian bug 47951)
- A patch to allow SSL environment variables to be accessible when using
mod_ssl and suExec. (similar to PR#9163)
- A patch to cause Apache to not run if user or
group directives are found within a
VirtualHost but suExec is not configured correctly.
(Debian bug 21525)
Debian added the same patches as Gentoo and additionally:
- A fix for a htdigest buffer overflow if arguments passed to it are too long.
This is only a security issue if htdigest is used setuid
- Changes to ApacheBench to support round-robin DNS
SCO added:
- A patch to mod_proxy needed for
mod_backhand
- A patch to add a new API function, ap_call_execute, needed
by the old mod-frontpage-VR module
- the "Accelerating Apache" performance patches from SGI.
The "Accelerating Apache" performance patches were first submitted
to the Apache Group by SGI in 1999. We
reported that they were designed to improve the performance of
Apache when measured specifically by the SPECweb96 benchmark. The
patches were named after the ten fold increase in speed they gave over
regular Apache on a dual processor SGI IRIX machine. Some of the
patches were folded in to Apache in 2000, but other parts were
rejected by the Apache developers. The Accelerating Apache project
was dropped by
SGI in February 2001.
Security Consequences
In March 2003 a vulnerability was found in the Oracle modifications to mod_dav.
This was not the first security hole that has been introduced by
third party modifications to Apache by vendors. However our own
research based on
issues listed in the CVE dictionary shows that the majority of these
vulnerabilities are due to poor configuration defaults rather than patches for new functionality that
went wrong:
CVE |
Type of Issue |
Severity |
Affected |
CAN-2002-0842 |
Remote attacker can run arbitrary commands |
High |
Oracle |
CAN-2002-0842 |
Remote attacker can run arbitrary commands |
High |
SCO (briefly) |
CAN-2000-1168 |
Remote attacker can run arbitrary commands |
High |
IBM |
CVE-2000-1016 |
Remote attacker can see files in /usr/doc |
Low |
SuSE Linux |
CVE-2000-0883 |
Remote attacker can see files in /perl |
Medium |
Mandrake Linux |
CVE-2000-0869 |
Remote attacker can read and write any file in docroot |
High |
SuSE Linux |
CVE-2000-0868 |
Remote attacker can obtain the source to CGI scripts |
Medium |
SuSE Linux |
CVE-2000-0234 |
Remote attacker can read .htaccess files |
Medium |
Cobalt |
CVE-1999-0678 |
Remote attacker can see files in /usr/doc |
Low |
Debian Linux |
Conclusion
What we found in our survey was that no two of the ten vendors were
alike; some vendors like OpenPKG made only the expected build and
configuration changes, whilst others made fairly substantial changes
including affecting the ABI. ABI changes mean that you can't reliably
take a module precompiled for one distribution and start using it on
another.
Third party modifications to Apache have
been known to cause bugs and security issues. This is often
frustrating for the Apache Software Foundation who end up receiving
all the bug reports for issues that don't even exist in the official
Apache releases. This is one of the reasons why the Apache Software
Foundation insists that when vendors make modifications to Apache that
they change the name of their version so it is not confused with
official Apache releases.
One thing that impressed us was how easy it was to identify the changes
that the vendors had made. In almost all cases the vendor's source package
contained a pristine copy of Apache along with one or more patch files for
the various changes. Working out what those changes did and where they came
from was another issue though, vendors could do a much better job of
labelling the origin of, and reason for, each of the patches they make.
|