At Octopuce, we are Debian professionals and are using Debian everywhere we can. For one of our customers, we maintained a mirror of Debian repositories, which are used by internal Linux servers.
This mirror is using apt-mirror since we cannot access the Internet by rsync or ftp protocol : we have to use http to mirror the repositories.
A few days ago, I saw the following error message on a Linux Debian server using this internal mirror :
W: GPG error: http://fr-debianmirror lenny/updates Release: The following signatures were invalid: BADSIG 9AA38DCD55BE302B Debian Archive Automatic Signing Key (5.0/lenny) ftpmaster@debian.org; W: You may want to run apt-get update to correct these problems
This issue was critical : the GPG signature of Debian security repository was incorrect! There were 3 places where this could happen:
- At the Debian repository security.debian.org : I don’t think so, or Google would have told me ;)
- During the mirroring process : maybe …
- At the final server location : each Debian machine has a GPG keyring of allowed keys for repository signature. The apt-key tool is used to manage this keyring, located at /etc/apt/trusted.gpg and /etc/apt/trustdb.gpg
Check apt-key configuration
First, check if the command apt-key list tell you that the faulty key is allowed. If it is not, you may add it by installing the proper Debian package, for example :
aptitude install debian-archive-keyring
Some people in the Internet tell you to use
gpg --keyserver keyserver.fr --recv-key 0x9AA38DCD55BE302B | apt-key add -
But using the debian package should work « the right way » :)
Check the mirror
Of course, our Debian servers were properly installed, so we already had the ftpmaster Debian archive GPG key in our servers.
Next step: I checked the faulty file : it was located at /var/spool/apt-mirror/mirror/security.debian.org/debian-security/dists/lenny/updates/Release.gpg
This file should be a GPG signature of the Release file, located in the same directory.
There, I saw this :
.../debian-security/dists/lenny/updates/$ ls -l -rw-r--r-- 1 apt-mirror 835 2010-10-20 10:13 Release.gpg -rw-r--r-- 1 apt-mirror 40K 2010-11-22 21:12 Release
And here it looks obvious to me we have a problem: a signature cannot be older than the file it is signing !
In fact, our apt-mirror (which is using wget to download files) is using a proxy server to connect to the official Debian mirrors. As such, I put the following in /etc/wgetrc :
http_proxy = http://10.42.12.12:8080/
And here is the problem: this proxy is unable to detect when the release.gpg file has been modified, and as such, it often returns an old version of this file !
To check this theory, I used wget with -S to see the HTTP headers returned by the server and the proxy :
fr-debianmirror:/tmp# wget http://security.debian.org/debian-security/dists/lenny/updates/Release.gpg -S --2010-11-26 12:08:43-- http://security.debian.org/debian-security/dists/lenny/updates/Release.gpg Connecting to 10.42.12.12:8080... connected. Proxy request sent, awaiting response... HTTP/1.1 200 OK Date: Fri, 26 Nov 2010 10:57:06 GMT Server: Apache Last-Modified: Mon, 20 Oct 2010 10:13:36 GMT ETag: "343-e08fc449a1532" Accept-Ranges: bytes Content-Type: text/plain Content-Length: 835 Connection: close Age: 855 Length: 835 [text/plain]
the lines in bold are the guilty one : the last modified date is not the right one, and the Age header tells us how old in the proxy cache is this entry.
Our solution
So, the solution was to tell wget to ask the proxy for a fresh version by using the –no-cache directive. A better way to use this is to add this line to /etc/wgetrc :
cache = off
And here we are :
wget http://security.debian.org/debian-security/dists/lenny/updates/Release.gpg -S --2010-11-26 12:17:39-- http://security.debian.org/debian-security/dists/lenny/updates/Release.gpg Connecting to 10.42.12.12:8080... connected. Proxy request sent, awaiting response... HTTP/1.1 200 OK Date: Fri, 26 Nov 2010 11:05:58 GMT Server: Apache Last-Modified: Mon, 22 Nov 2010 20:12:36 GMT ETag: "343-495a9e08a1500" Accept-Ranges: bytes Content-Type: text/plain Content-Length: 835 Connection: close Age: 0 Length: 835 [text/plain]