Long Exim break-in analysis

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Long Exim break-in analysis

Vladislav Kurz
Hello all,

first, I apologize for a long mail. Don't read if you don't like long e-mails.
But as Thorsten was already affected by exim exploit I thought this might be
interesting for all debian-exim users:

one of my friends asked me for help with his server, and I discovered that it
was rooted through unpatched exim. System is being reinstalled now, and I
decided to write something about this exploit. I hope you will find the info
interesting. It won't be anything exact, because the machine is offline now,
but anyway here it goes:

First sign was that mail did not get through. Server was overloaded and a
process named syslogd was using most of CPU. On the first sight "top" was
looking a bit different than usual. ps showed processes /sbin/syslogd and
syslogd (without path). First one was ok, the second one was doing something
nasty and using the CPU. /proc/PID/exe was symlink to perl...

After I killed (-9) this rogue syslog, exim spawned new one! So I killed them
both. There were some interesting files in /var/spool/exim4/ - two configs
that download binary named setuid into /var/spool/exim4/ and make it setuid
and try to run it. The other config did the sam ine /var/spool/exim/.
I think it was the same as shown on exim mailing list.

However /var/ was mounted nosuid so it failed (few days ago). But the bad guy
was able to get shell as debian-exim user, and compiled another binary. He
left us the source ;) - it was supposed to install his public key
into /root/.ssh/authorized_keys. I checked this file and found there a public
key but it was different then the one in /var/spool/exim/. Removed.

It seems that the first attack was uncuccessfull, but then some other attacker
found that /tmp was not on separate partition, and setuid worked there. He
left some evidence in /var/spool/exim/.bash_history - downloading and running
some rootkit. Further search for suspicious processes found sshd on port
above 55000. Killed immediately.

Then I started to get annoyed by ls, because it was spewing errors. It was
because I have alias l='ls --color=auto'. Pure ls was ok. So I started
looking for modified binaries, and found that some are owned by UID=122 which
was not present in /etc/passwd:

find /bin/ /sbin/ /usr/bin/ /usr/sbin/ -not -user root -ls

-rwxr-xr-x   1 122      114         54152 Dec  4  2005 /bin/netstat
-rwxr-xr-x   1 122      114         39696 Jan 30  2007 /bin/ls
-rwxr-xr-x   1 122      114         62920 Sep 13  2006 /bin/ps
-rwxr-xr-x   1 122      114        212747 Jan 30  2007 /sbin/ttyload
-rwxrwxr-x   1 122      114         93476 Jan 30  2007 /sbin/ttymon
-rwxr-xr-x   1 122      114         31504 Dec  4  2005 /sbin/ifconfig
-rwxr-xr-x   1 122      114         33992 Sep 13  2006 /usr/bin/top
-rwxr-xr-x   1 122      114         31452 Jan 30  2007 /usr/bin/md5sum
-rwxr-xr-x   1 122      114         12340 Aug  9  2006 /usr/bin/pstree
-rwxr-xr-x   1 122      114         59536 Jul 30  2007 /usr/bin/find

so now it explained why ls and top behaved differently than usual. Of course
we cannot trust these results because ls and find are modified as well...

Further idea was, they must have done something to start after reboot,
check /etc/inittab and there was something like this:

# standard tty stuff
0:2345:respawn:/sbin/ttyload

nice comment eh? Intersting is that mtime was probably preserved, but ctime
was recent (few hours).

ps did not show that ttyload is running, but killall killed something
anyway ;) because on first run it did not complain, but second time it said:
no process killed. Then I compared netstat (hacked) with nmap from outside,
and found that lots of ports are missing. Apache is running but not listening
according to netstat... so there might be further backdoors hidden.

Thats almost all. Machine is now offline, replaced by another one. I'll try to
get the hacked machine booted from live-cd, so I can examine it with
trustworthy tools, and if i find more interesting thing i'll post a follow
up.

Lessons learned:
1. subscribe to DSA and run apt-get
2. /var/spool, /var/tmp, /tmp and other places where unprivileged users can
write, should be mounted nosuid and even better noexec. It seems that this
could prevent the attack, or at least make it much more difficult.

As for point 2. it's a pity that dpkg is using /tmp and /var/lib/dpkg/ to run
scripts during installation and removal of packages. It would be nice if
whole /var could be mounted noexec.

That's all folks
--
Regards
        Vladislav Kurz


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/201012212307.37241.vladislav.kurz@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Martin Zobel-Helas
Hi,

On Tue Dec 21, 2010 at 23:07:37 +0100, Vladislav Kurz wrote:

>
> Lessons learned:
> 1. subscribe to DSA and run apt-get
> 2. /var/spool, /var/tmp, /tmp and other places where unprivileged users can
> write, should be mounted nosuid and even better noexec. It seems that this
> could prevent the attack, or at least make it much more difficult.
>
> As for point 2. it's a pity that dpkg is using /tmp and /var/lib/dpkg/ to run
> scripts during installation and removal of packages. It would be nice if
> whole /var could be mounted noexec.
>

# cat apt.conf.d/01remount
DPkg::Pre-Invoke {"if mount | awk '{print $3}' | grep -q '^/tmp$'; then /bin/mount -o remount,exec /tmp; fi";};
DPkg::Post-Invoke {"if mount | awk '{print $3}' | grep -q '^/tmp$'; then /bin/mount -o remount,noexec /tmp; fi";};


--
 Martin Zobel-Helas <[hidden email]>  | Debian System Administrator
 Debian & GNU/Linux Developer           |           Debian Listmaster
 Public key http://zobel.ftbfs.de/5d64f870.asc   -   KeyID: 5D64 F870
 GPG Fingerprint:  5DB3 1301 375A A50F 07E7  302F 493E FB8E 5D64 F870


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20101221221937.GS1822@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Bernhard R. Link-2
In reply to this post by Vladislav Kurz
* Vladislav Kurz <[hidden email]> [101221 23:09]:
> As for point 2. it's a pity that dpkg is using /tmp and /var/lib/dpkg/ to run
> scripts during installation and removal of packages. It would be nice if
> whole /var could be mounted noexec.

AFAIK dpkg does not run things in /tmp. The only thing running things in
/tmp on a normal system is debconf's dpkg-preconfigure, which you can
disable by editing /etc/apt/apt.conf.d/70debconf (which means that
you will get asked questions not at the beginning but while installing
stuff, but as servers usually do not have that many packages that is
easy to bear).

That said, having /tmp noexec,nosuid and /var nosuid will only make some
script-kiddies slower and the more people use it the less it helps.
As long as you have things like /dev/shm world-writeable and not
mounted nosuid there are trivial other ways for attackers. And history
show that there were often ways around noexec and nosuid and though many
of the known ones should be closed by now, there will always appear new
ones. So having those flags set might be some nice stumbling block for
script kiddies, but not much more. (Others include not installing
compilers or things like wget ftp or netcat, blocking outgoing and
incoming connections but a small whitelist in the firewall, installing
kernels without modules and [k]mem support, using some of the more
'obscure' architectures, ...) They do not increase your safety,
but sometimes one at least sees someone stumble....

        Bernhard R. Link


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20101222091850.GA25719@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Bastian Blank
On Wed, Dec 22, 2010 at 10:18:50AM +0100, Bernhard R. Link wrote:
> That said, having /tmp noexec,nosuid and /var nosuid will only make some
> script-kiddies slower and the more people use it the less it helps.

It is a start.

> As long as you have things like /dev/shm world-writeable and not
> mounted nosuid there are trivial other ways for attackers.

/dev/shm _is_ mounted nosuid by default.

>                                                            And history
> show that there were often ways around noexec and nosuid and though many
> of the known ones should be closed by now,

Around noexec: not much, at least for real binaries. Around nosuid:
please show me. Hijacking other suid-binaries is a different story and
calls for a reduction of privileges.

Bastian

--
No more blah, blah, blah!
                -- Kirk, "Miri", stardate 2713.6


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20101222103010.GA1256@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Izak Burger
In reply to this post by Vladislav Kurz
This is a me too email.

I found one overlooked machine that was compromised on 16th of December.

The usual process related things replaced:

free  pgrep  pmap  skill    snice   tload  uptime  w
kill  pkill  ps    slabtop  sysctl  top    vmstat  watch

All of these were chattr +ai, as if that was going to stop someone who
knows what's going on :-)

One process hidden, called dropbear. It was easy to find when
comparing the output of the hacked ps with the actual content of
/proc, and then checking the /proc/pid/exe symlink. Since kill was
also replaced, I quickly wrote a wrapper in C for the kill() system
call, and sent it a KILL signal.

The rest of the machine appears untouched, but I'll probably reinstall anyway.

Cheers,
Izak


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/AANLkTi=XOTx6CowzQzjhy-X1M+Mae6RJazeKQWGrvFy7@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Bastian Blank
On Wed, Dec 22, 2010 at 01:42:03PM +0200, Izak Burger wrote:
> The usual process related things replaced:
> free  pgrep  pmap  skill    snice   tload  uptime  w
> kill  pkill  ps    slabtop  sysctl  top    vmstat  watch

This looks like the rootkit I found somewhere in the internet:
| 137a3bbda16034d34307a9d686e6fdb45b3c8683  procps/free
| 5db25350dd15d3f1e63a4ff44fa85b72c21df72d  procps/kill
| eeab165a2cf06feb327fa996f35271c076e992bc  procps/pgrep
| eeab165a2cf06feb327fa996f35271c076e992bc  procps/pkill
| a6569d433351bba70ae55738b47267bf2514e27e  procps/pmap
| 074896d923ec652046c60cdcd254ff01c497bee9  procps/ps
| bbb33300c5d8f53a60fe472b6b879c9853b26c57  procps/pwdx
| 5db25350dd15d3f1e63a4ff44fa85b72c21df72d  procps/skill
| bd8e998354f28f5f7216688f3a4b6e4007170d63  procps/slabtop
| 5db25350dd15d3f1e63a4ff44fa85b72c21df72d  procps/snice
| bbf9b74494b4669c663c19cc53fd1fef9e585d2a  procps/sysctl
| c32f4ed4efa1305a2e9876b640e90fb9836a9f05  procps/tload
| 3c84c94470376612507d39fbe7a227465a516525  procps/top
| eb17b3b64913e7fa0d4b43a467a2548f96670a2e  procps/uptime
| 9815f97ed37553c7915e2e35dfaadab796aac864  procps/vmstat
| f7754627d890a393f0a917eaebbffdf458b6ce4d  procps/w
| c480eefa72eb62183fb6e26cd8d68c58fefc26e0  procps/watch

The initial checks shows 32-bit static binaries, built on RHEL 4, update
7 and 8.

But it also adds this to the startup scripts:
| /usr/sbin/iptables -I OUTPUT 1 -p tcp --dport 45295 -j DROP

The only reference I found is another remote shell somewhere in 2003.

> The rest of the machine appears untouched, but I'll probably reinstall anyway.

Something left behind in /var/spool/exim4?

Bastian

--
You're too beautiful to ignore.  Too much woman.
                -- Kirk to Yeoman Rand, "The Enemy Within", stardate unknown


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20101222120613.GA2794@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Izak Burger
On Wed, Dec 22, 2010 at 2:06 PM, Bastian Blank <[hidden email]> wrote:

> This looks like the rootkit I found somewhere in the internet:
> | 137a3bbda16034d34307a9d686e6fdb45b3c8683  procps/free
> | 5db25350dd15d3f1e63a4ff44fa85b72c21df72d  procps/kill
> | eeab165a2cf06feb327fa996f35271c076e992bc  procps/pgrep
> | eeab165a2cf06feb327fa996f35271c076e992bc  procps/pkill
> | a6569d433351bba70ae55738b47267bf2514e27e  procps/pmap
> | 074896d923ec652046c60cdcd254ff01c497bee9  procps/ps
> | bbb33300c5d8f53a60fe472b6b879c9853b26c57  procps/pwdx
> | 5db25350dd15d3f1e63a4ff44fa85b72c21df72d  procps/skill
> | bd8e998354f28f5f7216688f3a4b6e4007170d63  procps/slabtop
> | 5db25350dd15d3f1e63a4ff44fa85b72c21df72d  procps/snice
> | bbf9b74494b4669c663c19cc53fd1fef9e585d2a  procps/sysctl
> | c32f4ed4efa1305a2e9876b640e90fb9836a9f05  procps/tload
> | 3c84c94470376612507d39fbe7a227465a516525  procps/top
> | eb17b3b64913e7fa0d4b43a467a2548f96670a2e  procps/uptime
> | 9815f97ed37553c7915e2e35dfaadab796aac864  procps/vmstat
> | f7754627d890a393f0a917eaebbffdf458b6ce4d  procps/w
> | c480eefa72eb62183fb6e26cd8d68c58fefc26e0  procps/watch
>
> The initial checks shows 32-bit static binaries, built on RHEL 4, update
> 7 and 8.

Mine's a little different, but its an exact match on the file names at
least. The files are hige, at least 400k. Might be unstripped, haven't
checked yet.

> Something left behind in /var/spool/exim4?

Yup. I found the rkit.tgz tarball and the setuid.c code in
/var/spool/exim. The setuid.c program drops to root and adds a key to
root's authorized_keys.

The rootkit includes something called mig, which seems to be a tool to
clean wtmp with. Possibly to avoid detection of the login with the
newly added key.

The rootkit has an install.sh that is actually very well put together.
It checks for the usual rc.local scripts and adds itself to those, and
failing that, it adds /etc/init.d/xfs3 and installs it with
update-rc.d. Nice touch. This init script simply starts the dropbear
process and opens the firewall by inserting a rule at line 1.

But there is one bit that gets me. It does this:

mkdir -p /usr/include/mysql
echo dropbear >> /usr/include/mysql/mysql.hh1

It never does anything with that file, and that file does not exist on
a real system, so its almost like its leaving behind its business
card?

Now that the adrenaline rush is over, time to get back to other income
generating activities :-)

Cheers,
Izak


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/AANLkTinE=WG_6L4LVzHewJAgMZKjwwB5N595LU=TzYmk@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Izak Burger
http://www.reddit.com/r/netsec/comments/en650/details_of_the_root_kit_that_go

With the exception of replacing /etc/exim4/exim.conf, its pretty much
exactly what happened to me :-)


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/AANLkTikGvysh5i+wnNzoOYH7UQi9FRvF11uxdCT5z1Yi@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Maximilian Wilhelm
In reply to this post by Izak Burger
Anno domini 2010 Izak Burger scripsit:

Hi!

Nice reports :)

> But there is one bit that gets me. It does this:

> mkdir -p /usr/include/mysql
> echo dropbear >> /usr/include/mysql/mysql.hh1

> It never does anything with that file, and that file does not exist on
> a real system, so its almost like its leaving behind its business
> card?

Might that be a "all your passwords belong to us" file? I've had one
cracked ssh(d) once, which wrote all passwords from clent and server
connections to /usr/include/ssh.h IIRC. Maybe this on is something
similar?

Ciao
Max
--
Gib Dein Bestes. Dann übertriff Dich selbst!


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20101222124549.GA25060@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Karl Goetz-4
In reply to this post by Vladislav Kurz
On Tue, 21 Dec 2010 23:07:37 +0100
Vladislav Kurz <[hidden email]> wrote:

> Hello all,
>
> first, I apologize for a long mail. Don't read if you don't like long
> e-mails. But as Thorsten was already affected by exim exploit I
> thought this might be interesting for all debian-exim users:
>

Very interesting,
thanks for letting us know.
kk

--
Karl Goetz, (Kamping_Kaiser / VK5FOSS)
Debian contributor / gNewSense Maintainer
http://www.kgoetz.id.au
No, I won't join your social networking group

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Bernhard R. Link-2
In reply to this post by Bastian Blank
* Bastian Blank <[hidden email]> [101222 11:30]:
> On Wed, Dec 22, 2010 at 10:18:50AM +0100, Bernhard R. Link wrote:
> > That said, having /tmp noexec,nosuid and /var nosuid will only make some
> > script-kiddies slower and the more people use it the less it helps.
>
> It is a start.

I'd not call it a start. It is more little a pillow at the ground of the
pit. It's nice to have if someone falls but only helps once it is
already to late.

> > As long as you have things like /dev/shm world-writeable and not
> > mounted nosuid there are trivial other ways for attackers.
>
> /dev/shm _is_ mounted nosuid by default.

Indeed. Since lenny (and perhaps etchnhalf) it is nosuid by default.
Sorry, I sometimes lose track of the many little things I let my
installer patch after installing.

> >                                                            And history
> > show that there were often ways around noexec and nosuid and though many
> > of the known ones should be closed by now,
>
> Around noexec: not much, at least for real binaries.

In the past there was the ld.so trick. That is said to be closed now.
But I would make no bet that on a full desktop-system there is nothing
that cane still be used to execute something (perhaps some of those
start-programs-with-libraries already loaded tricks or things like
that).

> Around nosuid:
> please show me.

In the past there was perl-suid. I hope noone will do something stupid
as that again. But then I was already quite perplex something like that
existed.

        Bernhard R. Link


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20101223115444.GA9824@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Bastian Blank
On Thu, Dec 23, 2010 at 12:54:44PM +0100, Bernhard R. Link wrote:
> * Bastian Blank <[hidden email]> [101222 11:30]:
> > On Wed, Dec 22, 2010 at 10:18:50AM +0100, Bernhard R. Link wrote:
> > > That said, having /tmp noexec,nosuid and /var nosuid will only make some
> > > script-kiddies slower and the more people use it the less it helps.
> > It is a start.
> I'd not call it a start. It is more little a pillow at the ground of the
> pit. It's nice to have if someone falls but only helps once it is
> already to late.

I like to hack appliances. And they try to defend against people with
login rights. noexec on the correct locations makes it a lot more
dificult to go on.

> > >                                                            And history
> > > show that there were often ways around noexec and nosuid and though many
> > > of the known ones should be closed by now,
> > Around noexec: not much, at least for real binaries.
> In the past there was the ld.so trick. That is said to be closed now.

The kernel just does not allow any executable mappings of files on
noexec filesystems.

> But I would make no bet that on a full desktop-system there is nothing
> that cane still be used to execute something (perhaps some of those
> start-programs-with-libraries already loaded tricks or things like
> that).

For native code, they usualy rely on the possiblity to do executable
mappings.

> > Around nosuid:
> > please show me.
> In the past there was perl-suid. I hope noone will do something stupid
> as that again. But then I was already quite perplex something like that
> existed.

Lets try it. As the kernel explicitely disallows suid scripts for
security reasons, it needs to do more checks, not less.

Bastian

--
You can't evaluate a man by logic alone.
                -- McCoy, "I, Mudd", stardate 4513.3


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20101223142203.GA3010@...

Reply | Threaded
Open this post in threaded view
|

Re: Long Exim break-in analysis

Marc Haber-9
In reply to this post by Martin Zobel-Helas
On Tue, Dec 21, 2010 at 11:19:37PM +0100, Martin Zobel-Helas wrote:
> # cat apt.conf.d/01remount
> DPkg::Pre-Invoke {"if mount | awk '{print $3}' | grep -q '^/tmp$'; then /bin/mount -o remount,exec /tmp; fi";};
> DPkg::Post-Invoke {"if mount | awk '{print $3}' | grep -q '^/tmp$'; then /bin/mount -o remount,noexec /tmp; fi";};

Ugly workaround for a dpkg issue which still has a five-digit bug
number, which is TWELVE years old, and the only reaction to it was a
tag wontfix FOUR years after the issue was filed (#21941).

This is too sad to comment any further.

Greetings
Marc

--
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20101226152824.GL24207@...