Did I Just Get Owned?

A very popular question on the Ubuntu Forum is "I saw this output on a log. Is it bad?" This section will cover the basics of log auditing so that you can begin to understand which log output is concerning and which is probably harmless. We will also show you how to keep an eye on different aspects of your system in order to make sure you haven't gotten "owned". This guide will complement the basic security measures outlined in the Basic Security Wiki.

Logging is an incredibly powerful feature. It can give you intelligence on how an attack was carried out and the extent of the compromise. As such if your system is cracked, the first place a cracker will go after gaining root on the system will be to sanitize your log files. One of the quickest ways to determine if your system was compromised is if your key log files are tampered with. It's often glaringly obvious that a log file has been altered. Important signs to note in your log files when checking for tampering are the following:

If you see any of the above signs it's time to do some further digging. It would be safe to assume the machine may be compromised.

Now let's take a look at some of your key logs and auditing methods for determining a system compromise.

Log File Viewer

Ubuntu provides a convenient graphical log viewer for newer users. It can be found in Administration > Log File Viewer (On 10.04.3 LTS). Otherwise all the log files mentioned can be found in /var/log and viewed using terminal commands such as cat, more, less, head, and tail.

syslog

Syslog can be invaluable when trying to detect a compromised service or a possibly rooted system. It logs all events at the system level, including kernel behaviour and activity. Often, when a service is compromised it may cause the service to crash. This crash may or may not yield remote code execution. You may notice your system is behaving weirdly. If you are running several services on your machine, particularly services frequent to buffer overflow type attacks (applications that accept user input, either in the form of files or data), this file can be helpful in determining if a "crash" or segmentation fault has occurred. Now it's important to note that a seg fault does not mean your system was compromised. Sometimes an application will seg fault all on its own with no help. However, if you notice other interesting phenomena that are unexplained, it might be a good place to start.

Here is an example seg fault from the mysql service:

You can also notice naughty activity when a rootkit is hooking kernel space memory in syslog. Here is an example of the phalanx rootkit hooking the Linux kernel and the last syslog entry associated with it :

Syslog is also one of the most popularly tampered log files, so if you see any missing time, or any anomalies it is a good idea to start looking into the potential for a system compromise having occurred.

auth.log

This file contains logs on all user authentication. This can be very helpful in determining if someone has gained unauthorized access to your system either locally or remotely. This file is also useful to determine if a brute force attack has been successful against a remote administration service such as SSH. Due to the incriminating nature of this log file, this will be one of the first logs sanitized by a potential attacker. As such if this log appears to be tampered with, it is almost a dead give away that your system has been compromised.

An example log snippet from a brute force against SSH might look like this :

Also random guessing of usernames is an indication of a brute force attempt. Particularly if services and common usernames are being tried. The following are commonly brute forced usernames, if you see these names trying to authenticate against your system this is a good indicator that someone may be trying to brute force one of your services (usually SSH).

ufw.log

This is your Uncomplicated Firewall log. Blocked traffic will show up here. If you have configured strong firewall rules, you may notice UFW blocking traffic to an unauthorized port. Random arbitrary ports are often used by different malicious applications, they use oddball ports to try and obfuscate their existence as they make a remote connection back to their owner. If your firewall's outbound rules are configured properly and the creator of the malicious application wasn't very diligent you should be able to notice the traffic being blocked. Here is an example of UFW blocked traffic:

Here we see blocked outbound traffic. It is UDP originating at port 5353 bound for port 5353. It is broadcast traffic as it is bound for 224.0.0.251 which is a multicast address. It is important to pay attention to the destination and source ports here as there are quite a few ports which are frequented by malicious applications. Here are a few examples of the more common ones:

There are others but if you start seeing odd traffic on any unregistered port, there is a chance that it is malicious if you did not authorize it.

If you are using iptables instead of UFW, the same can be seen in either syslog or kern.log.

Watching Network Traffic

When being mindful of your system's security it is important to watch network traffic. Often times malicious applications or unauthorized access will lead to the opening of a connection that is equally malicious and unauthorized. A quick way to view your active connections, both inbound and outbound, is with the netstat command. Netstat can give you a near real-time look at the connections your system has.

For instance, the command watch sudo netstat -anp will provide a fairly verbose output for netstat that updates every 2 seconds.

Here is example output from that command:

Let's take a look at what that means for us. The first column tells us what the protocol is. The Recv-Q and Send-Q is how much data queued for the connection and isn't really that important. The local address is the listening address of the service, this is 127.0.0.1. The :631 indicates we are listening on port 631. The Foreign Address is the address we are connected to. 0.0.0.0:* means any host any port. We are in state Listen which means we do not have an active connection but are waiting for one. Our process ID and program name is 1122 and this is the common unix printing system daemon. Now here is an example of output that might be considered slightly more malicious in nature:

This output indicates a connection to a service running /bin/sh on 31337. This is a nearly direct indication that this machine has been compromised.

rkhunter & chkrootkit

rkhunter and chkrootkit are two applications that are designed to aid in the detection of a compromised system. They function by doing two things. First they check the integrity of commonly hooked system files. These files are often backdoored by an attacker in order to gain special access or glean credentials from a compromised system. An example of a frequently backdoored command in the Linux world is /bin/su.

It is important to understand that rkhunter and chkrootkit function best if they are given a benchmark standard. Meaning that you run them following your initial installation so that they may get a "base line" for what your system should look like. That way if changes are made, they will be able to detect the changes as potentially unauthorized. Note, sometimes updates may throw false positives due to the way these applications work.

The second thing that these applications do is they attempt to determine if your system has been compromised with a known, or signatured, root kit. They will look for files and process associated with known root kits, as well as known malicious communications ports associated with them. Additionally they will look for hidden pid's and hidden tcp ports, as this is often a sign of root kit like activity.

Bash Profiles

Yet another method by which an attacker can maintain access to a compromised machine is by loading a backdoor at login. Often times these can be noted in the following places:

There are other locations but these are the three most targeted. An example entry that might be found in a modified .profile might include:

this would start a netcat listener that executes a shell when an attacker tried to connect. The persistence is there because it is in your .profile, and will be loaded every time your shell is loaded.

Cron Jobs

Another common way an attacker will maintain access to a compromised system is to add a cron job with a backdoor. This backdoor might be a simple netcat listener or it could be something else. In any case, if you suspect that you have been compromised, checking cron is a good idea.

The following output from sudo crontab -l indicates that something not so kosher is happening every 5 hours.

Additional places to check for cron jobs being placed are as follows:

Tampered Environment & LD_PRELOAD

Another method a system may be backdoored is by adding the LD_PRELOAD= attribute to the environment. This will load a library often used to hook system calls and escalate privileges on a system. You can view your current environment by utilizing the printenv command. (Note that each user has their own environment.)

Additionally, if you suspect this you may wish to check /etc/ld.so.conf.d for any malicious links.

Hooked Run Level Scripts

An attacker will also commonly hook the rc.local on a root compromised machine. They will do this to maintain their presence on the machine.

If we look at the contents of our /etc/rc.local file we may find there is something not quite right on our compromised machine. An example entry in this file might be something along these lines:

This indicates that whatever libowned-4.0.0.so is, it is being executed at boot time generally by default. This file will not contain anything but a single exit 0 line. So if you see some strange additions here, particularly something as obvious as this line, it is safe to say your machine may be compromised.

Additional areas to check for potentially malicious scripts are the following directories:

Additional Users

Often times when a system is compromised an attacker will create an additional user to allow them return access in the event you change your password. If you suspect a compromise has occurred you can and should check for the creation of additional users.

You may notice something similar to the following in your auth.log file if a user has been created.

In your /etc/passwd file you may also notice an entry similar to this:



BasicSecurity/DidIJustGetOwned (last edited 2020-07-10 15:55:25 by ravexina)