Tag Archives: Fail2Ban

Strengthen Your IDS/IPS With Known WordPress Plugins

I maintain a dashboard in my Splunk environment to monitor for potential hacking attempts against my web servers. One way that I do this is to monitor for sources that are generating repeated 404 errors looking for exploitable web pages on the sites. During this recent review, I became aware of a number of new WordPress plugins that hackers are attempting to exploit. The URLs are easy to spot. They all begin with /wp-content/plugins/ followed by the name of plugin. The plugins that I saw are not even installed on my server, so these are obviously hackers looking for a way in.

This got me thinking about how I can use this information to strengthen my security with my intrusion detection and prevention systems.

Controlled WordPress Environment

For my particular installation, I am the only administrator for all of the hosted WordPress sites. This means that I have complete control over which plugins are installed. Because of this, I can use the solution that follows.

Identify Installed Plugins

The first step in my solution is to identify the list of currently installed plugins across all of my sites. From the directory where all of my sites are stored, I was able to use the following Linux command to get a list of plugins:

find . -type d -name plugins -exec ls {} \; | sort -u | grep -v "\.php$"

This command does the following:

  • find – Looks for all directories with the name of “plugins” and lists the contents of each directory.
  • sort – The sort command takes all of the plugin directory names, sorts them and removes duplicates.
  • grep – The plugins directory in WordPress may contain some PHP files in the directory. We can ignore these. This grep statement removes anything found to end with a .php extension.

Configure a Fail2Ban Filter

In order to secure my environment from these malicious invaders, I decided to again use Fail2Ban. If someone attempts to access a plugin that isn’t in a list of known plugins, then I want to immediately block any further access from that IP address. Fail2Ban makes this process very each.

Assuming that my list of installed plugins was pluginA, pluginB and pluginC, my filter file would look like the following:

[Definition]
failregex = ^<HOST> -.* \[.*\] "(GET|POST) \/wp-content\/plugins\/
ignoreregex = ^<HOST> .* "(GET|POST) \/wp-content\/plugins\/pluginA\/
  ^<HOST> .* "(GET|POST) \/wp-content\/plugins\/pluginB\/
  ^<HOST> .* "(GET|POST) \/wp-content\/plugins\/pluginC\/

This filter starts by triggering on any attempt made against the wp-content/plugins URL, but then provides exclusions for each plugin that is allowed.

I store this in a filter file that I named wp-plugin.conf in /etc/fail2ban/filter.d. I then add the following to my jail.local in /etc/fail2ban:

[wp-plugin]
enabled  = true
port     = http,https
filter   = wp-plugin
logpath  = /var/log/web/*access.log
maxretry = 1
bantime  = 2592000

This creates a jail with the wp-plugin filter. It monitors all of my web access logs and blocks a sender after 1 malicious attempt. Once triggered, the user is blocked for 2592000 seconds (30 days).

With this in place, restart the Fail2Ban service and a new layer of protection will be added to your security defenses.

Timthumb.php Remote Execution Vulnerability in WordPress

This morning, while going through my e-mails, I saw that my IDS system was seeing a lot of attempts against a timthumb.php file on my web sites. This seemed a little suspicious, so I headed out to Google to see what was going on. I started searching on “timthumb.php” and very quickly, Google gave me a suggestion of “timthumb.php exploit”. Yep, my suspicions were warranted.

Vulnerability Explained

This apparently isn’t a new vulnerability. It was a zero-day attack identified back in 2014. However, the fact that hackers are still trying to exploit it probably means that there are still web sites out on the web that haven’t been patched for this vulnerability.

The PHP script is used in a number of different themes for WordPress. The hacker exploits a bug in the software that allows the code to source an “image” from a remote web site. The following video does an excellent job of showing you the exploit.

Am I At Risk?

First task for me was to see if this is a vulnerability that affects me. I use Linux for all of my web servers. I was able to use one of the following commands to see if this code was implemented in any of my sites…

updatedb
locate timthumb.php
cd /PATH_TO_WEB_SERVER_ROOT (fill in with the actual path to your web server)
find . -name timthumb.php

Luckily, I did not find this being used on any of our themes. If you find this file in your environment, you might want to checkout the post by Sucuri Blog with tips for protecting your environment.

https://blog.sucuri.net/2014/06/timthumb-webshot-code-execution-exploit-0-day.html

Block Them Anyway!

I am a strong advocate about the fact that even though an exploit doesn’t exist on your system, you should still take action to block attempts. Someone was obviously maliciously attempt to attack your site and this is a clear piece of evidence as to how they do it. This attempt failed, but you can be sure they will keep trying until they find a way.

Another reasons to do this is to protect against site administrators loading themes with this vulnerability without your knowledge in the future. By putting the protection in place now, you can protect against future vulnerabilities created in your environment.

With that in mind, I decided to again use my Fail2Ban system to block individuals attempting to access this URL. The filter configuration file was very simple.

[Definition]
failregex = ^<HOST> - .*\/timthumb\.php
ignoreregex =

I added this filter to my jail configuration and restarted Fail2Ban. Now if anyone else attempts to access this URL that we don’t use, they will be automatically blocked from accessing our sites any further!

Intelligence in 404 Errors

I recently found myself in a conversation about Splunk. During the conversation, I was asked about which types of logs I found easiest and and most useful to ingest into the Splunk environment. Without giving it much thought, I immediately responded that web access logs were very easy to ingest and there is a lot of data that can be seen if you know where to look. Well, of course I set myself up for the next question . . . “can you give me an example?”

Understanding Access Logs

First, for those that aren’t familiar with web access logs, let’s take a moment to look at one. Below is a sample log entry from an Apache web server log:

103.249.31.189 - - [15/May/2016:21:43:02 -0400] "GET /wpfoot.php HTTP/1.1" 404 14975 "http://www.googlebot.com/bot.html" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

At a first glance, this can look a little intimidating. But let’s break down the various parts:

  • 103.249.31.189 – This is the IP address of the computer that is making the request to your web site.
  • 15/May/2016:21:43:02 -0400 – This is the date, time and timezone of when the request was made.
  • GET – When communicating with a web site, there are a number of different actions that can be requested against that page. For most standard web traffic, those requests are for either a GET or a POST. A GET can be thought of as a request to get data from a site. When you are simply clicking around a web site, you are most likely using GET requests to retrieve that data.  A POST is used when you are trying to submit data to a site. When you are filling out a contact form or logging into a site, you are most likely sending a POST request with that data.
  • /wpfoot.php – This is the page on your site that the user was trying to access.
  • HTTP/1.1 – This is simply telling you what HTTP protocol was used by the client when requesting the page.
  • 404 – We now come to the status code for this particular request. In this example, we see that the server responded with a status code of 404. This will be important to our conversation because a status code of 404 means that the server could not find the page that was requested.
  • 14975 – This number gives you the size of the response in bytes that was sent back to the requestor.
  • http://www.googlebot.com/bot.html – Often, we see a web page here and this is the address of the referring web page. This tells us that someone tried to get to /wpfoot.php from the bot.html site.
  • Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) – Finally, we see information about the browser that was used to make the connection.

That’s A Lot Of Data!

In just that one request, we identified 9 pieces of information. Imagine trying to review a web site’s access log with hundreds, thousands, even millions of log entries! How will you ever find anything useful in this data?

This is where one of my favorite pieces of software comes into the picture – Splunk. Splunk does a phenomenal job of ingesting all types of log and data sources and giving you simple yet powerful tools to analyze that data. Start sending your web server logs to your Splunk server and let’s begin to analyze.

A Simple Query

Once Splunk has started reading your data, we can begin to develop some searches against that data. For this topic, I decided to talk about gaining intelligence based on 404 status codes. Our Splunk search is very simple:

sourcetype=access_combined status=404 | top limit=10 uri

Simple, right? This search says, “Grab all of my access_combined log data (Apache access logs) and look for any record with a status of 404. Then show me the top 10 most requested web pages that received that status code.”

When you run this search, you will see something that might look like the following:

Splunk 404 Search

You will see the top 10 web pages requested, the number of times it was requested, and what percent of the total count of requests that is made up by this count.

Now that you have this list . . . what intelligence can we gather from it? Let me give you two scenarios to consider.

Scenario #1 – Web Coding Issue

Most people are familiar with the concept of broken links. This occurs when a web site directs you to a page that doesn’t exist. Nothing is more frustrating than trying to find a resource on the web to answer a question that you have, just to be taken to that “Page Cannot Be Found” message. If your site has any broken links, they will quickly be seen with a search like this and you can begin to find and correct these.

For example, I recently came across a result that looked something like:

/www.domain.com/page.html

At first glance, this looks like a perfectly legitimate URL. But you have to remember that what you are seeing in these logs is the part of the web page that comes after the web site address. Therefore, this was actually a link for:

http://www.domain.com/www.domain.com/page.html

We quickly found the pages in our site that were coded incorrectly.

Scenario #2 – Hackers Knocking At Your Front Door

Our 2nd scenario can actually be the most important of the two scenarios. Analyzing your 404 errors can give you a huge amount of insight into the activity of hackers on the Internet.

There are a large number of sites on the Internet that advertise vulnerabilities found in software. (See reference below for CVE) The intent of these sites are to make you aware of the vulnerability and urge you to upgrade software to remedy the issue, or possibly provide work arounds until a software patch is released. This is a great tool for admins to use to monitor issues that are released about the software they administer. But, admins aren’t the only individuals using these sites. The hackers know about them too!

Often times, you will see patterns in your logs where hackers are testing your site to see what software you have installed. Maybe they are looking for certain pieces of software. Maybe they are looking for installed components in the software. Regardless, this is a trial and error effort. But the good news is, we can see this in our logs.

An Example

I recently came across this exact URL in one of my searches:

/magazine/js/mage/cookies.js

This struck me as odd because there is nothing in any of my web sites about a magazine. So my suspicion level was already pretty high. I grabbed this URL and pasted it into a Google search. It didn’t take me long to discover that this is a component of the open source Magento e-commerce system. I took this knowledge and looked to see if there were any recent vulnerabilities discovered in the software. Sure enough, there is a bulletin on the Magento site asking users to upgrade because of vulnerabilities recently found in their software.

Safe, Right?

Luckily for me, I wasn’t running the Magento software on my system, so I was safe from being hacked. Or am I?

Let me give you something to think about. You are home with your family and a stranger comes to your front door. They jiggle the door knob to see if it’s unlocked and they find that it isn’t. So they walk away. The next night, you notice this same person come to your house and try to open the front window. Again, it was locked, so they leave. The third night, you find them snooping around your back door. Lucky for you, that was locked too. How many times are you going to let this happen before you take action?

This example is no different. You have concrete evidence of someone “jiggling the door knob” and “opening your front window” on your web site. Obviously, this wasn’t going to work because you don’t use the software. But they tried anyway and left evidence of them doing something they shouldn’t be doing. What if other hackers attempt to do the same thing? You now know what to look for so that you can park that 100 lb. German Shepard at the window and door to keep them away.

This is valuable information that you should now use to protect your network. If a hacker was willing to find an exploit this way, then you can be sure they will try other ways as well. As soon as we have a good way of knowing that someone is up to no good, we should be blocking them immediately for any further access to our sites.

Fail2Ban

One good resource that I personally work with is the open source Fail2Ban project. This is an extremely simple and yet very powerful piece of software. One of the many things this software can do is look for patterns in a web log and then alter the firewall of the server in real time to block further attacks from the source IP address. I created a new filter rule:

[Definition]
failregex = ^<HOST> - .*\/magazine\/js\/mage\/cookies\.js
ignoreregex =

With this rule, I can now monitor for future attempts against this specific URL and block the offender from making any other attempts against our systems.

Conclusion

This is just one of the millions of ways that Splunk can bring valuable intelligence into your environment with very little effort. Once you start identifying sources for this data and building out the searches to aggregate that data, you will find that the data mining options are endless.

Resources

 

Simple Protection for WordPress From Malicious PHP Files

While recently auditing my web server logs for 404 errors, I came across a new pattern of hacker attempts. There were a number of attempts at opening PHP files with a specific name.  A quick search of the internet turned up an article, WordPress Security – Arbitrary File Upload in Gravity Forms by Rodrigo Escobar. The article does a fantastic job of explaining this particular exploit.

To summarize how the attack works, the hacker attempts to exploit a vulnerability identified in the Gravity Forms plugin in order to upload a malicious PHP file to the web site. The file is loaded into the wp-content/uploads path of WordPress. Upon successful upload, they can call the PHP file directly and the Apache server will execute the code. Luckily for me, I don’t have this plugin installed and the attempts on my site were all failed attempts.

The Pattern

As I researched this issue further, it dawned on me that a number of exploits against WordPress work in a similar fashion. The identify a means of exploiting a file upload mechanism within WordPress. These mechanisms are usually designed to place the uploaded file somewhere within the wp-content/uploads directory. The hacker then tries to call the file directly.

As far as I can tell, there should NEVER be a legitimate case where a PHP file needs to be uploaded to this directory and run directly from this directory. The directory was designed to be a place for storing images, videos and documents (Word, PowerPoint, PDFs, etc.). Therefore, if a PHP file does exist in this directory structure, a successful hacker attack is probably already underway.

Solution #1 – Fail2Ban

My first reaction was to setup a new rule in my Fail2Ban system. This software does a great job of monitoring system logs for recurring patterns and then taking automatic action to remedy the problem. In the case of my web applications, it directly manipulates the iptables (software firewall) to block the offender.

I wrote a new rule to identify the access of PHP files in the uploads directory. The regex expression was fairly straightforward:

^<HOST> -.*"(GET|POST).*(?i)\/wp-content\/uploads\/[^"+]*php.*".*"$

I configured the service to block a source IP after just one attempt of this URL pattern. I restarted the Fail2Ban service and did some testing. The filter worked as designed.

The problem with this solution is that if the hacker is successful the first time, they will still be able to perform the hack before the software blocks them for good. It’s unlikely that the hacker will be successful in just one shot, but I don’t want to take that risk.

Solution #2 – Deny Access in Apache Configuration

I decided that I wanted to configure Apache to deny access to any attempt at a PHP file in that directory. A quick search of the internet gave me a couple of good resources. I could make use of the FilesMatch directive in the Apache configuration file to identify a file pattern and block access to that file. I did some testing and I developed the following solution:

        <Directory "/path/to/wordpress/htdocs/wp-content/uploads">
                .............
                <FilesMatch "\.php$">
                        Order allow,deny
                        Deny from all
                </FilesMatch>
        </Directory>

This information goes directly within the VirtualHost configuration for the WordPress web site in the Apache configuration files. After making the configuration change and restarting Apache, I dropped a test PHP file into various directory levels in wp-content/uploads. I tried accessing the PHP file and Apache always gave me a Forbidden access message.

Solution #3 – All of the Above!

The second solution is obviously the better approach. Even if the hacker is successful at uploading a file through an exploited upload mechanism, Apache will make sure that they never actually run the uploaded script. However, security works best in layers. Even though solution #2 is best, keep solution #1 in place too. Even though the hacker can’t succeed in running their script, allow the Fail2Ban software to recognize the attempt and block that hacker from any further exploit attempts on your site. They are obviously up to no good, so keep them from attempting any further harm.