A friend of mine contacted me recently after his site had crashed several times in the same 24-hour period. I reviewed the raw log files from the site, and found about 200+ instances of someone looking for files that didn’t exist. The requested file list read like this:
/wp-content/themes/modularity/functions/thumb.php /wp-content/themes/abstract/functions/timthumb.php /wp-content/themes/DailyNotes/functions/thumb.php /wp-content/themes/modularity/functions/scripts/thumb.php /wp-content/themes/synthetik/functions/timthumb.php /wp-content/themes/groovyphoto/functions/thumb.php /wp-content/themes/woostore/functions/thumb.php /wp-content/themes/wootube/functions/thumb.php /wp-content/themes/cushy/functions/thumb.php /wp-content/themes/sealight/functions/scripts/timthumb.php /wp-content/themes/purevision/library/functions/timthumb.php /wp-content/themes/retreat/functions/thumb.php /wp-content/themes/wedding/lib/functions/timthumb.php /wp-content/themes/prosto/functions/thumb.php
And on and on and so on for another several hundred lines! To most people, this would look like a confused search engine or bot, as all of these inquires were returning a 404 (file not found) error. But what I see here is actually a coordinated attack.
You see, the file this automated script is looking for is TimThumb, a php file that has known vulnerabilities in earlier versions. It is basically running through a list of every theme that used the file, trying to sniff out if my friend’s site is using a theme with the vulnerable version. And if it found that he was, it would instantly exploit the vulnerability and gain access to his server.
My friend isn’t worried about TimThumb though, because he already updated his vulnerable version ages ago. So this malicious script isn’t going to gain access to his server. What it is going to do, however, is slow his server to a crawl and eventually crash it. (Which it did today, multiple times.) This means even updated and secured sites should be concerned with their security, as security can be closely tied to performance and unplanned downtimes.
The attack came with other clues, however, and it’s important to recognize these clues early on so that you can defend against the attack and prevent damage or downtime to your server. The biggest clues were the graphs of the server’s activity. (These graphs happen to be available through a LiquidWeb hosting account — please ask your own hosting company what kind of tracking their provide.)
In the top graph, you can see what his server was doing — he’s basically getting a surge of activity (WAY higher than normal) and then the server crashes, over and over. In the bottom, you’ll see what a more normal pattern looks like, with highs and lows throughout the day and peaks at certain times of day. (The bottom graph is actually the load patterns of my own managed hosting server.) A third measure of activity is incoming/outgoing bandwidth chart, as seen here:
In this chart, you’ll see what I would consider to be normal activity — the initial spike of incoming activity (the blue line) when I first set up the server and put all the files on it, the regular small spikes of incoming activity when the admins are batch-loading posts, and the constant stream of outgoing activity (the green lines) which represent web activity. Where this chart goes very wrong is the huge spike of incoming activity and a smaller (but also huge) spike of outgoing activity which indicates when the site was being attacked.
The point of telling you all of this is for you to know that you don’t have to be a techie to know that something is going wrong. For my friend, he noticed that the server was crashing. Someone else might notice that they are seeing signs that the account has been hacked. No server company, web developer, or part-time employee is going to know the site as well as you do, so you are the best defense your site has when it comes to WordPress attacks.
So What’s The Fix?
The solution to his problem was to first install iThemes Security and then to ban the particular IP addresses responsible for this attack. This should stabilize his server performance and prevent such issues in the future. To create the ban list, we used the server logs of both the IPs sniffing around for TimThumb and the servers that were repeatedly hitting the same page from different locations like this:
To prevent problems like these, I now install and configure this plugin for all my managed hosting clients and every security audit that I do. It will continue to monitor and ban IPs for 404s, failed logins and brute force attacks. The plugin itself has many options that can get the average user in trouble though, so please be careful if you are installing it yourself!
In this case, each element was important to the restoration of the site: responsive support staff at the hosting company to provide these clues, a developer who could read and interpret them, and a site owner who was monitoring his site for signs of unusual behavior. Without one of these parts, the site may have suffered more damage or been hacked into.
Questions about WordPress Security?
Have you experienced these kinds of outages in an attack, or do you have questions about security in general? Please feel free to ask in the comments below!