I run several websites on my server – nothing heavy, just some various vhosts for {{Apache}}.<\/p>\n
Many (but not all) of them run {{WordPress}}.<\/p>\n
At some unknown point (and I haven’t kept the crap that was being used around), over 100,000 files were uploaded to the root directory of one of the websites (the only one, apparently, I did\u00a0not\u00a0<\/em>have cron’d to keep up-to-date with the latest-and-greatest version of WordPress) – most of these were random-named {{HTML}} or {{JavaScript}} files. Sometime late Thursday night \/ early Friday morning of last week, some number of those were triggered which launched a {{DDoS}} (distributed denial-of-service) attack against a hosting company in England.<\/p>\n After a relatively short period of time (on the order of a couple hours at most), this otherwise-low-traffic site generated 48MB in Apache httpd logs (normal for a given day is on the order of a few dozen to couple hundred kilobytes).<\/p>\n My hosting provider, with no warning, “locked” my server, and sent me an administrative message with the following cryptic email:<\/p>\n Your server with the above-mentioned IP address has carried out an attack on another server on the Internet.<\/p>\n This has placed a considerable strain on network resources and, as a result, a segment of our network has been adversely affected.<\/p>\n Your server has therefore been deactivated as a precautionary measure.<\/p>\n A corresponding log history is attached at the end of this email.<\/p><\/blockquote>\n Gee, thanks, hosting company – that was informative.<\/p>\n After several hours of back-and-forth with their support group, I was finally able to get a rescue boot environment enabled, a KVM session to that environment, and could start diagnosing the problem(s). First, of course, were the normal checks of dmesg, \/var\/log\/messages, and the like. there was running dig to find out who was being attacked (how I found the target IP belonged to a hosting provider in the UK). Nothing. I was also Googling similar error messages, and finally found a clue (though cannot recall where) that malicious {{JavaScript}} can cause messages like those provided to me to be trapped by external logging systems. <\/p>\n This led me to look in \/var\/log\/httpd instead of just \/var\/log. And there is where I found the unusual log file for my LUG’s website here in Kentucky – bglug-access_log was 48 megabytes. And bglug-error_log was 4.3 MB. As I mentioned above, a typical access_log for that site is closer to ~100 KB.<\/p>\n Opening the ginormous log file showed a host of HTTP 200 response codes for things that looked nothing like {{WordPress}} files (things like “qdlrdi-casio-parliament-90treaty.html”). There shouldn’t be HTTP 200 (OK) response codes for non-WordPress files, because it’s a WordPress-powered website.<\/p>\n Running a file listing to screen failed (in the rescue boot environment) – but doing an ls -l > files.out, and then a wc -l files.out showed over 105,000 files in the root directory of the BGLUG<\/a> website. <\/p>\n To get my server back up and online as quickly as possible, I edited the Apache vhosts.conf and disabled the Blue Grass Linux User Group site and contacted my hosting company as to what the root cause of the issue was, and what I had done to fix it (both needed for them to reenable my system).<\/p>\n After getting the server back online normally, I was able to clear-out all the junk that had been transparently uploaded into the LUG’s site.<\/p>\n One of the biggest annoyances of the whole process (after not having been given any warning from my hosting provider, but just a summary disconnect) was that permissions on the directory for the website were “correct” to have disallowed uploading random junk to the server: The user bglug had not been compromised (it hasn’t even logged-in in a few months) – and neither was the apache group (which, of course, cannot login, but still). <\/p>\n Apparently<\/em>, some part of the version of WordPress the site was running (or a plugin) was compromised, and allowed a malicious attacker to upload junk to the server, and spawn this DDoS on my server.<\/p>\n Moral of the story? Keep all your software up-to-date, and monitor your logs for suspicious activity – not sure monitoring would’ve done me good in this<\/em> case, but it’s a Good Practice\u2122 anyway.<\/p>\n","protected":false},"excerpt":{"rendered":" I run several websites on my server – nothing heavy, just some various vhosts for {{Apache}}. Many (but not all) of them run {{WordPress}}. At some unknown point (and I haven’t kept the crap that was being used around), over 100,000 files were uploaded to the root directory of one of the websites (the only …10:00:21.645887 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
\n10:00:21.646166 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
\n10:00:21.649166 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
\n10:00:21.649416 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
\n10:00:21.649421 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74.54988 > 85.233.160.139.8888: UDP, length 8192<\/code><\/p>\n
\ndrwxr-xr-x 6 bglug apache 5611520 Apr 11 13:24 bglug<\/code><\/p>\n
Continue reading keep your wordpress installs up-to-date<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[3,41,47],"tags":[227,387,424,450,668,830,854],"keyring_services":[],"class_list":["post-2554","post","type-post","status-publish","format-standard","hentry","category-bglug","category-technical","category-warning","tag-ddos","tag-hosting","tag-javascript","tag-linux","tag-security","tag-vulnerabilities","tag-wordpress"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/2554","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2554"}],"version-history":[{"count":0,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/2554\/revisions"}],"wp:attachment":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2554"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2554"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2554"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fkeyring_services&post=2554"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}