When hackers strike...

I am a very old fashioned webmaster who learned to craft simple websites with basic HTML and CSS. I have kept my own website afloat for almost 20 years and along the way also created a similar website for the slrn newsreader and a larger site for my father's family history efforts. Nothing commercial and all of it hobbyist efforts with a perverse pride easily demonstrated in the writing of clean HTML with a completely CSS generated layout.

When I heard that the family history site was suddenly inaccessible I was puzzled but not alarmed. I had left the site to my father's care for many years as the static pages were relatively easy to create and update. It was a solidly built site that attracted some interest in family history circles but not the sort of site I thought that would attract the attention of anyone but other family history buffs. In this I was wrong...

This page then is the story of how this a very ordinary webmaster faced up against a website hacker. I hope that it may be of some use to others who have faced a similar problem and perhaps, unlike myself, cleanly ejected the hacker and regained full control of their sites. And for those who lost the battle, or perhaps had a partial victory, you are not on your own!

The plot thickens...

So my father called around that day and told me that seemingly out of the blue Google had put the site on a blacklist and any attempt to access the site was met with a big, red warning screen. So my assistance was called for, and with a little detective work I eventually unravelled what had happened and constructed the following sequence of events.

The timeline...

The following sequence of events appears most likely:

  1. The logs show that about 6 months previously visitors to the site had gone from a healthy 500 or so per month down to zero. This apparent lack of visitors and lack of email generated from the site continued for the oncoming months. At this time I surmise that a hacker has managed to access the site and gain control of the CPanel.
  2. My father was from this time intermittently locked out of the site and when communicating these issues with the hosting company was met only with password resets. Unfortunately he never managed to get beyond a 'Tier 1' help desk employee, an employee who never looked any deeper than the narrowest requirements of the ticket.
  3. Using their CPanel access the hacker established several subdomains and downloaded a big zipped file to the server which was then decompressed to place a big set of phishing scripts into these subdomains. 'yahoo.zip' was the name of the zip file if that helps anyone? The hacker also at this time took control of the email system.
  4. Again using the CPanel access the hacker altered the DNS records to accomplish DNS hijacking of requests. I am not sure if this worked in tandem with some sort of random URI generator I found amongst the hacker's detritus. I confess I did not dig too deeply into the mire of scripts I discovered.

So this must have been in progress for some months before Google justifiably clamped down on the site. I hope that the phishing scripts and redirects did no damage although going through the email records for the domain I suspect that some damage was actually done.

Website weaknesses...

So how did this initially happen to a relatively sleepy family history site with an 85 year old man at the helm? Looking back I can see a few possibilities that I discuss below, I rank them in what I consider the order of culpability:

  1. I suspect that the server itself was compromised. This was a big, Singapore based hosting company that had gobbled up many smaller companies and subsequently fallen down on supporting both the server infrastructure and the customer base.
  2. There was an ancient Perl script in place that allowed for a site search. This was old when I put it in place and I believe it would have eventually became a significant vulnerability. Interestingly enough when I first logged into the site this script had been removed by the hacker.
  3. FTP used for file transfer. The Internet was a kinder place many years ago and all file transfers used plain old FTP. The world moved to a more secure transfer method but the site did not. As well I suspect that the password was either weak, irregularly changed, or more than likely both.

Overarching this is the absence of somebody apart from the actual author of the web files to oversee such things. And I will take the responsibility for that weakness in the site, I had the skills and interest for such work but life moved me in other and busier directions for some time.

Remedial work...

Faced with the mess of scripts and extra folders on the site I took up tools and started a manual cleanup. Initially I had great success but there was eventually a loss of website control again to the hacker. But first my enthusiastic first run at the hacker!

Success...

With a degree of overconfidence I launched into my plans of defeating this hacker.

  1. I contacted the hosting company for a password reset. With this done I generated a 16 character password, logged out and then logged back in. How hard could this be, I was wondering...
  2. I removed the two subdomains that the hacker had created and then painstakingly removed the mess of directories, files and scripts that had been installed. Perhaps some hackers are tidy but this one was not.
  3. Eventually removed the rogue DNS settings, this was something I would not have looked for without some prompting. But I am now well aware of Domain Name hijacking.
  4. Went through the email accounts and deleted non-essential email accounts as well as having a good look at some of the returns to the Postmaster account. It looked very bad for usernames and passwords unfortunately and I sincerely hope that this hacked site did not do too much damage.
  5. Added the IP addresses that looked really suspect to a block list set in .htaccess.
  6. Took a copy of the remaining files on the site and spent some time poring through these files looking for any damage, any scripts being called etc. All clean! It looks like the hacker preserved the HTML files and then kept them as a front for the subsequent redirects and phishing.

Feeling a little cocky I declared that the drama was over and that web development could proceed as normal. I was, of course, completely wrong...

Failure...

Did I inadvertently trigger a small war with this hacker, or was it simply a script or backdoor activated automatically when I cleaned out the site? The next week however was a true nightmare...

  1. The hacker, or the script, had the ability to take back control of the CPanel easily, no matter what password was set. Often this was done within 30 minutes of the new password being set.
  2. Any blocks to IP addresses that I instituted were removed during the period where I had no access to CPanel.
  3. The hacker now removed all subfolders in the website each time that I lost control of the website. With a new passsword I would reload the site and after a period of time the hacker would simply remove the files again.
  4. During one longer outage the hacker also reinstituted a subdomain, this time using a wildcard. I am not completely clear where this fits in with the hacking effort but I am sure it was a deliberately chosen strategy.

Throughout this difficult week the hosting Help Desk people were not helpful. I had a feeling that their answers were scripted and perhaps even coached along by an AI utility? And certainly at no time was there an acknowledgement that there could be an issue with the server itself. Frustrating!

A new beginning...

Eventually I realised that it was better to cut the Gordian Knot and make a fresh start, rather than go in circles with the 'Help' Desk. So it was that I removed the Domain Name and website files and recreated the whole project with my own host, a company that has hosted 'Andrew's Corner' for some decades, with absolutely no trouble. But after this brush with site takeovers I have substantially tightened up security:

  1. This host does not use CPanel and there is a degree of separation between the user accounts and the Control Panel. Difficult for a hacked web site to access DNS, mail settings, Domain Name ownership etc. I have set a 16 character password on Admin centre and also started using 2FA with a newly purchased Yubikey.
  2. The family history site now transfers files with SFTP from within the preferred Dreamweaver editor. There will be a mandatory password change every 90 days with a 16 character password. I transfer my own site with rsync over SSH but for this account I have chosen not to grant SSH access.
  3. Both sites are using a 'Let's Encrypt' SSL certificate which is offered free with all accounts with this hosting company.
  4. I have paid for a 'Malware Remover' which is a malware scanner with a few extra features, provided by the hosting company. This runs on each Domain for an annual cost and while I could probably cobble together a similar resource I am attracted to a 'set and forget' option.
  5. I am in the process of writing up a small script to routinely backup the family history site so in the event of any further hacking action I will have a guaranteed clean copy. This will be tagged into the script I already use to back up my own work to external HDD.

And so far my father's site has running without a hiccup! I have learnt a great deal while resurrecting the hacked site and I have hopefully now set it so securely that hackers will move on to easier targets. However in the back of my mind there is a cynical old nursing joke that I know from my Intensive Care days: 'What could go wrong??'...

Postscript...

I have now put together a very simply script that I run periodically to ensure I have a recent copy of the website:

#!/bin/sh

# A quick and dirty script to keep a local copy
# of my father's family history site:

set -e

wget --mirror \
	 --page-requisites \
	 --directory-prefix=/home/andrew/html \
	 https://www.strong-family.org

I prefer simple things but I will embelish this script over the next little while...

And in conclusion...

This page represents my experiences when I attempted to recover a website from a solid hacking effort. Please feel free to contact me with any errors of fact that you have found on this page, any errors of opinion will probably remain uncorrected. In the meantime I am still having a great time creating original content with HTML and CSS, what about you?