This morning, February 19, 2013 at about 3 AM an email arrived which set off alarms. Monitoring software on one of our servers had discovered a suspicious file: /lib64/libkeyutils.so.1.9.
Investigation revealed that this file is part of a server compromise. How the attacker is able to put this file in place is as yet unknown. What we do know at this point:
- RedHat Enterprise servers including CentOS and Scientific Linux are affected.
- Logins via SSH are recorded, including login name and passwords.
- Other logins, such as to email and cPanel are not affected
- Only 1 IP address has yet been recorded as the recipient of information
- More than 10,000 servers have been affected so far
- The goal of the attacker has so far been limited to sending spam email
To mitigate the threat we have set up scanning to find and remove suspicious files at 5 minute intervals and send alert emails when any suspicious file is found. This will trigger further investigation.
Since the source of the infection is unknown, the only prudent course is to assume the worst. We have set up firewall rules to prevent communication with the single IP address known to be receiving information. However, it would be naive to assume that this walls off the problem.
If you notice that your service is running more slowly than usual, the likely cause is actions we are taking to deal with this threat. It is possible that your service will be interrupted. Some counter measures are disruptive. For example, when server load becomes very high it can appear that a server is down because response is so slow. The fastest way to regain control in this case is a reboot.
If you call or email and do not get an immediate response, the reason is apt to be that we are working on a problem. We sometimes need to choose between solving a problem and explaining to 20 or 30 people that we are working on a problem. Frankly, it makes more sense to fix now and explain later.
If you observe problems with your service while this threat remains active, please be patient. We are all over it.