Who ya gonna call?
Do you have an approved incident response procedure in place? Getting your IRP set up before something happens could save you a lot of headaches -- not to mention your job
You're sitting at your desk, browsing through the system and notice something isn't right. Further investigation shows that there may be an intruder in your system. This wakes you up faster than a can of Jolt Cola, and you're all primed to take action. The question is: what kind of action? A system administrator's perception of an appropriate incident response may be very different from corporate management's. An approved incident response procedure can save you from major troubles -- and embarrassment. Here's how to do it right. (3,600 words)
et's consider how you would react to the following scenarios:
In each of these cases, your first response should be to inform management. Unfortunately, this is usually the last thing a system administrator actually does. The administrator will often make an assumption about the appropriate response. The problem is that there are political and legal issues to consider that the administrator is not qualified to address. In the precommercial days of the Internet, system administrators in research organizations freely exchanged information without informing management. Most, if not all, of the decision making was left up to the administrator. Today, corporations have strict policies regarding the release of proprietary information. In a commercial environment, even releasing information to CERT could get an administrator in a lot of trouble with management.
System administrators often complain that management doesn't understand technical issues. If management can't understand the technical issues, perhaps they need to be explained in clearer terms. The movie Armageddon has a scene in which the scientist is attempting to explain to the president the meteor's trajectory, mass, and velocity. The president is confused until the explanation is reworded to "a rock the size of Texas is going to hit Earth and destroy everything." Management just needs to know the issues so they can make a decision. The administrator's job is to understand the technology and produce the results management asks for. Don't give management data paralysis -- when they start giving you that deer-in-the-headlights look, you know you've gone too far.
The system administrator can make the process easier by interpreting the technical issues into business terms and developing an incident response procedure that has management's approval. This way, the administrator knows what actions are preapproved. For example, you may be authorized to call the technical contact for an address that is conducting repeated probes of your site.
Incident response procedures are nothing new in the industry. Many of the guidelines for developing an IRP were established during the precommercial days of the Internet and focus on how to proceed if your site has been compromised. While you should certainly have such a procedure in place, hopefully you'll be prepared to respond to an incident before it gets that far. Many sites are installing intrusion detection systems -- the technical equivalent of a burglar alarm. But what good is such a system if you're unprepared to react to the alarms?
The sidebar at the end of this column includes an outline I use for preparing an incident response procedure. Often, the issues that must be considered are the same across sites, but the details may vary, depending on the site's security policy and configuration. Hopefully, this outline will help you develop your own procedure.
Examples of response to above scenarios
While the appropriate response to a situation depends on your site's policies and procedures, it's very important to keep accurate chronological logs of events with details such as file names and the names and phone numbers of all people involved in any security incident. Check with your legal department for guidelines on evidence gathering. This is how I might respond to the hypothetical situations I mentioned earlier:
If there is an incident response team, contact them and let them
handle it. Otherwise, search the log files for her IP address and
gather technical evidence. Usually, I would
grep for her IP
address in the log files and save the output in another file to try
to establish a pattern. Save the original versions of all log files
as well as all working copies on protected backup media. Prepare a
file for management that summarizes the conclusion and attach
signed printouts of the log files as proof. For example:
Investigation has shown that the user's computer accessed pornographic sites on the Internet every night between the hours of 11:00 p.m. and 1:00 a.m. At no other time were these sites accessed from this computer. It is recommended that building access logs be checked to see who may have had physical access.
An admin at another site contacts you regarding the actions of one of your users. This could involve anything from nasty e-mail to attacks on his site.
Regardless of the administrator's attitude, you should politely inform him that you are not authorized to release any information without management approval, but you will pass along all the information. Ask for technical details such as his log data or a copy of the e-mail. Make sure you get his name, title, phone number, company, and e-mail address. Inform your management or appropriate security group.
You're experiencing repeated attacks from the same address. You've contacted the registered technical admin and you don't seem to be getting anywhere.
Inform your management and appropriate internal security organizations. Search all your log files for the offending addresses and save to a file to help with the analysis. Prepare a summary of incident times to assist in investigation. The real evidence will be the original log file, so you must save an untouched copy on backup media that is write-protected. If you have any intention of legal recourse, print out a copy of the original log, sign and date it, and also have a witness sign and date it. Make sure you have logs or copies of any e-mail to show all attempted correspondence with the technical contact. Again, check with your legal department for appropriate evidence gathering procedure.
An authoritative agency such as the FBI or CERT contacts you directly and informs you (sounding very official) that it is conducting an investigation of one of your users and expects your full cooperation.
Even the FBI has to have a search warrant to get company data and, unless you're an officer of the company, you're not the appropriate person to be served a warrant. Use the same guidelines as you would for an obnoxious administrator demanding information: be polite and gather information to be forwarded to management. The point is, you can't be certain who you're really speaking to and only management may authorize the release of company information. If it really is an FBI investigation, it must be escalated to senior management.
You notice that an executive in the company is browsing child-porn sites, and you suspect he may be saving images on company systems.
This is difficult because it's such a sensitive and emotional issue. However, if it isn't your company policy to monitor Internet usage, you shouldn't be. If you happened to come across it, you may have grounds to make a hostile work environment complaint to Human Resources. It isn't your place to audit the manager's system without prior approval, no matter what you suspect. Go to a manager you know and trust and seek his or her advice. Don't discuss it with anyone else in the company. For all you know, someone else may be using the executive's system, and he may not even be aware of it. Reputations are easily damaged in this kind of situation, and the damage is almost impossible to fix. It's also quite easy to lose your job in this kind of situation.
A new employee brags to you about a back door she left in her last company.
Because there's really no way to confirm whether or not it's true, this scenario is difficult to act on. I wouldn't want such a person working for me, because she could do the same thing to me, but that's a management decision. Tell your management and let them decide how to handle it.
You think your system may have been compromised.
Obviously, there is quite a lot involved in responding to this, and it requires a special response procedure. It's very easy to panic in this situation and very important that you do not. Strictly follow your response procedures and document everything. Things can happen very quickly, and it's easy to get confused later about exactly what happened. If it's at all serious you will inevitably be asked to produce a report. An accurate log is invaluable in such a situation. If there's a possibility that this may be needed in a legal case, make sure the original log is dated (with timestamps) and signed or initialed. Don't worry about the log being presentable; it just needs to be consistent and reasonably understandable. I have logs that are mostly scribbled, but they're at least semilegible.
You may disagree with my suggested responses to these security scenarios. If so, that's all the more reason to develop an incident response procedure that has the approval of your management. If getting started is your problem, use the attached IRP and edit it for your site. Come up with some of your own scenarios and discuss with management how they should be handled -- then put it in writing. Once you have a response procedure, it should be tested by the audit department as part of the regular security assessment. Whatever you do, don't wait for a major incident to occur before you write one.
Just as I was about to write about installing Sendmail 8.9.2 on a firewall, an announcement was made about the release of Sendmail 8.9.3. Oh, well -- that saves me from explaining how to install the patch. Next month, I'll describe the installation steps for a firewall and internal mail gateway and include sample configuration templates.
About the author
Carole Fennelly is a partner in Wizard's Keys Corporation, a company specializing in computer security consulting. She has been a Unix system administrator for more than 15 years on various platforms and has particularly focused on Sendmail configurations of late. Carole provides security consultation to several financial institutions in the New York City area. Reach Carole at email@example.com.
If you have technical problems with this magazine, contact firstname.lastname@example.org
Last modified: Tuesday, March 02, 1999