Generating content-based signatures for detecting bot-infected machines

Abstract

A botnet is a network of compromised machines that are remotely controlled and commanded by an attacker, who is often called the botmaster. Such botnets are often abused as platforms to launch distributed denial of service attacks, send spam mails or perform identity theft. In recent years, the basic motivations for malicious activity have shifted from script kiddie vandalism in the hacker community, to more organized attacks and intrusions for financial gain. This shift explains the reason for the rise of botnets that have capabilities to perform more sophisticated malicious activities. Recently, researchers have tried to develop botnet detection mechanisms. The botnet detection mechanisms proposed to date have serious limitations, since they either can handle only certain types of botnets or focus on only specific botnet attributes, such as the spreading mechanism, the attack mechanism, etc., in order to constitute their detection models. We present a system that monitors network traffic to identify bot-infected hosts. Our goal is to develop a more general detection model that identifies single infected machines without relying on the bot propagation vector. To this end, we leverage the insight that all of the bots get a command and perform an action as a response, since the command and response behavior is the unique characteristic that distinguishes the bots from other malware. Thus, we examine the network traffic generated by bots to locate command and response behaviors. Afterwards, we generate signatures from the similar commands that are followed by similar bot responses without any explicit knowledge about the command and control protocol. The signatures are deployed to an IDS that monitors the network traffic of a university. Finally, the experiments showed that our system is capable of detecting bot-infected machines with a low false positive rate.