Repository logo

Detecting advanced botnets in enterprise networks




Zhang, Han, author
Papadopoulos, Christos, advisor
Ray, Indrakshi, committee member
Pallickara, Shrideep, committee member
Hayne, Stephen C., committee member

Journal Title

Journal ISSN

Volume Title


A botnet is a network composed of compromised computers that are controlled by a botmaster through command and control (C&C) channel. Botnets are more destructive compared to common virus and malware, because they control the resources from many compromised computers. Botnets provide a very important platform for attacks, such as Distributed Denial-of-Service (DDoS), spamming, scanning, and many more. To foil detection systems, botnets began to use various evasion techniques, including encrypted communications, dynamically generated C&C domains, and more. We call such botnets that use evasion techniques as advanced botnets. In this dissertation, we introduce various algorithms and systems to detect advanced botnets in enterprise-like network environment. Encrypted botnets introduce several problems to detection. First, to enable research in detecting encrypted botnets, researchers need samples of encrypted botnet traces with ground truth, which are very hard to get. Traces that are available are not customizable, which prevents testing under various controlled scenarios. To address this problem we introduce BotTalker, a tool that can be used to generate customized encrypted botnet communication traffic. BotTalker emulates the actions a bot would take to encrypt communication. To the best of our knowledge, BotTalker is the first work that provides users customized encrypted botnet traffic. The second problem introduced by encrypted botnets is that Deep Packet Inspection (DPI)-based security systems are foiled. We measure the effects of encryption on three security systems, including Snort, Suricata and BotHunter (BH) using the encrypted botnet traffic generated by BotTalker. The results show that encryption foils these systems greatly. Then, we introduce a method to detect encrypted botnet traffic based on the fact that encryption increases data's entropy. In particular, we present two high-entropy (HE) classifiers and add one of them to enhance BH by utilizing the other detectors it provides. By doing this HE classifier restores BH's ability to detect bots, even when they use encryption. Entropy calculation at line speed is expensive, especially when the flows are very long. To deal with this issue, we introduce two algorithms to classify flows as HE by looking at only part of a flow. In particular, we classify a flow as HE or low entropy (LE) by only considering the first M packets of the flow. These early HE classifiers are used in two ways: (a) to improve the speed of bot detection tools, and (b) as a filter to reduce the load on an Intrusion Detection System (IDS). We implement the filter as a preprocessor in Snort. The results show that by using the first 15 packets of a flow the traffic delivered to IDS is reduced by more than 50% while maintaining more than 99.9% of the original alerts. Comparing our traffic reduction scheme with other work we find that they need to inspect at least 13 times more packets than ours or they miss about 70 times of the alerts. To improve the resiliency of communication between bots and C&C servers, bot masters began utilizing Domain Generation Algorithms (DGA). DGA technique avoids static blacklists as well as prevents security specialists from registering the C&C domain before the botmaster. We introduce BotDigger, a system that detects DGA-based bots using DNS traffic without a priori knowledge of the domain generation algorithm. BotDigger utilizes a chain of evidence, including quantity, temporal and linguistic evidence to detect an individual bot by only monitoring traffic at the DNS servers of a single network. We evaluate BotDigger's performance using traces from two DGA-based botnets: Kraken and Conflicker, as well as a one-week DNS trace captured from our university and three traces collected from our research lab. Our results show that BotDigger detects all the Kraken bots and 99.8% of Conficker bots with very low false positives.


Rights Access


network security
intrusion detection system


Associated Publications