Detecting and modeling polymorphic shellcode: A new approach

The problem of modeling and detecting polymorphic engines shellcode is adressed in this book. By polymorphic engines, we mean programs having the ability to transform any piece of malware into many instances consisting of different code but having the same functionality as the original malware. Typically, polymorphic engines work by encrypting the target malware using various encryption techniques and providing a decryption module in order to execute the newly encrypted instance. Moreover, those engines have the ability to mutate their decryption routine making them unique from one instance to another and hard to detect. We propose a new concept of signatures, shape signatures, which cope with the highly mutated nature of those engines. The shape signatures try to identify the constant part as well as the mutated part of the deciphering routines. This combination is able to cope with the highly mutated nature of those engines in a much more efficient way compared to traditional signatures used in most intrusion detection systems. We also aim at modeling those polymorphic engines by showing that they exhibit a specific byte composition.

MSc graduate student in Information Security at Concordia University Montreal, Omar Nbou is a professional within the security field, expert in different domains such as malware detection, network security protocols, vulnerability assessment and exploitation. He currently works as a Consultant in IT Security.