Automatic Intelligent Analysis of Malware Behaviour

Authors: H. Dornhackl, K. Kadletz, R. Luh, P. Tavolato

Abstract:

In this paper, we describe the use of formal methods to model malware behaviour. The modelling of harmful behaviour rests upon syntactic structures that represent malicious procedures inside malware. The malicious activities are modelled by a formal grammar, where API calls’ components are the terminals and the set of API calls used in combination to achieve a goal are designated non-terminals. The combination of different non-terminals in various ways and tiers make up the attack vectors that are used by harmful software. Based on these syntactic structures a parser can be generated which takes execution traces as input for pattern recognition.

Keywords: Malware behaviour, modelling, parsing, search, pattern matching.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1109285

References:

[1] Cohen, F.: Computer Viruses: Theory and Experiments. In: Computer and Security 6/1, 1987, pp. 22-35.
[2] Filiol, E.; Helenius, M; Zanero, S.: Open Problems in Computer Virology. In: Journal of Computer Virology 1(3-4), 2006, pp. 55-66.
[3] Kramer, S.; Bradfield, J.C.: A General Definition of Malware. In: Journal in Computer Virology, 6/2, 2010, pp. 105-114.
[4] Jacob, G.; Debar, H; Filiol, E.: Functional polymorphic engines: formalisation, implementation and use cases. In: Journal in Computer Virology, 5/3, 2009, pp. 247-261.
[5] Jacob, G.; Debar, H; Filiol, E.: Malware Behavioural Detection by Attribute-Automata Using Abstraction from Platform and Language. In: Lecture Notes in Computer Science 2009, Vol. 5758/2009, pp. 81-100.
[6] Jacob, G.; Debar, H; Filiol, E.: Formalization of Malware through Process Calculi. In: Journal in Computer Virology, 5/3, 2009, pp. 247- 261.
[7] Beaucamps, P.; Gnaedig, I.; Marion, J.: Behaviour Abstraction in Malware Analysis. In Lecture Notes in Computer Science 2010, Vol. 6418/2010, pp. 168-182.
[8] Bayer, U.; Kirda, E.; Kruegel, C.: Improving the Efficiency of Dynamic Malware Analysis. 25th Symposium On Applied Computing, Lausanne, 2010.
[9] Bayer, U.; Moser, A.; Kruegel, C.; Kirda, E.: Dynamic Analysis of Malicious Code. Journal in Computer Virology 2/1, Springer, 2007.
[10] Kirda, E.; Kruegel, C.: Large-Scale Dynamic Malware Analysis. PhD Dissertation, Technical University of Vienna, 2009.
[11] Christodorescu, M.; Jha, S.; Kruegel, C.: Mining Specifications of Malicious Behaviour. ESEC/FSE’07, September 3–7, 2007, Cavtat near Dubrovnik, Croatia, 2007.
[12] Dornhackl, H.: Syntaktische Musterdefinition von ausgewählten Malwareverhalten und Implementierung eines Parsers. Master thesis, FH St. Pölten, 2013 (in German).
[13] Luh, R.; Tavolato, P.: Behaviour-based Malware Recognition. Forschungsforum der österreichischen Fachhochschulen, 2012.
[14] Fukushima, Y.; Sakai, A.; Hori, Y.; Sakurai, K.: A Behaviour Based Malware Detection Scheme for Avoiding False Positive. Secure Network Protocols (NPSec), 2010 6th IEEE Workshop on Secure Network Protocols, 2010.
[15] Dornhackl H., Kadletz K., Luh R., Tavolato P.: Using Formal Methods for Malware Behaviour Modelling, to be published.
[16] Batra, R.: API Monitor. retrieved from http://www.rohitab.com/ apimonitor, last accessed 2013-10-14.
[17] Gonzalez, C.; Thomason, M.: Syntactic Pattern Recognition. Addison- Wesley, 1978.
[18] Bayer, U.: Large-Scale Dynamic Malware Analysis. PhD thesis, Technische Universität Wien, 2009.
[19] Oracle Corporation, Oracle Virtual Box retrieved from https://www.virtualbox.org, last accessed 2013-10-14.