Detecting and mitigating malware in virtual appliances

Update Item Information
Publication Type thesis
School or College College of Engineering
Department Computing
Author Nayak, Prashanth
Title Detecting and mitigating malware in virtual appliances
Date 2014-12
Description System administrators use application-level knowledge to identify anomalies in virtual appliances (VAs) and to recover from them. This process can be automated through an anomaly detection and recovery system. In this thesis, we claim that application-level policies defined over kernel-level application state can be effective for automatically detecting and mitigating the effects of malicious software in VAs. By combining user-defined application-level policies, virtual machine introspection (VMI), expert systems, and kernel-based state management techniques for anomaly detection and recovery, we are able to provide a favorable environment for the execution of applications in VAs. We use policies to specify the desired state of the VA based on an administrator's application-level knowledge. By using VMI we are able to generate a snapshot that represents the true internal state of the VA. An expert system evaluates the snapshot and identifies any violations. Potential violations include the execution of an irrelevant application, an unauthorized process, or an unfavorable environment configuration. The expert system also reasons about appropriate recovery strategies for each of the violations detected. The recovery strategy decided by the expert system is carried out by recovery tools so that the VA can be restored to an acceptable state. We evaluate the effectiveness of this approach for anomaly detection and repair by using it to detect and recover from the actions of different types malicious software targeting a web server VA. The system is shown to be effective in guarding the VA against the actions of a kernel-exploit kit, a kernel rootkit, a user-space rootkit, and an application malware. For each of these attacks, the recovery component was able to restore the VA to an acceptable state. Although, the recovery actions carried out did not remove the malicious software, they substantially mitigated the harmful effects of the malicious software.
Type Text
Publisher University of Utah
Subject Detection; Expert systems; Malware; Mitigation; Virtual appliance; Virtual machine introspection
Dissertation Institution University of Utah
Dissertation Name Master of Science
Language eng
Rights Management Copyright © Prashanth Nayak 2014
Format Medium application/pdf
Format Extent 1,742,339 bytes
Identifier etd3/id/3303
ARK ark:/87278/s6z92mpg
Setname ir_etd
ID 196868
Reference URL https://collections.lib.utah.edu/ark:/87278/s6z92mpg
Back to Search Results