Technical white paper

HPE next-generation intrusion detection system

Introduction

The kernel is the most privileged and sensitive part of the operating system running on your server. It controls and interposes on all interactions by applications. This includes interactions with other applications sharing the same server and all hardware resources such as the network, keyboard, and screen. For this reason, the kernel is often targeted by attackers. By inserting a special kernel module into the kernel, they are able to provide themselves with a hidden back door. The kernel module is the critical component of a set of tools called a rootkit, which gives the attacker persistent access to the server - they can access it at any time. It is called a rootkit because it gives the attacker complete power over the system, equivalent to root or administrator access.
The rootkit enables attackers to monitor all activity on the server and retrieve all data and files from it. A well-crafted rootkit is able to hide all attacker activity from users of the server, even system administrators or root users. So it is extremely challenging even for the most skilled system administrator to detect the compromise of the operating system by running tools and applications on the operating system. Relying on a compromised operating system to detect its own compromise is problematic.
Hewlett Packard Labs initiated a research project to investigate the development of a Distributed Intrusion Monitoring Engine (DIME) to protect the kernel from malicious code. This white paper explores how it is significantly more difficult for a rootkit to hide from DIME. This outcome is achievable, because the DIME scanning engine does not share the processor, so it is outside and independent of the compromised operating system. Instead, it runs on HPE Integrated Lights Out (iLO). So with DIME we are no longer relying on the operating system to detect its own compromise. DIME does not rely on signatures. All rootkits are different. Instead DIME detects changes to kernel APIs and code, which any rootkit must make to hide its presence.