Till Studer's Notes

Recent Notes

18 min read

Intrusion Detection System

Theoretical Background

Intruders

  • Three classes of intruders:
    • Masquerader is an individual who is not authorized to use the computer and who penetrates a system’s access controls to exploit a legitimate user’s account, typically an outsider.
    • Misfeasor is a legitimate user who accesses data, programs, or resources for which such access is not authorized, or who is authorized for such access but misuses his or her privileges, typically an insider.
    • Clandestine user is an individual who seizes supervisory control of the system and uses this control to evade auditing and access controls or to suppress audit collection, either outsider or insider.
  • The attacks vary:
    • Performing a remote root compromise of an e-mail server, or defacing a Web server
    • Copying a database containing credit card numbers
    • Viewing sensitive data, including payroll records and medical information, without authorization
    • Running a packet sniffer on a workstation to capture usernames and passwords
    • Using a permission error on an anonymous FTP server to distribute pirated software and music files
    • Posing as an executive, calling the help desk, resetting the executive’s e-mail password, and learning the new password
    • Using an unattended, logged-in workstation without permission

Formal Definition of an Intrusion

Systems under attack do NOT meet at least one of these:

  • User, process actions conform to statistically predictable pattern
  • User, process actions do not include sequences of actions that subvert the security policy
  • Process actions correspond to a set of specifications describing what the processes are allowed to do

Attacker’s Side

  • Intruder’s Goal is to insert a backdoor in a system.
    • Therefore, she will modify system configuration files or programs.
    • That requires privilege (i.e. attacker enters system as an unprivileged user and must acquire privilege)
    • Therefore the attacker may break in using sequence of commands that violate security policy or she may cause program to act in ways that violate program’s specification.
  • Denning’s General Model: Exploiting vulnerabilities requires abnormal use of normal commands or instructions
    • Includes deviation from usual actions
    • Includes execution of actions leading to break-ins
    • Includes actions inconsistent with specifications of privileged programs

Goals of an IDS

  1. Detect wide variety of intrusions:
    • Previously known and unknown attacks
    • Need to learn/adapt to new attacks or changes in behavior
  2. Detect intrusions in timely fashion
    • Be real-time, especially when system responds to intrusion
    • Or report intrusion occurred a few minutes or hours ago
  3. Present analysis in simple, easy-to-understand format
  4. Be accurate: Minimize wrong alerts and time spent on verifying attacks

Why is an IDS Needed?

  • Packet filtering firewalls provide a good basis to protect networks and hosts, but against “intrusion” threats, firewalls are virtually useless:
    • Firewalls only block “unwanted” traffic
      • But if one allows traffic to e.g., an internal mail or web server, the corresponding data gets through the firewall without being inspected
      • So, an attacker that tries to exploit vulnerabilities won’t be detected
    • Firewall do not protect from internal users “importing” malware
      • By installing dangerous applications, unwittingly downloading malicious code or using a laptop that was previously infected at home (or someplace else)

Practical

  • Tasks / Goals of Intrusion Detection Systems (IDS):
    • Monitoring of network and/or system activity to detect attacks, attack attempts and general abuse
    • Difficulty: Detect the true security-critical activities among a potentially huge amount of legitimate activity while minimizing the false alerts
  • Two main types depending on sensor-placement:
    • Host-based IDS (HIDS) monitoring individual hosts
    • Network IDS (NIDS) monitoring entire (or parts of) networks
    • Sometimes used in combination: Hybrid IDS

Components and Terminology

  • Sensor: A sensor is a hardware and/or software component that directly monitors system or network activity
  • Event: A sensor generates an event when it detects “noteworthy” activity
  • Alert: Depending on configuration settings, an event or a combination of events may result in an alert; an alerts usually means that a human or automated intervention must take place
  • Database and Correlation Engine: A centralized host that collects events from different sensors, correlates them, and generates alerts if necessary
  • Management Console: A host used by the IDS operator for configuration, report generation etc.
  • False Positive: A false positive is a false alert reported by an IDS
  • False Negative: A false negative is a security-critical event that was missed by the IDS

Host-based IDS

  • Monitor individual hosts
  • Sensors (software component) are directly installed on the hosts that are monitored
  • Analyses system calls, application logs, file modifications but also network data from/to this host
  • Examples: OSSEC, Sentry, Tripwire, Snort (when used on a host)
  • Compared to Network IDS:
    • Advantages
      • Effects of an attack are clearly visible and therefore detectable (e.g. detect file modifications)
      • No problems with encrypted network data
      • Few false positives due to context information (OS, apps)
    • Disadvantages
      • Limited view on one host
      • Puts additional load on the monitored system
      • Cannot be easily hidden and are therefore attackable
      • Trustworthiness of a sensor is questionable after successful attack

Network IDS

  • Monitor network segments
  • Sensors are dedicated devices in the network that see the desired traffic (often attached to the monitor port of a switch)
  • Analyzes network traffic for malicious activity such as scanning traffic or attack attempts
  • Examples: HP TippingPoint, ISS RealSecure, Snort (dedicated sensor)
  • Compared to Host-based IDS:
    • Advantages
      • Broad view of entire activity in monitored segment
      • No additional load for monitored systems
      • Sensors can be well protected (passive monitoring, no IP address needed)
    • Disadvantages
      • Problems with encrypted traffic
      • Result of an observed attack attempt are hard to see
      • Require additional hardware components
      • Difficult to cope with large traffic volumes (Gigabit Ethernet)
      • Difficult to get all traffic in switched networks

Hybrid IDS

  • Basic idea: combining host-based and network sensors
  • Can combine the advantages to detect attacks with high precision, e.g.
    • A network sensor first detects a possible attack against a host and generates an event
    • If the host sensor reports a suspicious file modification briefly afterwards, the probability is high that the attack was indeed successful and generating an alert is certainly justified

Deployment Scenarios

  • IDS operation range: from very small to large environments
  • A “small” IDS can be used to monitor just one host
    • Everything (sensor, database (log file), alert generation, console) is on the same system, sometimes even on the monitored host
  • A large, hybrid IDS can be used to monitor large corporate environments
    • Consists of several host-based and network sensors that report their events to a centralized database and correlation engine
    • Correlation engine examines all events and raises alerts if necessary
    • Aggregated view and correlation of all events can result in more precise alert generation and detect attacks a separate sensor is likely to miss

Challenges

  • Configuring and operating an IDS is a challenging task and several problems must be solved
    1. Minimize both the number of false positives and false negatives
    2. Cope with large amounts of data (monitoring of e.g. Gigabit links)
    3. Where to best place sensors
    4. Keeping system (esp. signature database) up to date
    5. A policy must be defined which describes what to do in the case of alerts
    6. 24x7 monitoring needs people (unless the IDS is used for forensic purposes only)
  • Using an IDS does not mean “buying, installing, and configuring once” but is a continuous effort
    1. Fine-tuning takes time and requires know-how
    2. Continuous adaptation to cope with the current situation is required
    3. Outsourcing the operation of an IDS as a managed security service (MSS) maybe an option

Models

Three models are used in intrusion detection:

  1. Anomaly detection
    • What is usual, is known
    • What is unusual, is bad (possible intrusion)
  2. Misuse detection
    • What is bad (possible intrusion), is known
    • What is not bad, is good (no intrusion)
  3. Specification-based detection
    • What is good, is known
    • What is not good, is bad (possible intrusion)

Comparison of three models:

  • Anomaly detection:
    • Detects unusual events, but these are not necessarily security problems
  • Misuse detection:
    • If all policy rules are known, its easy to construct rulesets to detect violations
      • The usual case is that much of policy is unspecified, so rulesets describe attacks, and are not complete
  • Specification-based vs. Misuse:
    • Specification assumes: if specifications followed ⇒ policy not violated
    • Misuse assumes: if policy as embodied in rulesets followed ⇒ policy not violated

Anomaly Detection

  • Anomaly detection analyses a set of characteristics of system and compares their values with expected values. It reports when computed statistics do not match expected statistics.
  • For the determination of a deviation, the following approaches are used:
    • Threshold Metrics
    • Statistical Methods
    • Markov Model

Threshold Metrics

  • Counts number of events that occur
    • Between and events (inclusive) expected to occur
    • If number falls outside this range ⇒ anomalous
  • Example: Windows locks a user out after sequential failed login attempts
    • Range is .
    • or more failed logins are deemed anomalous
  • Appropriate threshold may depend on non-obvious factors, illustrated in example
    • If keyboards are US keyboards, and most users are German, typing errors get very common
      • ⇒ low threshold results in a large number of false alarms

Statistical Methods

  • The analyzer computes the mean and standard deviation (first two moments) and possibly other measures of correlation (higher moments).
    • If the measured values fall outside the expected interval for particular moments, we speak of an anomaly.
    • Because the profile, or description of the system, may evolve over time, anomaly-based intrusion detection systems take these changes into account by aging (or weighting) data or altering the statistical rule base on which they make decisions.
  • Challenges:
    • It is assumed that behavior of processes and users can be modeled statistically
    • Ideal: matches a known distribution such as Gaussian or normal
      • Otherwise, must use techniques like clustering to determine moments, characteristics that show anomalies, etc.
    • Real-time computation are also an issue

Markov Model

  • Past state affects current transition
  • Anomalies based upon sequences of events, and not on occurrence of single event
  • Issue: Must train the system to establish valid sequences
    • Use known, training data that is not anomalous
    • The more training data, the better the model
    • Training data should cover all possible normal uses of system
  • Example: Time-based Inductive Learning (TIM)
    • Valid Sequence of events is
    • TIM derives following rules:
    • So if is seen, an alert triggers as always follows in rule set
    • So if is seen, no alert is triggered as multiple events can follow
      • that can not happen was not learned
      • thus no rule was derived
      • and since no rule was derived, it can’t be broken

Ground Truth:

Protocol Anomaly

  • Goal: Detect protocol violations (in-compliant with specification)
  • Why?
    • In the past, several attacks occurred by using a protocol in a way that does not conform to the specifications (e.g. Ping of Death)
    • As a large number of protocols are relatively small and well-defined, both false positives and negatives can be kept small
      • Mainly because legitimate users / programs typically don’t violate the protocols
      • Does usually not require fine-tuning because the protocols are standardized
  • Examples:
    • Look for unusual combination of flags in TCP packets
      • e.g. all or none of the flags are set as used in Xmas or NULL port scans
    • IP fragments with non-reasonable combinations of header field values
      • e.g. fragment offset + fragment payload length > 65’535 bytes (max. IP packet length)
    • Make sure that during a POP3 session, attempts to download e-mails (RETR) are only made after a successful login (USER, PASS with OK responses)

System Call Tracing

  • Traces are defined as normal behavior in terms of sequences of system calls.
  • A database of the traces will be built for a process (e.g. sendmail) by observing process behavior during normal runs of the program.
  • Example:
    • The trace length is 4 and the process has the trace:
    • So the traces table (length 4) looks the following:
    • So if the following trace is later seen:
    • We get a mismatch rate of 5/18 (26%) in this trace.

Execution Path Profiling

  • Anomaly detection by execution path profiling:
    • Analyze normal execution path of programs on a system without infections
    • Generate a state diagram that e.g. contains all transitions between functions during normal operation
    • During program execution, compare the actual transitions with this state diagram

Machine Learning

  • All the methods used so far assumed some kind of statistical distribution of underlying
  • Machine learning techniques does not assume a priori distribution of data to classify data as anomalous.
  • Types of learning:
    1. Supervised learning methods:
      • begin with data that has already been classified
      • split it into “training data”, “test data”
      • use first to train classifier, second to see how good the classifier is
    2. Unsupervised learning methods:
      • no pre-classified data, so learn by working on real data
      • implicit assumption that anomalous data is small part of data
    • Measures used to evaluate methods based on:
      • TP: true positives ➔ correctly identify anomalous data
      • TN: true negatives ➔ correctly identify non-anomalous data
      • FP: false positives ➔ identify non-anomalous data as anomalous
      • FN: false negatives ➔ identify anomalous data as non-anomalous

Clustering

  • Clustering does not assume a priori distribution of data, but
    • Obtains data and groups into subsets (clusters) based on some property (feature)
    • Then the clusters are analyzed instead of individual data points.
  • It is often used for network traffic anomaly detection.
  • Example features:
    • Total amount of traffic (overall and per system)
    • Frequency distribution of individual protocols
    • Byte frequency distribution per protocol and/or protocol field
    • Packet length distribution
    • Fraction of fragmented packets
    • Typical content of certain protocol fields (e.g. IP addresses)
    • Traffic activity depending on time of day

Misuse Modeling or Signature Based Detection

  • Determines whether a sequence of instructions being executed is known to violate the site security policy
    • Descriptions of known or potential exploits grouped into rule sets (Use of a large signature, regularly updated database (usually provided by IDS provider) to detect known attacks
    • IDS matches data against rule sets; on success, potential attack found
  • Cannot detect attacks unknown to developers of rule sets, as it does not have any rules to cover them
  • Challenges:
    • Signatures should unspecific enough so that minor variations of an attack make it possible to avoid detection
      • e.g. access /etc/passwd is detected but /etc/./passwd is not
    • Signatures should also not be too generic as this likely results in many false positives
      • /etc/passwd may be completely legitimate in some packets
    • It should not easily be possible to evade the IDS by
      • Encoding of characters (e.g. Unicode instead of ASCII)
      • Fragmenting the packets (e.g. /etc/pa in first and sswd in second packet)

Effectiveness

Three metrics are used for indicating effectiveness:

  1. Accuracy is the percentage (or fraction) of events classified correctly
  2. Detection rate is the percentage (or fraction) of reported attack events that are real attack events.
    • Its also called the true positive rate.
  3. False alarm rate is the percentage (or fraction) of non-attack events reported as attack events
    • It is also called the false positive rate

Architectures

  • Intrusion detection systems and logging & audit architectures are basically the same:
  • Agent like logger: it gathers data for analysis
    • Obtains information and sends to director
    • May put information into another form (Preprocessing of records to extract relevant parts)
    • May delete unneeded information
    • Director may request agent send other information
  • Director like analyzer: it analyzes data obtained from the agents according to its internal rules
  • Notifier obtains results from director, and takes some action
    • May simply notify security officer
    • May reconfigure agents, director to alter collection, analysis methods
    • May activate response mechanism

Network Issues

  • Network architecture dictates agent placement (!)
    • Ethernet or broadcast medium: one agent per subnet
    • Point-to-point medium: one agent per connection, or agent at distribution/routing point
  • Focus is usually on intruders entering network
    • If few entry points, place network agents behind them
    • Does not help if inside attacks to be monitored
  • Agents produce information at multiple layers of abstraction
    • Application-monitoring agents provide one view (usually one line) of an event
    • System-monitoring agents provide a different view (usually many lines) of an event
    • Network-monitoring agents provide yet another view (involving many network packets) of an event

Intrusion Detection Responses

  • There are various options about what to do if an alert is raised
    • Do nothing (only reasonable if IDS is used for forensics)
    • Manual intervention (slow)
    • Automated intervention
      • Close connection
      • Dynamically adapt firewall rules
  • Prevent malicious traffic from reaching target systems in first place
    • Intrusion Prevention Systems

Send TCP Reset

  • Option: terminate TCP connection
    • IDS sends a TCP reset to both machines
  • Problems and limitations
    • Will often not work: TCP resets must use the correct sequence numbers while the end-to-end traffic is ongoing
    • Response will always be late (some packets already exchanged), especially with “single packet attacks”
    • Attacker can simply re-establish the connection and try again (problem is not “gone”)

Block Attacker at Firewall

  • Option: IDS blocks attacker permanently (or for a certain time period)
    • IDS dynamically modifies firewall to block any traffic from the attacker’s host
    • Works for all protocols and prevents future attack attempts
  • Problems and limitations
    • Response will always be late, ineffective against “single packet attacks”
    • Attacker can simply switch IP address for future attacks
    • Can be abused for Denial of Service (DoS): block a legitimate user by “attacking” from his host using source IP address spoofing

Limitations of Reactive Actions

  • All reactive actions have some inherent, fundamental weaknesses
    • Some packets always get through until the IDS response happens and is effective
    • Single-packet attacks get always through (e.g. SQL Slammer 2003)
    • The same attack can be executed again by the same or other attacker using the same or another attacking host
    • Blocking with firewall rules is prone to DoS abuse
  • Can we do better?
    • Yes, do not just passively monitor the traffic but “stop” it and only forward it when it is legitimate
    • This should allow to block malicious traffic before it can reach the target system
    • ➔ Intrusion Prevention System

Intrusion Prevention System (IPS)

  • Option: Unlike with IDS, the traffic does not flow past a sensor (passive monitoring), but through a sensor (active interception)
    • If a sensor generates an event, the traffic is blocked temporarily
    • If no alert is generated, the traffic is forwarded
    • If the event results in an alert (e.g. by correlating it with other events from other sensors), the traffic is blocked
      • Blocked traffic is logged and discarded
  • Benefits
    • If operated correctly, no attack traffic reaches the target system
      • Because suspicious packets are blocked and only forwarded if no alert is generated
    • It can protect against single-packet attacks
  • Challenges / Limitations compared to IDS:
    • Packets can only be blocked for a very short time to avoid that legitimate communication relationship are significantly delayed
      • This implies that IPS are resource intensive because traffic must be analyzed in near-real time
      • This limits correlation-possibilities with other events from the same or other sensors
    • False positives lock out legitimate user (with IDS, the event/alert is just “registered”)

Evasion

  • There are many options for an attacker to fool an IDS or IPS sensors
  • Typical strategy: make sure the IDS sees data differently than the target system
  • Options:
    • Fragmentation
    • Change ordering of packets
    • Encoding
    • Chaff
    • Combinations thereof

Fragmentation

  • The attacker fragments the IP packet with the attack traffic “hoping” that the IDS does not (correctly) reassemble IP packets before analyzing them

  • Simple fragmentation

  • Send fragments out of sequence

  • Fragmentation overlap

  • The same attack principle can also be used by spreading the attack across multiple IP packets

  • Countermeasures

    • Primary countermeasure against fragmentation attacks:
      • Don’t inspect every single IP fragment or IP packet for itself but use the entire context
    • Minimal solution: Reassemble IP packets before analyzing them
      • This protects from attacks that spread the attack pattern across multiple fragments of an IP packets
      • Make sure to cope correctly with reordered and overlapping fragments
    • Even better: Take the entire session into account (rather than individual packets)
      • This makes it possible to detect attacks that are spread against multiple IP packets
      • It also helps detecting attacks that require more than one step

Encoding

  • Example of a vulnerability in Outlook (CVE-2004-0121) allows executing arbitrary commands

  • When a user opens a mailto-URL in a HTML document, Outlook is started and the e-mail address is passed as an argument

    • OUTLOOK.EXE /m "mailto:email@address"

  • Specifically crafting the mailto-URL allows an attacker to pass additional arguments to Outlook.

    • This includes JavaScript code, which allows an attacker to e.g. download and start a desired EXE program

  • Normal attack string (ASCII encoded):

    <html> <body>
<img src="mailto:aa&quot; /select javascript:alert ('vulnerable') ">
</body> </html>

  • Character encoding of the document with 7-bit Unicode

    HTTP/1.1 200 OK
Date: Wed, 29 Jun 2020 22:59:39 GMT
Server: Apache/1.3.20 (Unix) PHP/4.0.6
Last-Modified: Wed, 29 Jun 2020 22:59:39 GMT
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Content-Length: 145
Content-Type: text/html; charset=utf-7

+ADw-html+AD4 +ADw-body+AD4
+ADw-img src+AD0AIg-mailto:aa+ACY-quot; /select javascript:alert('vulnerable')+ACIAPg
+ADw-/body+AD4 +ADw-/html+AD4

  • Base64 transfer-encoding with chaff

    HTTP/1.1 200 OK
Date: Wed, 29 Jun 2020 23:05:05 GMT
Server: Apache/1.3.20 (Unix) PHP/4.0.6
Content-Length: 703
Content-Type: message/rfc822; charset=iso-8859-1
MIME-Version: 1.0
Content-Type: multipart/related;
boundary="----=_NextPart_000_0009_98F1ECB0.631DDD4F";
type="text/html“
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.5600

This is a multi-part message in MIME format.

------=_NextPart_000_0009_98F1ECB0.631DDD4F
Content-Type: text/html
Content-Transfer-Encoding: base64
P[G;.?h0bW_{#w_+%_~&%]I<Dxib!&2$R'5|Pg,^o8(;aW1nI:$H );_N'-?
yYz$0i\(*~?bWF>p^b.&HRv}OmF#.hJn%#:F1b3Q`7IC{9(#@z#.ZW}__xl
_Y&3Qg[amF*2YX#N^}|^?^`j() cm$]>_l%w,dD"$p](hb.\^#GVy'>d_@!
!_~Cgnd`n[ Vsb](m'VyYWJsZS#c` !)#"p'I@%j4KP'C9i`~b.:2]R5_'{
P?$i';A_8L *,2)h}0)@bWw_+Cgo=
------=_NextPart_000_0009_98F1ECB0.631DDD4F--

  • HTTP offers further encoding options

    • Data compression (none, gzip, or deflate)
    • Character sets
      • ASCII, UTF-8, UTF-7, UTF-16LE, UTF16BE, UTF-32LE, UTF-32BE
      • Specified in HTTP header, initial bytes of document, or HTML tag

  • All this can be combined in literally thousands of ways…

  • Countermeasures:

    • IDS should be able to cope with various encodings
    • Typical approach: first decode (normalize) the data to a base encoding scheme before comparing it to the signatures
    • The large number of encoding possibilities makes this a difficult task (we have just looked at HTTP here…)

Relevant Note(s):

Graph View

Backlinks