Complex event processing

Complex event processing (CEP) consists of processing many events happening across all the layers of an organization, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time.

Complex event processing refers to process states, the changes of state exceeding a defined threshold of level, time, or value increment or just of a count as the event. It requires the respective event monitoring, event reporting, event recording and event filtering. An event may be observed as a change of state with any physical or logical or otherwise discriminated condition of and in a technical or economical system, each state information with an attached time stamp defining the order of occurrence and a topology mark defining the location of occurrence.

Conceptual description

Among thousands of incoming events, a monitoring system may for instance receive the following three from the same source:

1. church bells ringing.
2. the appearance of a man in a tuxedo with a woman in a flowing white gown.
3. rice flying through the air.

From these events the monitoring system may infer a complex event: a wedding. CEP as a technique helps discover complex events by analyzing and correlating other events:^[1] the bells, the man and woman in wedding attire and the rice flying through the air.

CEP relies on a number of techniques,^[2] including:

• event-pattern detection
• event abstraction
• modeling event hierarchies
• detecting relationships (such as causality, membership or timing) between events
• abstracting event-driven processes

Commercial applications of CEP include algorithmic stock-trading, the detection of credit-card fraud, business activity monitoring, and security monitoring.^[3]

Related concepts

One of many fields where CEP is used is Operational Intelligence (OI) solutions to provide insight into business operations by running query analysis against live feeds and event data. OI solutions use real-time data to collect and correlate against historical data to provide insight and analysis of the current situation. Multiple sources of data can be combined from different organizational silos to provide a common operating picture that uses current information. Wherever real-time insight has the greatest value, OI solutions can be applied to deliver the information and need.

In network management, systems management, application management and service management, people usually refer instead to event correlation. As CEP engines, event correlation engines (event correlators) analyze a mass of events, pinpoint the most significant ones, and trigger actions. However, most of them do not produce new inferred events. Instead, they relate high-level events with low-level events.^[4]

In artificial intelligence, inferred information is typically produced by inference engines, e.g. rule-based reasoning engines. However, new information is usually not produced in the form of complex (i.e., inferred) events.

Example

A more systemic example of CEP involves a car, some sensors and various events and reactions. Imagine that a car has several sensors—one that measures tire pressure, one that measures speed, and one that detects if someone sits on a seat or leaves a seat.

In the first situation, the car is moving and the pressure of one of the tires moves from 45 psi to 41 psi over 15 minutes. As the pressure in the tire is decreasing, a series of events containing the tire pressure is generated. In addition, a series of events containing the speed of the car is generated. The car's Event Processor may detect a situation whereby a loss of tire pressure over a relatively long period of time results in the creation of the "lossOfTirePressure" event. This new event may trigger a reaction process to note the pressure loss into the car's maintenance log, and alert the driver via the car's portal that the tire pressure has reduced.

In the second situation, the car is moving and the pressure of one of the tires drops from 45 psi to 20 psi in 5 seconds. A different situation is detected—perhaps because the loss of pressure occurred over a shorter period of time, or perhaps because the difference in values between each event were larger than a predefined limit. The different situation results in a new event "blowOutTire" being generated. This new event triggers a different reaction process to immediately alert the driver and to initiate onboard computer routines to assist the driver in bringing the car to a stop without losing control through skidding.

In addition, events that represent detected situations can also be combined with other events in order to detect more complex situations. For example, in the final situation the car was moving normally but suffers a blown tire which results in the car leaving the road and striking a tree and the driver is thrown from the car. A series of different situations are rapidly detected. The combination of "blowOutTire", "zeroSpeed" and "driverLeftSeat" within a very short space of time results in a new situation being detected: "occupantThrownAccident". Even though there is no direct measurement that can determine conclusively that the driver was thrown, or that there was an accident, the combination of events allows the situation to be detected and a new event to be created to signify the detected situation. This is the essence of a complex (or composite) event. It is complex because one cannot directly detect the situation; one has to infer or deduce that the situation has occurred from a combination of other events.

Types

Most CEP solutions and concepts can be classified into two main categories:

1. Computation-oriented CEP
2. Detection-oriented CEP

A computation-oriented CEP solution is focused on executing on-line algorithms as a response to event data entering the system. A simple example is to continuously calculate an average based in data on the inbound events.

Detection-oriented CEP is focused on detecting combinations of events called events patterns or situations. A simple example of detecting a situation is to look for a specific sequence of events.

Integrating CEP with Business Process Management

Of course, rarely does the application of a new technology exist in isolation. A natural fit for CEP has been with BPM.^[5] BPM very much focuses on end-to-end business processes, in order to continuously optimize and align for its operational environment.

However, the optimization of a business does not rely solely upon its individual, end-to-end processes. Seemingly disparate processes can affect each other significantly. Consider this scenario: In the aerospace industry, it is good practice to monitor breakdowns of vehicles to look for trends (determine potential weaknesses in manufacturing processes, material etc). Another separate process monitors current operational vehicles life cycle and when appropriate decommissions at the end of their useful lives. Now one use for CEP is to link these separate processes, so that in the case of when the initial process (breakdown monitoring) discovers a malfunction based on metal fatigue (a significant event) an action can be created to exploit the second process (life cycle) to issue a recall on vehicles using the same batch of metal discovered as faulty in the initial process.

The integration of CEP and BPM must exist at two levels, both at the business awareness level (users must understand the potential holistic benefits of their individual processes) and also at the technological level (there needs to be a method by which CEP can interact with BPM implementation).

Computation-oriented CEP's role can arguably be seen to overlap with Business Rule technology.

Academic research

• Aurora (Brandeis University, Brown University and MIT)
• Borealis (Brandeis University, Brown University and MIT)
• Cayuga (Cornell University)
• ETALIS (Forschungszentrum Informatik Karlsruhe and Stony Brook University)
• Global Sensor Networks (EPFL)
• NiagaraST (Portland State University)
• PIPES (University of Marburg)
• SASE (UC Berkeley/UMass Amherst)
• STREAM (Stanford University)
• Telegraph (UC Berkeley)
• epZilla (University of Moratuwa)
• ACAIA.org Acoustic Event Detection (AED) (Acoustic Computing for AI/AmI Applications)

See also

• Event correlation
• Event-driven architecture — (EDA) is a software architecture pattern promoting the production, detection, consumption of, and reaction to events.
• Event Processing Technical Society — (EPTS) is an event processing community of interest
• Event stream processing — (ESP) is a related technology that focuses on processing streams of related data.
• Real-time computing — CEP systems are typically real-time systems
• Real time enterprise
• openPDC — A complete set of applications for processing streaming time-series data in real-time.
• Operational intelligence — Both CEP and ESP are technologies that underpin operational intelligence.
• Pattern matching
• Rulecore
• WebSphere Business Events
• Esper - A Java and .NET complex event processing engine

References

1. ^ D. Luckham, "The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems", Addison-Wesley, 2002.
2. ^ O. Etzion and P. Niblett, "Event Processing in Action", Manning Publications, 2010.
3. ^ Details of commercial products and use cases
4. ^ J.P. Martin-Flatin, G. Jakobson and L. Lewis, "Event Correlation in Integrated Management: Lessons Learned and Outlook", Journal of Network and Systems Management, Vol. 17, No. 4, December 2007.
5. ^ C. Janiesch, M. Matzner and O. Müller: "A Blueprint for Event-Driven Business Activity Management", Lecture Notes in Computer Science, 2011, Volume 6896/2011, 17-28, DOI: 10.1007/978-3-642-23059-2_4

External links

• Separating the Wheat from the Chaff Article about CEP as applied to RFID, appeared in RFID Journal
• The Event Processing Technical Society
• Complex event processing: still on the launch pad in Computerworld
• Processing Flows of Information: From Data Stream to Complex Event Processing - Survey article on Data Stream and Complex Event Processing Systems
• S4 Distributed Event Processing Framework from Yahoo