Skip to content

Window Correlator

Window correlator detects incoming events based on the predicate section and stores them in data structures based on the evaluate section. If then the test rule in analyze section is matched, trigger section is called.

Sigma Correlations

The Window Correlator is also used as an engine for Sigma Correlations, whose syntax can thus be used natively in LogMan.io. For more information about Sigma Correlations, see Sigma Correlations Documentation.

The following sample correlation detects more than or equal to 5 error connections between two IP addresses:

Sample

---
define:
    name: "Network T1046 Network Service Discovery"
    description: "Detects more than or equal to 5 error connections between two IP addresses"
    type: correlator/window

logsource:
    type: "Network"

mitre:
    technique: "T1046"
    tactic: "TA0007"

predicate:
    !OR
    - !EQ
        - !ITEM EVENT log.level
        - "error"
    - !EQ
        - !ITEM EVENT log.level
        - "critical"
    - !EQ
        - !ITEM EVENT log.level
        - "emergency"

evaluate:
    dimension: [source.ip, destination.ip]  
    by: "@timestamp"
    resolution: 60

analyze:
    window: hopping
    aggregate: sum
    span: 10
    test:
        !GE
        - !ARG
        - 5

trigger:
    - event:
            threat.indicator.confidence: "Medium"
            threat.indicator.ip: !ITEM EVENT source.ip
            threat.indicator.port: !ITEM EVENT source.port
            threat.indicator.type: "ipv4-addr"

Section define

This section contains the common definition and meta data.

Item name

Shorter human-readable name of this declaration.

Item type

The type of this declaration, must be correlator/window.

Item description (optional)

Longed, possibly multiline, human-readable description of the declaration.

Section logsource

Specifies the types of event lanes that the incoming events are read from. You can use a scalar (base, complex, activity, …) or an object with vendor, product, category, service, and other keys aligned with event lane templates. For allowed values and the difference between rule logsource and template logsource, see Logsource in correlation rules.

Section predicate

The predicate filters incoming events using an expression. If the expression returns True, the event will enter evaluate section. If the expression returns False, then the event is skipped.

Other returned values are undefined.

Include of nested predicate filters

Predicate filters are expressions located in a dedicated file, that can be included in many different predicates as their parts.

If you want to include an external predicate filter, located either in the library, use !INCLUDE statement:

!INCLUDE /predicate_filter.yaml

where /predicate_filter is the path of the file in the library. The content of predicate_filter.yaml is an expression to be included, like:

---
!EQ
- !ITEM EVENT category
- "MyEventCategory"

Section evaluate

The evaluate section specifies primary key, resolution and other attributes that are applied on the incoming event. The evaluate function is to add the event into the two dimensional structure, defined by a time and a primary key.

Item dimension

Specifies simple or compound primary key (or dimension) for the event. The dimension is defined by names of the input event fields.

Example of the simple primary key:

evaluate:
    dimension: [source.ip]

Note

Tenant is added automatically to the dimension list.

Example of the compound primary key:

evaluate:
    dimension: [source.ip, destination.ip]

If exactly one dimension like DestinationHostname is a list in the original event and the correlation should happen for each one of the dimension values, the dimension should be wrapped in [ ]:

evaluate:
    dimension: [source.ip, destination.ip, [DestinationHostname] ]

Item by

Specified the name of the field of the input event that contains a date/time information, which will be used for evaluation. Default is: @timestamp.

Item event_count (optional)

Name of the attribute, that specifies the count for correlation within one event, hence influencing the "sum of events" in analysis. Defaults to 1.

Item resolution

Specifies the resolution of the time aggregation of the correlator. The unit is seconds.

evaluate:
    resolution: 3600  # 1 hour

Default value: 3600

Item saturation (optional)

Specifies the duration of the silent time interval after the trigger is fired. It is specific for the dimension. The unit is resolution.

Default value: 3

Section analyze (optional)

The section analyze contains the configuration of the time window that is applied on the input events. The result of the time window analysis is subjected to the configurable test. When the test is successful (aka returns True), the trigger is fired.

Note: The section is optional, the default behavior is to fire the trigger when there is at least one event in the tumbling of the span equals 2.

Item when (optional)

Specifies when the analysis of the events in the windows should happen.

Options:

  • event (default): Analysis happens after an event comes and is evaluated, usually useful for match and arithmetic correlation
  • periodic/...: Analysis happens after a specified interval in seconds, such as periodic/10 (every 10 seconds), periodic/1h (every 3600 seconds / one hour) etc. Usually useful for UEBA evaluation.

Periodic analysis requires the time window resolution and span to be set properly, so the analysis does not happen too often.

Item window (optional)

Specifies what kind of time window to use.

Options:

  • tumbling: Fixed span (duration), non-overlapping, gap-less contiguous time intervals
  • hopping: Fixed span (duration), overlapping windows contiguous time intervals

Default value: hopping

Item span

Specifies the width of the window. The unit is resolution.

Item aggregate (optional)

Specifies what aggregation functions to be applied on events in the window.

Aggregate functions

  • sum: Summation
  • median: Median
  • average: The (weighted) average
  • mean: The arithmetic mean
  • std: The standard deviation
  • var: The variance
  • mean spike: For spike detection. The baseline is mean value, return the percentage.
  • median spike: For spike detection. The baseline is median value, return the percentage.
  • unique count: For a unique count of the event attribute(s): dimension has to be provided.
  • distance: Behavior depends on analyze.dimension: for a geo point field (e.g. source.geo.location), computes geographic distance between locations in the window (meters)—typical for impossible travel. For a numeric field, computes cumulative path length between successive samples. See below.
  • velocity: Measures rate of change of a numeric value over time in the window. Requires dimension in analyze naming the numeric field to track. See below.

Default value: sum

distance and velocity

distance

Set analyze.dimension to the field whose dynamics you want to measure:

  • Geo point (geo_point in ECS, e.g. source.geo.location): The aggregate is a geographic distance in meters (great-circle distance between locations observed in the window—e.g. successive authentications from different places). Use in evaluate.dimension the entity to correlate on (often user.name or user.id), set predicate so the geo field exists, and tune resolution and span to define how long a “short window” is. Compare analyze.test to a threshold in meters (e.g. 1000000 for 1000 km) for impossible travel–style rules. If geo-IP is coarse (country-level only), consider complementary detections (e.g. different countries in a short interval) instead of distance-only logic.

  • Numeric field (e.g. a counter or measurement): Aggregates the total absolute change between consecutive samples within the window (path length). Large distance means the value moved a lot between samples, even if it returned near the start—useful for volatility in metrics.

velocity

Applies to numeric fields only: aggregates how quickly the value changes over the window (rate of change). Use for burst or ramp detection on numeric series.

Compare with sum, which adds event weights or counts: distance on geo points measures spatial separation; distance on numerics and velocity describe dynamics of a scalar series.

Example (impossible travel, distance on geo point; threshold in meters):

evaluate:
    dimension: [user.name]
    by: "@timestamp"
    resolution: 300  # 5 minutes

analyze:
    window: hopping
    aggregate: distance
    dimension: source.geo.location
    span: 2  # 10 minutes of cells (2 × resolution)
    test:
        !GT
        - !ARG
        - 1000000  # e.g. 1000 km

Example (numeric path length: trigger when cumulative movement of network.bytes in the window exceeds a threshold):

analyze:
    window: hopping
    aggregate: distance
    dimension: network.bytes
    span: 10
    test:
        !GE
        - !ARG
        - 1000000

Example of the unique count:

analyze:
    window: hopping
    aggregate: unique count
    dimension: client.ip
    span: 6
    test:
        !GE
        - !ARG
        - 5

Trigger when 5 and more unique client.ips are observed.

Item test (optional)

The test is an expression that is applied on the output of the aggregate calculation. If the expression returns True, the trigger will be fired if a dimension is not already saturated. If the expression returns False, then no action is taken.

Other returned values are undefined.

Top-level test (rule fixtures)

The test inside analyze is the condition above. A separate, optional top-level test section in the rule YAML (with named cases, input events, and expected output) is used to validate the rule with synthetic data. See analyst: Test (optional, for rule validation).

Section trigger

The trigger section specifies what kinds of actions to be taken when the trigger is fired by test in the analyze section. See correlator triggers chapter for details.