Event Pattern Authoring

The majority of pattern detection logic is configured in the event pattern view. In this environment, you define how to find matches from incoming data via elements, either by themselves, or in relation with other elements by means of constraints. Once a pattern is detected, outcomes are used to produce output data based on the input data or computational results of the input data.

Working with Elements

An element is a unit of pattern detection logic that defines what parts of the incoming data constitute a match. In other words, elements tell the system what kinds of things to look for in the data. For more information about elements, see Model Artifacts.

Creating Elements

To create an element:

1. In the tool palette, click the Element (⬭) icon.
2. Click the event type to associate with the new element. Click New Event Type to create new event types, and then follow the instructions in Creating New Event Types.
3. Click any open space in the diagramming area to place the element.
4. In the details panel, edit the element’s properties as needed. For more information, see Editing Elements.

Elements may also be created directly on the canvas as follows:

1. Right-click any empty space in the canvas.
2. In the selection menu, click Add Element.
3. In the dialog, select an event type. To create a new event type, click New, then follow the instructions in Creating New Event Types.

note

Depending on the source of the input data, elements in the event pattern view may have differently shaded colors:

• An element associated with a raw event type is deep blue.
• An element associated with a derived event type (that is, one that comes from an output of a lower-level event pattern) is brighter blue.

Editing Elements

When an element is selected in the event pattern view, its information is displayed in the details panel, where the element's details can be viewed and modified as needed.

To edit an element:

1. In the diagramming area, click the element to edit.
2. In the details panel, click the Contents tab if it is not already selected.
3. Edit the element's details as necessary:
   1. Under Attributes:
      1. From the Event Types dropdown menu, select the event type to modify the event type associated with the element.
      2. In the Alias field, enter an alias for labeling the element. (It is recommended that the alias describe the element's unique role in its event pattern that differentiates it from other elements using the same event type.)
      3. In the Min field, specify the minimum number of events the element must receive.
      4. In the Max field, specify the maximum number of events the element must receive.
      5. Under Time to Live, define an expiration period for the element (i.e., how long it should exist in the system before being discarded) if needed:
         1. Click +. The Create Time-To-Live dialog opens. (The Create Time-To-Live dialog is also accessible by right-clicking an element and selecting Add Time to Live from the context menu.)
         2. In the Time Value field, enter a number indicating how long the element should remain valid.
         3. From the Unit Type dropdown menu, select the appropriate unit of time for the specified time value.
         4. Check the Apply to Matched Events checkbox to drop any matched records after the configured time-to-live period. Leave it unchecked to let matched records persist after the configured time-to-live period.
         5. Click Create to create the expiration period, or Cancel to discard it.
         6. Edit the expiration period by changing the field values, if needed.
         7. Click the Delete (trash can) icon to clear the entered values, if necessary.
   2. Under the Event Type section (which uses the name of the event type selected for the element):
      1. Click the pencil (✎) icon to edit the details.
      2. Follow the procedure in Editing Event Types.

Working with Outcomes

An outcome is a unit of pattern detection logic that represents output data. In other words, once the system finds a pattern established in an element, an outcome tells the system what to do after that discovery. For more information about outcomes, see Model Artifacts.

Creating Outcomes

To create an outcome:

1. In the tool palette, click the Outcome (▭) icon.
2. Select the event type to associate with the new outcome. If a new event type should be created for the outcome, follow the process in Creating New Event Types.
3. Click any open space in the diagramming area to place the outcome.
4. In the details panel, edit the outcome’s properties as needed. For more information, see Editing Outcomes.

Outcomes may also be created directly on the canvas as follows:

1. Right-click any empty space in the canvas.
2. In the selection menu, click Add Outcome.
3. In the dialog, select an event type. If a new event type should be created, click New, then follow the instructions in Creating New Event Types.

Editing Outcomes

When an outcome is selected in the event pattern view, its information is displayed in the details panel, where the outcome's details can be viewed and modified as needed.

To edit an outcome:

1. In the diagramming area, click an outcome to edit.
2. In the details panel, click the Content tab if it is not already selected.
3. Edit the outcome’s details as needed:
   1. Under Attributes:
      1. Under Sources, click a source to edit its details. For more information, see Editing Elements.
      2. Under Computations, click a computation to check its details. Click the Delete (trash can) icon beside a computation to delete it. For more information about computations, see Built-In Computation Functions.
      3. Under Risk, choose the appropriate risk type:
         1. If the risk factors (elements) are independent from one another, select Weighted to assign a weight to each element.
         2. If the risk factors (elements) are dependent upon one another, select Bayesian to specify the conditions used to compute risk.
      4. Under Batch Output, click the Enable Batch Output toggle to enable batch output if desired. (By default, the toggle is set to "Off.") For more information, see Configuring Batch Output.
   2. Under the Outcome section (which uses the name of the selected outcome):
      1. Click the pencil (✎) icon to edit the details.
      2. Follow the procedure in Editing Event Types.

Understanding Risk Types

Cogynt models can calculate risk based on two different approaches: weighted risk and Bayesian risk.

Risk types and weights are assigned when editing outcomes.

Using Weighted Risk

Weighted risk is used when the probability (weight) of elements or risk factors does not change based on the presence or absence of other risk factors. In other words, weighted risk is used for independent risk factors.

tip

Suppose you want to calculate a population's cancer risk based on a number of personal and environmental factors, such as whether they smoke, whether they work with toxic chemicals, and so on. Since the risk of each factor remains more or less static, weighted risk would be useful for this model.

Using Bayesian Risk

Bayesian risk is modeled after the Bayes' theorem. It is used when the probability (weight) of elements or risk factors can change based on the presence or absence of other risk factors. In other words, Bayesian risk is used for dependent risk factors.

tip

Suppose you want to calculate the risk of a wildfire occurring in a particular region. The wildfire risk might increase if the region has seen a drought, and it might decrease if there has been an unusual amount of rain. In this case, Bayesian risk would be useful to model the wildfire probability.

Configuring Batch Output

"Batching" refers to an increment of time during which the system waits to publish any updates for an event pattern until the next increment begins.

Batch output defines the increment of time during which the system listens to incoming data streams, and only publishes the data's updated (latest) state acquired during that interval.

note

The batch output interval refers to server time (time elapsed during the operation of the server), not data time (time specified in the data's timestamps or metadata).

To configure batch output for event patterns:

1. In the diagramming area, click the desired event pattern's outcome.
2. In the details panel, click the Contents tab if not already selected.
3. Click the Batch Output section to expand it.
4. Click the Enable Batch Output toggle to set it to "On".
5. In the Time Value field, enter the interval of time that the output window should span.
6. From the Unit Type dropdown menu, select the appropriate unit for the time value.
7. Click outside of the details menu to save the changes.

Working with Constraints

A constraint is a condition placed upon an input event for an event pattern to be considered a match. For more information about constraints, see Model Artifacts.

Creating Constraints

To create a constraint:

1. In the diagramming area, click an element.
2. Click the Connection (🔀) button, then drag the line to another element.
3. In the Create a Constraint dialog:
   1. From the first field's dropdown menu (labeled with the name of the first element's event type), select the appropriate field name. If the list is too long, type the first few characters of the field's name to narrow down the list.
      • To use a value computed from the element's fields instead, select Custom Field at the top of the dropdown menu. For more information, see Custom Fields Authoring.
   2. From the Comparison Operator dropdown menu, select a comparison operator for the two elements.
   3. From the second field's dropdown menu (labeled with the name of the second element's event type), select the appropriate field name, or select Custom Field to use a computed value. (The second field cannot be selected until the first field is selected.)
   4. If available, expand Advanced Settings to configure the following as needed:
      1. In the Offset field, specify a value to add to the source event for the comparison. (For example, if you knew that all the source event's dates would be five minutes behind due to a server outage, you could use Offset to add five minutes to the source value.)
      2. In the Tolerance field, specify a value to add and subtract from the source event to create a range of valid values for comparison operations. (For example, if you knew that the source event's dates would be either five minutes ahead or five minutes behind, and wanted to cover all possible dates within that range, then you could use Tolerance to establish this as the acceptable range.)
4. Click Save to create the constraint, or click Cancel to discard it.

Selecting Constraints

Constraint options vary depending on the datatypes involved. For example, if a constraint is created between two scalar strings, the constraints are limited to "equal" and "not equal."

The available constraints are as follows:

• Contains – Checks whether one element is contained within the other.
• Count equal – Checks whether the number of elements in an array is equal to the specified number.
• Count greater than – Checks whether the number of elements in an array exceeds the specified number.
• Count greater than or equal – Checks whether the number of elements in an array exceeds or matches the specified number.
• Count less than – Checks whether the number of elements in an array is less than the specified number.
• Count less than or equal – Checks whether the number of elements in an array is less than or equal to the specified number.
• Count not equal – Checks whether the number of elements in an array does not match the specified number.
• Equal – Checks whether both elements have the same value.
• Greater than – Checks whether one element represents a greater number than the other.
• Greater than or equal – Checks whether one element represents a greater or equivalent number than the other.
• In network – Checks whether one element exists in the same network as the other.
• Inside – Checks whether one element is situated inside the other.
• Less than – Checks whether one element represents a lesser number than the other.
• Less than or equal – Checks whether one element represents a lesser or equivalent number than the other.
• Not equal – Checks whether the elements are not the same.
• Outside – Checks whether one element is situated outside the other.

The possible datatypes are as follows, and can be either scalars or arrays:

• Boolean – A true or false value.
• Date/Time – A datatype that represents a date and time together as a single data unit.
• Float – A floating point number (decimal value or fractional number).
• Geo Coordinate – A GeoJSON-formatted pair of numerical coordinates indicating a single geographical point. For more information about using this datatype, see Working with GeoJSON Datatypes.
• Geo Polygon – A GeoJSON-formatted set of coordinate pairs describing a geographical area. For more information about using this datatype, see Working with GeoJSON Datatypes.
• Integer – A whole number, such as 25 or -25.
• IP – A unique address that identifies a device on the internet or a local network.
• IP Range – A set of consecutive IP addresses.
• JSON – A set of objects consisting of attribute-value pairs and arrays.
• Null – An empty value.
• String – A sequence of (often alphabetical) characters.
• Unique ID (UUID) – An alphanumeric string for labeling information in a computer system.
• URL – An address for a unique resource on the Internet.

For certain Cogynt builds, several datatypes beginning with the prefix COG_ may be available. These datatypes represent Cogynt-level metadata or extra features that invoke particular Cogynt behaviors. The COG_ datatypes include:

• COG_attachments: Metadata for files that are manually attached to events via external submission mechanisms. (For example, if your Cogynt setup is designed to accept incident reports from an external interface, COG_attachments collects an attached incident report's metadata.) The metadata is passed into HCEP as a JSON object. The attached file itself is then viewable in Workstation.
• COG_confidence: The place where risk scores are published. Manually mapping values here allows bypassing risk tables if desired.
• COG_filter: A Boolean field that, if it receives True, publishes the event to the output topic. If it receives False, it suppresses the event instead.
• COG_id: A unique identifier for items. COG_id only exists in input events, and cannot be overwritten from the UI.
• COG_matches: The behavior of COG_matches varies depending on whether it is an input or output element:
  • As input: If the element or event type has a lexicon filter on it, then COG_matches produces an array of words that matches the lexicon. Similarly, for matched nodes, it lists the name of the lexicon node.
  • As output: If mapped to an outcome, words indicating lexicon matches are highlighted when shown in Cogynt Workstation.
• COG_timestamp: An optional field. Datetime values can be mapped to COG_timestamp. When ingested, Cogynt Workstation interprets the mapped value to determine the timestamp used for denoting events in chronological order. (COG_timestamp has no impact on how Authoring patterns behave.)

The constraints available for the various datatype combinations are outlined in the following tables.

note

Only valid datatype pairings are listed in the tables. If a pairing is not listed, then there are no constraints available for it.

The same constraints are available regardless of which datatype is the first and which is the second in the pairing.

Scalar-to-Scalar Constraints

First Datatype	Second Datatype	Available Constraints
Boolean	Boolean	equal not equal
Date/Time	Date/Time	equal not equal less than less than or equal greater than greater than or equal
Float	Float	equal not equal less than less than or equal greater than greater than or equal
Geo Coordinate	Geo Coordinate	equal not equal
Geo Coordinate	Geo Polygon	inside outside
Geo Polygon	Geo Polygon	equal
Integer	Integer	equal not equal less than less than or equal greater than greater than or equal
IP	IP	equal not equal in network
IP Range	IP	equal not equal in network
IP Range	IP Range	equal not equal in network
JSON	JSON	equal not equal
String	String	equal not equal
Unique ID	Unique ID	equal not equal
URL	URL	equal not equal

Array-to-Array Constraints

First Datatype	Second Datatype	Available Constraints
Geo Coordinate	Geo Coordinate	equal not equal
Geo Coordinate	Geo Polygon	inside outside
Geo Polygon	Geo Polygon	equal
JSON	JSON	equal not equal

Array-to-Scalar Constraints

First Datatype (Array)	Second Datatype (Scalar)	Available Constraints
Boolean	Boolean	contains
Boolean	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
Date/Time	Date/Time	contains
Date/Time	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
Float	Float	contains
Float	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
Geo Coordinate	Geo Coordinate	equal not equal
Geo Coordinate	Geo Polygon	inside outside
Geo Polygon	Geo Coordinate	inside outside
Geo Polygon	Geo Polygon	equal
Integer	Integer	contains count equal count not equal count less than count less than or equal count greater than count greater than or equal
IP	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
IP	IP Range	contains
String	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
String	String	contains
Unique ID	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
Unique ID	Unique ID	contains
URL	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
URL	URL	contains

Working with GeoJSON Datatypes

GeoJSON-formatted datatypes must adhere to the following conditions.

GeoJSON Formatting

A valid GeoJSON object must be represented in either Geometry or Feature style.

Geometry Style

A GeoJSON object in Geometry style must contain two fields: type and coordinates. The following are examples of valid GeoJSON objects in Geometry style:

"my_point": {
    "type": "Point",
    "coordinates": [125.6, 10.1]
}

"my_polygon": {
    "type": "Polygon",
    "coordinates": [
        [
            [0, 0], [10, 10], [10, 0], [0, 0]
        ]        
    ]
}

Feature Style

Feature style is a wrapper of Geometry style. It is designed to provide additional information.

On the root layer, a valid Feature-style GeoJSON object must contain three fields: type, geometry, and properties. The following are examples of valid GeoJSON objects in Feature style:

"my_point": {
    "type": "Feature",
    "geometry": {
        "type": "Point",
        "coordinates": [102.0, 0.5]
    },
    "properties": {
        "some_prop": "some_value"
    }
}

"my_polygon": {
    "type": "Feature",
    "geometry": {
        "type": "Polygon",
        "coordinates": [
            [
                [0, 0], [10, 10], [10, 0], [0, 0]
            ]            
        ]
    },
    "properties": null
}

Working with Geo Polygons

Geo polygons must be closed (i.e., the first and last coordinate listed in the coordinates field must be the same).

Multi-polygons are not accepted, even though such constructs are valid GeoJSON forms.

Editing Constraints

To edit a constraint:

1. In the diagramming area, click a constraint to edit.
2. In the details panel, click the Contents tab if it is not already selected.
3. Under Fields, from the first field's dropdown menu, select the appropriate field name. If the list is too long, type the first few characters of the field’s name to narrow down the list.
   • To use a value computed from the element's fields instead, select Custom Field at the top of the dropdown menu. For more information, see Custom Fields Authoring.
4. From the Comparison Operator dropdown menu, select a comparison operator for the two elements.
5. From the second field's dropdown menu, select the appropriate field name, or select Custom Field to use a computed value.
6. If available, expand Advanced Settings to edit the Offset and Tolerance values.

Changes to a constraint are saved automatically.

Creating HTTP Parameter Mappings

note

Available only when one of the elements in a constraint references an HTTP or Cogynt AI event type.

Parameter mappings for Cogynt AI event types are created and edited in the same way as those for HTTP event types. The mapped input parameters are the ones defined in the Cogynt AI event type's Input Parameters section. For more information, see Cogynt AI Event Types.

To create an HTTP parameter mapping:

1. In the diagramming area, click an element.
2. Click the Connection (🔀) button, then drag the line to another element.
3. In the HTTP Parameter Mapping computation modal:
   1. Map input values from the joined element to input parameters of the HTTP element. For more information, see Custom Fields Authoring.
4. Click Save to save the parameter mapping, or click Cancel to discard it.

Editing HTTP Parameter Mappings

To edit an HTTP parameter mapping:

1. In the diagramming area, click on the constraint to edit.
2. In the details panel, click the Contents tab if it is not already selected.
3. Click the pencil (✎) icon to load up the HTTP Parameter Mapping computations modal.
4. Make the desired changes in the computation diagram.
5. Click Save to save the changes, or click Cancel to discard them.

Editing Groups

The only editable attribute of a group is its color.

note

Groups are a legacy feature in Cogynt, and no longer serve any necessary functions.

To change the color of a group:

1. In the diagramming area, click the group to edit.
2. In the details panel, click the Contents tab if it's not already selected.
3. Click the color bar above the Members section.
4. From the color palette, select the desired color.
5. Click anywhere outside the color palette to close it and save the change.

For more information about groups, see Model Artifacts.

Working with Group By

Group By instructs the system how to group the analysis of an event pattern's records. (In earlier versions of Cogynt, Group By was named "Partitions.")

tip

Consider a pattern that analyzes a stream of bank transactions from a bank, containing the magnitude of position or negative flow of cash value and the bank account to which it pertains. The pattern computes the balance for each account.

In this case, the pattern is grouped by the bank account field, so the system knows to compute a balance for each bank account, rather than treat the entire stream as a single bank account.

Choosing a Grouping Mode

Each event pattern has a grouping mode that determines whether the pattern maintains state, and if so, how that state is grouped.

To choose a grouping mode for an event pattern:

1. In the diagramming area, click an event pattern.
2. In the details panel, click the Contents tab if it is not already selected.
3. Under Attributes, click the Group By section to expand it.
4. Select the desired grouping mode:
   • No Grouping: Runs the pattern statelessly. The system assumes that every record has no state, and performs computations without maintaining state. This mode is optimal for running ETL processes.
   • Group All: Groups all results into a single shared state (or solution). When this mode is selected, a warning appears, because grouping all records into a single group may significantly reduce performance.
   • Group by Fields: Groups results by selected fields from the ingested records. Each group maintains its own state. For more information, see Creating Group By Fields.

note

If Group by Fields is selected but no groups have been defined, a warning icon appears beside the Group By section.

Creating Group By Fields

Group by fields define the groups that an event pattern's results are split into. They are available only when the grouping mode is Group by Fields.

To create a group by field for an event pattern:

1. In the diagramming area, click an event pattern to give a group by field.
2. In the details panel, click the Contents tab if it is not already selected.
3. Under Attributes, click the Group By section to expand it.
4. Select the Group by Fields grouping mode.
5. Click Add Group By Field. (If the event pattern already has groups, click Add Group instead.)
6. In the Create Group dialog:
   1. From the Element dropdown menu, select an element containing the field to group by.
      • To treat windows as their own discrete entities (by granting each window a unique ID within Cogynt), select the window (Sliding Window or Tumbling Window) instead. The window appears in the dropdown menu only if windowing is enabled. For more information, see Setting Windowing.
   2. From the Field dropdown menu, select the field to group by. (The Field dropdown menu cannot be opened until a selection is made from Element.)

      • To group by a value derived from the element's fields instead, select Custom Field at the top of the dropdown menu. A computations view appears, where you define the custom field:

        1. Add computation elements (such as functions, constants, and operators) to the diagram, and map the fields of the selected element to their inputs.
        2. Map the result of the computations to the output. The value produced at the output becomes the field that records are grouped by, in the same way that a field selected from the dropdown menu would.
        3. Click Save in the footer of the computations view to save the custom field, or click Cancel to discard it. The custom field appears in the Create Group dialog in place of the Field dropdown menu, where it can be edited or removed.

        Defining a custom field works the same way as defining a custom field in a constraint. For more information, see Custom Fields Authoring.

   3. If the selected field (or custom field) is a date and time, select the granularity to group by from the Time Granularity dropdown menu: Millisecond, Second, Minute, Hour, Day, Month, or Year. For example, select Month to group records by the month of their date and time.
      • Millisecond is selected by default.
      • For fields in the Cogynt Date format, only Day, Month, and Year are selectable, and Day is selected by default.
   4. If the selected field is an array, click the Split Array toggle to activate or deactivate array splitting. If set to "On," a unique group is created for each element in the input array.
7. Click Save to create the group, or click Cancel to discard it.

Editing Group By Fields

Existing group by fields can be modified as needed.

To edit a group by field:

1. In the diagramming area, click an event pattern that contains a group by field.
2. In the details panel, click the Contents tab if it is not already selected.
3. Under Attributes, click the Group By section to expand it.
4. Click the pencil (✎) icon beside the group to edit.
5. In the Edit Group dialog, edit the group details as needed. For more information about the group details, see Step 6 of Creating Group By Fields.
6. Click Save to save the changes, or click Cancel to discard them.

note

The Split Array toggle may be adjusted from the group itself, without having to click the pencil icon. For more information about splitting arrays, see Step 6 of Creating Group By Fields.

Deleting Group By Fields

Group by fields that are no longer necessary can be removed.

To delete a group by field:

1. In the diagramming area, click an event pattern that contains a group by field.
2. In the details panel, click the Contents tab if it is not already selected.
3. Under Attributes, click the Group By section to expand it.
4. Click the trash can (🗑) icon beside the group to delete.

Setting Windowing

"Windowing" is the process of splitting a stream of data into portions of finite size, which can then be subjected to computations. Windowing separates the data from an input stream into discrete sections, making it possible to process and calculate data over time. For more information, refer to Windows in the official Apache Flink documentation.

To add windowing to an event pattern:

1. In the diagramming area, click an event pattern to set windowing for.
2. In the details panel, click the Contents tab if it is not already selected.
3. Under Attributes, click the Windowing section to expand it.
4. From the Window Type dropdown menu, select the desired window type ("Sliding" or "Tumbling"). For more information about these windowing types, refer to Tumbling Windows and Sliding Windows in the official Apache Flink documentation.
5. From the Input Event Type dropdown menu, select the event type to use as the input.
6. From the Referenced Timestamp dropdown menu, select the timestamp for the window to use as its referencing timeframe.
7. If the Referenced Timestamp selection is anything other than "Kafka Ingestion Time", the Lateness field appears. In the Lateness field, specify the length and unit of time that the system should continue accepting and processing events from a window after it has closed.
8. In the Window Length field, specify the minimum length of time that the window should span.
9. In the Slide Length field, specify the length of time for each sliding window. (Note: This field appears only if Window Type is set to "Sliding".)
10. From the Unit Type dropdown menu to the right of the Minimum Length field, specify the unit of time for the Minimum Length value.
11. In the Timeout field, specify how long the system should wait before closing a window if no new input data is being received. (Note: The Timeout value should not exceed the Minimum Length value.)
12. From the Unit Type dropdown menu to the right of the Timeout field, specify the unit of time for the Timeout value.
13. Click outside of the details menu to save the changes.