Workflow
Apache Kafka - Example Use Cases
Overview
This Pre-Built Automation bundle contains example use cases that are applicable when Itential Automation Platform is integrated with Apache Kafka using the kafka-node npm library. Because every environment is different, these use cases are fully functioning examples that can be easily modified to operate in your specific environment. These workflows have been written with modularity in mind to make them easy to understand and simple to modify to suit your needs.
Example Workflows
Name | Overview |
---|---|
Consume Message - Kafka - Library - Example | This example use case consumes Kafka messages on a given topic and triggers a job from an IAP Operations Manager Event Trigger along with passing message data to that job's context. The Apache - Kafka - Library Pre-Built provides a modular worfklow that writes to a topic in Apache which then writes to a message queue in IAP's Event System. In order to subscribe to the relevant event topic, one must configure the adapter-kafka service configuration's topic properties correctly. The following topic properties is an example of setting the adapter-kafka service configuration so that the adapter subscribes to the Kafka event topic
Note when a new topic is set in the service configuration, a new topic is written to file in the adapter-kafka directory in the file On setting the adapter-kafka service configuration as shown above, navigate to the To then produce a message to the topic set, run the modular workflow from the Apache - Kafka - Library Pre-Built
Upon the Note the Trigger
|
For further technical details on how to install and use this Example Project, please click the Technical Documentation tab.
Table of Contents
Getting Started
This section is helpful for deployments as it provides you with pertinent information on prerequisites and properties.
Helpful Background Information
Workflows and processes often include logic that varies from business to business. Our Example Projects are more complex processes that include several of our modular components to build a more complete process.
While these can be utilized, you may find more value in using them as a starting point to build around.
Prerequisites
Itential Example Projects are built and tested on particular versions of IAP. In addition, Example Projects are often dependent on external systems and as such, these Example Projects will have dependencies on these other systems. This version of Apache - Kafka - Library - Example has been tested with:
- IAP 2023.2
External Dependencies
Name | OS Version | API Version |
---|---|---|
Apache Kafka |
Adapters
Name | Version | Configuration Notes |
---|---|---|
adapter-kafka | ^0.5.0 |
How to Install
To install the Example Project:
- Verify you are running a supported version of the Itential Automation Platform (IAP) as listed above in the Supported IAP Versions section in order to install the Example Project.
- Import the Example Project in Admin Essentials.
Testing
Cypress is generally used to test all Itential Example Projects. While Cypress is an opensource tool, at Itential we have internal libraries that have been built around Cypress to allow us to test with a deployed IAP.
When certifying our Example Projects for a release of IAP we run these tests against the particular version of IAP and create a release branch in GitLab. If you do not see the Example Project available in your version of IAP please contact Itential.
While Itential tests this Example Project and its capabilities, it is often the case the customer environments offer their own unique circumstances. Therefore, it is our recommendation that you deploy this Example Project into a development/testing environment in which you can test the Example Project.
Using this Example Project
Example Projects contain 1 or more workflows. Each of these workflows have different inputs and outputs.
Consume Message - Kafka - Library - Example
This example use case consumes Kafka messages on a given topic and triggers a job from an IAP Operations Manager Event Trigger along with passing message data to that job's context.
The Apache - Kafka - Library Pre-Built provides a modular worfklow that writes to a topic in Apache which then writes to a message queue in IAP's Event System.
In order to subscribe to the relevant event topic, one must configure the adapter-kafka service configuration's topic properties correctly.
The following topic properties is an example of setting the adapter-kafka service configuration so that the adapter subscribes to the Kafka event topic kafka-example-topic
, does not filter messages based on any text, and subscribes as well to the IAP Event System topic iap-example-topic
:
"topics": [
{
"name": "kafka-example-topic",
"always": true,
"subscriberInfo": [
{
"subname": "default",
"filters": [],
"rabbit": "iap-example-topic",
"throttle": {}
}
]
}
]
Note when a new topic is set in the service configuration, a new topic is written to file in the adapter-kafka directory in the file .topics-<kafka-adapter-name>.json
. If the Consume Message - Kafka - Library
worfklow is not being triggered, this file may not have yet been updated with the expected topic.
On setting the adapter-kafka service configuration as shown above, navigate to the Consume Message - Kafka - Library - Example
Operations Manager automation, click on the Trigger Consume Message - Kafka - Example - Trigger
and select under Event
the topic that was set in the adapter service configuration. The event topic based on service configuration above would be iap-example-topic - @itentialopensource/adapter-kafka
.
To then produce a message to the topic set, run the modular workflow from the Apache - Kafka - Library Pre-Built Produce Message - Kafka - Library
. To produce a message to the topic configured above, provide inputs such as the following:
{
"adapterId": "kafka",
"topic": "kafka-example-topic",
"messages": [
"This is a message sent to topic kafka-example-topic and will then be picked up eventually by IAP's Event System in the topic iap-example-topic"
],
"suppressMessage": true
}
Upon the Send Message
task completing in this workflow, there will be a job started from the workflow Consume Message - Kafka - Library
of this Pre-Built that can be viewed in Operations Manager in the Jobs panel.
Note the Trigger Consume Message - Kafka - Example - Trigger
defaults to the kafka - @itentialopensource/adapter-kafka
event topic, which would correspond to the following adapter-kafka service configuration:
"topics": [
{
"name": "kafka-example-topic",
"always": true,
"subscriberInfo": [
{
"subname": "default",
"filters": [],
"rabbit": "kafka",
"throttle": {}
}
]
}
]
Capabilities include:
- Trigger a workflow on consuming a Kafka message from event topic written to by an adapter-kafka sendMessage call
Entry Point IAP Component
The primary IAP component to run Consume Message - Kafka - Library - Example is listed below:
IAP Component Name | IAP Component Type | Consume Message - Kafka - Library - Example | Operations Manager Automation |
---|
Inputs
There are no inputs for Consume Message - Kafka - Library - Example.
Outputs
There are no outputs for Consume Message - Kafka - Library - Example.
Query Output
There are no query output examples for Consume Message - Kafka - Library - Example.
Example Inputs and Outputs
No example inputs or outputs found.
API Links
API Name | API Documentation Link | API Link Visibility |
---|---|---|
Apache Kafka API Documentation | https://kafka.apache.org/documentation/#gettingStarted | Public |
IAP Operations Manager Event Trigger Overview | https://docs.itential.com/docs/triggers-2023-1#event-triggers | Public |
IAP Event System Overview | https://docs.itential.com/docs/event-system-4 | Public |
Itential adapter-kafka Properties Configuration | https://gitlab.com/itentialopensource/adapters/notification-messaging/adapter-kafka/-/blob/master/PROPERTIES.md?ref_type=heads#adapter-properties-and-descriptions | Public |
Additional Information
Support
Please use your Itential Customer Success account if you need support when using this Workflow Project.