Big companies, especially those that are merged between two or more companies are always deciding to use integration because of its main benefits such as rapid development, big data throughputs, scalability and many more. How much the projects are increasing, their monitoring and being notified about everything is becoming a nightmare. Notifications everywhere, most of them are going in your Junk folder because they are useless to you, but you are too busy to turn them off or restructure them and all is becoming a one big mess.
SnapLogic has a good notification handled solution on a pipeline level where notifications are based on some rules and are sending notification emails to a predefined list of email addresses. But, as I said, if we have a high volume system of multiple integrations, well packed with a framework, error handling, logging etc. and have a complex architecture then sending notifications for every single pipeline will get you a full inbox and personal ignorant attitude to this type of emails which will result in losing the main purpose of these notifications.
We, as integration developers, faced many of these difficulties and for that matter we decided to develop a unified monitoring solution for SnapLogic specific integrations. Below I will present the tool in a short-described version, but you can always email us for more detailed explanation, even a demo of it.
A Unified Monitoring Solution for SnapLogic
With only using a JSON configuration file where you are defining list of parameters that are needed for monitoring your environment/integration/business logic and etc, the person that will monitor, who won’t need to be an IT person, will do their complete job.
Figure 1: JSON configuration file
The file above contains parameters which are defining the logic by which the process is deciding what should monitor. Dates parameters are defined so the tool can make a call to the SnapLogic REST monitoring API and look for historic executions for a predefined time period. In the pipeline_name parameter we are defining a list of pipelines which we want to monitor, the recipient_list is the list of the email addresses that will receive a notification (actually a whole report) for what happened and in environment we are defining the organization on SnapLogic level where the pipelines are located.
In order to add more value to all this, we made a custom Client specific business logic to the process because we thought that checking only SnapLogic logs is not enough for a complete monitoring tool. Almost every integration project is logging in a DB for what happened. So, in order to be profound and useful, we are adding a logic where the user will define DB account and query a table by which the tool can decide if a Completed transaction in SnapLogic is really completed by Business rules, or it should be in failed state. This will help and add value to the non IT people who will receive the reports, to know what really happened in their system.
Figure2: Monitoring tool in SnapLogic
After we defined the JSON configuration file and did some data transformation of it, the next step is to call the REST Monitoring API provided by SnapLogic with a REST Get snap. For more information how this API should be invoked, visit SnapLogic Documentation portal where everything is very well explained in details.
Figure 3: REST Get snap calling SnapLogic Monitoring API
Following to the set of results we are getting, based on the API response, we are routing the status of the pipelines according to their status of execution. Processes with failed status are directly sent to the report, but completed ones are sent to another set of validation logic where it will be decided whether the integration is really completed successfully or not.
As I said earlier, we are sending a query and account in the configuration file by which we will check if the pipeline completed its execution properly. That is a centralized sub pipeline where we are using the query to get the real status of the pipeline execution.
Figure 4: Pipeline execute snap calling sub pipeline with query business logic
After we have all the results we need, we are making some XSLT transformations in order to get a good looking report that will be understandable for the end user.
Example of XSLT transformation that we use for creating the failed table in the report:
The end user will receive an email notification with content like the following report:
Figure 5: Email report sent to the end users from the Monitoring tool
This email report is listing multiple information that is useful for the end user who will use this tool. Of course, this info can be edited based on the end user needs. The monitoring tool can be exposed as an API and called from any pre-developed application where the end user will trigger the monitoring tool by clicking some button in on an UI or it can be scheduled to run on some predefined interval and generating results, all this depends on the project and user demands.
We all know what are the main benefits of continuous monitoring to the environment and measuring system performance. With this tool, end users will get more insights of what is happening in the system, what they can do to improve something, measuring capacity and similar. Note that this is only one way of implementing a monitoring tool in SnapLogic, there are plenty of possibilities for improving this. So, if you have one, we will be more than happy to discuss about it with you.