Client Overview
Global Provider of HR & Financial, Health & Wealth Solutions with over 25 years of experience
Business Need
Client is redesigning their solutions platform using the next generation architecture. They wanted to develop a Log Aggregation Framework that could monitor & detect failures ahead of time. The framework should
- Capture distributed microservices logs into a centralized location
- Acquire data from diverse set of data sources (UI, database, microservices)
- Provide capability of visualize and search desired logs via GUI
- Analyze log patterns to determine different workflows of application
- Identify failure pattern, exports and sends report to respective teams
Key Features
- ELK (Elasticsearch-Logstash-Kibana) based Log Processing framework with key components including
- Logstash – a server-side data processing pipeline that ingests data from a multiple sources simultaneously, transforms it, and sends it to Elasticsearch
- Elasticsearch - a distributed analytics engine to manage data acquired from Logstash
- Kibana
- Provides visualization on the data
- Monitors ELK nodes – Node health, request/response time etc.
- Email alerts on predefined conditions
- High Availability – Maximizing uptime of ELK nodes via internal and external load balancers
- Security - SSL/OpenSSL encryption across each communication between ELK nodes
- HDFS – Compressed raw data backup for future use
Benefits
Secure centralized Log Processing Framework based on ELK stack with powerful visualization and search functionalities enabling them to deliver high availability through
- Real time monitoring of core micro services
- Real time access to failures leading to faster turnaround time
- Integrated framework to analyze failures/errors from diverse applications through a single interface
