Streaming Analytics for Mobile Network
The ability to analyze data in real time for mobile network is crucial for diagnostics and ensuring the quality of the service for end customers. To achieve this we have built a real-time ingestion and analytics platform that processes 2.2 billions messages a day from mobile networks hardware. During the talk we will show how we used Flink and Flink SQL to build this platform. The solution includes calculation of more than 5000 KPIs and 1500 aggregation defined in SQL, on 750 Kafka topics. We will describe how we manage Flink jobs at scale using Ververica and Kubernetes, how we monitor the platform using Clickhouse and what problems we need to overcome in the project.
View this presentation to learn:
- How to minimize data processing latency when we talk about almost 100 mln events per hour from 750 different data sources with around 500 thousands devices?
- How to provide a data analytics solution that would be able to standardize data models, perform aggregations and compute KPI for end users?
- Can Apache Flink be a main technology used in the infrastructure to ingest, process data streams and load data for future analysis?
This is the performance given by GetInData | Part of Xebia and NetWorks! during the Big Data Technology Warsaw Summit