US national wireless service provider had large amount of daily data coming from a cableÂ modemÂ termination system equipment. The client had two-step requirement for the network traffic analysis:
- Analyze if the daily data is valuable for business custom analysis,
- If it is, propose solution for data extraction and make it available for further ETL processes.
- The development team provided detailed and deep understanding of IPDR protocol format in order to get familiar and define correct formatting;
- Secured data transfer to S3 bucket;
- Provided solution has failover workload where errors in processing or missing files are caught and retry mechanism is introduced;
- Define specific roles with at least privileges to run the AWS services and the scripts;
- Define CI/CD process that will include several teams: developers, QA and DevOps;
- Establish and transfer the knowledge for using CloudFormation specific deployment.
InterWorksâ€™ analysis and development team proposed and built two-step solution, providing:
- First phase: Information that data is valuable and has significant meaning for networks traffic analyses on daily, monthly and yearly levels.
- Second phase: Optimized and secured data transfer serverless solution that provides readable data on Redshift cluster for further ETL analyses.
Our solution is based on Amazon Web Services serverless Lambda functions that are processing incoming IPDR data files. Incoming files are saved in data lake and reside in several S3 buckets, based on different processing stages providing long term storage of network data. Processed data is available on Redshift as warehouse suitable database for further ETL analyses.
Solution is built under custom secured VPC, deployed in separate public, private and data subnets. Assigned execution roles and policies have minimum required permissions.
Using CloudFormation templates the solution can be quickly deployable to different AWS accounts or as different environments in the same AWS account.
Benefits and Results
- The client gets a low-cost reliable solution running entirely on AWS infrastructure making it fully scalable and upgradable at any moment.
- Easy integration with third party systems allowing usage of various notification and monitoring tools that provide efficient support process.