Breaking Through the Challenges With Data Ingestion at Scale

Client Background

A renowned American Company leads in technology and communication services, providing voice, data, and video solutions through its highly regarded networks and platforms. It meets customers'dependable network connectivity, security, mobility, and control needs.

The organization's responsible business strategy aims to drive economic, social, and environmental progress. The company pioneered commercial 5G for mobility, fixed wireless, and mobile edge computing. Its operating model centers on two customer-oriented domains: Consumer and Business.

Project Outcome

The Telecom Giant experienced a significant reduction in operating costs and a corresponding increase in the number of files to be processed per day. After implementing the custom IoT solution built on Flex83 Middleware, the company scaled from 5000 files per day to a massive 40,000+ files per day. The project goals were achieved at one-third of the expected time of completion.

Introduction

A Fortune 500 company with over 100,000 employees and several global establishments needed help in scaling their application related to processing files consisting of geospatial network signal data and improving their cellular networks. They were required to collect numerous data packets (at milliseconds granularity) through the signals corresponding to each location using their devices or network scanners, creating 500mb-1gb-sized files. It impeded the application's processing capabilities, and they could not fully utilize their hardware resources, negatively influencing the overall operating costs.

To address these issues, the company partnered with IoT83 Ltd. to effectively augment the file processing technique.

Technical Challenges

The company encountered challenges in efficiently scaling the processing of critical data files—such as DML_DLF, DLF, and SIG—collected from tower network scanners and other signal-capturing devices. Despite the large volume of data, including millions of signal readings per file, the existing infrastructure was underutilized, leading to inflated operational costs. To address this, the company aimed to process these high-density files using a Spark-based distributed computing system.

Steps Involved in the Workflow

Stage 1:

Device-generated files in formats such as DML_DLF, DLF, and SIG/Scanner were initially transformed into a spark compatible format. Leveraging Spark and custom parsing logic, each file—often containing millions of records—was converted into a sequence file format. However, due to the intensive processing involved, this stage could only handle 250–300 files per hour.

Stage 2: 


The previously generated sequence files were further processed through a robust parsing layer that mapped encoded data fields to meaningful business-specific values. This transformation made the data suitable for advanced geospatial processing in the next phase. The processed data was then stored using Hive, enabling schema definitions and facilitating standardized querying through SQL.

Stage 3: 


The analyzer stage involves reading data from Hive tables and executing thousands of business logic operation son each individual record across all files. This stage performs complex aggregations and joins across multiple datasets, ultimately producing an output of approximately 300,000 to 700,000 records for every batch of 30–60 files. The results are indexed and stored in Elasticsearch for efficient querying and analysis.

Challenges in the Workflow

The client could process only 5000–7000 files per day. Here, the average file size was around 250–400 Megabytes, which means the total Amount of data transferred every 24 hours was between 1.2 to 2.8 Terabytes. The company wanted to scale up this number to 40,000 files per day. It required a decent infrastructure, code optimization, and spark Parameters/config tunings that could process 16 Terabytes every 24 hours.

The Solution

The company partnered with IoT83 to significantly scale and enhance its existing Spark-based file processing system. The team at IoT83 leveraged the Flex83 Middleware to accelerate the custom IoT solution development, utilizing pre-built microservices to streamline the process while enabling the client to maintain full ownership of the application IP.