KAS: um framework de autoadaptação de parâmetros para o Kafka Streams

Abstract

The proliferation of internet-connected applications and Internet of Things (IoT) devices has led to an exponential increase in data volumes requiring real-time processing. Data Stream Processing systems, such as Kafka Streams, emerge as a solution for processing this data continuously and with low latency. However, prolonged execution under dynamic conditions, including variations in workloads, input throughput, and resource availability, often results in performance degradation due to suboptimal initial configurations. To address this challenge, this work describes how Kafka Streams works internally, identifies the most relevant parameters for performance and proposes a self-adaptive framework for Kafka Streams that dynamically adjusts system parameters in response to changing operational conditions. The Kafka Adaptive Streams (KAS) framework leverages self-adaptive rules derived from Apache Kafka’s documentation and source code to monitor system metrics, compute optimal parameter values, and apply reconfigurations aimed at improving latency, throughput, and resource efficiency. To validate these rules, the efficiency of a system without self-adaptation is compared to that of a selfadaptive system subjected to a workload greater than its processing capacity. Experimental validation under controlled workloads demonstrates that the KAS framework allows for an increase in consumer throughput of at least 64% compared to static configurations, confirming its effectiveness in maintaining high performance in dynamic environments. The findings contribute to the advancement of adaptive stream processing systems, offering a scalable solution for real-time data processing in evolving operational contexts.