About

The amount of data and the change in behaviours happens very fast in this interconnected world. These repeated changes and the amount of data make machine learning algorithms lose accuracy because they don’t know about these new patterns. This change in pattern of data is known as Concept Drift and there are already many approaches for treating these drifts. Usually these treatments are costly to implement because they require knowledge of drift detection algorithms, software engineering and need maintenance for new drifts.

The proposal of Driftage is build a framework using multi-agent systems to simplify the implementation of concept drift detectors for dynamic environments.

Driftage is a modular framework where:

Monitor

Captures data integrated to any framework you want: Spark, Flink, or even a Python function.

Analyser

Analyses data collected by Monitor using a customized Predictor for Concept Drift detection. Fast classifiers from Scikit-Multiflow or Facebook Prophet are great projects for that.

Planner

Observes for new predictions and based on them chooses if the drift is really valid. If it’s a real drift, it should be sent to the Executor. A custom Predictor can be done for that too, like a voting one or a more time-consuming algorithm from Scikit-Learn, TensorFlow, PyTorch, or others.

Executor

Receives from Planner new Drifts and sends them to a custom Sink. It can be a Apache Kafka, RabbitMQ, API, and so on…

Knowledge Base

Store data collected and predicted by Analyser. Can be queried by Analyser for retraining or Planner for predictions.

If you want to know how more how Driftage is structured, take a look at DataFlow and Sequence Diagram.