Automatic detection of data and concept drift in ML systems for chess broadcasting
As machine learning systems evolve, they face the challenge of operating in dynamic environments where model performance can degrade over time due to model drift. Therefore, the work continues once the model is in production; continuous monitoring and adjustment are essential to maintain optimal per...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2024 |
| País: | España |
| Institución: | Universitat Politècnica de Catalunya (UPC) |
| Repositorio: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | inglés |
| OAI Identifier: | oai:upcommons.upc.edu:2117/419868 |
| Acceso en línea: | https://hdl.handle.net/2117/419868 |
| Access Level: | acceso abierto |
| Palabra clave: | SQL (Computer program language) Machine learning deriva de dades deriva de conceptes codificadors automàtics aprenentatge automàtic SQL NoSQL data drift concept drift autoencoders machine learning SQL (Llenguatge de programació) Aprenentatge automàtic Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic |
| Sumario: | As machine learning systems evolve, they face the challenge of operating in dynamic environments where model performance can degrade over time due to model drift. Therefore, the work continues once the model is in production; continuous monitoring and adjustment are essential to maintain optimal performance. As a model receives new data, it might exhibit a different probability distribution than the data used for training, necessitating strategies to detect and address drift effectively. This thesis addresses these challenges within a chess broadcasting system developed by the GESSI research group. This system leverages machine learning for real-time chess game state detection. Despite its capabilities, it is not immune to data and concept drift challenges, which can diminish its effectiveness in production environments. Moreover, this work aims to enhance the efficiency and scalability of the chess game state recognition system by optimizing both the data storage solution and the processing pipeline. These enhancements enable the system to perform consistently and reliably in production, with improved data accessibility and streamlined operations. To address these issues, this thesis presents three main contributions: (1) an enhanced pipeline for image validation that reduces human error and increases efficiency, (2) the migration of the existing data storage from CSV files to a MongoDB database, optimizing scalability and data accessibility, and (3) the integration of drift detection mechanisms, utilizing both autoencoders and production model embeddings, to monitor and mitigate drift in the system's pipeline. Through this research, we demonstrate that embedding-based drift detection is viable for high-dimensional data such as images. Furthermore, we show that the production model can serve as a reliable and resource-efficient alternative to custom autoencoders for feature extraction, offering comparable performance in drift detection tasks. Our findings highlight the value of embedding-based drift detection for unstructured data, with applications extending beyond chess broadcasting to other machine learning systems in dynamic domains. |
|---|