Managing Risks Associated with the Evolution of Systems with AI Components

Abstract

Background: As organizations increasingly incorporate Artificial Intelligence (AI) components into their software systems, managing their long-term development has become a major challenge. The data-driven and flexible nature of AI systems creates specific types of technical debt that can result in significant risks to both the system and the business.  Objective: This study aims to investigate (1) the risks linked to technical debt in evolving AI components, (2) the methods organizations use to evaluate these risks, and (3) the strategies they adopt to reduce them.  Methods: An exploratory qualitative research design was used. Data was gathered through semi-structured interviews with six industry professionals, including Software Engineers, Machine Learning Engineers, Data Scientists, and Engineering Managers. The data was analyzed through thematic analysis to find recurring patterns and themes from practitioners' experiences.  Results: The findings identify two main categories of risk: system-level risks, which include degraded performance, model drift, feedback loops, configuration issues, and challenges with maintainability; and business-level risks, which include damage to reputation, increased operational costs, violations of service level objectives, and loss of customer trust. Continuous monitoring such as performance tracking, data drift detection, and output validation emerged as the main risk assessment strategy. Mitigation strategies include version control, testing, schema validation, documentation, modular system design, and automated threshold optimization. Clear differences were noted between academic literature and industry practices, especially regarding the use of operational and MLOps-driven approaches.  Conclusion: This study offers valuable insights into how technical debt affects evolving AI systems and highlights practical strategies used in the industry to manage related risks. Although the findings are limited due to the small sample size and the exploratory nature of the study, they help bridge the gap between theory and practice and point out areas for future research, including quantitative validation and scalable AI risk management frameworks.