ML: Technical Debt & Cardinal Sins

• Machine learning (ML) offers significant potential but can lead to substantial technical debt.
• ML, initially functional and locally successful, can develop widespread problems due to technical debt accumulation.

Rules of ML

• Rule #1: Know your Cardinal Sins.
• Rule #2: Split Machine Learning and Traditional Code.
• Rule #3: Apply solid engineering practices at the system level.
• Rule #4: Know your ML facts.
• Rule #5: ML is a great tool, not a panacea.

Common ML Mistakes

• Launching an ML project without fully understanding its implications
• Neglecting system testing
• Failing to monitor the data pipeline

Rule #1: Know Your Cardinal Sins

Cardinal Sins

• Dependency Debt
• Data Debt
• Configuration Debt
• Glue Code Debt
• Reproducibility Debt
• Abstraction Debt
• Process Debt
• Anti-Pattern Debt
• Test Debt
• Monitoring Debt

Dependency Debt

• ML systems are particularly prone to dependency debt.
• Undeclared consumers cause unintended consequences when shared data dependencies change.
• Data dependencies are unstable and can change without warning, resulting in silent model degradation.

Data Debt

• Flawed models and inaccurate predictions result from poor data quality.
• Common data quality issues include missing values, inconsistent formats, outliers, and biases.

Data Debt: Example

• Introducing a new signal can lead to a drop in prediction quality.
• Possible causes include:
  • Skew in new data is representing a new population
  • Feature was engineered based on a now broken assumption
  • The system is now exploiting the deployed model through a feedback loop.

Configuration Debt

• It accounts for the cost of managing and maintaining an ML system's configuration.
• ML systems have many configuration parameters that require careful tuning for optimal performance.
• Tracking and reproducing configuration changes is difficult, which can lead to errors.

Glue Code Debt

• It is code connecting different components of an ML system.
• It can be challenging to test and maintain, leading to errors and inefficiencies.
• Dependencies can arise between components, complicating changes or upgrades.

Reproducibility Debt

• It is the cost of reproducing the results of an ML experiment.
• Reproducibility is essential for debugging, validating results, and sharing research.
• ML experiments are hard to reproduce due to complex dependencies, randomness, and inadequate documentation.

Abstraction Debt

• It is the cost of creating and maintaining abstractions in an ML system.
• Abstractions hide complexity, promote reuse, and improve maintainability.
• Abstractions can be difficult to understand, costly to create, and inflexible.

Process Debt

• It is the cost of developing and maintaining processes for building and deploying ML systems.
• ML projects often need new processes for data collection, labeling, model training, deployment, and monitoring.
• These processes can be complex and time-consuming to develop and maintain.

Anti-Pattern Debt

• It is the cost of using anti-patterns in an ML system.
• Anti-patterns are common mistakes that cause problems with performance, scalability, and maintainability.
• Examples include using excessively complex models, overfitting, and ignoring data quality.

Test Debt

• This is the cost of not testing ML systems adequately.
• ML systems can be difficult to test because of their complexity and the difficulty of defining clear test cases
• Testing ensures that ML systems are accurate, reliable, and robust.

Monitoring Debt

• Lack of adequate ML systems monitoring incurs costs for the project.
• ML systems can degrade over time due to changes in data, environment, or the model itself.
• Monitoring detects problems early, enabling corrective action.

Rule #2: Splitting Machine Learning and Traditional Code

• Separate ML code from traditional code to improve modularity and maintainability.
• Treat ML code as a black box with well-defined inputs and outputs.
• Eases testing, debugging, and upgrading ML code without affecting the broader system.

Rule #3: Solid Engineering Practices at the System Level

• Apply solid engineering practices to the entire ML system.
• Includes:
  • Version control
  • Testing
  • Documentation
  • Monitoring
  • Automation
• Helps ensure that the ML system is reliable, scalable, and maintainable.

Rule #4: Know Your ML Facts

• Understand the limitations of ML.
• ML is not a silver bullet.
• Understanding strengths and weaknesses of different algorithms is the right approach.
• Be aware of potential biases in the data and the model.

Rule #5: ML is a Great Tool, Not a Panacea

• Don't use ML for everything.
• ML is a powerful tool, not always the best one.
• Consider other approaches before using ML.

Conclusion

• Minimizing technical debt and maximizing ML benefits can be achieved by adhering to outlined rules.
• Machine learning offers significant potential, coupled with the risks of accumulating massive technical debt.