The model is rarely the constraint
Ask a team why their machine learning project stalled and the answer is almost never that the model underperformed. It is that the training data did not match production data, that nobody agreed what a correct prediction meant, or that the pipeline feeding the model broke quietly and no one noticed for six weeks.
This is why our first phase on an ML engagement is usually not modelling. It is establishing what we call a data contract: an explicit, versioned agreement about what each upstream field means, what its valid range is, who owns it, and what happens when it changes. Without that, a model is a very expensive way to encode assumptions nobody wrote down.
Define the failure you can tolerate
Accuracy is a weak target. What matters is the cost asymmetry between a false positive and a false negative, and that is a business decision rather than a technical one. In fraud screening, a missed fraud and a blocked legitimate customer have wildly different costs, and they change by segment and by season.
Getting that written down before training starts changes the model choice, the threshold, the monitoring and the escalation path. Teams that skip it end up optimising a metric that nobody in the business recognises.
Build the feedback loop before the model
A model in production without a feedback loop is a system that degrades invisibly. Input distributions drift, upstream systems change their defaults, a partner starts sending a field in a different unit. You want to detect that from monitoring rather than from a customer complaint.
Practically, that means logging every prediction with its input features and model version, sampling for human review on a fixed cadence, and alerting on distribution shift rather than only on error rates. It is unglamorous infrastructure and it is what separates a model that lasts three years from one that lasts three months.
When the answer is not machine learning
We regularly recommend against it. If the rules are known, stable and auditable, a rules engine is cheaper to build, far cheaper to explain to a regulator, and it does not degrade. If you have fewer than a few thousand labelled examples, or if the underlying process changes faster than you can retrain, machine learning will disappoint you expensively.
The strongest signal that ML is the right tool is that the pattern is real, repeated and too complex to enumerate — and that you already have clean, labelled history of it. If those three things are not true, fix them first. That work is valuable regardless of what you build on top.