"If you can't measure it, you can't improve it." — Peter Drucker
In 2002, I was part of a SaaS service that created one of the first tools to mine data from online news sources. What we had not accounted for was the rate at which online publications modified their structure, breaking the scripts we had written to harvest their content. As the source list grew from hundreds to thousands, the publications being missed exploded. We had failed to build any mechanism to automatically detect when a source went offline.
The result was that our own subscribers were alerting us to coverage gaps before we discovered them ourselves. The solution was a monitoring and alerting system that allowed one person to track over 20,000 sources in real time. What had been a reactive embarrassment became a proactive operational advantage.
KPIs must be defined for every meaningful dimension of the business — not just the obvious ones. In one SaaS service, sales team members had been creating "free trials" and, in a remarkable percentage of cases, simply never ending them. These were perpetual free trials — a polite term for non-paying subscribers. One salesperson actually requested commission on non-paying accounts. These things do not fix themselves.
Roadmap Considerations
◆ Identify mission-critical service components — the ones whose failure affects the customer most severely. ◆ Ensure real-time monitoring and measurement of those components. ◆ Automate as much of the monitoring and alerting as possible. Manual checks will eventually fail.