Beyond workflows, the issue of data is salient. AI solutions have no benefit if there’s not enough data. Often, a medical algorithm shows great results in development and validation but when it gets to the real world, the results are disappointing. That is because it may not be getting the data it needs in a timely manner or the data it’s getting is too different than what it was trained on. Both of these issues stem from the fact that training or validation data is usually limited in size and/or not as diverse as the data in the real world. The other key issue is that the necessary data often sits in different information systems and creating interfaces with each system is difficult, technically and politically. As such, when an AI solution is launched in a real-world environment, its success may be undermined by the data issues. If you’re a buyer of such a system, you want to make this a key part of your business case assessment. If you don’t and the system you end up choosing does not perform up to expectations, it will be a wasted effort and will make future AI investments more difficult.
The main issues to assess here are 1. what data will the system need to create its output, 2. where does that data reside in your environment, 3. How easy will it be to direct that data to the AI system, 4. Will the flow of data into the AI system be in time for the system to create its output without disrupting current workflows?, 5. What are the potential risks (e.g., permissions, technical difficulty, unreliable interfaces, etc) that may disrupt the availability of data, 6. Can those risks be proactively mitigated to ensure reliable flow of data into the AI system on a consistent basis?.
If the answer to any of these issues is not satisfactory, then the business case will be shaky at best. You may say that these types of issues such as workflows, data availability, implementation efforts, etc are not really the business case issues. You may say these are technical issues. Although these issues are not usually part of a classic business case analysis, once you’re dealing with technologies such as AI that could be game-changing for a business, not factoring these key issues into your business analysis would be foolish. Any of these issues can make the project a failure and destroy the forecasts you’ve built based on the expected revenues or costs. In technology business case analysis, the performance of the technology is not a given. Identifying the key factors that can affect the performance of the technology and factoring that into your analysis is wise. This is especially true of AI in healthcare since the risks in case of failure are high, data required is often fragmented and in poor format, trust and literacy is low on the part of the users, and the buyers have tight budgets.
Ultimately, the lack of such robust analysis may explain why so many of the health AI projects have not lived up to expectations to date. A careful analysis of these projects indicates that the data issues discussed in this post are responsible for many of these failures. As the buyers of these technologies develop a playbook for how to evaluate the business case for these technologies, it is important that these issues are fully examined. Anyone of these issues can sink a promising technology even if everything else is in place.
