Besides issues in getting a hold of large and diverse datasets, annotation or labeling, and sexy new methods to train models (synthetic data and federated learning!,) transparency also relates to model interpretability—in other words, humans should be able to understand or interpret how a given technology reached a certain decision or prediction. AI technologies will need transparency to justify a particular diagnosis, treatment recommendation, or outcome prediction. Or, will they? If we observe that by feeding more and better data into models, new or existing, and their pattern recognition and predictability keeps improving, do we have to fully understand how the model came to its conclusions? After-all, we can test a model and investigate its output to see if it is making good predictions or spotting issues accurately. If so, and perhaps even better than humans, do we not use it because we can not explain how the black box came to its conclusions?
This type of transparency is increasingly difficult with deep learning models that are more powerful, can absorb large amounts of multi-dimensional data, and provide output that is beyond human processing capability and not easy to explain. There will be powerful debate about whether these outputs should be used in managing patients. After-all, we are giving medications to patients that have demonstrable benefits but we do not exactly know how they create that benefit. So, why should the algorithms be any different? On the other hand, it will take time to prove that the recommendations from an algorithm are superior in improving patient outcomes and safe. I think once that happens, clinicians will become more comfortable using algorithms without being able to fully explain how they arrived at their outputs. You can see in the Figure that as the accuracy of these models increases (due to the use of deep learning algorithms,) the explainability has been going in the opposite direction.
Another reason why transparency is important is that AI technologies have the potential for algorithmic bias, thereby reinforcing discriminatory practices based on race, sex, or other features. Transparency of training data and of model interpretability would allow examination for any potential bias. Ideally, machine learning might even help resolve healthcare disparities if designed to compensate for known biases. Once again, all of this is aspirational and if we see models that are improving patient outcomes with low explainability, would we not use those? This debate is especially salient now since we are observing that although some of these models have performed very well in isolated validation studies, once they are put to work in the real clinical setting, their performance has been less than stellar. That means that the real test of whether the black box produces output superior to humans is not in controlled settings (where many companies now are validating their algorithms) but in the real world setting. In that situation, we can carefully study whether the models’ output is accurate, unbiased, and repeatable with different types of real world data. This type of study has not been done in large scale with most of the models so far and this issue will remain a barrier to adoption of these models for the near future.
