Damian Sendler: A multidisciplinary strategy that is very technical and difficult, but seems to yield results that resonate, is quickly being adopted by the field of psychiatry as the current “wild west.” To assess these papers, the techniques are frequently unclear and it is difficult to identify the right reviewers. If there isn’t a clear framework for evaluating these research and fostering credibility, this problem will only become worse. To address this urgent need, our research focuses on developing a framework for evaluating complicated methodologies such that the process is done objectively, equitably and scientifically. There are three essential factors to consider when evaluating the usefulness of artificial intelligence in psychiatry: explainability, transparency and generalizabilty.. We explain how difficult it may be to define these three concerns since the phrases have diverse meanings in medical, computer science, and law. We explore how specifying these three difficulties helps to establish a framework to assure trustworthiness. To summarize, we believe it is critical to begin the conversation so that a policy can be established and that the scientific community is extra cautious when evaluating clinical uses of these models.
Damian Jacob Sendler: Artificial Intelligence (AI) and machine learning (ML) in psychiatry have recently seen a boom in popularity. In clinical decision-making, automated analysis may give additional information and identify subtle patterns that people have a hard time noticing. There is, however, a lack of knowledge of the underlying processes and how and when it should be used, even when the outcomes seem to be positive. There’s a risk that people may stop trusting human judgment in favor of computer algorithms because of this. 1,2 It is true that the mystique surrounding artificial intelligence (AI) does not promote the kind of dependability that is so important when using medical technologies.
Dr. Sendler: In several fields, including education, politics, and medical, the role of AI and ML in society has sparked heated disputes. We want to start a discussion on how to verify that ML techniques are legitimate, effective, and implemented and utilized appropriately, given that a rising number of psychiatric articles use ML methods3–9. Prior to implementing any new methods, we must first address the steps necessary to guarantee that research is of the highest quality, that findings are subjected to comprehensive and fair evaluation, and that any new findings have clinical use. Using AI in psychiatry may be possible in the near future if the following three principles can be established: (1) explainability, (2) transparency, and (3) generalizability. Psychiatry may be able to use these technologies in the near future.
Damian Sendler
Google AI’s lung cancer screening model, for example, performed on par with six radiologists in detecting the disease (10). However, IBM Watson’s choice to treat a cancer patient overlooked contraindication evidence (11). In both successes and failures, we are forced to analyze who is ultimately accountable, how to reduce mistakes in the future, and how best to continue the road of AI help. In order to answer these issues, the AI’s decision-making process must be described.
What would be helpful to physicians is an explanation of behavior (e.g., no vocal modulation in psychosis) that links it to clinical concepts (e.g., schizophrenia) (an indicator of a worsening clinical state13). As stated by DSM-V, these components are phenomenology or symptoms.14 However, the NIMH Research Domain Criteria15 recommends a more basic and granular level of characterisation, and ML-based modeling typically does not operate at this feature level (eg, frequency counts, word choice, time of task completion). Several models integrate many characteristics (hundreds or thousands) and learn how to weight them to best describe the clinical constructs.
Models may work for bogus reasons16, which in medicine can have disastrous effects since clinical judgments may be based on a poor basis, even if they don’t always correlate directly with clinical characteristics, thus understanding the classes of information the model is employing is crucial.
Each application area and ML model has its own unique explanation. Explanations from classification models may need a counterfactual example in order to explain why a certain situation is labeled differently by the model. When it comes to analyzing data, regression models are ideal since they allow you to focus on certain attributes and give weights to them. If we had more or less of a certain variable, we may infer that our result would have been different. This set of variables may not be connected in any meaningful way, but they are at least statistically significant in some way. 17,18 It is crucial to know both the statistical data used to train the model and the distributions and probabilities given to features and classes in order to get accurate results.
Damian Jacob Markiewicz Sendler: It’s also important to distinguish between models that are easy to understand (like decision trees or linear regression/classification models) and ones that are difficult to understand (eg, complex, deep neural networks). An interpretable model has a small number of characteristics, and the model learns weights for each of those features as it is built up. There are many ways to measure the quality of a person’s memory recall, but a simple example is the number of words remembered, which could be weighted at 2.0 and semantic closeness of a recall to the original tale, which could be weighted at 3.0. Often at the expense of accuracy, these basic models may give more in-depth explanations than more complicated, deep neural networks can. Tradeoff between model performance and model explainability is well-known in the industry 19 As a result, the most effective machine learning models can’t be explained. Recent breakthroughs in machine learning (ML) have made it possible for us to see inside the “black box,” yet the perspective may still be limited. While achieving explainability is a worthy objective, achieving transparency and generalizability is a more practical one.
There must be clarity in the objective of a model, how it is built and how it was trained, as well as any assumptions that were made in the process. The model’s characteristics should also be linked to the constructions of interest being analyzed, and this should be made clear.
Damian Jacob Sendler
Trust in machine learning models must meet the aforementioned standards. In contrast, a model that is promoted as a money-saver and is clear about its techniques and assumptions and presented as reliably forecasting a variable of interest would instill higher faith in the model’s claims. In order to better understand model behavior, it is important to understand common assumptions (eg, the data are independent and identically distributed, cross validation is used in training, or model choice [eg, selecting a regression classifier implies that the data are believed to be linearly separable].
A ML model’s output should be designed to complement the clinical decision-making process rather than be a final conclusion that is afterwards challenged by the scientific community. Including clinical recommendations for usage is as crucial as being open and honest about the model’s complexities. The doctor must be aware of the system’s specifics and how to utilize it effectively. Open-source code isn’t implied since it’s impossible in many circumstances, but rather that high-level functions and details are accessible.
Good science and the advancement of knowledge are built on generalizability. Because complex ML techniques tend to overgeneralize, they need huge datasets. Because of the absence of established cross-validation techniques, many research in psychiatry have been published with small and insufficient sample sizes (e.g., less than 50 training samples and less than 20 generalization sets). 9 A “matched” dataset is impossible to produce in the clinical sense because of the enormous datasets that have become the norm in machine learning. In spite of the fact that the datasets are unlikely to be matched, the benefit of ML is that it can use the diversity inherent in huge data sets to offer accurate characterizations.
Due to the biases and prejudices contained in the data used to train ML systems, they reflect past patterns of racial, economic, and gender inequality.
20–23 It is imperative that the scientific community produce big and reliable datasets that are representative of the target populations in order to evaluate algorithms without bias or generalizability difficulties. A benchmark that all models must achieve before they can be used in the actual world would guarantee that there is less discrimination against certain groups. It is essential that the database include samples from the whole desired audience. This will raise the likelihood of early and accurate detection and diagnosis by increasing the accuracy of assaying the proper measurement constructs. In order to build the next generation of ML-based solutions, it is essential to ensure that the underlying algorithms are valid across genders, ages, ethnicity, and sociodemographics.
For every product of computer science, thorough testing is required before it can be put into use in a therapeutic context. In testing, edge cases and corner cases are concepts that must be taken into account. It is important that a model be able to handle the most extreme clinical cases, which is why the first example is at the extremes of acceptable operating parameters (eg, minimum and maximum values of features) and the second example is at the extremes of expected operating parameters (eg, the combination of features at their outer limits). Breaking a machine learning model is the best technique to assure that it will behave as predicted in all circumstances..
Damien Sendler: ML integration into psychiatry faces a number of challenges, many of which may be addressed using the aforementioned paradigm. There are, however, additional long-term issues to consider. How can sensitive data be shared in a consortium-like approach with AI technologies that function best with massive datasets? It’s hard to utilize state-of-the art, publicly accessible techniques to identify patients (even if the PHI is deleted from the data, numerous sources may be triangulated to keep identity), so how can we go around this? Finally, who bears the brunt of the blame for mistakes? Ultimately, who is responsible for making sure the model is safe to use, and who is responsible for making sure the model is safe to use? These are the kinds of topics that should be debated across the board.
There is a long way to go before we can use what we’ve learned about machine learning in the field of psychiatry in therapeutic settings. We must turn our attention to make ML models as resilient as feasible so that they can be used in the real world after accuracy advances in ML models have reached a plateau. Machine learning models should not be used as final decision-makers in medicine, but should instead be used to benefit from the unique strengths of machines. We’ve laid out a framework that may be used by both researchers and clinicians who want to incorporate machine learning into their practice. ML models must be able to explain themselves, be transparent, and be able to be generalized. This framework offers the foundation for this evaluation.
