Advanced Methodology and Statistics Seminars (AMASS)

Presented by:

Nur Hani Zainal, Ph.D., MS, Presidential Young Professorship (PYP) Assistant
Professor, National University of Singapore (NUS)

Natalia Van Doren, Ph.D., MS, NIDA T32 Postdoctoral Research Fellow,
University of California San Francisco (UCSF)

Participants earn 4 continuing education credits

Categories: Technology/Digital Health; Research Methods and Statistics

Keywords: Clinical Decision Making; Psychotherapy Outcome; Substance Abuse

Basic to moderate level of familiarity with the material.

Perennial clinical science questions to inform treatment targets and targeted treatments involve identifying risk factors and prescriptive predictors. Multivariate models are central to these questions. However, typical general linear models are limited in their ability to test intricate interactions among variables with diverse distributional patterns, identify relevant predictors for specific subgroups, and deduce complex and possibly non-linear relations (Shatte et al., 2019). To tackle these challenges, the ascent of machine learning (ML) is evident as it develops algorithms capable of categorizing persons with a wide array of unevenly distributed risk factors (Dwyer & Krishnadas, 2022). Predictive ML uses algorithms to gauge the predictive strength of multiple variables for future unseen data. Employing flexible non-linear and higher-order interaction ML algorithms can enhance model development and optimize predictive accuracy (Deisenhofer et al., 2023). Further, the Shapley additive explanations (SHAP) method helps to intuitively visualize predictor-outcome relations (Lundberg & Lee, 2017) while adjusting for other predictors (Molnar, 2022). However, precision medicine techniques are often overlooked in psychology. Our workshop introduces ML methods and applications to digital and precision mental health.

In this workshop, participants will be introduced to the following topics and tutorials:

• Preprocess data using R packages (e.g., dplyr, missRanger) for clinical predictive ML analyses
• Define training and testing data sets using caret, nestedcv, and related R packages
• Performing clinical predictive ML analyses with the caret, nestedcv, and related R packages
• Identify appropriate ML algorithms for their data and research questions
• Evaluate ML model performance using a classic suite of ML models
○ Classic models: LASSO regression, ridge regression, elastic net regression
○ Tree-based models: bagging, boosting, decision trees/CART, random forest
○ Other semi-parametric ML models: support vector machine, Super Learner
○ Other packages used: rpart, ranger, e1071, and sl3
• Visualize predictor-outcome relations in a multivariate model using the fastshap R package
○ Create partial dependence plots
○ Interpret bee swarm plots using the Shapley additive explanations (SHAP) method
• Apply these skills to various clinical data sets
○ Cross-sectional psychiatry epidemiological surveys
○ Electronic health records (EHR) data
○ Randomized controlled trial data
○ Prospective-longitudinal observational data
• Report on results for peer-reviewed academic scientific publications
• Discuss the clinical and ethical implications of ML models

At the end of this session, the learner will be able to:

1. Define ML and related precision medicine approaches
2. Articulate various guidelines for using ML, such as the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD; Collins et al., 2015; Moons et al., 2015)
3. Download actionable tutorials to conduct multivariate predictive ML modeling
4. Apply precision medicine methods in teaching, research, and clinical work
5. List five ethical considerations in applying precision medicine methods to clinical data

Long-term goals:

1. Optimize clinical care and delivery through the application of precision medicine in stepped-care and stratified care settings
2. Apply precision medicine methods to plug treatment gaps for underserved populations, such as ethnic/racial minorities and low and middle-income countries

Recommended Readings:

Zainal, N.H., Bossarte, R.M., Gildea, S.M. et al. Developing an individualized treatment rule for Veterans with major depressive disorder using electronic health records. Molecular Psychiatry (2024). https://doi.org/10.1038/s41380-024-02500-0

Shatte, A. B. R., Hutchinson, D. M., & Teague, S. J. (2019). Machine learning in mental health: a scoping review of methods and applications. Psychological Medicine, 49(9), 1426-1448. https://doi.org/10.1017/S0033291719000151

Dwyer, D., & Krishnadas, R. (2022). Five points to consider when reading a translational machine-learning paper. British Journal of Psychiatry, 220(4), 169-171. https://doi.org/10.1192/bjp.2022.29

Deisenhofer, A.-K., Barkham, M., Beierl, E. T., Schwartz, B., Aafjes-van Doorn, K., Beevers, C. G., . . . Cohen, Z. D. (2023). Implementing precision methods in personalizing psychological therapies: Barriers and possible ways forward. Behaviour Research and Therapy.https://doi.org/10.1016/j.brat.2023.104443

Lewis, M. J., Spiliopoulou, A., Goldmann, K., Pitzalis, C., McKeigue, P., & Barnes, M. R. (2023). nestedcv: an R package for fast implementation of nested cross-validation with embedded feature selection designed for transcriptomics and high-dimensional data. Bioinformatics Advances, 3(1), vbad048. https://doi.org/10.1093/bioadv/vbad048

Benjet, C., Zainal, N. H., Albor, Y., Alvis-Barranco, L., Carrasco-Tapias, N., Contreras-Ibáñez, C. C., . . . Kessler, R. C. (2023). A precision treatment model for internet-delivered cognitive behavioral therapy for anxiety and depression among university students: A secondary analysis of a randomized clinical trial. JAMA Psychiatry.https://doi.org/10.1001/jamapsychiatry.2023.1675

Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., … & Goldenberg, A. (2019). Do no harm: a roadmap for responsible machine learning for health care. Nature Medicine, 25(9), 1337-1340. https://doi.org/10.1038/s41591-019-0548-6

Presented by:

Ruben G. Martinez, Ph.D., Assistant Professor, The Warren Alpert Medical School of Brown University

Hannah E. Frank, Ph.D., Assistant Professor, The Warren Alpert Medical School of Brown University

Participants earn 4 continuing education credits

Categories: Dissemination & Implementation Science, System Stakeholder Issues

Keywords: Community-Based, Implementation, Health Care System

Moderate to advanced level of familiarity with the material.

Engaging community partners in identifying and prioritizing determinants to implementing evidence-based interventions (EBIs) is critical to promoting equitable implementation processes and improving care in health and mental health settings. Recent advancements in implementation science methods provide guidance on how to engage in these activities and ensure that community partners are active participants. Our expert implementation team represents the Brown Research on Implementation and Dissemination to Guide Evidence Use (BRIDGE) Program and the NIMH-funded IMPACT ALACRITY center, through which we provide training and consultation to researchers across the country on best practices for engaging community practice members in diverse settings. Through this work, we have refined several methods for identifying and prioritizing determinants (barriers and facilitators) in ways that honor community partners’ values and expertise.

Conducting community-engaged research has been increasingly acknowledged as a necessary step to advance health equity, but many administrators and researchers are left without concrete steps to understand community partners’ perspectives on EBIs. We propose a skills-based workshop that gives participants an experiential view into the process of identifying and prioritizing determinants to implementing EBIs in health and mental health settings. We will begin by highlighting the importance of determinants in implementing EBIs and methods to identify and prioritize determinants. We will focus on participatory methods for identifying determinants (e.g., rapid ethnography, process mapping). We will provide guidance for and examples of creating interview guides and preparing for interviews. Participants will engage in experiential exercises related to barrier identification methods throughout the workshop. We will then work through a group barrier prioritization exercise informed by a toolkit developed by the IMPACT center.

Outline:

1. Introductions and gauging familiarity with methods and implementation science
2. Setting the stage – introduce guiding concepts for this workshop:
   1. Presentation and facilitated discussion of equity and community engagement in implementation science
   2. Importance of including community partners in identifying and prioritizing barriers and facilitators (i.e., determinants)
   3. Identifying and selecting determinant frameworks
3. Methods overview – Introduction to determinant identification methods
   1. IMPACT toolkits
   2. Process mapping
   3. Participatory interviewing/qualitative methods
4. Break
5. Experiential exercise/case study application
   1. Developing an interview guide
   2. Training your implementation team in reflexivity, recognizing positionality and intersectionality
   3. Experiential interview exercise
   4. Brief overview of rapid qualitative analysis methods
6. Break
7. Live barrier prioritization exercise
8. You have identified and prioritized barriers… Now what?
9. Facilitated group discussion/Q&A

At the end of this session, the learner will be able to:

1. Describe the importance of including community partners in identifying determinants and prioritizing barriers when implementing evidence-based interventions.
2. Identify appropriate implementation frameworks to guide the process of identifying and prioritizing determinants in health and mental health settings.
3. Apply methods for identifying determinants and prioritizing barriers to implementation in health and mental health settings.
4. Describe how rapid qualitative methods can provide detailed and timely information about community partners’ perspectives.
5. List criteria by which barriers can be prioritized and create a clear plan for prioritizing barriers with community partners.

Long-term Goals:

1. Design a set of determinant identification activities to inform an implementation effort as part of a research project or implementation practice effort
2. Design and apply participatory barrier prioritization exercises with community partners

Recommended Readings:

Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022). The updated Consolidated Framework for Implementation Research based on user feedback. Implementation science, 17(1), 75.

Gale, R. C., Wu, J., Erhardt, T., Bounthavong, M., Reardon, C. M., Damschroder, L. J., & Midboe, A. M. (2019). Comparison of rapid vs in-depth qualitative analytic methods from a process evaluation of academic detailing in the Veterans Health Administration. Implementation Science, 14(1), 1-12.

Kim, B., McCullough, M. B., Simmons, M. M., Bolton, R. E., Hyde, J., Drainoni, M. L., … & McInnes, D. K. (2019). A novel application of process mapping in a criminal justice setting to examine implementation of peer support for veterans leaving incarceration. Health & Justice, 7(1), 1-11.

Lewis, C. C., Scott, K., & Marriott, B. R. (2018). A methodology for generating a tailored implementation blueprint: an exemplar from a youth residential setting. Implementation Science, 13(1), 1-13.

The IMPACT Center (2023). Prioritizing Implementation Barriers: Toolkit for Designing an implementation Initiative. https://impscimethods.org/toolkits/prioritizing-implementation-barriers-toolkit