COVID-19 Drug Efficiency Estimation

During my M.Sc. in Bioinformatics at Sharif University of Technology, my research in the Robust and Interpretable Machine Learning (RIML) Lab focused on evaluating potential prophylactic treatments against SARS-CoV-2. We utilized the RxRx19a dataset from Recursion Pharmaceuticals, which contains high-throughput, five-channel fluorescent microscopy images of cells treated with various FDA/EMA-approved drugs.

The primary goal of this project was to extract cellular morphological features at the single-cell level and aggregate them to estimate the antiviral efficacy (hit score) of different drug-dose combinations.

Representative five-channel fluorescence microscopy images showing the nucleus, RNA, Golgi apparatus, actin cytoskeleton, and endoplasmic reticulum.

The Challenge: Out-of-Distribution (OOD) Phenotypes

A major hurdle in current drug discovery methods is that neural networks often identify promising hits that later fail in preclinical trials. In the context of the RxRx19a dataset, models were trained solely on healthy (negative control) and infected (positive control) cells, as their true efficacy score is given and trivia (healthy is 1 and infected is 0).

Consequently, when predicting the efficacy of a treatment, the morphological features of the drugged cells were completely Out-of-Distribution (OOD) because the model had never seen them during training. This lack of exposure meant that previous algorithms couldn’t be fully trusted and falsely flagged many toxic or ineffective treatments as positive hits.

A Novel Solution: Confidence-Based Estimation

To solve this, we developed a first-in-field, weakly-supervised hit prediction pipeline that calculates both a disease score and a confidence score for its predictions.

Our pipeline: Image preprocessing (illumination correction and cell segmentation), feature embedding, data transformation (simulating drug effects), and the confident hit prediction network.

Instead of blindly trusting the model’s judgment on unknown samples, our approach acknowledges its uncertainty. We introduced a unique data augmentation step using a mixup methodology, creating a transformed sample via a convex combination of positive and negative controls, and adding orthogonal noise to simulate both drug efficacy and side effects.

During training, the model receives “hints” about the disease score. If the model relies heavily on these hints (indicating high uncertainty regarding OOD samples), it outputs a lower confidence score.

Results & Impact

By applying this confidence mechanism to both CellProfiler features and deep neural network embeddings (ResNet18), we significantly improved the reliability of hit discovery.

a) Replicate reproducibility test. The blue distributions are the null distribution. The red vertical line indicates the upper 95% of the null distribution. b) Stability test of disease scores with respect to the percentage of missing data. We can see our proposed method (orange) is more stable against missing data, while the RxRx19a origin algorithm fluctuates adversely after missing more than 15% of data. c) Disease score distributions of our weakly supervised confident hit predictions for negative and positive controls. We observed that the model properly predicts unseen inactive ultraviolet-irradiated SARS-CoV-2 scores in a zero-shot learning process. d) Calibration of Confidence scores. We bucketize disease scores into 10 bins, and calculate median of confidence score in each bin. This plot show model uncertainty and high entropy for disease score between [0.3, 0.7].

Higher Consensus: The mean Jaccard similarity between the top 10 discovered drugs across different representation spaces increased from 0.13 to 0.20 when utilizing our confidence score.
Enhanced Stability: Our model proved to be highly stable against missing data compared to the original, unstable algorithms used in RxRx19a.
Zero-Shot Robustness: The pipeline accurately identified unseen, inactive ultraviolet-irradiated SARS-CoV-2 samples as healthy, proving its robustness in a zero-shot learning setup.

This research highlights the importance of mitigating shortcut learning and reducing reliance on spurious correlations in safety-critical medical AI (Mirzaie et al., 2023).

The COVID-19 pandemic has prompted a surge in drug repurposing studies. However, many promising hits identified by modern neural networks failed in the preclinical research, which has raised concerns about the reliability of current drug discovery methods. Among studies that explore the therapeutic potential of drugs for COVID-19 treatment is RxRx19a. Its dataset was derived from High Throughput Screening (HTS) experiments conducted by the Recursion biotechnology company. Prior research on hit discovery using this dataset involved learning healthy and infected cells’ morphological features and utilizing this knowledge to estimate contaminated drugged cells’ scores. Nevertheless, models have never seen drugged cells during training, so these cells’ phenotypic features are out of their trained distribution. That being said, model estimations for treatment samples are not trusted in these methods and can lead to false positives. This work offers a first-in-field weakly-supervised drug efficiency estimation pipeline that utilizes the mixup methodology with a confidence score for its predictions. We applied our method to the RxRx19a dataset and showed that consensus between top hits predicted on different representation spaces increases using our confidence method. Further, we demonstrate that our pipeline is robust, stable, and sensitive to drug toxicity.

The Challenge: Out-of-Distribution (OOD) Phenotypes

A Novel Solution: Confidence-Based Estimation

Results & Impact

References

2023