Snuffy for Whole Slide Images

As part of my research at the Rohban Lab, we developed Snuffy, an efficient Whole Slide Image (WSI) classifier published at ECCV 2024. Analyzing WSIs is notoriously difficult because a single slide can contain billions of pixels, requiring architectures that can efficiently process massive sequences of tissue patches without losing critical diagnostic data.

My primary contribution to this project was bridging the gap between tumor biology and transformer mathematics—specifically, designing the Snuffy sparsity patterns and formulating the theoretical proof that guarantees the model functions as a universal approximator.

The Biological Concept: The Tumor Microenvironment

Present sparsity patterns are predominantly designed with a focus on NLP tasks, often relying on strict locality windowed patterns. However, in the domain of WSIs, this NLP-centric approach falls short.

Biologically, a tumor’s behavior and the detection of cancer depend heavily on the broader tissue microenvironment. Pathologists frequently depend on the identification of non-related tissue embedded within other tissues as a pivotal biomarker for cancer detection. Crucially, this remains true even in instances where the tissue itself does not exhibit overt signs of malignancy. Therefore, to accurately model WSIs, the attention mechanism must be able to capture these critical, non-local insights rather than being artificially restricted to immediate neighbors.

The Solution: Snuffy Sparsity Patterns

Inspired by the inherent characteristics of WSI analysis, we proposed the Snuffy sparsity patterns. Instead of standard local windows, the Snuffy sparsity concept comprises three targeted key elements:

Class-related Global Attentions (Λ_top): These are crucial for the final classification task. We leverage our max-pooling component to identify these top patches; after several iterations, the scores converge and these patches remain constant across layers.
Random Global Attentions (Λ_lr): We incorporate random patches into the design to integrate insights from the broader tissue microenvironment, enhancing the capture of critical information across the slide.
Diagonal Attentions (k): This ensures that even if contextual attention embeddings are not computed within the self-attention framework for a specific patch, the patch’s original embedding remains preserved in the output.

Figure 1. Overview of the proposed method. a) The WSIs are segmented into 256 × 256 patches at 20X magnification, followed by embedding extraction via a pre-trained ViT. Subsequently, these embeddings are inputted into the Snuffy for patch and WSI classification. (b) The connectivity matrix illustrates the Snuffy attention spar- sity patterns, with Class-related Global Attentions, highlighted in darker colors either vertical or horizontal (the darker the more important), Diagonal Attentions depicted with pink, and Random Global Attentions shown in the lightest pink.

The Proof: Universal Approximation

My core theoretical contribution was demonstrating that our sparse transformer serves as a universal approximator for sequence-to-sequence functions.

To satisfy the necessary mathematical criteria, we had to demonstrate the existence of a Hamiltonian path in the graph corresponding to the union of our Snuffy sparsity patterns. Drawing on graph theory (specifically, that a graph where the maximum independent set is less than or equal to its chromatic number has a Hamiltonian cycle), we proved that covering half of the patches in all layers satisfies these properties and leads to the formation of a Hamiltonian path.

I modeled this observation as a generalized version of the coupon collector problem, where the goal is to collect half of the “coupons” (patches) in a uniform group setting.

Ultimately, the proof establishes that our transformer does not necessitate Ω̃(n) layers as suggested in previous studies. Instead, it requires only O(n log 2 / λ_r) layers to ensure universal approximation with high probability, achieving the most stringent probabilistic limit of the layer count to date.

Impact and Results

Figure 2. Qualitative view of ROIs recognized by Suffy through its Patch Classification. (a) An example WSI from the test set of the CAMELYON16 dataset. (b) ROIs are identified by Snuffy with black lines delineating the ground truth ROIs.

By combining this biologically grounded attention mask with Snuffy’s architecture, we achieved:

State-of-the-Art Accuracy: Superior performance on major computational pathology benchmarks like TCGA-BRCA and TCGA-LUNG.
Biological Interpretability: Attention maps that actually correlate with localized, meaningful tissue structures rather than scattered noise (Jafarinia et al., 2025).

Figure 3. Performance (AUC) vs. efficiency (size and time) trade off on CAMELYON16.</b>

Whole Slide Image (WSI) classification with multiple instance learning (MIL) in digital pathology faces significant computational challenges. Current methods mostly rely on extensive self-supervised learning (SSL) for satisfactory performance, requiring long training periods and considerable computational resources. At the same time, no pre-training affects performance due to domain shifts from natural images to WSIs. We introduce Snuffy architecture, a novel MIL-pooling method based on sparse transformers that mitigates performance loss with limited pre-training and enables continual few-shot pre-training as a competitive option. Our sparsity pattern is tailored for pathology and is theoretically proven to be a universal approximator with the tightest probabilistic sharp bound on the number of layers for sparse transformers, to date. We demonstrate Snuffy’s effectiveness on CAMELYON16 and TCGA Lung cancer datasets, achieving superior WSI and patch-level accuracies. The code is available on https://github.com/jafarinia/snuffy.

The Biological Concept: The Tumor Microenvironment

The Solution: Snuffy Sparsity Patterns

The Proof: Universal Approximation

Impact and Results

References

2025