A Multimodal Generative AI Framework for Cancer Pathology Classification

Abstract

We propose a multimodal generative AI framework for cancer pathology that combines three capabilities: (i) histopathology picture classification, (ii) clinical note classification, and (iii) prompt-driven clinical captioning using retrieval-augmented generation (RAG). The approach establishes accurate vision baselines on PatchCamelyon (PCam) and creates a complementing text pipeline by curating and categorizing TCGA BRCA clinical notes via two distinct routes to assure trustworthy supervision.

We investigate caption utility by creating image-based descriptions and constructing a lightweight caption-label classifier to assess RAG's downstream worth. Together, these components establish a uniform data and model foundation, explain interfaces for combining picture and text representations, and give an evidence-based approach to short, clinically grounded summaries.

The main objective is to provide a practical, auditable procedure that improves diagnostic support by matching visual findings with structured language while still allowing for parameter-efficient adjustment and eventual inclusion into clinical decision tools.

System Architecture

Datasets

Histopathology Images — PatchCamelyon (PCam)

Experiments use the PatchCamelyon (PCam) histopathology patch dataset via the torchvision PCAM interface. Each example is a color image patch paired with a binary label (benign vs. malignant). Official splits are preserved: 262,144 training, 32,768 validation, and 32,768 testing images. No relabeling, rebalancing, or re-splitting was performed. All reported metrics are computed on the fixed splits for strict reproducibility.

Clinical Notes — TCGA BRCA

Clinical notes were provided via the Cancer Genome Atlas (TCGA) portal, focused on the Breast Invasive Carcinoma (BRCA) cohort. 1,105 BRCA-specific pathology reports were extracted from ~48K reports on various cancer types. Two annotation approaches were used: Approach 1 (PyTesseract OCR + keyword/negation rules) and Approach 2 (TCGA metadata morphological codes). The dataset exhibited considerable class imbalance (1,190 malignant vs. 89 benign), which was addressed by random upsampling to achieve a 1:1 ratio. The final balanced dataset of 2,380 reports was split: train 1,428 / val 476 / test 476.

Methodology

Histopathology Classification

Three modern image backbones were benchmarked on PCam using a unified classification pipeline. Each backbone was instantiated from standard libraries and adapted to binary prediction with a single linear classification head. Decision thresholds were selected on the validation set (ResNet-50 τ=0.22, ViT-B/16 τ=0.36) and then fixed for test-time reporting.

ViT-Base/16 delivers the most consistent performance across all metrics; ResNet-50 is a close second; EfficientNet-B0 remains a strong lightweight baseline even without threshold tuning. These fixed numbers establish robust image baselines to be held unchanged in subsequent fusion and caption-aware experiments.

Clinical Note Classification — ClinicalBERT Fine-Tuning

The balanced clinical notes are used as input to a ClinicalBERT model for benign and malignant classification. We evaluate the pretrained model as a baseline, then sequentially apply Supervised Fine-Tuning (SFT) and Task-Adaptive Pretraining (TAPT) to increase domain adaptation, followed by LoRA to fine-tune parameters efficiently.

The optimum LoRA configuration (rank r=4, α=8, learning rate 2e-3, 10 epochs) had the highest stable validation F1-score (0.94 ± 0.05) and was chosen for clinical note classification in the multimodal fusion experiments. The experiments were conducted in two environments: SJSU GPU Lab (RTX 5090) and Google Colab (A100 GPU).

Caption Generation — LiquidAI RAG + Prompting Strategies

Caption generation addresses limited caption supervision by first comparing three zero-shot LLMs (GPT-4o, GPT-3.5-turbo, Claude 3.5 Sonnet) on 300 stratified notes. A fourth model (DeepSeek) served as an objective judge and selected GPT-4o as the best performer; its captions were adopted as ground truth for the LiquidAI RAG pipeline. RAG was then trained and fine-tuned with LoRA (rank r, scale α) while tuning output token limit and input context length. Four prompting strategies were evaluated within the RAG setup under identical hyperparameters: Zero-Shot, Self-Reflection, Self-Ask, and Tree-of-Thought.

Tree-of-Thought (ToT) achieved the highest overall lexical and semantic alignment (R1=0.409, RL=0.320, BF1=0.862) under the LoRA-RAG setup (r=16, α=64, LR=7e-6, max_len=2048), confirming that structured reasoning enhances factual completeness in medical captioning.

Conclusion

Together, the PCam image baselines, ClinicalBERT fine-tuning pipeline, BLIP-2 caption analysis, and LoRA-RAG captioning system establish a uniform data and model foundation for multimodal cancer pathology classification. The framework provides evidence-based approaches to short, clinically grounded summaries while remaining parameter-efficient and extensible to future fusion architectures.