CAMEL: Class-Aware Medical Vision-Language Alignment
Problem Statement
Standard contrastive learning frameworks, such as those used in CLIP-style Vision-Language Pretraining (VLP), rely on binary supervision: each image-text pair is either a perfect match (score = 1) or a complete mismatch (score = 0). While effective for generic web-scale image-text alignment, this harsh supervision fails to capture the nuanced semantic relationships between intra- and inter-class samples inherent in the medical domain.
In clinical imaging, diseases often exhibit overlapping visual or textual characteristics. For instance, pneumonia and COVID-19 both present with diffuse opacities in chest radiographs. Standard contrastive loss treats such semantically related pairs as hard negatives, forcing their embeddings apart despite shared pathological features. This rigid separation results in a loss of fine-grained clinical structure in the learned representation space, limiting the model's ability to capture subtle diagnostic distinctions.
We address this with two contributions: (1) C-CAS — a class-aware soft similarity metric that assigns continuous alignment targets reflecting semantic relatedness, and (2) C-CAP — a pretraining framework that uses C-CAS targets as supervision, replacing binary cross-entropy with soft alignment objectives. Together, these enable the model to distinguish subtle disease-specific patterns while maintaining awareness of shared clinical features, without relying on external knowledge bases or specialized feature extractors.
Patient Profile Generation
A key challenge in medical VLP is the scarcity of paired image-text data. To address this, we construct patient profiles by pairing lesion images with structured clinical metadata, then employ mPLUG-Owl3 to generate clinically grounded narrative reports. Each profile combines visual observations (morphology, color, texture) with patient context (age, anatomical site, symptoms), yielding rich supervision signals that go far beyond simple diagnostic labels.
Samples of patient profiles with synthetically generated clinical narratives using mPLUG-Owl3.
Synthetic Report Generation
While this approach is domain-agnostic and applicable to any medical image-text pairing, we demonstrate its effectiveness in dermatology using synthetically generated clinical reports. To address the scarcity of paired dermatological image-text data, we employ mPLUG-Owl3, a state-of-the-art vision-language model, to generate clinically grounded narratives from lesion images and patient metadata. Unlike simple diagnostic labels or generic descriptions, these synthetic reports capture the richness of clinical language, incorporating visual observations (e.g., morphology, color, texture) alongside contextual patient information (e.g., age, location, symptoms). This process yields a high-quality training corpus that provides stronger supervision signals while maintaining clinical realism and diagnostic relevance.
Samples of synthetic dermatology reports generated using mPLUG-Owl3.
Method
To capture the nuanced relationships between semantically related medical samples, we propose two complementary components: C-CAS (Class-aware Contrastive Alignment Score) for computing soft similarity targets, and C-CAP (Class-aware Contrastive Alignment Pretraining) for optimising the vision-language encoder using those targets.
C-CAS: Soft Similarity Targets
Unlike standard contrastive learning that treats all non-matching pairs as equally dissimilar, C-CAS computes soft alignment targets by combining three complementary signals:
where $\mathrm{Sim}(T_i, T_j)$ represents the cosine similarity between text embeddings, $\mathrm{Sim}(I_i, I_j)$ between image embeddings, and $\sum_k(C_i \land C_j)$ counts shared class labels between samples $i$ and $j$. The hyperparameters $\alpha, \beta, \gamma$ balance the contribution of each component. This formulation encodes both visual and textual consistency while explicitly incorporating class structure, enabling the model to distinguish hard negatives (e.g., visually similar but distinct diseases) from semantically related pairs (e.g., different manifestations of the same condition).
C-CAP: Class-Aware Contrastive Alignment Pretraining
C-CAP uses the C-CAS targets as continuous supervision during pretraining. Given a batch of $N$ image-text pairs, we extract image embeddings $\mathbf{I}_i$ and text embeddings $\mathbf{T}_j$ from their respective encoders, then compute pairwise cross-modal similarities $S_{i,j} = \mathrm{cosine}(\mathbf{I}_i, \mathbf{T}_j)$. The pretraining objective minimises the mean squared error between predicted similarities and C-CAS targets:
This soft alignment objective encourages the encoders to produce embeddings that reflect not only exact matches but also the degree of semantic relatedness across samples. Compared to binary cross-entropy used in standard CLIP, C-CAP avoids over-penalising semantically related pairs by assigning them intermediate targets rather than forcing them to zero similarity. By replacing binary supervision with continuous, class-aware targets, CAMEL addresses the coarse-grained alignment limitations of standard CLIP-based methods, resulting in representations that preserve fine-grained clinical structure and improve generalisation across diagnostic tasks.
References
- Learning Transferable Visual Models from Natural Language Supervision. ICML, 2021
- MPLUG-OWL3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models. 2024
- PAD-UFES-20: A Skin Lesion Dataset Composed of Patient Data and Clinical Images Collected from Smartphones. Data in Brief, 2020