Document detail
ID

oai:arXiv.org:2403.15593

Topic
Computer Science - Computer Vision... Computer Science - Machine Learnin...
Author
Dehdashtian, Sepehr Wang, Lan Boddeti, Vishnu Naresh
Category

Computer Science

Year

2024

listing date

5/22/2024

Keywords
zero-shot computer learning representations
Metrics

Abstract

Large pre-trained vision-language models such as CLIP provide compact and general-purpose representations of text and images that are demonstrably effective across multiple downstream zero-shot prediction tasks.

However, owing to the nature of their training process, these models have the potential to 1) propagate or amplify societal biases in the training data and 2) learn to rely on spurious features.

This paper proposes FairerCLIP, a general approach for making zero-shot predictions of CLIP more fair and robust to spurious correlations.

We formulate the problem of jointly debiasing CLIP's image and text representations in reproducing kernel Hilbert spaces (RKHSs), which affords multiple benefits: 1) Flexibility: Unlike existing approaches, which are specialized to either learn with or without ground-truth labels, FairerCLIP is adaptable to learning in both scenarios.

2) Ease of Optimization: FairerCLIP lends itself to an iterative optimization involving closed-form solvers, which leads to $4\times$-$10\times$ faster training than the existing methods.

3) Sample Efficiency: Under sample-limited conditions, FairerCLIP significantly outperforms baselines when they fail entirely.

And, 4) Performance: Empirically, FairerCLIP achieves appreciable accuracy gains on benchmark fairness and spurious correlation datasets over their respective baselines.

;Comment: The Twelfth International Conference on Learning Representations (ICLR) 2024

Dehdashtian, Sepehr,Wang, Lan,Boddeti, Vishnu Naresh, 2024, FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs

Document

Open

Share

Source

Articles recommended by ES/IODE AI

Exploring Scalability in Large-Scale Time Series in DeepVATS framework
series visual scalability deep learning embeddings time module
Detection of a major Lynch Syndrome-causing MLH1 founder variant in a large-scale genotyped cohort
lynch syndrome mismatch repair deficiency biobank genotype association personalized medicine identified genotyped individuals mlh1 founder cancer variant