CARF: Self-Supervised Contrastive Learning Using Random Feature Corruption
Paper Link
Self-Supervised Learning (SSL) and Unsupervised-Learning are hot topics in recent advances in the ML community.
Today, we will cover a very interesting paper, published in ICLR – 2022 (Link to Conference home page) which is considered a top-tier AI and ML Conference.
The paper suggests a Self-supervised learning method that is suitable for Tabular data!
Why is this an interesting and impactful paper you ask?
Application of Tabular Data has considered one of the fields in ML that Deep learning is not a game changer like CV and NLP.
In addition, SSL which usually lives under Deep Learning is even less common when Tabular data is in place.
Now, I hope I managed to convince you that the following paper is one worth knowing and understanding.
let’s understand what is under the hood 🙂
The Paper
In a nutshell:
- generate a view (distortion) for a given input by selecting a random subset of its features and replacing them with random draws from the features’ respective empirical marginal distributions.
- Train with the contrastive loss paradigm, specifically – InfoNCE loss.
Let’s dive into the paper and its details:
The paper is based on the paradigm of Self-supervised contrastive learning, which enables state-of-the-art performance with orders of magnitude less labeled data. However, most works done in this area are domain-specific (Vision and NLP).
The key idea is to learn representations that are robust to different views or distortions of the same input. Often achieved by maximizing the similarity between views of the same input and minimizing the similarity between those of different inputs via a contrastive loss.
SCARF takes a very similar approach to was is done in VIME, which I covered In a previous post and can be found here, the key difference using Contrastive learning loss vs. mask and feature prediction auxiliary task.
The Algorithm resembles many other Self-supervised and Semi-supervised algorithms
For All Papers Click here
For Computer Vision Paper Click here
Other Articles that you may find interesting:
