Skip to content

machinelearning:: introduce a class-conditional InfoNCE loss alongside supervised contrastive learning to learn spread out feature clusters paper link Improving Transfer and Robustness in Supervised Contrastive Learning · Hazy Research (stanford.edu)

Ideas

  • [[SSL Methods]]: i could do my contrastive matching idea with multiclass. Converse of where we have coarse and fine grained labels.
  • apply infoNCE to #Project_Barretts [[SSL Methods]]
  • use an autoencoder trained on both p53 and h&E reconstruction? that would push it to learn more general features!
    • could concatenate rather than replace the output

Summary

  • In supervised contrastive learning (supcon), we minimize the distance between classes with the same label

  • Dissecting Supervised Contrastive Learning: supervised contrastive loss works well because it encourages each class to be represented as a single point (mega overfitting). Which is great if you're testing on the same dataset that you're running supcon on, but terrible for learning general features. ![[Pasted image 20221121001420.png|400]]

  • Info NCE loss is a popular contrastive loss function that optimizes the log probability of classifying the positive sample correctly given a set of negative samples and one positive example
    • ![[Pasted image 20221121002421.png|400]]
    • where \(f(x,c)\) is a scoring function that takes the anchor sample \(x\) and a context vector \(c\) (indicating the class of the anchor) and outputs a score indicating the likelihood that \(x\) is from \(c\) \(f(x_{pos},c) = \frac{p(x_{pos}|c)}{p(x_{pos})}\) , where \(p(x_{pos}|c)\) is approximated by our model through sigmoid or softmax
  • By applying SupCon alongside a traditional unsupervised contrastive loss (in this case infoNCE), we get a wider spread of features between classes. This paper sought to explore why that is
    • ![[Pasted image 20221121003937.png|400]]
    • (tSNE plots)
  • This paper posits the two challenges to learning transferable representations: balancing the proper amount of spread, and breaking class-fixing permutation invariance

Balancing Spread

  • SupCon trends towards class collapse, while InfoNCE encourages uniformity independent of class (since we treat all non-anchors as negative examples, even if they are in the same class)
  • ![[Pasted image 20221121004356.png|500]]

  • this balance is extremely hard. Both terms are non-convex, so an optimal distribution may not have sufficient spread. Even in an optimal distribution according to the 2 loss terms, our representations could flip from collapsed to completely uniform depending on the relative weights.

    Given these technical challenges, we take a different approach. We construct a simple family of relatively spread out point distributions on which it is easy to compute the loss value. We compare this loss value with the loss of the collapsed or uniform distributions and find that when you add a weighted class-conditional InfoNCE loss to SupCon, the spread out distributions have lower loss than either extreme.

  • In other words, I think they take the InfoNCE loss within each class (to encourage the data within a class to spread out more). The graphic below shows when we weight class-conditional info-NCE more, the cosine similarity within a class decreases. ![[Pasted image 20221121010053.png|500]]

Class-Fixing Invariance

![[Pasted image 20221121010201.png|600]] We could permute the points within the same class and end up with the same loss, despite a much worse representation that doesn't separate the classes at all. - to address this, they show conventional contrastive data augmentations helps subclasses cluster - they concatenate the contrastive model's representations with a class-conditional autoencoder (what is class-conditional? some kind of class-aware autoencoder)

Ideas

  • Bi-encoders take multimodal data as input (image and text pairs) and learn representations of them in a shared embedding space. This alignment of pairs of image and text is analogous to our contrastive task, where we want our representation to be invariant to the modality. They apply their class-aware findings to bi-encoders TABi: Type-Aware Bi-Encoders for Open-Domain Entity Retrieval · Hazy Research (stanford.edu)
  • In general, it seems like bi-encoders could be highly relevant to #Project_Barretts [[SSL Methods]]

Abstract

TL;DR: In this blog post, we take a look at how to learn a representation with good transfer and robustness properties using supervised contrastive learning. In short, we identify two key points: we need to ensure the right amount of “spread” in the geometry of the representation, and we need to break a particular invariance that naturally arises from analysis of the contrastive loss function. We present two modifications to supervised contrastive learning that result in significant improvements in transfer and robustness.