Metric Learning과 트리플 손실 퀴즈

Deep Learning Week 14-2

• Deep Learning course, week 14-2 content
• Homework 2 due December 11th, 11:59pm (Wednesday)
• Lecture ends December 11th
• Final exam December 18th (Wednesday)
• Final exam format similar to midterm (multiple choice, true/false, etc.)
• No sample questions provided for final exam

Content

• Metric Learning
  • Learning to Rank
  • Triplet Loss
• Contrastive Learning
  • Pre-trained Language-Vision Model
    • CLIP
    • BLIP
    • LLaVA

Supervised, Unsupervised, Weak Supervised Learning

• Supervised learning: explicit labels (e.g., sentiment classification, machine translation)
• Unsupervised learning: no labels (masked language modeling, autoregressive language modeling)
• Weak supervision: labels in relative relationships

Weak Supervision for Language-Vision Domain

• Example image descriptions:
  • A man preparing desserts in a kitchen covered in frosting.
  • A restaurant has modern wooden tables and chairs.

Metric Learning

• Metric Learning aims to quantify semantic closeness (similarity) between samples
• It learns a "distance function" for semantic distance/similarity.
• Similarity is defined with respect to the given dataset
• "Relative Similarity" is a key concept

Metric Learning (continued)

• Applying supervised machine learning algorithms is difficult
• Easier to collect datasets showing similarity
• Examples include YouTube (watched vs. not watched videos) and search engine results (initial vs. later results)

Learning to Rank: Build a Ranking Model

• Ranking model encodes relative similarity of data
• Training data are lists of items
• Format variations include: point-wise, pair-wise, list-wise
• Ranking model examples: document retrieval (web search, collaborative filtering, recommendation systems)

How can we build a ranking model?

• Point-wise: predict scores for items
• Pair-wise: train models to rank items A over items B in a query
• List-wise: preserve order minimizing the inversions of items within a query list

Triplet Loss

• Training data includes (anchor, positive, negative) examples

• Aim: build a model where the anchor is closer to the positive example than to the negative.

• Measure distances anchor-positive and anchor-negative.

• Minimize the difference between these distances

• "Margin" is used to maintain distance between negative samples

• Constructing negative samples randomly is not ideal

• Challenging negative samples are crucial for effectively learning.

Contrastive Learning

• Contrastive learning uses a pairwise loss function to learn better representations of image and text.
• Similar pairs given label 1, dissimilar pairs label 0
• Pull similar pairs closer and push dissimilar pairs further apart.
• Distance between (anchor, positive) should be small.
• Distance between (anchor, negative) should be large.
• Triplet Loss and Contrastive Learning are similar but differ in how the loss is calculated.

Contrastive Learning: SimCLR, InfoNCE

• SimCLR: loss calculations are given for positive image-text pairs.
• InfoNCE: Inspired by Negative Sampling for word2vec.
• Efficient pairs are generated with random words.

CLIP: Contrastive Language Image Pre-training

• Multiple pre-trained language-vision models exist (e.g., VL-BERT, CoVIRT, ViT-L/14, and VirTex)
• A large dataset of 400M (image, text) pairs from the internet was collected and used for pre-training
• Trial #1 method explained for pre-training.

CLIP: Contrastive Language Image Pre-training (inference)

• Zero-shot prediction method described.
• Training Details (e.g., image encoder, text encoder, batch size, training time)

BLIP: Bootstrapping Language-Image Pre-training...

• BLIP uses a pre-trained encoder-decoder model designed for unified understanding and generation tasks
• Method called MED (Multimodal Mixture of Encoded-Decoder)
• Images and texts are pre-processed to allow the encoder-decoder structure to perform both tasks in a unified model
• The process uses noisy data and a filtering method called "CapFilt"

BLIP (continued)

• Multimodal mixture of encoder-decoder system
• Images and text encoded to produce synthetic captions
• The objective function described in detail
• Model pre-training method, detailed
• Performance achieved on tasks

LLaVA: Large Language and Vision Assistant

• LLaVA (Large Language and Vision Assistant) employs instruction tuning (e.g., prompt completion, fine-tuning based on FLAN-T5)

• Uses Vicuna (language model) and Visual Encoder