Generative Adversarial Networks Quiz

Inference and Training

• Inference can be performed with YOLOv8 and DeepLabv3+
• DeepLabv3+ has a demo available on Google Colab
• Training can be done with YOLOv8 and Unet on ISBI (Image Segmentation Benchmark on ISBI dataset)

Homework

• Train Unet on GTA5 dataset using TensorFlow
• Choose specific parameters for training: number of epochs, batch size, loss function, optimizer, and learning rate
• Use custom data generator and custom data augmentation (random translation, random flip)
• Evaluate the model using scikit-learn functions: confusion matrix, precision, recall, F-score, and accuracy

Agenda

• Artificial Intelligence and Computer Vision Application Domains
• Artificial Intelligence and Computer Vision tasks
• Machine Learning and Deep Learning
• Neural Networks
• Neural Networks for Classification in Computer Vision
• Evaluation and Metrics
• Training Neural Networks
• Implementation challenges
• Neural Networks for other Computer Vision tasks
• More Neural Networks

3D Deep Learning

• PointNet: a deep neural network for 3D classification and segmentation
• PointNet++: a hierarchical feature learning method for 3D point sets
• Neural Radiance Fields (NeRFs): a fully-connected network for 3D scene reconstruction
• Instant-NGP: a library for 3D neural rendering

Audio

• Possible approaches to audio classification: take spectrograms of slices of input and treat them as a sequence or take spectrogram of the input and treat it as an image
• Use a Deep Neural Network to process the input
• Hershey et al. (2015) introduced human-level control through deep reinforcement learning

Autoencoders

• Autoencoders are used for dimensionality reduction, anomaly detection, and generative modeling

GANs

• Generative Adversarial Networks (GANs) consist of a generator and discriminator
• Applications: DeepFakes, style transfer, image-to-image translation, and super resolution
• Nixon DeepFake Clips: In Event of Moon Disaster

DL4NLP

• Probabilistic modeling of word occurrences
• Word embeddings – distributed representation
• Word2Vec is a popular embedding

Transformers

• Probabilistic modeling of word occurrences
• Self-Attention Layer: computes attention over the other positions in the sequence
• Multiple heads (K = 8)

Stable Difusion

• Denoising approach
• Text-to-image task
• A text encoder turns prompt into a latent vector
• A diffusion model repeatedly "denoises" a 64x64 latent image patch

Visual Content Generation

• DALL-E: text-to-image
• SORA: text-to-video
• Zero123: image-to-3D
• DreamFusion: text-to-3D using 2D Diffusion
• Magic3D: Text-to-3D

Deepfakes

• Deepfake: video generated by AI, voice by human imitator
• Morgan Freeman

Sound Generation

• AudioCraft: a library for generative audio models
• MusicGen: text-to-music
• AudioGen: text-to-sound
• EnCodec: neural audio codec
• Multi Band Diffusion: decoder using diffusion
• MAGNeT: text-to-music and text-to-sound

Music Generation

• UDIO.com: generates 30-second segments with lyrics
• Suno.com: generates ~2-minute songs with lyrics