23-24 - M2AI - DL4CV - 1 - Deep Learning 201-229.pdf

Demo Inference with YOLOv8 Inference with DeepLabv3+ https://colab.research.google.com/github/google- research/deeplab2/blob/main/DeepLab_Demo.ipynb Training YOLOv8 Training Unet on ISB...

Demo Inference with YOLOv8 Inference with DeepLabv3+ https://colab.research.google.com/github/google- research/deeplab2/blob/main/DeepLab_Demo.ipynb Training YOLOv8 Training Unet on ISBI José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 201 201 Homework Train (Tensorflow): Data (GTA5 part 1) Unet https://download.visinf.tu- Explicitly choosing: darmstadt.de/data/from_games/ Number of epochs Split (60%-20%-20%) Batch size Custom data Generator Loss function Custom data Augmentation Optimizer Random translation (random crop) Learning rate Random flip (horizontal) Learning rate schedule With callbacks: Evaluate (scikit-learn functions): Tensorboard Confusion matrix Checkpoint Precision Early stopping Recall F-score Accuracy José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 202 202 1  Agenda Artificial Intelligence and Computer Vision Application Domains Artificial Intelligence and Computer Vision tasks Machine Learning and Deep Learning Neural Networks Neural Networks for Classification in Computer Vision Evaluation and Metrics Training Neural Networks Implementation challenges Neural Networks for other Computer Vision tasks More Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 203 203 3D Deep Learning for point clouds PointNet: Qi, Su, Mo, Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, CVPR 2017 https://github.com/charlesq34/pointnet https://github.com/fxia22/pointnet.pytorch PointNet++ Qi, Yi, Su, Guibas, “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space”, NeurIPS 2017 https://github.com/charlesq34/pointnet2 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 204 204 2  3D Neural Radiance Fields (NeRFs) represents a scene using a fully-connected (non-convolutional) network input is a single continuous 5D coordinate (spatial location (x, y, z) and viewing direction (θ, φ)) output is the volume density and view- dependent emitted radiance at that spatial location Allows to use 2D images and their camera poses to reconstruct a volumetric radiance- and-density field, visualized using ray marching https://www.matthewtancik.com/nerf Instant-NGP: https://nvlabs.github.io/instant-ngp/ Nerfstudio: https://docs.nerf.studio/ José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 205 205 SmartPhoneHeadScanner José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 206 206 3  Audio Possible approaches: Take spectrograms of slices of input and treat them as a sequence Possibly with overlapping slices Take spectrogram of the input and treat it as an image Use a Deep Neural Network to process the input https://towardsdatascience.com/audio- classification-using-fastai-and-on-the-fly-frequency- transforms-4dbe1b540f89 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 207 207 Audio Hershey et al. “CNN Architectures for Large-Scale Audio Classification”, International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017 https://research.google/pubs/pub45611/ AudioSet: https://research.google.com/audioset/ ; https://github.com/tensorflow/models/tree/master/research/audioset ; https://github.com/IBM/audioset-classification https://towardsdatascience.com/tagged/audio-classification Audio classification using transfer learning approach: https://medium.com/@krishna_84429/audio-classification-using- transfer-learning-approach-912e6f7397bb Audio Classification using FastAI and On-the-Fly Frequency Transforms: https://towardsdatascience.com/audio-classification-using-fastai-and- on-the-fly-frequency-transforms-4dbe1b540f89 Audio Classification with Pre-trained VGG-19 (Keras): https://towardsdatascience.com/audio-classification-with-pre-trained- vgg-19-keras-bca55c2a0efe Urban Sound Classification using Convolutional Neural Networks with Keras: Theory and Implementation: https://medium.com/gradientcrescent/urban-sound-classification-using- convolutional-neural-networks-with-keras-theory-and-486e92785df4 Raw Waveform-based Audio Classification Using Sample-level CNN https://towardsdatascience.com/audio- Architectures: https://arxiv.org/pdf/1712.00866.pdf classification-using-fastai-and-on-the-fly-frequency- transforms-4dbe1b540f89 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 208 208 4  Speech Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. Thai-Son Nguyen, Sebastian Stueker, Alex Waibel (2020), Super-Human Performance in Online Low-latency Recognition of Conversational Speech, https://arxiv.org/abs/2010.03449 The Babel Fish - Speech-to-Speech Translator https://github.com/ibro45/Speech-to- Speech-Translator Christian Bartz, Tom Herold, Haojin Yang, Christoph Meinel (2017), Language Identification Using Deep Convolutional Recurrent Neural Networks , ICONIP 2017 https://github.com/HPI-DeepLearning/crnn-lid Language Identification From Speech https://github.com/ibro45/Language-Identification- Speech spoken language identification, https://github.com/tomasz- oponowicz/spoken_language_identification José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 209 209 RNNs Recurrent Neural Networks https://towardsdatascience.com /recurrent-neural-networks- b7719b362c65 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 210 210 5  RNNs Recurrent Neural Networks take inputs as time sequences The layer output for t-1 is used as an input for t Useful for video or NLP Popular choices are LSTM and GRU https://classroom.udacity.com/courses/ud187/lessons/6d543d5c-6b18-4ecf-9f0f- 3fd034acd2cc/concepts/1b8a6e07-b2d3-4e4c-904c-822217cd8335 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 211 https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21 211 Architecture Search Automatic learning of network structures - AutoML NASnet – first Deep Neural Network created by a Deep Neural Network Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le, “Learning Transferable Architectures for Scalable Image Recognition”, CVPR 2018 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 212 212 6  Multimodal Learning Learn from multiple types of inputs (Video + Audio, Video + Audio + Text) Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman Mohamed (2022), Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction, ICLR 2022 https://github.com/facebookresearch/av_hubert?tab=readme-ov-file Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, Antonio Torralba (2018), “The Sound of Pixels”, ECCV 2018 Zhang ICMR16 https://github.com/hangzhaomit/Sound-of-Pixels https://www.youtube.com/watch?v=LjPEAn6ehwo Ruohan Gao and Kristen Grauman, “2.5D Visual Sound”, CVPR 2019 https://github.com/facebookresearch/2.5D-Visual-Sound VisualBERT: A Simple and Performant Baseline for Vision and Language https://github.com/uclanlp/visualbert Shiqing Zhang, Shiliang Zhang, Tiejun Huang, Wen Gao (2016), Multimodal Deep Convolutional Neural Network for Audio-Visual Emotion Recognition, ICMR2016 Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency (2019), Multimodal Machine Learning: A Survey and Taxonomy, PAMI 2019 Zhao CVPR2019 https://ieeexplore.ieee.org/document/8269806, https://arxiv.org/abs/1705.09406 Reading List for Topics in Multimodal Machine Learning https://github.com/pliang279/awesome-multimodal-ml CMU 11-777: Multimodal Machine Learning https://piazza.com/cmu/fall2018/11777/resources Bowen Shi ICLR 2022 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 213 213 Deep Q Learning Reinforcement Learning (Q-Learning) Use a DL network to learn Q-values Google DeepMind's Deep Q- learning playing Atari Breakout Mnih, V., Kavukcuoglu, K., Silver, D. et al. (2015). Human-level control through deep reinforcement learning, Nature 518, 529–533 https://doi.org/10.1038/nature14236 https://www.youtube.com/watch?v= V1eYniJ0Rnk&t=22s https://github.com/kuz/DeepMind- Atari-Deep-Q-Learner José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 214 214 7  Autoencoders Slide Andreas Geiger José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 215 215 Autoencoders Slide Andreas Geiger José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 216 216 8  GANs Generative Adversarial Networks Generator trained to produce fake synthetic images Discriminator trained to classify as Real/Fake Alternate between training the generator and the discriminator Applications DeepFakes - Generate realistic synthetic faces/images/sound/video Style Transfer Image-to-image translation Super resolution Nixon DeepFake Clips: In Event of Moon Disaster https://www.youtube.com/watch?v=yaq4sWFvn AY https://developers.google.com/machine-learning/gan José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 217 217 GANs Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. NIPS, 2014 StyleGAN - A Style-Based Generator Architecture for Generative Adversarial Networks https://github.com/NVlabs/stylegan https://github.com/NVlabs/stylegan2 - TensorFlow 1.14 Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila (2020) “Analyzing and Improving the Image Quality of StyleGAN” https://arxiv.org/abs/1912.04958 https://www.youtube.com/watch?v=SWoravHhsUU Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, “Image-to-Image Translation with Conditional Adversarial Networks”, CVPR 2017 https://github.com/phillipi/pix2pix Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”, ICCV 2017 https://github.com/junyanz/CycleGAN Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, Xiaoou Tang (2018), ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks, ECCV 2018 Workshops https://arxiv.org/abs/1809.00219 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 218 218 9  DL4NLP https://uni-tuebingen.de/fakultaeten/mathematisch-naturwissenschaftliche- fakultaet/fachbereiche/informatik/lehrstuehle/autonomous-vision/lectures/deep-learning/ Probabilistic modeling of word occurrences Models are typically trained to output the probability of the next word in the sentence Word embeddings – distributed representation Words are represented by a vector RN in a lower dimensional space Word2Vec is a popular embedding José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 219 219 Transformers Probabilistic modelling of word occurrences Models are typically trained to output the probability of the next word in the sentence Self-Attention Layer For each position t in a sequence we compute an attention over the other positions in the sequence Multiple heads (K = 8) Self-attention constructs a tensor A[k; t1; t2] – the strength of the attention weight from t1 to t2 for head k http://jalammar.github.io/illustrated-transformer/ https://nlp.seas.harvard.edu/annotated-transformer/ José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 220 220 10  Stable Difusion Denoising approach Text-to-image task A text encoder turns prompt into a latent vector A diffusion model repeatedly "denoises" a 64x64 latent image patch A decoder turns the 64x64 latent patch into a 512x512 image Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models”, CVPR 2022 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 221 221 Visual Content Generation DALL-E: text-to-image https://openai.com/index/dall-e-3/ Used by copilot SORA: text-to-video https://openai.com/index/sora/ Zero123: image-to-3D https://github.com/cvlab-columbia/zero123 DreamFusion: text-to-3D using 2D Diffusion https://dreamfusion3d.github.io/ https://github.com/ashawkey/stable-dreamfusion Magic3D: Text-to-3D https://research.nvidia.com/labs/dir/magic3d/ José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 222 222 11  Deepfakes Morgan Freeman Deepfake Video generated by AI Voice by human imitator José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 223 223 Sound Generation AudioCraft Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexandre https://about.fb.com/news/2023/08/audiocraft-generative-ai-for-music-and-audio/ Défossez, “Simple and Controllable Music Generation”,NeurIPS2023, https://papers.nips.cc/paper_files/pa per/2023/hash/94b472a1842cd7c56 dcb125fb2765fbd-Abstract- Conference.html MusicGen: text-to-music AudioGen: text-to-sound EnCodec: neural audio codec Multi Band Diffusion: decoder using diffusion MAGNeT: text-to-music and text-to- sound https://github.com/facebookresearch/ audiocraft https://www.techspot.com/news/99659-audiocraft-open-source-audio-generative-ai-model-meta.html José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 224 224 12  Music Generation UDIO.com Text prompt -> 30 second segments with lyrics Suno.com Text prompt -> ~2 minute songs with lyrics José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 225 225 Other optimization applications Thomas Geijtenbeek, Michiel van de Panne, A. Frank van der Stappen (2013), Flexible Muscle- Based Locomotion for Bipedal Creatures, SIGGRAPH Asia 2013 https://www.goatstream.com/res earch/papers/SA2013/ José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 227 227 13  Demo Audio Recognition https://www.tensorflow.org/tutorials/audio/simple_audio Autoencoder https://www.tensorflow.org/tutorials/generative/autoencoder Generative Adversarial Network https://www.tensorflow.org/tutorials/generative/dcgan Stable Diffusion José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 228 228 Summary Artificial Intelligence and Computer Vision Application Domains Automotive, Manufacturing, Security and Defense, Agriculture, Retail, Healthcare, Media Artificial Intelligence and Computer Vision tasks AI, ML, Deep Learning, Computer Vision tasks, Traditional Approach vs Deep Learning Approach Machine Learning and Deep Learning Supervised Learning, Evaluation and Metrics overview, Features and Classifiers Neural Networks Neurons and Neural Networks, Deep Neural Networks, Activations and Label Encoding, Convolutional Neural Networks Neural Networks for Classification in Computer Vision LetNet, AlexNet, GoogLeNet, VGG, ResNet Evaluation and Metrics Classification, Object detection/Segmentation, Tracking Training Neural Networks Gradient descent and parameter updates, Forward pass and backward pass, Normalization, Loss functions, Optimizers, Learning rate, Generators, Callbacks Implementation challenges Training challenges, Transfer Learning, Data Augmentation, Synthetic Datasets, Inference challenges, Model Compression Neural Networks for other Computer Vision tasks Classification, object detection, semantic segmentation, instance segmentation More Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 229 229 14