23-24 - M2AI - DL4CV - 1 - Deep Learning 93-134.pdf

Agenda Artificial Intelligence and Computer Vision Application Domains Artificial Intelligence and Computer Vision tasks Machine Learning and Deep Learning Neural Networks Neural Networks for Classificat...

Agenda Artificial Intelligence and Computer Vision Application Domains Artificial Intelligence and Computer Vision tasks Machine Learning and Deep Learning Neural Networks Neural Networks for Classification in Computer Vision Evaluation and Metrics Classification Object detection/Segmentation Tracking Training Neural Networks Implementation challenges Neural Networks for other Computer Vision tasks More Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 93 93 Evaluation strategy Dataset split Training set (~60%) Validation set (~20%) Test set (~20%) Stratified split Considering the classes Random shuffle https://tarangshah.com/blog/2017-12-03/train-validation-and-test-sets/ José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 94 94 1  Evaluation strategy Cross-validation Successively train and evaluate on different sets of data k-fold cross-validation https://en.wikipedia.org/wiki/Cross-validation_(statistics) José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 95 95 Evaluation Metrics Classification – binary (2 class) problems True Positives – correctly identified positives (class 1) instances True Negatives – correctly identified negatives (class 0) instances False Positives – incorrectly classified as positives (class 1) that are really negatives (class 0) False Negatives – incorrectly classified as negatives (class 0) that are really positives (class 1) José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 96 96 2  Evaluation Metrics Confusion Matrix José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 97 97 Evaluation Metrics Confusion Matrix - normalized José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 98 98 3  Evaluation Metrics 𝑇𝑃 Precision = 𝑇𝑃+𝐹𝑃 𝑇𝑃 Recall = 𝑇𝑃+𝐹𝑁 𝑇𝑃+𝑇𝑁 Accuracy = 𝑇𝑃+𝑇𝑁+𝐹𝑃+𝐹𝑁 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛∗𝑅𝑒𝑐𝑎𝑙𝑙 F1score = 2 ∗ 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑅𝑒𝑐𝑎𝑙𝑙 https://en.wikipedia.org/wiki/Pr ecision_and_recall José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 99 https://en.wikipedia.org/wiki/Precision_and_recall 99 Evaluation Metrics Multiclass problems José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 100 100 4  Regression Metrics MSE – Means Squared Error 1 𝑛 σ (𝑦 − 𝑦 ′ )2 𝑁 1 MAE – Mean Absolute Error 1 𝑛 σ 𝑦 − 𝑦′ 𝑁 1 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 101 101 Evaluation Metrics Object detection Intersection-over-Union (IoU) - Jaccard index 𝐴∩𝐵 𝑇𝑃 𝐼𝑜𝑈 = 𝐴∪𝐵 = 𝑇𝑃+𝐹𝑃+𝐹𝑁 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 102 102 5  Evaluation Metrics Object detection 2(𝐴∩𝐵) 2𝑇𝑃 Dice index = = 2𝑇𝑃+𝐹𝑃+𝐹𝑁 |𝐴|+|𝐵| José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 103 103 Evaluation Metrics Object detection metrics (1 class): Intersection-over-Union [email protected] / [email protected] [email protected] / [email protected] Average Precision (AP) / Average Recall (AR) – IoU=0.50:0.05:0.95 Object detection metrics (multiclass): mean Average Precision (mAP)/mean Average Recall (mAR) mean of AP/AR for all classes Keras implementation MSCOCO Python Toolbox implementation José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 104 104 6  Evaluation Metrics Semantic Segmentation metrics: Pixel-wise classification metrics Precision, Recall, F-Score, Accuracy Segmentation Area Metrics Mean Intersection-over-Union IoU for each class Average over classes Keras implementation José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 105 105 Evaluation Metrics Tracking metrics (50% IoU threshold): σ𝑖,𝑡 𝑑𝑡𝑖 MOTP - Multiple object tracker precision (mm) - = σ𝑡 𝑐𝑡 error in estimated position for matches over all frames, averaged by total number of matches tracker ability to estimate position, independent of skill at recognizing object or keeping consistent trajectories σ𝑡 𝐹𝑁𝑡 +𝐹𝑃𝑡+𝑀𝑀𝑡 MOTA - Multiple object tracker accuracy (%) = 1 − σ𝑡 𝐺𝑇𝑡 Bernardin, Keni & Elbs, Alexander & Stiefelhagen, Rainer. (2006), “Multiple object tracking performance metrics and evaluation in a smart Room environment”, IEEE International Workshop on Visual Surveillance MOTChallenge.net/devkit py-motmetrics José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 106 106 7  Evaluation Metrics σ𝑖,𝑡 𝑑𝑡𝑖 MOTP - Multiple object tracker precision (mm) - = σ𝑡 𝑐𝑡 σ𝑡 𝐹𝑁𝑡 +𝐹𝑃𝑡 +𝑀𝑀𝑡 MOTA - Multiple object tracker accuracy (%) = 1 − σ𝑡 𝐺𝑇𝑡 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 107 107 Evaluation Metrics Tracking metrics: Track quality measures mostly tracked (MT) >80%, partially tracked (PT) 20%, mostly lost (ML) α (threshold) Each predicted detection may overlap with more than one ground-truth detection Hungarian Algorithm for one-to-one matching Use matched prediction-GT pairs to compute DetA José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 110 110 9  Evaluation Metrics HOTA: Higher Order metric for Tracking Accuracy AssA: Association Accuracy Association measures how well a tracker links detections over time into the same identities (IDs) Take each detection-GT pair and measure alignment between prediction track and GT track José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 111 111 Evaluation Metrics HOTA: Higher Order metric for Tracking Accuracy Overview e implementation Luiten, J., et al. HOTA: A Higher Order Metric for Evaluating Multi-object Tracking. IJCV 129, 548– 578 (2021) https://autonomousvision.github.io/hota-metrics/ https://github.com/JonathonLuiten/TrackEval José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 112 112 10  Homework Evaluate performance metrics of 5 different pretrained classification networks with ImageNet calibration set Precision, Recall, F-Score, Accuracy Confusion Matrix (absolute+normalized) Inference Frame rate (fps) José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 113 113 Agenda Artificial Intelligence and Computer Vision Application Domains Artificial Intelligence and Computer Vision tasks Machine Learning and Deep Learning Neural Networks Neural Networks for Classification in Computer Vision Evaluation and Metrics Training Neural Network Gradient descent and parameter updates, Forward pass and backward pass Normalization, Loss functions, Optimizers, Learning rate Generators, Callbacks Implementation challenges Neural Networks for other Computer Vision tasks More Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 114 114 11  Training Neural Networks “Training means optimizing the parameters so that the network’s output is equal (or close) to the groundtruth” But how do we do this? José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 115 115 Training Neural Networks Training means iteratively changing the weights, so that the output is closer and closer to a known correct result (Ground Truth label) Initialize the weights randomly Define a loss function E.g. error = |a3 – a3gt| Apply Gradient Descent on the weight values to minimize the sum of errors Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural computation, vol. 1, no. 4, pp. 541–551, 1989. José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 116 https://towardsdatascience.com/gradient-descent-algorithm-and-its-variants-10f652806a3 116 12  Training Neural Networks Training is iteratively changing each parameter by looking at examples José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 117 117 Learning rate Low leaning rate will make the network learn too slowly High learning rate will not allow the network to converge https://www.jeremyjordan.me/nn-learning-rate/ José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI http://cs231n.github.io/neural-networks-3/ 118 118 13  Training Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 119 119 Training Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 120 120 14  Training Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 121 121 Training Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 122 122 15  Training Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 123 123 Training Neural Networks Remember, training is updating the parameters W José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 124 124 16  Training Neural Networks 𝜕𝐿 We need: 𝜕𝑊 3 to update W3 𝜕𝑦 𝜕𝑦 𝜕𝑢 𝜕𝐿 Chain rule: = ∙ 𝜕ℎ2 to update layer 2 𝜕𝑥 𝜕𝑢 𝜕𝑥 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 127 127 Training Neural Networks 𝜕𝑦 𝜕𝑦 𝜕𝑢 = ∙ 𝜕𝑥 𝜕𝑢 𝜕𝑥 José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 128 128 17  Training Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 130 130 Training Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 131 131 18  Training Neural Networks José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ BAIT 132 132 Training summary To train a network you need: Labeled dataset Network structure definition Training settings Loss function Optimizer Optimizer parameters Batch size Epochs Learning rate/Learning rate scheduler José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 133 133 19  Demo Train a fully connected network on the FashionMNIST dataset Train a convolutional neural network on the FashionMNIST dataset Train a convolutional neural network on the FashionMNIST dataset with PyTorch José Henrique Brito | 2Ai - EST - IPCA, PT | DL4CV @ M2AI 134 134 20