Deep Learning Quiz

Deep Learning: A Branch of Machine Learning

• Deep learning is a part of machine learning that uses artificial neural networks with representation learning.

• It allows for supervised, semi-supervised, and unsupervised learning.

• Deep neural networks, deep belief networks, deep reinforcement learning, recurrent neural networks, convolutional neural networks, and transformers are some deep-learning architectures.

• These have been applied to various fields, including computer vision, speech recognition, natural language processing, medical image analysis, and material inspection.

• ANNs have various differences from biological brains, and deep learning is concerned with an unbounded number of layers of bounded size.

• Deep learning eliminates feature engineering, and deep learning algorithms can be applied to unsupervised learning tasks.

• The word "deep" refers to the number of layers through which the data is transformed.

• There is no universally agreed-upon threshold of depth dividing shallow learning from deep learning, but most researchers agree that deep learning involves CAP depth higher than 2.

• Deep learning architectures can be constructed with a greedy layer-by-layer method.

• Deep neural networks are generally interpreted in terms of the universal approximation theorem or probabilistic inference.

• Deep learning has a rich history, with many different neural networks and architectures being developed over the years.

• Long short-term memory (LSTM) has become the most cited neural network of the 20th century, and residual neural network is the most cited neural network of the 21st century.The Evolution of Deep Learning in Speech Recognition and Computer Vision

• In 1995, Brendan Frey demonstrated that it was possible to train a network containing six fully connected layers and several hundred hidden units using the wake-sleep algorithm, co-developed with Peter Dayan and Hinton.

• Sven Behnke extended the feed-forward hierarchical convolutional approach in the Neural Abstraction Pyramid by lateral and backward connections in order to flexibly incorporate context into decisions and iteratively resolve local ambiguities.

• Deep learning methods for speech recognition were not as successful as non-uniform internal-handcrafting Gaussian mixture model/Hidden Markov model (GMM-HMM) technology based on generative models of speech trained discriminatively because of gradient diminishing and weak temporal correlation structure in neural predictive models.

• The speaker recognition team at SRI International reported significant success with deep neural networks in speech processing in the 1998 National Institute of Standards and Technology Speaker Recognition evaluation, and the SRI deep neural network was then deployed in the Nuance Verifier, representing the first major industrial application of deep learning.

• The principle of elevating "raw" features over hand-crafted optimization was first explored successfully in the architecture of deep autoencoder in the late 1990s, showing its superiority over the Mel-Cepstral features that contain stages of fixed transformation from spectrograms.

• Speech recognition was taken over by LSTM, which started to become competitive with traditional speech recognizers on certain tasks in 2003.

• CNNs were superseded for ASR by CTC for LSTM, but are more successful in computer vision.

• Advances in hardware have driven renewed interest in deep learning, with GPUs well-suited for the matrix/vector computations involved in machine learning.

• Deep learning started to outperform other methods in machine learning competitions in the late 2000s.

• In 2011, the DanNet achieved superhuman performance in a visual pattern recognition contest, outperforming traditional methods by a factor of 3.

• In 2014, Sepp Hochreiter's group used deep learning to detect off-target and toxic effects of environmental chemicals in nutrients, household products and drugs and won the "Tox21 Data Challenge" of NIH, FDA and NCATS.

• In March 2019, Yoshua Bengio, Geoffrey Hinton and Yann LeCun were awarded the Turing Award for conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing.

• Deep neural networks can model complex non-linear relationships, but they are prone to overfitting and computation time issues.Deep Learning: Techniques, Applications, and Hardware

• Deep neural networks (DNNs) are a type of machine learning algorithm that can learn from large amounts of data and can be used in a variety of applications, including automatic speech recognition, image recognition, and natural language processing.

• DNNs are composed of many layers of non-linear hidden units and a very large output layer, which can lead to overfitting. To combat this, regularization techniques such as ℓ2-regularization or sparsity (ℓ1-regularization) can be applied during training, or dropout regularization can randomly omit units from the hidden layers during training.

• DNNs require many training parameters, such as the size (number of layers and number of units per layer), the learning rate, and initial weights. Various tricks, such as batching and the use of GPUs, have been developed to speed up computation.

• Engineers may also look for other types of neural networks with more straightforward and convergent training algorithms, such as the cerebellar model articulation controller (CMAC).

• Advances in both machine learning algorithms and computer hardware have led to more efficient methods for training DNNs. GPUs have displaced CPUs as the dominant method of training large-scale commercial cloud AI. Deep learning processors, such as neural processing units (NPUs) and tensor processing units (TPUs), have been designed to speed up deep learning algorithms.

• Deep learning has been successfully applied in various fields, including automatic speech recognition, image recognition, natural language processing, drug discovery and toxicology, medical image analysis, and financial fraud detection.

• Deep learning has also been used in recommendation systems, bioinformatics, customer relationship management, mobile advertising, image restoration, and military applications.

• Deep learning has been shown to produce competitive results in medical applications such as cancer cell classification, lesion detection, organ segmentation, and image enhancement.

• Deep learning has been successfully applied to inverse problems such as denoising, super-resolution, inpainting, and film colorization.

• Deep learning is being used to train robots in new tasks through observation by the United States Department of Defense.

• Deep learning is closely related to a class of theories of brain development proposed by cognitive neuroscientists in the early 1990s. These developmental models share the property that various proposed learning dynamics in the brain (e.g., a wave of nerve growth factor) support the self-organization somewhat analogous to the neural networks utilized in deep learning models.

• Deep learning has commercial applications, with major commercial speech recognition systems, recommendation systems, and mobile advertising systems based on deep learning.Deep Learning: Theory, Criticisms, and Concerns

• Facebook's AI lab performs tasks such as automatically tagging uploaded pictures with the names of the people in them.

• Google's DeepMind Technologies developed a system capable of learning how to play Atari video games using only pixels as data input.

• In 2015, they demonstrated their AlphaGo system, which learned the game of Go well enough to beat a professional Go player.

• Google Translate uses a neural network to translate between more than 100 languages.

• Covariant.ai was launched in 2017, which focuses on integrating deep learning into factories.

• The University of Texas at Austin (UT) developed a machine learning framework called Training an Agent Manually via Evaluative Reinforcement (TAMER), which proposed new methods for robots or computer programs to learn how to perform tasks by interacting with a human instructor.

• A new algorithm called Deep TAMER was introduced in 2018 during a collaboration between the US Army Research Laboratory (ARL) and UT researchers.

• Deep TAMER used deep learning to provide a robot the ability to learn new tasks through observation.

• Deep learning methods are often looked at as a black box, with most confirmations done empirically, rather than theoretically.

• Deep learning consists of dozens and even hundreds of layers, while the brain itself consists of very few layers.

• Deep learning architectures display problematic behaviors, such as confidently classifying unrecognizable images as belonging to a familiar category of ordinary images (2014) and misclassifying minuscule perturbations of correctly classified images (2013).

• Most Deep Learning systems rely on training and verification data that is generated and/or annotated by humans.