BabyLM Challenge

• The presentation discusses the BabyLM Challenge, ELC BERT, the Loose Track winner, and various outstanding papers.
• The presentation was given by Lucas Georges Gabriel Charpentier from the Language Technology Group, University of Oslo on December 14th, 2023.
• Charpentier can be reached at [email protected].

BabyLM Challenge

• The challenge was proposed by Alex Warstadt et al.
• Aims to create a small, high-quality dataset matching the number of tokens a 13-year-old child is exposed to.
• It will be run as multiple iterations.
• The challenge aims to create more cognitively plausible models.
• Designed to optimize training pipelines before scaling, democratizing language model pre-training outside the industry.
• Includes a STRICT track with 100M words of developmentally plausible language.
• Includes Encyclopedic knowledge, complex written English, and subtitles.
• Includes a STRICT-SMALL track which is a scaled down version of the STRICT track with only 10M words.
• Includes a LOOSE Track which has a limit of 100M words but allows other data types like audio and images, and enables multimodality.
• LMs grammatical abilities are evaluated using BLIMP, which provides a minimal pair of sentences.
• Models that assign a higher probability to the acceptable sentence are marked as correct.
• Hypernyms and other question-answer congruence tasks are included in the BLIMP supplemental.
• Evaluation uses a mix of both Glue and SuperGlue Benchmarks.
• MSGS tests models for linguistic or surface feature biases.
• Score of -1 means surface bias, a score of 1 means linguistic bias.
• Surface features include lexical content, linguistic features include main verb form.
• Knowledge distillation from data pre-processing has been shown to be helpful.
• Curriculum learning and model scaling have mixed or unclear results.
• Multi-modal learning and training objectives have not been shown to be helpful.
• BabyLM 2024 is confirmed and will explore multi-modal tracks and standardize data preprocessing.
• A survey is available for providing ideas and suggestions for future iterations at https://babylm.github.io/.

ELC BERT

• Standard transformer-based models use standard resudiuals that weigh all layers equally.
• Aims to see whether learning layer weights can produce different weighings for each layer.
• Models are pre-trained constraints using using good quality datasets of 10M and 100M words.
• The approach of LTG-BERT is adapted for all other training choices.
• LTG-BERT was optimized for low-resource MLM.
• Several improvements are implemented by LTG-BERT, including: NormFormer layer normalization, disentangled attention mechanism with relative positions (DeBERTa), GEGLU activation function, high weight decay, no linear biases, and random span masking
• Datasets used to train base versions (~100M parameters) of LTG-BERT and ELC-BERT
• Datasets used to train small versions (~25M parameters) of LTG-BERT and ELC-BERT
• Pretraining datasets for the STRICT and STRICT-SMALL tracks are a mix of 10 different corpora.
• Light preprocessing that normalizes punctuation and whitespaces.
• Within the CHILDES subcorpus, the preprocessing capitalizes the first letter on each line, normalizes punctuation and whitespaces, and puts every line between double quotes as directed speech.
• Similar steps are done for other corpora, including replacing the penn tree format in Children's book test to replace -LRB- and -RRB- with '(' and ')'
• The original paragraphs are restored by Project Gutenberg, where text file is aligned into blocks by inserting a new line every 70 characters, in order to remove the sentence structure.
• Models original resudial connection formula uses a standard encoder flow.
• Models are then modified, new residual connection formula is implemented, and a new encoder flow is used.
• Ablation modifications are also made.
• Layer weighting varies between BERT layer contribution and ELC-BERT layer contribution.
• Not all layers are equally as important when fine tuning models.
• Focus on the previous layer for every layer and the embedding layer fort he first five and last layers.
• Improved performance was seen with (Super)GLUE results, comparable BLIMP performance.

Contextualizer

• Paper: "Towards More Human-like Language Models based on Contextualizer Pretraining Strategy."
• Authored by Chenghao Xiao, G. Thomas Hudson, and Noura Al Moubayed.
• Avoid exposing the knowledge of a domain surrounded by knowledge of the same domain ("contextualization trap").
• Can be seen through math diagrams of how a Contextualizer handles data.
• Shuffling the data and then concatenating and padding it (4b) leads to substantial improvements.
• Little gains were seen by doing a round of clean data before the round after.
• Results work better for the 100M dataset than the 10M dataset.
• Potentially leads to models learning less shortcuts.
• BLIMP results are on par with BERT and 1.2% lower than RoBERTa.

Outstanding papers

• "Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures."
• Written by Julius Steuer, Marius Mosbach, and Dietrich Klakow.
• Access whether GPT-like models can acquire formal and functional linguistic competence and be "cognitively plausible".
• GPT-like models either acquire formal and functional linguistic competence or be "cognitively plausible" but not both.
• Best models on MSGS, GLUE, and BLIMP are larger (over 50M parameters) while best models for reading time are smaller (less than 5M parameters.)
• Model size is not the only important factor in reading time; hidden size is also important.
• No positive effect on reading time of training for multiple epochs. -Developmentally plausible data samples are better for reading time.
• CLIMB—Curriculum Learning for Infant-inspired Model Building.
• Writen by Richard Diehl Martinez, Zébulon Goriely, Hope McGovern, Christopher Davis, Andrew Caines, Paula Buttery, and Lisa Beinborn.
• Aims to explore different types of curriculum learning and improve LM performance.
• A variety of curriculmns can be used with CLIMB that impact BLIMP.
• Different styles of volcabulary curriculum improve different tasks.
• Ordering by difficulty can be useful with multiple copora with data curriculum.
• When implementing objective curriculum, it is better to use a multitask objective rather than sequentially changing objectives.
• Combining curricula showed potential on BLIMP, but no other datasets that were evaluated.
• Noisy data leads to better models than clean datasets with small-corpora.

Other LTG submission

• "Mean BERTs Make Erratic Language Teachers: The Effectiveness of Latent Bootstrapping in Low-Resource Settings."
• Paper by David Samuel.
• The paper tests whether the success of latent supervision for computer vision can be carried to NLP.
• Student language models are compared with mean teacher models based on exponential moving average.
• Fine tuning SuperGlue tasks show improved preformance.
• Results are nuianced for Natural Language Processing, latent supervision is great for computer vision.
• MSGS results are poor, and BLIMP has mixed results.
• Pre-training time is increased by 50%.