Game Audio History - Introduction

7 - Game Audio - Introduction Sound Design - Games and Multimedia Miguel Negrão ©2022 CC BY-NC-ND 4.0 Quick history of sound in video games  The first video games were silent e.g. Spacewar! (1962) https://www.youtube.com/watch?v=Rmvb4Hktv7U Spacewa...

7 - Game Audio - Introduction Sound Design - Games and Multimedia Miguel Negrão ©2022 CC BY-NC-ND 4.0 Quick history of sound in video games  The first video games were silent e.g. Spacewar! (1962) https://www.youtube.com/watch?v=Rmvb4Hktv7U Spacewar! - PDP-1 - One of the First Video Games (MIT 1962)  Arcade Video Games The first mass-produced video games were coin-operated in arcades. These games had rudimentary sound, e.g. Pong (Atari, 1972). Sound was important to attract gamers in the arcade. Pong: https://www.youtube.com/watch?v=fiShX2pTz9A  Arcade Video Games - 1970s Extremely limited, custom hardware. Early arcade games had only introduction and game-over music. During game play only basic sound effects. "Space invaders" (1978) - one of the first with continuous music.  SPACE INVADERS PART 2 ARCADE MAME VIDEO GAME TAITO 1979 invadpt2 Space invaders: https://youtu.be/kR2fjwr-TzA?t=392  Arcade Video Games - 1980s Dedicated sound chips - programmable sound generators (PSGs). These are programmable synthesizers. Example: 3 oscillators + 1 noise generator. Sequences of notes or simple sounds are sent to the synthesizers. Eventually DACs were introduced which could play sound files (mostly very short percussive sounds). Some arcade games had multiple sound chips to be able to play more sounds.  Arcade Video Games - 1980s Arcade Game: Frogger (1981 Konami) Frogger: https://youtu.be/WNrz9_Fe-Us?t=181  Home consoles 1977 - Atari Video Computer System - 2 channel. 1983 - Nintendo Entertainment System - 5 channel PSG 2 pulse-waves 1 triangle-wave 1 noise generator 1 sampler (for voice, etc). (channels in this context = number of independent sounds) 1983 - NES - Super Mario Bros Super Mario Bros. - NES Gameplay https://youtu.be/ia8bhFoqkVE?t=178  First Personal Computers 1984 - IBM PCjr https://www.youtube.com/watch?v=s5a6jKUkjlA IBM PCjr Gam… Gam… Apple II (+ additional sound chips) Commodore 64 At the time quite high quality. Commodore …  16 bits consoles New advancements in sound hardware: 1988 - Sega Megadrive - FM synthesis 1990 - Super Nintendo Entertainment System - Wavetable synthesis  16 bits consoles Sega Megadrive/Genesis PSG 3+1 chip Yamaha FM synthesis chip (YM 2612) 6 channels of digitized stereo sound 1 channel PCM 8-bit usually used 3 channels (percussion, bass, melody), others were used for filler chords, or arpeggios. delays between channels to create a fatter sound, phasing or flanging.  16 bits consoles Sonic the Hedgehog - Green Hill Zone Sega Megadrive/Genesis Sonic The Hedgehog https://youtu.be/N6tSowlt7E4?t=37  FM synthesis Sound synthesis technique invented by John Chowning at Stanford University. PSG, hardware synthesizers. wider range of timbres and sounds. Yamaha DX7 https://www.youtube.com/watch?v=F3rrjQtQe5A Yamaha DX7 -… -…  16 bits consoles Super Nintendo Entertainment System Sony SPC-700 16-bit Sony digital signal processor - wavetable synthesis. Play waveform at given frequency. 16-bit stereo DAC. MIDI files could be used to create the music. 16 bits consoles - Super Nintendo Entertainment System The Legend of Zelda: https://youtu.be/Z6hjG6MCcZ8?t=10795 SNES Longplay The Legend of Ze… Ze… Star Ocean : https://youtu.be/k-ndSEMJM3E?t=79 Star Ocean (SNES) - Intro  PC + soundcard Adlib (1987), Sound Blaster (1989) - FM chip 1992 - Sound Blaster 16 - 16bit PCM + CD- ROM (FM still used). 1997 - AC'97 Full PCM streaming. (16- or 20-bit audio with 5.1 surround sound support) 2004 - Intel High Definition Audio - Up to 15 input and output streams, up to 16 PCM audio channels per stream.  Dune gameplay (PC Game, 1992) Dune (1992) https://www.youtube.com/watch?v=2Rvo_dJZDj0  1983 - Musical Instrument Digital Interface (MIDI) protocol In console and PC games MIDI started to be used for the music tracks in the 80s and 90s. Games music is stored as sequences of notes. Notes are converted to sounds in realtime using hardware synthesizers. Different sound card/synthesizer → Different sound produced. Quality of sound depends heavily on the sound card used.  PC - MIDI music Monkey Island 2: LeChuck's Revenge (1991) https://youtu.be/tY0OwHUS8r4?t=1250 Monkey Island 2: LeChuck's Revenge (1991) PC Playthrough - Ni… Ni…  Musical Instrument Digital Interface (MIDI) protocol A protocol of communication between digital musical instruments. Does not send audio. Sends only control information. Example: play the note 50, stop the note 50, change volume to 45, change modulation to 87, etc.  Musical Instrument Digital Interface (MIDI) protocol Was developed in 1980s to create standard for interfacing music synthesizers, keyboards, etc. Originally geared toward professional musicians. 16 channels (= 16 tracks) Low amount of data transmitted. E.g. a single note only transmits 3 bytes.  General MIDI Table with predefined sounds Up to the synthesizer how to create each particular sound. 1 Acoustic Grand Piano 2 Bright Acoustic Piano 3 Electric Grand Piano 4 Honky-tonk Piano 5 Electric Piano 1 6 Electric Piano 2 7 Harpsichord 8 Clarinet...  MOD Originally used for creating game music for the Amiga computer in 1987. Stores a small sample for each instrument as PCM data. Sample is loaded onto sound card, notes are still sent similarly to MIDI. Similar formats: SoundFonts, DLS  PCM audio Eventually sound cards were able to play arbitrary PCM sound files. Due to bandwidth/storage limitations, PCM audio was used only for (short) sound effects. Music continues to be played via MIDI + synthesizer (part of the soundcard).  CD-ROM With the introduction of the CD-ROM it was possible to have entire music tracks stored as PCM audio files. The quality of music in CD-ROM games became similar to any music recording. Music for games could be created by same means used to create music recordings (in studio). DVD-ROM → no more storage limitations for audio in games.  Current game engines Sounds for sound effects, dialogue and music are created and imported as PCM sound files. These sound files are played at appropriate times. (Ironically synthesizers are coming back, but this time around in software, for instance Unreal Engine 4.17 (2017), New Audio Engine introduces a modular synth, and 5.0 introduces Metasounds which can implement modular synths.)  3D games - positional audio With the introduction of 3D games (Doom, etc), the variation of sound due to the position of objects relative to the player started to be simulated with increasing realism. Usually in game engines this is called "2D audio" vs "3D audio".  Original Doom Gameplay [Nightmare Difficulty] Doom (1996): https://youtu.be/8mEP4cflrd4?t=71  3D games - Realism Acoustic properties such as room size are also sometimes simulated. Surround sound allows for the sound source to be in any direction in a horizontal plane. With VR helmets - binaural HRTF based 3D sound rendering. Occlusion, etc.  THE LAST OF US 2 Gameplay Walkthrough Part 1 FULL GAME [1080p HD PS4 PRO] - No C… C… The last of us 2: https://youtu.be/z1OESz4zpLs?t=13773  Algorithm music Lucas Arts iMusic (Monkey Island series , 90s) Spore (2008, generative music by Brian Eno) Uurnog https://connect.unity.com/p/articles-making-uurnogs- algorithmic-music Uurnog - Custom Music Software Characteristics of sound in digital games  Interactivity Games are interactive. Because the sequence of events that happen in a game is not know in advance, game audio cannot be just a single pre-mixed sound file (like in films). The mixing of the different sounds must be done in realtime as the game is played.  Examples of sounds which appear in a digital game: Music Sounds which happen due to some action which is visible on screen (shoot a gun, walk, etc). Dialogue with non-player characters. Voice of a narrator. Sounds from clicking on a 2D UI (e.g. pause menu). Notification sounds (e.g. powerup) Classification of sounds classification of sounds: Diegetic vs Non-Diegetic "diegesis" → from Latin "recounted history". Diegetic sound → comes from sources on screen or which are implied to be present. Non-diegetic sound → comes from sources which are not present on screen and are not present in the current action. Diegetic Shoot a gun, walk, open a door, collisions. Dialogue with non-player characters. Non-diegetic Music Voice of a narrator. Sounds from clicking on a 2D UI (e.g. pause menu). Notification sounds (e.g. power-up).  Classification of sounds: dynamic vs static Static audio → always plays the same, e.g. game over music. Dynamic audio → audio that does not play always the same way or at the same moment in time. Interactive audio → a sound event which occurs as a direct reaction to an action by the player. E.g. press a button, gun fires. Adaptive audio → audio which changes in reaction to the game play and the current state of the game. E.g. health deteriorates, music becomes somber. Classification of sounds in 3D games: by function Ambient sounds Environments in the real world always have some type of background sound. Interaction sounds Pressing button, firing gun. Physical simulation collisions, footsteps, falling into water, swimming. Classification of sounds in 3D games: by function Dialogue Narrator Notifications, HUD. Sounds from NPCs. Cutscenes. Classification of sounds: 2D vs 3D object In a 3D game the character moves in a virtual 3D world. It is possible to simulate the presence of a sound in this 3D world by changing d Volume according to the distance between player and the sound source, θ and panning according to the direction of the sound source relative to the player. character Classification of sounds: 2D vs 3D object Sounds with this type of simulation applied are called 3D sounds. Sounds without this simulation are called 2D Sounds. d 2D sounds will play without any attenuation or panning due to player movement. They θ will appear to come always from the center (in stereo systems). character  Classification of sounds: 2D vs 3D These terms are a bit confusing, as actually in most games simulated sound sources (what we called "3D sounds") can actually only be spatialized in a 2D plane, as most systems don't provide a way to simulate height (put a sound above your head). This has finally changed with VR and binaural which can place sounds above/below the player using HRTFs.  Classification of sounds: 2D vs 3D Some sounds can have only attenuation without panning. This is useful when the sound has a large size, and the player can "enter" into the sound. This is the case with most ambient sounds. object d θ character  Examples of 3D sounds: Collisions, footsteps, water splash. Ambient sound from small objects. NPC weapon sounds. NPC vehicles. Dialogue from on screen NPC. Music, dialogue coming from objects in the environment (TVs, radios, etc).  Examples of 2D sounds: All non-diegetic sounds UI sounds (e.g. menu, HUD) Music Narrator Character thinking voice. In first person games, possibly sounds emanating from the character, such as footsteps, gun shots, character dialogue.  Classification of sounds: constant vs one-shot Constant: the sound is continuously present. Implemented by looping sound file. One-shot: appears as consequence of event/action. Does not loop. Use of randomization.  Classification of sounds: constant vs one-shot Constant: Forest Sea Traffic Office noise Air conditioning Server box Projector Space station  Classification of sounds: constant vs one-shot One-shot: door opening dialogue weapons collisions dog barking. single bird sound.  3D game: Ambient sounds Area Loop Source Loop Source one-shot  3D game: Ambient sounds Area Loop The background sound of a location. Relatively long sound (2-3m) seamlessly looped. Usually only attenuation and no panning. Covers large area in map. Example: forest, cave, office, traffic.  3D game: Ambient sounds Source Loop A constant sound created by a specific object. Shorter duration (4-10s) seamlessly looped. Spatialized at specific location (attenuation + pannning). Covers small area in map. Examples: PC, refrigerator, water dripping.  3D game: Ambient sounds Source one-shot Sounds which appear occasionally. Contribute to give the sense of activity and life in the map. Play at random times. Can coincide with an actual object in the map or be imaginary (e.g. non existing birds). Spatialized at specific location (attenuation + panning). Examples: Animals, voice chatter behind a door. The asset list  The asset list A list of every sound that will occur in the game. Usually stored in a spreadsheet for reference.  The asset list Filename Type (music, sound effect or dialog) Description (e.g. car sounds) Indication of whether the sound is looping or one-shot. Dialog text. Trigger (what causes the sound to play). Location (which level/map)  The asset list Example Location Trigger Description Filename Building pressing button on mechanical Mechanical01.wav 1 door 1 sound