Spatial Audio Techniques Quiz

Game Audio - Sound Spatialization

• Sound spatialization is the placement of sound in specific locations or areas in space using speakers and other technologies.
• In the games and film industry, this is often called "surround" or "spatial audio".
• All sound is spatial because it propagates from a source to a listener through space.
• Sound perception is also inherently temporal, as it takes time to listen to a sound.
• Sound sources are typically idealized as point sources that radiate sound equally in all directions.
•  Real-world sound sources are more complex, affected by large objects, many small objects, varying radiation directions, refraction, interference, standing waves, and acoustics (the study of mechanical waves).
• In daily life, individuals are surrounded by multiple sound sources of varying sizes and positions.
• Due to evolutionary pressure, humans can detect the direction of (some) sound sources.
• The brain detects cues from the signals arriving at both ears to determine the position of a sound source.

Detecting a Single Sound Source

• The brain uses amplitude and time differences in the signals arriving at the two ears to locate a sound source:
  • Amplitude difference: High frequencies. Sound from the left or right becomes softer as it moves around the head.
  • Time difference: Low frequencies. Sound from the left or right takes extra time to reach the opposite ear.
• The brain also uses spectral cues from the head, outer ear, and torso to distinguish sounds above and below the head, and uses small head movements to help pinpoint the location.

Localization of a Single Sound Source

• By listening to the reverberation, one can determine the type of room they are in to some extent.
• Some individuals have developed human echolocation, which allows them to navigate by listening to the interaction of a sound with the environment.

Realistic Sound Reproduction

• To reproduce realistic sound, the position and direction of sound sources must be simulated, along with the spatial properties of room acoustics.
• Ambient sound, comprised of many individual sound sources, needs to be simulated in such a way that these sources combine into an immersive soundscape from all directions.

Approaches to Spatialization

• Spatializing a mono signal, sometimes called panning
• Stereo and multi-channel recording with multiple capsules or microphones.

Spatial Audio Techniques

• Sound Field Synthesis attempts to recreate an accurate physical sound field to create correct perceptual cues (Wave Field Synthesis, Ambisonics).
• Perception-based methods don't attempt to recreate a physical sound field but generate equivalent perceptual cues (stereophony, VBAP).

Stereo Panning

• Used widely.
• It's a perception-based method.
• Loudspeakers are often 60 degrees apart.
• Only the amplitude of the signal at each loudspeaker is manipulated.
• A mono signal is spatialized . Both speakers use the same signal with different levels.
• The head is equidistant from the speakers.
• The method can only simulate sound sources in an arc and 30 degrees to each side.

Stereo Linear Panning

• Sound intensity is lower at the center position.
• A “hole in the middle” effect occurs as the signal appears louder at the endpoints compared to the center.
• Constant power panning is a solution to this issue.

Stereo Constant power spanning

• Sound intensity remains constant for all angles.

Multi-Channel Systems (Horizontal Plane)

• To calculate the gains for loudspeakers, find the speakers that are nearest in angle to the source, and calculate the L1(theta) and L2(theta) to determine the gain for the channels.

Unreal Engine's Panning Method

• Unreal Engine's panning method uses equal-power panning law to compute panning values.
• It uses (azimuth – previous channel azimuth) / (next channel azimuth - previous channel azimuth) to compute fraction.
• This is the standard method used by Unreal Engine for surround sound systems.

Computer Games- Stereo Panning

• Game loudspeakers position is set as -90 degrees and 90 degrees (not the real position in the room).
• Source position and listener position are important to determine the sound direction of a sound effect.
• It isn't possible to distinguish between sound sources in front of or behind the listener.

3D Multi-channel Systems with Height- Vector Based Amplitude Panning (VBAP)

• Algorithm: only the tree closest loudspeakers will be used.
• Gains to each loudspeaker are calculated using information about their 3D position and the virtual source position .

Spatialization and Height in Digital Games

• Height is important in games.
• Positioning sound effects in height is difficult due to the difficulty of placing speakers in the ceiling without altering the room acoustics.
• It is very unlikely that gamers will install additional loudspeakers in the ceiling.
• Headphone-based spatialization is more practical for realism at this time.

Simulating Distance in Sound

• Free-field conditions occur in a space with no reflections, absorption, or other obstructions.
• In free field, doubling the distance from a point source results in a 6dB reduction in SPL.
• Real-world conditions often deviate from the free-field model.
• Output is calculated as input multiplied by a value that accounts for distance.
• At the maximum distance, sound is completely or almost completely silenced.

Amplitude panning Limitations

• It limits sound sources to the circle of the speakers.
• It doesn't allow placing sound sources closer than the loudspeakers.
• Localization only works when the listener has a defined spot. It affects how realism is perceived in cinema theaters and live concerts.

Sound Field Synthesis (Wave Field Synthesis, Ambisonics)

• Wave Field Synthesis, based on Huygens' principle, determines the position and magnitude of each source.
• A high number of densely packed loudspeakers around a listening area is needed to recreate the intended sound field.

Ambisonics

• A sound field reproduction technique
• It is based on spherical harmonic decomposition of the sound field.
• It can encode spatial sound for 2D and 3D multi-loudspeaker systems.
• The number of channels needed depends on the spherical harmonic order, used for encoding.

Binaural Synthesis

• Capture Head-Related Transfer Function (HRTF) for the location of the virtual source
• The HRTF captures how each frequency is affected in terms of gain and delay for each ear.
• Research centers have developed various HRTF sets like the one from IRCAM Listen, KEMAR to determine the frequency characteristics.
• For rendering a mono source, the HRTF for each position is convolved with the source to achieve a left- and right-channel output.
• For locations outside the recorded positions, interpolation of the recorded positions is used.
• Virtual Ambisonics use a decode to virtual loudspeaker setup with each speaker to apply their HRTF to achieve results for virtual, non-recorded sources.

Binaural Synthesis, continued

• Used in VR situations
• VR helmets usually have head-tracking.
• This allows sound to be moved whenever the user rotates their head while using the virtual reality experience..

Simulating Encased Spaces- Reverberation

• Different reverberation signals can be used by loudspeakers, based on the geometry of the space being simulated.
• Early reflections can be simulated on each loudspeaker.
• The ratio of dry signal to reverb signal can be adjusted based on listener and source position.

Steam Audio- Physics-Based Reverb

• Reflections and reverb are important in spatial audio and help users sense the scenario through clues in the sound.
• The actual scene geometry is used to simulate reverb, letting users feel all sounds around them.
• Effects like occlusion, reverb, and sound propagation are done using binaural rendering.

Microsoft- Project Acoustics- Physics-Based Reverb

• Ray-based acoustics methods, use one or a few calculated rays to check for occlusion, and drive reverb.
• These methods can be unreliable but a pebble can obstruct as much sound as a boulder.
• The methods don't account for sound bending around objects(diffraction).
• The simulation method captures diffraction using wave-based analysis, making the acoustics more predictable, accurate, and seamless.

Surround Standards and Formats

• Different standards have been adopted, from 5.1 up to 7.1, 10.2, 22.2,and others formats.
• Includes film, television, computer games, and others formats.

5.1 Surround Sound

• A standard multichannel audio reproduction setup,
• 5 full bandwidth channels including center, front left and right, back left and right, and a low-frequency channel used for subwoofers.
• The standards are now standardized by ITU-R BS.775. 3
• How these channels are created is up the to content creator, not standardized.
• Low-frequency sounds (like explosions) are allocated to the 6th channel.

5.1 Small Speakers- Bass Management

• If the main speakers can't handle the full low-frequency range, then a subwoofer will perform the task.
• The crossover between small and subwoofer speakers determine the distribution of the signals that cannot be handled by the first.

Additional Surround Formats and Systems

• Surround formats like 7.1 have additional loudspeakers for better spatialization, which is particularly important in cinema theaters.
• Different commercial standards for handling and reproducing surround sound.

Consumer and Cinema Theater Surround Formats

• Dolby Digital, DTS, Dolby TrueHD, and others standards exist.
• These formats use different lossy/lossless compression technologies.
• Dolby TrueHD/ DTS-MA are superior formats and are used for reproducing sounds with high clarity in surround sound formats.

Dolby Atmos (2012)

• An object-based audio format.
• Up to 128 independent audio objects can be used.
• Spatial audio metadata is used for each object.
• Each audio track can be assigned directly to a speaker or object.
• 10 channel 7.1.2 bed includes extra channels (like x) designed for ceiling channels.

Dolby Atmos for Headphones

• Dolby Atmos streams are rendered to headphones by converting them into a two-channel format.

Windows Sonic (2017)

• An audio platform for integrated spatial sound.

Support for Object-Based Spatialization in Game Engines and Audio Middleware

• Dolby Atmos supports middleware for Unity3D, Unreal Engine, FMOD, and Wwise
• The system supports different audio configurations (like windows sonic)

Games Using Dolby Atmos

• Several different games use surround formats that support Dolby Atmos including Shadow of the Tomb Raider, Assassin's Creed Origins, Gears of War 4, and Overwatch.

Game Using Windows Sonic

• The article suggests using the same API as Dolby Atmos.

Systems for VR Audio

• Binaural rendering techniques
• Other complex room simulations like occlusion are important to replicate real-world sounds and surroundings.

Additional Notes

• Several surround formats and technologies are currently employed for improving audio experiences and delivering clearer and more immersive soundtracks to the end-users.
• All mentioned technologies involve capturing, generating, processing, and rendering data to adjust the sound and experience based on real scenarios, including the location, position, direction, and other realistic characteristics to the users and sound effects.