# What can we learn about humans from looking at speech technology? ### Will Styler <https://savethevowels.org/talks/colloq_cogs_speechlg.html> --- ### First, the elephant in the room - I'm not actually in Cognitive Science! --- - <img class="r-stretch" src="img/impostor.jpg" alt="impostor.jpg The image displays a digital illustration set against a backdrop of deep space. At the top center, the word Impostor is written in large, red, pixelated letters. Below this text stands a cartoon character from the video game Among Us. The character has a bean-like shape with a thick black outline. Its suit is primarily bright red, but it features darker purple and magenta shading on the left side of its body and its right leg to suggest folds in the fabric. On the front of the character's head area is a large, light blue-grey visor with rounded corners and a white reflection in the center. A red backpack protrudes from the back of the figure. The background depicts outer space, filled with numerous small white stars against a dark blue and black sky. There are also nebula clouds visible in shades of purple, pink, and reddish-orange. Beneath the character's feet is a dark grey shadow on the ground. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Cognitive Science wasn't a separate program where I grew up - I have my BA, MA and Doctorate in Linguistics from the [University of Colorado at Boulder](https://www.colorado.edu/linguistics/) - CU Boulder had the [Institute of Cognitive Science](https://www.colorado.edu/ics/) - Collaborations between faculty in CS, Philosophy, Linguistics, Psych, Education, and more - I went to their talks and was advised by an affiliate, but there was not a COGS major or Ph.D Specialization - ... but I am bothered by COGS-flavored questions --- ### I'm a Computational Phonetician - This means I study human speech perception and production - ... using computational methods and models - This involves a mix of experiments, data analysis, recordings, and instrumental measurements - I collaborate lots with [Dr. Sarah Creel](https://quote.ucsd.edu/lasr/) --- ### I'm also Director of [Computational Social Science](https://css.ucsd.edu) at UCSD - We have [lots of Cognitive Scientists](https://css.ucsd.edu/people/faculty.html#Cognitive-Science)! - Including [Dr. Sean Tro] - So, this means I'm always thinking about computers as a tool for understanding humans! --- ## What is Speech Technology? --- ### We're getting very used to speech technology - Siri/Alexa/GoogleAssistant - ChatGPT Voice Mode - Speech-to-Text Keyboards - Text-to-Speech (e.g. in Twitch streams) --- ### There are many kinds of speech technology - Voice Activity Detection - Automatic Noise Filtering - Voice Compression and Encryption - Forced Alignment and Timestamping - Automatic Speech Recognition (ASR) - Speech Synthesis or Text-to-Speech (TTS) --- ### These are really interesting tools! - They allow new kinds of human-computer interactions - They are incredible tools for accessibility - They're great for processing large amounts of data - ... but the most interesting part of these tools? --- ### The most interesting part about them is that they work at all! --- ### Today's Plan - Why is speech so hard to produce? - How do computers produce speech? - Why is speech so hard to perceive? - How do computers perceive speech? --- ## Speech is Hard --- ### Human Speech is incredibly difficult - This is an incredibly intricate gestural dance in your mind and mouth - Let's try it --- ### "A Linguistics Major goes very well with Cognitive Science" - First, focus on your jaw - Now, on your tongue - Now, feel the vibes --- ### Ultrasound of Speech <img class="r-stretch" src="phonmedia/tools_ultrasound.jpg" alt="toolsultrasound.jpg The image shows a person engaged with an imaging device that appears to be used for medical purposes, possibly related to brain imaging given the context of the text on the screen. The individual is wearing large over-ear headphones and seems focused on the monitor in front of them. The monitor displays a 3D image of what looks like a human brain, with various labels such as P1 and P2, which might indicate different parts or perspectives of the brain being examined. There are also some numerical values next to these labels, suggesting measurements or coordinates related to the brain's structure. The person is seated in front of the monitor, which is mounted on a stand with multiple foam pads for stability. The individual appears to be wearing a light-colored top and has dark hair. They seem to have a microphone attached near their mouth, indicating that they might be communicating or providing feedback during this process. To the right side of the image, there's a piece of paper taped to the wall, which likely contains instructions or information related to the procedure being performed on the monitor. The background is plain and light-colored, suggesting an indoor setting, possibly a medical facility or research lab. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Ultrasound of Speech <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <br> <tiny>From University of Michigan Phonetics Lab</tiny> --- ### Many paths to the same sound --- ### Many paths to the same sound <img class="r-stretch" src="phonmedia/englishr.jpg" alt="englishr.jpg The image is a series of MRI (Magnetic Resonance Imaging) scans arranged in a grid format. These scans appear to be cross-sectional images taken at different levels along the length of an individual's neck and upper chest, showing various anatomical structures such as muscles, bones, and soft tissues. Each scan provides a detailed view of the internal anatomy from a side perspective. The scans are labeled with numbers on the right-hand side indicating which level they represent in terms of depth within the body. For example, 1 likely represents an image taken at one level above another, progressing downward as you move to higher numbers. The text at the bottom of the grid reads: Native speakers of American English producing sustained [J] as in 'purr'. This suggests that the scans are part of a study or research project focused on how native speakers of American English produce the sound represented by the phonetic symbol [J], which is often used to represent the r sound in words like purr. The MRI images show various anatomical structures, including what appears to be the larynx (voice box) and surrounding muscles. The scans are in grayscale, with different shades indicating varying densities of tissue. There are no people or characters depicted in this image; it is purely an anatomical study using medical imaging technology. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Speech is *hard* - Speech is flapping bits of meat around in your head and throat while you expel air. - This creates tiny vibrations in the air, ca --- ### So, what do these vibrations look like? --- <img class="r-stretch" src="phonmedia/sky.png" alt="sky.png The image is a graph that represents an audio signal over time. The x-axis of the graph is labeled Time (s) and ranges from 0 seconds to approximately 0.6278 seconds. The y-axis is labeled with values ranging from -0.3656 at the bottom to 0.3023 at the top, indicating a range of audio signal amplitudes. The graph shows an audio waveform that starts at zero amplitude and rises sharply in the first few milliseconds before dropping back down to near-zero amplitude. This pattern repeats several times throughout the duration of the recording, with each cycle being relatively short compared to the overall length of the recording. The text sky is written above the graph, suggesting this might be a spectrogram or waveform visualization for the word sky. There are no people depicted in this image; it solely focuses on the audio signal and its representation over time. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch" src="phonmedia/skyspectrogram.png" alt="skyspectrogram.png The image provided is a spectrogram, which is a visual representation of the spectrum of frequencies in a sound signal as it varies with time. This particular spectrogram shows the frequency content over time for the word sky. Key Elements: 1. Axes: - The horizontal axis represents time (in seconds), ranging from 0 to approximately 0.6278 seconds. - The vertical axis represents frequency, measured in Hertz (Hz), ranging from 0 Hz at the bottom to around 5000 Hz at the top. 2. Text: - At the center of the spectrogram, there is text that reads sky, indicating what sound was analyzed for this spectrogram. 3. Color Gradient: - The image uses a grayscale color gradient where darker shades represent higher intensity or louder sounds. - Lighter shades indicate lower intensity or quieter sounds. 4. Frequency Distribution: - At the beginning of the time axis (around 0 seconds), there is a burst of activity, indicating the initial sound onset. - As we move along the horizontal axis, the frequency content changes over time, showing how the frequencies vary during the duration of the word sky. - The spectrogram shows that higher frequencies are present at the beginning and then decrease as the sound progresses. 5. Intensity: - There is a noticeable increase in intensity (darker shades) around 0 seconds, which corresponds to the initial burst of sound. - As time progresses, there is a gradual decrease in intensity until it stabilizes for most frequencies by about 0.6278 seconds. This spectrogram provides a detailed view of how the frequency content changes over time during the pronunciation of sky. It's useful for analyzing speech sounds and understanding their acoustic properties. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### There is incredible complexity in this process - Fluid movement of your mouth and tongue - Careful planning of air and breathing - Control of pitch, gestures, and other aspects --- ### ... and we want to do *this* with software?!? - 'Speech Synthesis' or 'Text-to-Speech' (TTS) - *How do we do that?* --- ### The Task - "A linguistics major goes very well with Cognitive Science" <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Text Analysis - "OK, the human gave me text, what do they actually want me to say?" - This part is usually done in Python - This is actually hard - '1997' is many things --- ### "PG&E will file schedules on April 20." - <img class="r-stretch wide" src="comp/tts_phones.jpg" alt="ttsphones.jpg The image appears to be a phonetic transcription of a sentence using the International Phonetic Alphabet (IPA). The sentence is: Peggy and Eddie will file schedules on April twentieth. Here's a breakdown of the text in the IPA: - P is pronounced as /p/ - e is pronounced as /ɛ/ - g is pronounced as /ɡ/ - y is pronounced as /j/ - a is pronounced as /æ/ - n is pronounced as /n/ - d is pronounced as /d/ - E is pronounced as /iː/ - d is pronounced as /d/ - i is pronounced as /ɪ/ - w is pronounced as /w/ - l is pronounced as /l/ - f is pronounced as /f/ - i is pronounced as /ɪ/ - l is pronounced as /l/ - e is pronounced as /ɛ/ - s is pronounced as /z/ - k is pronounced as /k/ - h is pronounced as /h/ - a is pronounced as /æ/ - j is pronounced as /dʒ/ - ax is pronounced as /æks/ - s is pronounced as /s/ - o is pronounced as /ɑː/ - n is pronounced as /n/ - d is pronounced as /d/ - S is pronounced as /ɛ/ - k is pronounced as /k/ - h is pronounced as /h/ - a is pronounced as /æ/ - j is pronounced as /dʒ/ - ax is pronounced as /æks/ - s is pronounced as /s/ - o is pronounced as /ɑː/ - n is pronounced as /n/ - d is pronounced as /d/ - E is pronounced as /iː/ - y is pronounced as /j/ - w is pronounced as /w/ - h is pronounced as /h/ - a is pronounced as /æ/ - l is pronounced as /l/ - e is pronounced as /ɛ/ - s is pronounced as /z/ - k is pronounced as /k/ - h is pronounced as /h/ - a is pronounced as /æ/ - j is pronounced as /dʒ/ - ax is pronounced as /æks/ - s is pronounced as /s/ - o is pronounced as /ɑː/ - n is pronounced as /n/ - d is pronounced as /d/ - A is pronounced as /eɪ/ - P is pronounced as /p/ - r is pronounced as /ɜː/ - i is pronounced as /ɪ/ - l is pronounced as /l/ - e is pronounced as /ɛ/ - t is pronounced as /t/ - w is pronounced as /w/ - e is pronounced as /ɛ/ - n is pronounced as /n/ - d is pronounced as /d/ - T is pronounced as /t/ - h is pronounced as /θ/ The sentence also includes a percentage: -1.4%. This likely refers to the probability or likelihood of an event, but without more context, it's unclear what specific event this percentage relates to. There are no diagrams in the image; only phonetic transcriptions and text. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> - <img class="r-stretch wide" src="comp/tts_wave.jpg" alt="ttswave.jpg The image provided is a visual representation of an audio waveform. This type of graph shows how sound pressure changes over time in an audio signal. - Shape and Structure: The waveform consists of horizontal lines that represent the amplitude (loudness) of the sound at different points in time. The vertical axis represents the amplitude, while the horizontal axis represents time. - Amplitude Variations: The height of each line indicates how loud or soft a particular part of the audio is. Higher peaks indicate louder sounds, and lower troughs represent quieter moments. - Duration: The length of the waveform shows the duration of the sound. Each segment of the graph corresponds to a different section of the audio file. - Text and Numbers: There are no visible text or numbers in this image that provide additional information about the content of the audio, such as its title, artist, or genre. This type of visual is commonly used by audio editors and sound engineers to analyze and manipulate audio files. It helps them understand the dynamics of a recording—how loud it gets, how quiet it becomes, and where there might be unwanted noise or silence that could be removed for better quality. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> - (Thanks to Julia Hirschberg for this annotated chunk) --- ### Then, we turn that into audio - "I know what needs to be said, now, give me a wave I can play back for the humans" --- ### For a long time, we cheated using humans! - **Concatenative or 'Unit Selection' TTS** chops up bits and pieces of existing speech to create new speech - You record a huge database of speech from a voice actor, with optimum 'coverage' - You update as new words emerge (e.g. COVID, rawdogging, skibidi) - **You then combine these words into sentences to match the text** - ... and you use fancy algorithms to smooth the results out. --- ### This isn't easy - You have to choose the best recorded token - You might have 500 recordings of 'went' - Context matters a lot - "park" can sound very different in different places with words - You can't get full coverage - "Ruaridh", "Krivokapic", "simp", "La Jolla" --- ### The result can be imperfect <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### ... and you've only got one voice - Which means that each new person needs a new collection of data --- ### Then, Artificial Neural Networks arrived, and everything changed <img class="r-stretch" src="dalle/cuteneuralnetwork.jpg" alt="cuteneuralnetwork.jpg This image features a colorful, anthropomorphic character that resembles a friendly virus or cell set against a glowing, abstract background. The character is the central focus and appears to be walking or dancing towards the viewer. The character has a round, bulbous body covered in a texture that looks like a digital mesh or network of glowing white lines connecting small dots. This mesh creates a gradient of colors across its body, shifting from cool blues and teals on the left side to warm oranges and yellows on the right. Radiating outward from the main body are approximately twelve distinct spikes or antennae. Each spike ends in a shiny, spherical orb. These orbs vary in color, including metallic gold, bright orange, deep purple, and teal green. The character has a face with two very large, round eyes positioned side-by-side near the top of its body. The eyes are dark brown with thick black outlines and white highlights that suggest they are shiny or reflecting light. Below the eyes is a simple, curved black line forming a gentle smile. At the bottom of the character's body are two short, stubby legs that look like rounded feet. The left leg (viewer's left) is primarily orange and brown with a metallic sheen, while the right leg is blue and purple. They appear to be in mid-stride, suggesting movement. The background is soft and out of focus, filled with bokeh effects—circular blurs of light in shades of pink, white, yellow, and blue. Behind the character, there are faint, thin white lines connecting dots in a geometric pattern, resembling a constellation or a neural network map, which extends across the entire backdrop. There is no text or specific diagrams present in the image. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### The World's Worst Introduction to Neural Networks <img class="r-stretch" src="img/dnn.jpg" alt="dnn.jpg This image displays a schematic diagram of a feedforward neural network architecture, arranged horizontally from left to right. The diagram is divided into three distinct sections by vertical dotted lines, representing different layers of the network. On the far left is the Input layer, labeled at the top with the text Input layer and the variable $i$. This section consists of a vertical column of grey circular nodes. To the left of these nodes are labels indicating the inputs: Input 1 with an arrow pointing to the top node, Input 2 with an arrow pointing to the second node, followed by a vertical ellipsis (three dots), and finally Input n with an arrow pointing to the bottom node. This indicates that there are 'n' total input values entering the network. The middle section is labeled at the top as Hidden layers. It contains three distinct columns of grey circular nodes, representing different stages within the hidden part of the network. These columns are labeled above as $h1$, $h2$, and $hn$. Each column shows a top node, a second node, a third node, a vertical ellipsis indicating more nodes in between, and a bottom node. This signifies that there are multiple neurons in each hidden layer. On the far right is the Output layer, labeled at the top with Output layer and the variable $o$. This section features a column of grey circular nodes similar to the input layer but arranged vertically on the right side. To the right of these nodes are labels indicating the outputs: Output 1 with an arrow pointing away from the top node, followed by a vertical ellipsis, and Output n with an arrow pointing away from the bottom node. This indicates 'n' output values produced by the network. Connecting these layers is a dense web of straight black lines representing the connections or weights between neurons. Every node in the Input layer connects to every node in the first hidden column ($h1$). Similarly, every node in $h1$ connects to every node in $h2$, and so on through to $hn$. Finally, every node in the last hidden column ($hn$) connects to every node in the Output layer. This structure illustrates a fully connected network where information flows sequentially from left to right. At the very bottom left of the image, there is source text that reads: Source: https://doi.org/10.1186/s40537-020-00328-3. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Take COGS 181 to actually understand this! --- ### Neural TTS is quite powerful - Train a neural network with text and corresponding audio - Make it output something which can made into a wave very readily - Either make the wave directly, or make an intermediate representation which can be turned into a wave --- ### [TacoTron2](https://arxiv.org/pdf/1712.05884) is a relatively simple, open system - It takes text, and generates spectrograms, chunk-by-chunk, which can be turned into a waveform --- ### TacoTron 2 <img class="r-stretch" src="phonmedia/tts_tacotron2.png" alt="ttstacotron2.png The image is a block diagram illustrating the architecture of the Tacotron 2 system, which is used for speech synthesis. The diagram consists of various components connected by arrows indicating the flow of data. Components and Flow: 1. Input Text: - At the bottom left corner, there's an input labeled Input Text. This represents the text that will be converted into speech. 2. Character Embedding: - The text is fed into a block labeled Character Embedding, which converts each character in the input text into numerical vectors. 3. Bidirectional LSTM (Long Short-Term Memory): - The output from Character Embedding goes to a Bidirectional LSTM, which processes the sequence of characters bidirectionally. 4. Location Sensitive Attention: - The Bidirectional LSTM's output is then passed through Location Sensitive Attention, which helps in focusing on relevant parts of the input text at different positions. 5. 2 LSTM Layers: - The attention mechanism’s output goes into two layers of Long Short-Term Memory (LSTM) networks, labeled as 2 LSTM Layers. 6. Linear Projection: - After the 2 LSTM Layers, there are blocks for Linear Projection, which transform the outputs from the LSTMs. 7. 5 Conv Layer Post-Net: - The output of Linear Projection is then fed into a block labeled 5 Conv Layer Post-Net. This consists of five convolutional layers that further refine and smooth out the speech signal. 8. 2 Layer Pre-Net: - Another part of the system, labeled as 2 Layer Pre-Net, processes the input text before it reaches the main processing blocks. 9. Mel Spectrogram: - At the top center of the diagram is a visual representation called Mel Spectrogram. This shows the frequency content over time in a logarithmic scale and is used to represent the target output for speech synthesis. 10. WaveNet Samples: - On the right side, there's an input labeled WaveNet Samples, which represents the raw audio samples that are generated by WaveNet. Additional Details: - The diagram includes labels such as Location Sensitive Attention and Linear Projection, indicating specific processing steps. - There is a + symbol between the 2 Layer Pre-Net and the 5 Conv Layer Post-Net, suggesting an addition or concatenation of outputs from these two components before they are fed into the next part of the system. This diagram provides a clear overview of how Tacotron 2 processes input text to generate speech by converting it into audio waveforms. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### This allows us to go from text to speech! - We feed in text, and we get back a wave, with no humans involved past making training data! - The results are getting very, very good. <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### The State of the Art is Advanced, but closed - Current state of the art models from ElevenLabs, OpenAI, Google, and Amazon are all closed and proprietary - If you want the best TTS in the world, it has to happen on somebody else's computer - Details are often not published and considered "trade secrets" - They may well be open-source models with changes and tweaks - It's not currently possible to teach the state of the art in TTS! - ... and this should disturb us as a society --- ### Neural TTS can be trained using *any* voice - You can build a model from the ground up using any voice you'd like - [Except Scarlett Johansson](https://www.npr.org/2024/05/20/1252495087/openai-pulls-ai-voice-that-was-compared-to-scarlett-johansson-in-the-movie-her) - If all your training data are from a bored Bostonian, you'll end up with a bored Bostonian TTS voice - This is very expensive, though, and doesn't scale well at all - You also need *lots* of data from the new speaker --- > All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### ... but isn't linguistic information separate from talker information? - Why can't we just adapt to new voices? --- ### We can think of all speech as having 'content' and 'style' - Voices express linguistic 'content' - Phonemes, with ordering, and necessary tone/prosody for comprehension - This is 'all we need' to understand the utterances - 'Style' is everything else we've been talking about - 'Speaker' identity - Social components - Emotional content - Plus prosodic factors (e.g. speed, emphasis, prosodic 'tunes', sarcasm) --- ### Couldn't we just abstract out the 'style' component and apply it to whatever Linguistic content we'd like? - Yes! --- ### Here's a Multi-Speaker Version of TacoTron 2 <img class="r-stretch" src="phonmedia/tts_tacotron2_multispeaker.png" alt="ttstacotron2multispeaker.png The image is a detailed diagram illustrating a multi-speaker Text-to-Speech (TTS) system. The diagram includes various components and their interactions within this system. Components of the Multi-Speaker TTS System: 1. Text Input: - At the bottom left, there's an input labeled Text, which is the starting point for the process. 2. Speaker Embedding: - Below the text input, there are two blocks labeled Speaker Embedding. - These blocks represent the speaker-specific information that will be used to generate speech. 3. Speaker Encoder: - To the right of the Speaker Embedding block is a red box labeled Speaker Encoder. - This component processes the speaker embedding and prepares it for further use in generating speech. 4. CBH-LSTM (Conditional Bi-directional Hierarchical Long Short-Term Memory): - Below the Speaker Encoder, there's another block labeled CBH-LSTM, which is a type of neural network used to process sequences. - This component takes the speaker embedding and text input as inputs. 5. Phone Embedding: - To the right of CBH-LSTM, there’s an orange box labeled “Phone Embedding.” - This block converts the text into phonetic representations (phones) which are used to generate speech sounds. 6. Self Attention: - Below Phone Embedding is a block labeled Self Attention. - Self-attention mechanisms help in understanding the relationships between different parts of the input sequence, which aids in generating more natural-sounding speech. 7. Decoder RNN (Recurrent Neural Network): - To the right of Self Attention, there’s another orange box labeled “Decoder RNN.” - This component takes the phonetic representations and generates a sequence of audio frames that represent the speech sound. 8. Additive Attention: - Below Decoder RNN is an orange block labeled Additive Attention. - This mechanism helps in focusing on specific parts of the input to generate more accurate speech sounds. 9. Concat: - To the right of Additive Attention, there’s a block labeled “Concat.” - This component combines multiple outputs from previous blocks into one sequence for further processing. 10. Pre-Net and Post-NET Blocks (Green Boxes): - At the top left is a green box labeled Pre-Net. - To its right is another green box labeled Post-Net. - These blocks are used to preprocess and post-process the audio signal, respectively. 11. WaveNet: - At the far right of the diagram, there’s a yellow block labeled “WaveNet.” - WaveNet is a type of neural network that generates raw audio samples from the processed outputs. 12. Sigmoid Function and Stop Token: - Below WaveNet are two blocks labeled Sigmoid and Stop Token. - The Sigmoid function is used to normalize the output, while the Stop Token helps in determining when to stop generating speech. Flow of Information: - The process starts with text input. - This text goes through a Speaker Encoder which processes speaker-specific information. - The processed speaker embedding then flows into CBH-LSTM along with phone embeddings. - The outputs from these components are combined and passed through various attention mechanisms (Self Attention, Additive Attention). - These combined outputs go to the Decoder RNN, which generates phonetic representations of speech sounds. - Further processing involves concatenation and additional attention mechanisms before being fed into Pre-NET and Post-NET blocks for final audio processing. Visual Representation: - The diagram uses arrows to indicate the flow of information between different components. - Each block is labeled with its function or name, making it easier to follow the process step by step. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### The results of this are... terrifying - ... and has given rise to 'Deepfake' voices --- ### Neural Styler Transfer <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (TacoTron2) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p>(ElevenLabs) (Credit to Erick Amaro and Mia Khattar!) --- ### Multilingual Examples <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (English) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (French) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Spanish) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Mandarin) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Italian) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Russian) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Japanese) --- ### This system isn't perfect > Adenocarcinoma in Tubovillious Adenoma bona fide certiorari de jure collusion RICO ex post facto CVN AWACS Escapement Tourbillion Remontoir de Egalite <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Prosody is still hard <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### ... but OMG, this thing can do arbitrary speech, in an arbitrary voice - ... and it's never had a tongue, had phonics training, and doesn't actually know anything at all about mouths - Arguably, it doesn't know anything about English - ... although some systems use a language model too - *This is amazing!* - ... but it can model more complexity still --- ### Code Switching It's like sometimes mezclo un poco de español con my English, cuando me siento particularmente spicy, y tengo curiosidad to know cómo la TTS handles it. <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Wow. - Not only can exposure to data allow a deep neural network to learn to map written language into speech in one language - ... but it can do it for two languages - ... at once - ... with clear mixing of the two --- # This shouldn't work - Yet, here we are - Espicy! --- # What about Speech Perception? --- ### Speech Perception is *hard* - Speech is flapping bits of meat around in your head and throat while you expel air. - This creates tiny vibrations in the air - **Speech perception is turning the resulting vibrations in the air back into language** --- ### For computers, this is 'Automatic Speech Recognition' (ASR) - The task is to turn speech into equivalent text - This is *really, really hard* --- ### Let's focus on one of the really hard problems --- ## Vowel Perception --- ### What is a vowel? What kind of vowels are we talking about? --- <img class="r-stretch" src="phonmedia/aeiou.png" alt="aeiou.png The image contains a list of letters with some marked with red circles and crosses, indicating they are incorrect choices in what appears to be an educational context, possibly related to phonetics or spelling. The letters shown are: - a (marked with a red circle) - e (marked with a red circle) - i (marked with a red circle) - o (marked with a red circle) - u (marked with a red circle) Following these, there is text that reads and sometimes in black. Next to this phrase, the letter v has been crossed out with a red line and marked with a red cross. To the right of all the letters, there are two large words written in red: - The first word says NEVER!!!! - Below it is another word that reads sometimes but with the letter v crossed out. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch big" src="phonmedia/vowelcharts_english.png" alt="vowelchartsenglish.png The image is a phonetic chart used in linguistics to represent vowel sounds. It's called an articulation diagram which shows how vowels are formed by varying the position of the tongue and lips. Key Features: 1. Axes: - The horizontal axis represents the front-back dimension, with 'Front' on the left side and 'Back' on the right. - The vertical axis represents the high-low dimension, with 'Close' at the top and 'Open' at the bottom. 2. Vowel Sounds: - Each vowel is represented by a letter or symbol placed within the grid formed by these axes. - The vowels are categorized into different positions based on their articulation (front, back, central), and openness (close, open-mid, open). 3. Examples of Vowels: - Front Vowels: These include sounds like /i/, /e/, /æ/ which are placed towards the left side. - /i/ is close and front. - /e/ is close-mid and front. - /æ/ is open-mid and front. - Back Vowels: These include sounds like /u/, /o/, /ɑː/ which are placed towards the right side. - /u/ is close and back. - /o/ is close-mid and back. - /ɑː/ is open-mid and back. 4. Central Vowels: - These include sounds like /ɪ/, /ɜː/, which are placed in the middle of the chart, neither front nor back. - /ɪ/ is close and central. - /ɜː/ is open-mid and central. 5. Other Symbols: - The symbol '3' represents a neutral vowel sound that can be considered as a central vowel but not strictly classified under any specific category like front, back, or central. Detailed Description of the Chart: - The chart has a grid with horizontal lines representing different positions from front to back and vertical lines representing different positions from close to open. - Each cell in this grid contains a symbol that represents a particular vowel sound. For example, /i/ is placed at the top left corner (close and front), while /ɑː/ is located at the bottom right corner (open-mid and back). This chart helps linguists understand how different vowels are produced by varying the position of the tongue and lips in relation to these two dimensions. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Review: What is a vowel? * A vowel is voicing passing through (and resonating in) an unobstructed vocal tract! * If we change the position of the tongue, we change the resonances --- <img class="r-stretch big" src="phonmedia/voweltongue.png" alt="voweltongue.png The image is a diagram illustrating the positions of the tongue for different vowel sounds in speech. The diagram shows a side view of the human head with an emphasis on the mouth and throat area. The tongue's position relative to the palate (the roof of the mouth) is highlighted, indicating how it changes shape and position to produce various vowel sounds. The diagram includes labels for four different vowels: i, e, ɛ, and a. Each label corresponds to a specific line that shows where the tongue should be placed in your mouth when pronouncing these vowels. The lines are color-coded, with each vowel having its own distinct color: - i is represented by a red line. - e is shown as a blue line. - ɛ is depicted using a green line. - a is indicated by a purple line. The diagram also includes small dots on the tongue to show where it should be placed for each vowel sound. The i position shows the tongue positioned high in the mouth, close to the palate, which is typical for front and high vowels like /ɪ/. For e, the tongue is shown lower than for i, but still relatively high, closer to the middle of the mouth. The ɛ vowel has a slightly more open position compared to e, with the tongue positioned further back in the mouth. This is characteristic of front and mid vowels like /ɛ/. Finally, the a vowel shows the tongue at its lowest point, close to the throat, which is typical for back and low vowels such as /ɑː/ or /a/. The diagram provides a clear visual guide on how the position of the tongue changes when pronouncing different vowels. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Review: What is a vowel? A vowel is voicing passing through (and resonating in) an unobstructed vocal tract! If we change the position of the tongue, we change the resonances * Different resonances *filter* the sound differently and determine the vowel quality * **Different tongue shapes create different resonances, and different vowels!** --- <img class="r-stretch big" src="phonmedia/voweltongue2.png" alt="voweltongue2.png The image consists of four detailed line drawings that represent different vowel sounds produced by the human vocal tract. Each drawing shows a side view of the mouth and throat area, focusing on the position of the tongue and lips for various vowels. 1. Top Left (i): This diagram illustrates the position of the tongue when producing the vowel sound /i/, which is often used in words like see or she. The tongue is positioned high in the mouth with the tip near the front, and the lips are rounded slightly. 2. Top Right (u): Here, we see the diagram for the vowel sound /u/, as in boot or put. The tongue is placed low back in the mouth, close to the soft palate, while the lips are rounded tightly together. 3. Bottom Left (æ): This drawing shows the position of the tongue and lips when producing the vowel sound /æ/, which can be heard in words like cat or hat. The tongue is relatively low in the mouth with a slight forward movement, and the lips are slightly apart but not rounded. 4. Bottom Right (ɑ): This diagram depicts the position for the vowel sound /ɑ/, often used in words such as father or father. The tongue is positioned low back in the mouth near the soft palate, and the lips are relatively relaxed with a slight parting. Each drawing includes a small dot to indicate where the center of the tongue should be placed within the oral cavity for each vowel sound. The diagrams are simple line drawings without any additional colors or shading, focusing solely on illustrating the vocal tract positions necessary for producing these specific sounds. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### What do vowels sound like? * We talk about vowel quality in terms of "formants" * These are bands of the spectrum where the energy is strongest * The frequencies of these formants are our primary cues --- <img class="r-stretch big" src="phonmedia/iformants.png" alt="iformants.png The image appears to be a spectrogram, which is a visual representation of the spectrum of frequencies in a sound signal as it varies with time. The horizontal axis represents time, and the vertical axis represents frequency (in Hertz). The intensity of the color at each point indicates the amplitude or volume of the sound at that particular frequency and moment. In this spectrogram: - The y-axis is labeled Hz, which stands for Hertz, a unit of measurement for frequency. It ranges from 0 to approximately 4000 Hz. - The x-axis represents time, though it's not marked with specific intervals or labels indicating the duration in seconds or minutes. The spectrogram shows a sound that has a consistent pattern across different frequencies over time: - There is a band of activity around 2000 Hz and another at approximately 1000 Hz. - The intensity of these bands fluctuates slightly, suggesting variations in volume within those frequency ranges as the sound progresses. There are no people or characters depicted in this image. It focuses solely on the analysis of audio data through a spectrogram format. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch big" src="phonmedia/iformantslabeled.png" alt="iformantslabeled.png The image appears to be a spectrogram of speech, which is a visual representation of the spectrum of frequencies present in a sound over time. This type of graph is commonly used in phonetics and linguistics to analyze the frequency content of sounds. Key Features: 1. X-Axis (Horizontal Line): Represents time. 2. Y-Axis (Vertical Line): Represents frequency, measured in Hertz (Hz). The scale on the left side indicates that frequencies range from 0 Hz at the bottom to approximately 4000 Hz at the top. Spectrogram Details: - Frequency Bands: - There are three labeled bands of interest marked as F1, F2, and F3. These correspond to formant frequencies in speech sounds. - F1 (First Formant): Typically associated with vowels and is lower than F2 and F3. - F2 (Second Formant): Higher frequency band that varies depending on the vowel being produced. - F3 (Third Formant): Even higher in frequency compared to F1 and F2, often used for distinguishing between different vowels. - Intensity Bands: - The spectrogram shows varying intensities of sound energy over time. Darker areas indicate louder sounds or more intense frequencies at that point in time. Time Axis: - The horizontal axis represents the duration of speech. The left side is labeled with a number [1], which might be indicating the start of a specific segment. People and Characters: The image does not contain any people, characters, or text other than what has been described above. It focuses solely on the spectrogram representation of sound frequencies over time. If you need further assistance understanding this type of graph or have additional questions about it, feel free to ask! This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Vowel formants * F1 and F2 are generally considered to be the most important * F3 is good for rounding and rhoticity --- ### Formants alone can be enough for some perception! --- ### Let's listen to some sounds <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Let's listen to some sounds <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <br> ### Now let's play all three at once! <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Let's listen to some sounds <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <br> ### Now let's play all three at once! <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <br> ### Does this help? <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### So, vowels are basically formant patterns --- <img class="r-stretch big" src="phonmedia/vowelformants.gif" alt="vowelformants.gif This image displays a scientific chart containing eight individual spectrograms arranged in a grid of two rows and four columns. The background is white, and the graphical elements are black. There are no people or characters in this image. The vertical axis on the far left represents frequency in Hertz (Hz), with markings for 1000, 2000, 3000, and 4000 Hz increasing from bottom to top. The horizontal axis at the very bottom represents time in milliseconds (ms), with markings for 0, 300, and 400 visible. Each of the eight panels displays a spectrogram of a specific vowel sound, identified by an International Phonetic Alphabet (IPA) symbol centered below each graph. In these graphs, darker horizontal bands represent formants, which are concentrations of acoustic energy at specific frequencies. Small black arrows point from the left margin toward these dark bands in most panels to highlight them. Top Row (from left to right): 1. Panel labeled [ i ]: Shows a spectrogram with distinct dark bands. One band is visible near the top around 3000 Hz, another below it around 2500 Hz, and a lower one near 1000 Hz. Arrows point to these frequencies. 2. Panel labeled [ ɪ ]: Similar to the first panel but with slightly different band positions. The upper bands are clustered between 2000 and 3000 Hz. Arrows indicate specific formant locations. 3. Panel labeled [ ɛ ]: Shows bands that are generally lower in frequency than the previous two. There is a cluster of energy around 2000-2500 Hz and a distinct band near 1000 Hz. Arrows point to these features. 4. Panel labeled [ æ ]: This graph shows dark bands distributed across the vertical range, with significant energy visible between 2000 and 3000 Hz and another band lower down around 800-900 Hz. Bottom Row (from left to right): 1. Panel labeled [ a ]: This spectrogram shows three distinct dark bands indicated by arrows on the far left. One is low, near 500 Hz; one is in the middle around 1200 Hz; and one is higher up around 2800 Hz. 2. Panel labeled [ ɔ ]: Shows a pattern with energy concentrated between 1000 and 2500 Hz. Arrows point to bands near 600 Hz, 1200 Hz, and 2300 Hz. 3. Panel labeled [ ʊ ]: This graph shows a concentration of energy lower down on the frequency scale compared to the front vowels. There are arrows pointing to bands around 800 Hz and 1500 Hz. The texture is somewhat grainier than the others. 4. Panel labeled [ u ]: This final panel shows dark bands primarily in the lower half of the graph, below 2000 Hz. On the right side of this specific graph, there is a distinct curved line or hook shape rising from the middle frequency area up towards the top right corner. Arrows point to bands near 500 Hz and 1200 Hz. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> <small>Different American English vowels, as spoken by a male speaker</small> --- ### ... and vowel formants map to articulation! --- <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ## Speaker Variation! --- ### Speaker Vowel Space Variation * Different speakers produce different resonances, even for the “same” vowels * Vocal tracts can be shorter, longer, wider... --- <img class="r-stretch big" src="ling_memes/vocaltract.jpg" alt="vocaltract.jpg This image is a meme featuring a photograph of a small kitten lying on its back. The kitten has white fur covering its chest, belly, and paws, while its head and back have patches of orange and dark brown or black fur. It is captured in a dynamic pose with one front paw stretched straight up into the air towards the top right corner. Its mouth is wide open in a large yawn or meow, clearly showing its pink tongue and gums. The kitten's eyes are open but look slightly unfocused or squinted due to the expression. There is white text with a black outline overlaid on the image in a bold, sans-serif font typical of internet memes. At the very top, the text reads mai vocal track. At the bottom, the text reads iz dis long. This caption uses internet slang (dropping 'y's and using 'z' for 's') to say my vocal track is this long, humorously suggesting that the length of the kitten's stretched paw represents how long its singing voice can be. The background behind the kitten appears to be a dark fabric, likely a blanket or clothing, which has a pattern of red flowers with green leaves. In the bottom left corner, there is a small watermark URL in white text that reads ICANHASCHEESBURGER.COM. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Speaker Vowel Space Variation Different speakers produce different resonances, even for the “same” vowels * Speaker can have colds or allergies, can have more nasal voices... * Sociolinguistic factors galore * Every person has a different set of basic vowel formant positions * This is called the speaker’s “vowel space” --- <img class="r-stretch big" src="phonmedia/ipaformantsgraph.png" alt="ipaformantsgraph.png The image is a scatter plot titled F1 and F2 Values by Vowel. It displays data points representing different vowels on a graph with two axes: F1 (on the horizontal axis) and F2 (on the vertical axis). The values for F1 range from approximately 300 to 4,000 Hz, while the values for F2 range from about 50 to 1,500 Hz. Each data point on the graph is labeled with a vowel symbol. Here are the details of each label: - i: Located in the upper left quadrant. - I: Positioned slightly below and to the right of i. - e: Found near the center-left area. - æ: Situated close to the bottom-center part of the graph. - ε: Placed above and to the left of æ. - ə: Located in the lower middle section, closer to the horizontal axis. - ʊ: Positioned towards the upper right quadrant. - u: Found slightly below and to the right of ʊ. - ʌ: Situated near the bottom-right area. - ɒ: Placed close to the center-right part. - ɑ: Located in the lower middle section, closer to the vertical axis. The graph uses a Cartesian coordinate system with labeled axes for F1 (horizontal) and F2 (vertical). The data points are plotted as black dots. There is no additional text or diagram within the image that provides further context or explanation of the relationship between the vowels and their corresponding F1 and F2 values. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch big" src="phonmedia/clearspeech_speakeraverages.png" alt="clearspeechspeakeraverages.png The image is a scatter plot titled F1 and F2 Values by Vowel. It displays data points representing different vowels across two axes: F1 (on the horizontal axis) and F2 (on the vertical axis). The plot uses various symbols to represent different vowel sounds, as indicated in the legend on the right side of the image. Here's a detailed description: - Axes: - The x-axis is labeled F1 and ranges from approximately 0 to 5. - The y-axis is labeled F2 and ranges from approximately 0 to 1. - Legend: - The legend on the right side of the plot explains which symbol corresponds to each vowel sound. - A blue circle with a caret (^) represents /a/. - A green circle with an at sign (@) represents /æ/. - A purple circle with a lowercase a (a) represents /ɑː/. - An orange circle with the letter i in it represents /ɪ/. - A red circle with two dots above the i (ih) represents /ɨ̞/ or /ɨ/. - A cyan circle with a lowercase u (u) represents /ʊ/. - Data Points: - Each vowel sound is represented by multiple data points, which are scattered across the F1 and F2 axes. The distribution of these points varies for each vowel type. The plot provides a visual representation of how different vowels can be distinguished based on their F1 and F2 values in speech production. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Moment-to-moment Vowel Variation * Even the same speaker will have variation from moment to moment * Sometimes we misarticulate, accidentally making the wrong vowel quality * Or we talk with food in our mouths, producing different resonances * Or sometimes, we’re just plain lazy * This leads to constant and massive changes in vowel production --- <img class="r-stretch big" src="phonmedia/clearspeech_speakeraverages.png" alt="clearspeechspeakeraverages.png The image is a scatter plot titled F1 and F2 Values by Vowel. It displays data points representing different vowels across two axes: F1 (on the horizontal axis) and F2 (on the vertical axis). The plot uses various symbols to represent different vowel sounds, as indicated in the legend on the right side of the image. Here's a detailed description: - Axes: - The x-axis is labeled F1 and ranges from approximately 0 to 5. - The y-axis is labeled F2 and ranges from approximately 0 to 1. - Legend: - The legend on the right side of the plot explains which symbol corresponds to each vowel sound. - A blue circle with a caret (^) represents /a/. - A green circle with an at sign (@) represents /æ/. - A purple circle with a lowercase a (a) represents /ɑː/. - An orange circle with the letter i in it represents /ɪ/. - A red circle with two dots above the i (ih) represents /ɨ̞/ or /ɨ/. - A cyan circle with a lowercase u (u) represents /ʊ/. - Data Points: - Each vowel sound is represented by multiple data points, which are scattered across the F1 and F2 axes. The distribution of these points varies for each vowel type. The plot provides a visual representation of how different vowels can be distinguished based on their F1 and F2 values in speech production. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch big" src="phonmedia/clearspeech_alltokens.png" alt="clearspeechalltokens.png The image is a scatter plot titled Vowels from 8 clear-speech speakers. It displays data points representing different vowels produced by eight speakers of English. The x-axis represents F2 (fundamental frequency) in Hertz, ranging from approximately 350 to 1400 Hz. The y-axis represents F1 (first formant frequency), which ranges from about 200 to 800 Hz. Each data point is color-coded and marked with a symbol that corresponds to the type of vowel it represents: - Red circles represent @ - Green triangles represent @ - Blue squares represent u - Yellow diamonds represent a - Cyan pentagons represent e - Light blue plus signs represent ei - Brown stars represent au - Dark red diamonds represent ou - Purple hexagons represent i - Orange asterisks represent oi The plot includes 2802 tokens, all values are in Hertz. The data points are scattered across the graph, with each vowel type grouped together but spread out based on their respective F1 and F2 values. There is no additional context or background information provided about the speakers or the specific phonetic analysis being conducted. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch big" src="phonmedia/clearspeech_alltokensellipses.png" alt="clearspeechalltokensellipses.png The image is a scatter plot titled Vowels from 8 clear-speech speakers. It displays data points representing different vowels spoken by eight individuals, with each vowel represented by a specific color and symbol. The x-axis of the graph is labeled as F2 (Frequency 2) in Hertz (Hz), ranging from approximately 500 Hz to 1400 Hz. The y-axis is labeled as F1 (Frequency 1) also in Hz, ranging from about 3500 Hz to 800 Hz. The data points are color-coded and marked with symbols that correspond to different vowels: - Red circles represent the vowel @. - Green triangles represent the vowel @h. - Blue squares represent the vowel a. - Yellow diamonds represent the vowel u. - Light blue pentagons represent the vowel a. - Orange plus signs represent the vowel ei. - Brown squares represent the vowel au. - Purple diamonds represent the vowel ou. - Pink hexagons represent the vowel iu. - Magenta stars represent the vowel oi. The graph shows that each vowel has its own distinct cluster of data points, indicating different frequency patterns for F1 and F2. The clusters are spread across the graph, with some overlap but clear differentiation between vowels based on their color and symbol representation. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch big" src="humorimg/trainwreck.png" alt="trainwreck.png The image depicts a railway accident scene with several damaged train cars lying on their sides across multiple tracks. The train cars are in various states of disrepair; some have been bent and twisted, while others appear more intact but still show signs of impact. In the foreground, there is a green freight car that has derailed and is resting on its side over the tracks. Its roof is crushed, and it appears to be leaking or damaged at the top. Nearby, another train car with a brownish-green coloration lies on its side as well, with visible damage to its structure. Further back in the image, there are more train cars that have also derailed and are scattered across the tracks. Some of these cars appear to be intact but leaning at odd angles due to the collision or derailment. Several people can be seen near the scene of the accident. They seem to be inspecting the damage or possibly assessing the situation for safety reasons. The individuals vary in appearance, with some wearing darker clothing and others lighter attire. Their postures suggest they are engaged in conversation or observation. The tracks themselves appear to be part of a larger railway system, as there is another set of tracks running parallel to the damaged ones on the right side of the image. The ground around the tracks consists of gravel and dirt, typical for railway infrastructure. In the background, some greenery can be seen, indicating that this accident has occurred in an area with vegetation nearby. There are also some structures visible at a distance, possibly part of the railway station or related facilities. The sky is clear, suggesting it might have been taken during daytime under fair weather conditions. The overall scene conveys a sense of aftermath and investigation following a significant railway incident. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Every person you've ever talked with has had different vowel formant patterns * ... and yet, we understand each other, somehow --- <img class="r-stretch big" src="img/magic.jpg" alt="magic.jpg The image displays a cartoon illustration set against a plain white background. In the bottom left corner, a hand and forearm reach upward and slightly to the right. The figure appears to be wearing formal attire, indicated by a dark brown or black suit sleeve and a white cuff at the wrist. The hand is gripping the brim of a tall top hat. The top hat is tilted diagonally toward the upper left corner. It is decorated with vertical stripes in various colors, including purple, yellow, light blue-green, and dark blue. At the very top center of the hat, there is a small black circle or detail. Emerging from the opening of the tilted hat is a cascade of five-pointed stars that flow toward the right side of the image, simulating a magic trick. These stars are scattered in an arc pattern. They come in several bright colors: yellow, magenta pink, light blue, dark blue, green, and purple. Each star has a thick outline, primarily in blue or black. There is no text present in the illustration. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### There are a few ways this might work --- ### Speaker-intrinsic vowel space normalization * Normalization is a process that “happens” * You meet somebody, you create a model of their vowel space, and you move on * These models of speaker vowels are maintained in memory * One model per person, and a new model each time! --- ### Direct Realism - We're using our senses to form a model of reality, including inside the mouth - We don't really care about the acoustics per se, just estimating the gestures - "Based on everything I'm hearing, this seems like she's making the same tongue shape I hear in /i/" - This includes lots of adjustments 'for free' --- ### Speaker-extrinsic vowel space normalization * We store information from *every vowel we hear*! * Normalization is then just bulk comparison and probability * Vowel identities are probabilistically determined * Perhaps we also segment by speaker, dialect, language, etc --- <img class="r-stretch big" src="phonmedia/clearspeech_alltokens.png" alt="clearspeechalltokens.png The image is a scatter plot titled Vowels from 8 clear-speech speakers. It displays data points representing different vowels produced by eight speakers of English. The x-axis represents F2 (fundamental frequency) in Hertz, ranging from approximately 350 to 1400 Hz. The y-axis represents F1 (first formant frequency), which ranges from about 200 to 800 Hz. Each data point is color-coded and marked with a symbol that corresponds to the type of vowel it represents: - Red circles represent @ - Green triangles represent @ - Blue squares represent u - Yellow diamonds represent a - Cyan pentagons represent e - Light blue plus signs represent ei - Brown stars represent au - Dark red diamonds represent ou - Purple hexagons represent i - Orange asterisks represent oi The plot includes 2802 tokens, all values are in Hertz. The data points are scattered across the graph, with each vowel type grouped together but spread out based on their respective F1 and F2 values. There is no additional context or background information provided about the speakers or the specific phonetic analysis being conducted. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Humans are able to do this process - ... but what about ASR? --- ## ASR --- ### ASR builds mappings from audio to text - We feed the system lots of text, and lots of corresponding audio - It learns the patterns of sound associated with a given text - Some use language models to give better predictions --- ### Vintage ASR used to require explicit speaker normalization - In the HMM days, ASR software required personalization and 'training' - Setup began with "Read these texts aloud" - It would then process for a little while as it 'customized' to your voice - The model *simply wouldn't work* without this level of customization --- ### ... but when Neural Networks happened, things changed --- ### Whisper's architecture is complicated <img class="r-stretch" src="phonmedia/asr_whisper.png" alt="asrwhisper.png The image is a detailed diagram illustrating the process of sequence-to-sequence learning using transformer models in natural language processing (NLP). The diagram is divided into two main sections: the left side represents the encoder part of the model, and the right side shows the decoder part. Encoder Section: - Input: The input to the encoder section is a log-Mel spectrogram. This is represented at the bottom-left corner with a visual depiction showing frequency bands over time. - Transformer Encoder Blocks: These blocks are responsible for processing the input sequence. There are three such blocks, each containing two components: - MLP (Multi-Layer Perceptron): A neural network layer that processes information in multiple layers. - Self Attention: This mechanism allows different parts of the input to interact with each other based on their relevance and importance within the sequence. It is represented by a blue box labeled self attention. - Sinusoidal Positional Encoding: This is added to the input at the bottom-left corner, which helps the model understand the position of elements in the sequence. - Cross Attention: This mechanism allows information from the encoder blocks to be used during decoding. It connects with the decoder section through a connection labeled cross attention. Decoder Section: - Transformer Decoder Blocks: These blocks are responsible for generating output sequences based on the encoded input. There are three such blocks, each containing two components similar to those in the encoder: - MLP (Multi-Layer Perceptron). - Self Attention: This mechanism allows different parts of the generated sequence to interact with each other. - Learned Positional Encoding: This is added at the bottom-right corner and helps the model understand the position of elements in the output sequence. Output: - The final output consists of tokens in a multitask training format. These tokens are labeled as SOT, EN, TRANSCRIBE, followed by numerical values (e.g., 0.0, 1.0) and words like The quick brown. - The process involves predicting the next token based on the current input sequence. Text: - At the top of the diagram, there is a title: Sequence-to-sequence learning. - On the right side, above the decoder blocks, there are labels indicating different stages or components such as next-token prediction, cross attention, and self attention. This detailed breakdown should help you understand the structure and function of sequence-to-sequence learning using transformer models. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### ... but it is wildly effective - It works relatively quickly - On relatively low-end hardware - ... and most amazing of all... ### Whisper can get human-like performance in speech transcription* <img class="r-stretch" src="phonmedia/asr_whispervshumans.png" alt="asrwhispervshumans.png The image is a bar chart that compares word error rates for different methods of transcription: ASR (Automatic Speech Recognition) and human transcription with computer assistance. The x-axis represents various transcription methods labeled as Whisper, A through I. The y-axis shows the Word Error Rate in percentage. Each method has two bars representing its performance under ASR and human transcription with computer assistance. The bars are color-coded for clarity: - ASR: - Whisper: Blue - Others (A to I): Different colors (orange, green, red, purple, pink, gray, yellow, cyan) - Human Transcription with Computer Assistance: - Whisper: Light blue - Others (A to I): Different colors (light orange, light green, light red, light purple, light pink, light gray, light yellow, light cyan) Each bar has a box plot at the bottom showing the distribution of error rates. The median value is indicated by a horizontal line inside each box. The numbers above some bars represent specific values: - Whisper: 8.81 (ASR), 9.66 (human transcription with computer assistance) - A: 9.74 (ASR), 10.9 (human transcription with computer assistance) - B: 9.74 (ASR), 12.2 (human transcription with computer assistance) - C: 8.65 (ASR), 8.14 (human transcription with computer assistance) - D: 8.65 (ASR), 8.14 (human transcription with computer assistance) - E: 7.61 (ASR), 8.14 (human transcription with computer assistance) - F: 8.65 (ASR), 8.14 (human transcription with computer assistance) - G: 8.65 (ASR), 8.14 (human transcription with computer assistance) - H: 8.96 (ASR), 10.5 (human transcription with computer assistance) - I: 10.5 (ASR), 12.2 (human transcription with computer assistance) The chart is designed to compare the performance of different transcription methods in terms of word error rates, highlighting how human transcription with computer assistance generally performs better than ASR for most methods except Whisper and B. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Wait, what was that asterisk? ---