# Speech Technology is Incredible ### Will Styler - CSSA Talk Series <https://savethevowels.org/talks/cssa2025.html> --- ### Introductions - I'm an Associate Teaching Professor of Linguistics - I'm a Computational Phonetician - I study human speech perception and production using computers - This involves a mix of experiments, data analysis, recordings, and instrumental measurements - I've also done Natural Language Processing work - I work to develop new tools for measurement and teaching --- ### I'm also Director of [Computational Social Science](https://css.ucsd.edu) at UCSD - We're an interdisciplinary program uniting people across Social Science who work with and care about computation and data - We believe that **data matters because people matter** - I'm here because I'm always thinking about computers as a tool for understanding humans! --- ### But there remains an elephant in this CSSA-filled room - I'm not actually in Cognitive Science! --- <img class="r-stretch" src="img/impostor.jpg" alt="impostor.jpg The image displays a digital illustration set against a backdrop of deep space. At the top center, the word Impostor is written in large, red, pixelated letters. Below this text stands a cartoon character from the video game Among Us. The character has a bean-like shape with a thick black outline. Its suit is primarily bright red, but it features darker purple and magenta shading on the left side of its body and its right leg to suggest folds in the fabric. On the front of the character's head area is a large, light blue-grey visor with rounded corners and a white reflection in the center. A red backpack protrudes from the back of the figure. The background depicts outer space, filled with numerous small white stars against a dark blue and black sky. There are also nebula clouds visible in shades of purple, pink, and reddish-orange. Beneath the character's feet is a dark grey shadow on the ground. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Cognitive Science wasn't a separate program where I grew up - I have my BA, MA and Doctorate in Linguistics from the [University of Colorado at Boulder](https://www.colorado.edu/linguistics/) - CU Boulder had the [Institute of Cognitive Science](https://www.colorado.edu/ics/) - Collaborations between faculty in CS, Philosophy, Linguistics, Psych, Education, and more - I went to their talks and was advised by an affiliate, but there was not a COGS major or Ph.D Specialization - ... but I am bothered by many COGS-flavored questions --- ### But I do have a lot of COGS ties - I collaborate lots with [Dr. Sarah Creel](https://quote.ucsd.edu/lasr/) in COGS - Who is an incredible person to talk to if you're interested in speech in COGS - CSS has [lots of Cognitive Scientists](https://css.ucsd.edu/people/faculty.html#Cognitive-Science)! - Including [Dr. Sean Trott](https://seantrott.github.io/), one of our Core Faculty - And [Dr. Shannon Ellis](https://shannon-ellis.com/) is on our steering committee - ... and I know that Language folks over here are just as awesome and effective --- ### No matter where my office is, Language is the best - Speech is even cooler than Language - ... and all the more awesome when we put computers into the process --- ## Speech Technology --- ### Speech Technology is pervasive in the US - Siri/Alexa/GoogleAssistant - ChatGPT Voice Mode - Speech-to-Text Keyboards - Text-to-Speech (e.g. in GPS or Twitch streams) - ... and much more! --- ### Speech technology is absolutely fascinating - **... but the most interesting part is that it works at all!** --- ### Today's Plan - Why is speech so hard to produce? - How can computers produce speech? - Why is speech so hard to perceive? - How can computers perceive speech? --- ## Speech is Hard --- ### Human Speech is incredibly difficult - This is an incredibly intricate gestural dance in your mind and mouth - Let's try it --- ### "A Linguistics Major goes very well with Cognitive Science" - First, focus on your jaw - Now, on your tongue - Now, feel the vibes --- ### Speech is *hard* - Fluid movement of your mouth and tongue - Careful planning of air and breathing - Control of pitch, gestures, and other aspects - All to create tiny pressure variations in the air --- <img class="r-stretch" src="phonmedia/sky.png" alt="sky.png The image is a graph that represents an audio signal over time. The x-axis of the graph is labeled Time (s) and ranges from 0 seconds to approximately 0.6278 seconds. The y-axis is labeled with values ranging from -0.3656 at the bottom to 0.3023 at the top, indicating a range of audio signal amplitudes. The graph shows an audio waveform that starts at zero amplitude and rises sharply in the first few milliseconds before dropping back down to near-zero amplitude. This pattern repeats several times throughout the duration of the recording, with each cycle being relatively short compared to the overall length of the recording. The text sky is written above the graph, suggesting this might be a spectrogram or waveform visualization for the word sky. There are no people depicted in this image; it solely focuses on the audio signal and its representation over time. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch" src="phonmedia/skyspectrogram.png" alt="skyspectrogram.png The image provided is a spectrogram, which is a visual representation of the spectrum of frequencies in a sound signal as it varies with time. This particular spectrogram shows the frequency content over time for the word sky. Key Elements: 1. Axes: - The horizontal axis represents time (in seconds), ranging from 0 to approximately 0.6278 seconds. - The vertical axis represents frequency, measured in Hertz (Hz), ranging from 0 Hz at the bottom to around 5000 Hz at the top. 2. Text: - At the center of the spectrogram, there is text that reads sky, indicating what sound was analyzed for this spectrogram. 3. Color Gradient: - The image uses a grayscale color gradient where darker shades represent higher intensity or louder sounds. - Lighter shades indicate lower intensity or quieter sounds. 4. Frequency Distribution: - At the beginning of the time axis (around 0 seconds), there is a burst of activity, indicating the initial sound onset. - As we move along the horizontal axis, the frequency content changes over time, showing how the frequencies vary during the duration of the word sky. - The spectrogram shows that higher frequencies are present at the beginning and then decrease as the sound progresses. 5. Intensity: - There is a noticeable increase in intensity (darker shades) around 0 seconds, which corresponds to the initial burst of sound. - As time progresses, there is a gradual decrease in intensity until it stabilizes for most frequencies by about 0.6278 seconds. This spectrogram provides a detailed view of how the frequency content changes over time during the pronunciation of sky. It's useful for analyzing speech sounds and understanding their acoustic properties. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### ... and we want to do *this* with software?!? - 'Speech Synthesis' or 'Text-to-Speech' (TTS) - *How do we do that?* --- ### The Task "A Linguistics major goes very well with Cognitive Science" <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Historically, the steps were simple - Analyze what the text needs to sound like ('Text Analysis') - Jelena saw 1985 listings in La Jolla CA for over $2 million - /jɛlɛnə sɑ najntin hʌndɹɪd . . . / - Now, transform that into a wave we can play back for the humans --- ### For a long time, we cheated using humans! - **Concatenative or 'Unit Selection' TTS** chops up bits and pieces of existing speech to create new speech - You record a huge database of speech from a voice actor, with optimum 'coverage' - You update as new words emerge (e.g. COVID, rizz, skibidi) - **You then combine these words into sentences to match the text** - ... and you use fancy algorithms to smooth the results out. --- ### The result can be imperfect <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Then, Artificial Neural Networks arrived, and everything changed <img class="r-stretch" src="dalle/cuteneuralnetwork.jpg" alt="cuteneuralnetwork.jpg This image features a colorful, anthropomorphic character that resembles a friendly virus or cell set against a glowing, abstract background. The character is the central focus and appears to be walking or dancing towards the viewer. The character has a round, bulbous body covered in a texture that looks like a digital mesh or network of glowing white lines connecting small dots. This mesh creates a gradient of colors across its body, shifting from cool blues and teals on the left side to warm oranges and yellows on the right. Radiating outward from the main body are approximately twelve distinct spikes or antennae. Each spike ends in a shiny, spherical orb. These orbs vary in color, including metallic gold, bright orange, deep purple, and teal green. The character has a face with two very large, round eyes positioned side-by-side near the top of its body. The eyes are dark brown with thick black outlines and white highlights that suggest they are shiny or reflecting light. Below the eyes is a simple, curved black line forming a gentle smile. At the bottom of the character's body are two short, stubby legs that look like rounded feet. The left leg (viewer's left) is primarily orange and brown with a metallic sheen, while the right leg is blue and purple. They appear to be in mid-stride, suggesting movement. The background is soft and out of focus, filled with bokeh effects—circular blurs of light in shades of pink, white, yellow, and blue. Behind the character, there are faint, thin white lines connecting dots in a geometric pattern, resembling a constellation or a neural network map, which extends across the entire backdrop. There is no text or specific diagrams present in the image. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### The World's Worst Introduction to Neural Networks <img class="r-stretch" src="img/dnn.jpg" alt="dnn.jpg This image displays a schematic diagram of a feedforward neural network architecture, arranged horizontally from left to right. The diagram is divided into three distinct sections by vertical dotted lines, representing different layers of the network. On the far left is the Input layer, labeled at the top with the text Input layer and the variable $i$. This section consists of a vertical column of grey circular nodes. To the left of these nodes are labels indicating the inputs: Input 1 with an arrow pointing to the top node, Input 2 with an arrow pointing to the second node, followed by a vertical ellipsis (three dots), and finally Input n with an arrow pointing to the bottom node. This indicates that there are 'n' total input values entering the network. The middle section is labeled at the top as Hidden layers. It contains three distinct columns of grey circular nodes, representing different stages within the hidden part of the network. These columns are labeled above as $h1$, $h2$, and $hn$. Each column shows a top node, a second node, a third node, a vertical ellipsis indicating more nodes in between, and a bottom node. This signifies that there are multiple neurons in each hidden layer. On the far right is the Output layer, labeled at the top with Output layer and the variable $o$. This section features a column of grey circular nodes similar to the input layer but arranged vertically on the right side. To the right of these nodes are labels indicating the outputs: Output 1 with an arrow pointing away from the top node, followed by a vertical ellipsis, and Output n with an arrow pointing away from the bottom node. This indicates 'n' output values produced by the network. Connecting these layers is a dense web of straight black lines representing the connections or weights between neurons. Every node in the Input layer connects to every node in the first hidden column ($h1$). Similarly, every node in $h1$ connects to every node in $h2$, and so on through to $hn$. Finally, every node in the last hidden column ($hn$) connects to every node in the Output layer. This structure illustrates a fully connected network where information flows sequentially from left to right. At the very bottom left of the image, there is source text that reads: Source: https://doi.org/10.1186/s40537-020-00328-3. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### For today, Neural Networks learn to transform input data into a desired output data - Training involves presenting the network with both input and output - Then we change the network between to make the output closer to the desired output - Then you feed in new input, and get new output - They are wildly complex, and wildly powerful - **Take COGS 181 to actually understand this!** --- ### [TacoTron2](https://arxiv.org/pdf/1712.05884) is a relatively simple, neural TTS system - For now, we input text, we get speech - It's trained using speech with paired text - It takes text, and generates spectrograms, chunk-by-chunk, which can be turned into a waveform --- ### TacoTron 2 <img class="r-stretch" src="phonmedia/tts_tacotron2.png" alt="ttstacotron2.png The image is a block diagram illustrating the architecture of the Tacotron 2 system, which is used for speech synthesis. The diagram consists of various components connected by arrows indicating the flow of data. Components and Flow: 1. Input Text: - At the bottom left corner, there's an input labeled Input Text. This represents the text that will be converted into speech. 2. Character Embedding: - The text is fed into a block labeled Character Embedding, which converts each character in the input text into numerical vectors. 3. Bidirectional LSTM (Long Short-Term Memory): - The output from Character Embedding goes to a Bidirectional LSTM, which processes the sequence of characters bidirectionally. 4. Location Sensitive Attention: - The Bidirectional LSTM's output is then passed through Location Sensitive Attention, which helps in focusing on relevant parts of the input text at different positions. 5. 2 LSTM Layers: - The attention mechanism’s output goes into two layers of Long Short-Term Memory (LSTM) networks, labeled as 2 LSTM Layers. 6. Linear Projection: - After the 2 LSTM Layers, there are blocks for Linear Projection, which transform the outputs from the LSTMs. 7. 5 Conv Layer Post-Net: - The output of Linear Projection is then fed into a block labeled 5 Conv Layer Post-Net. This consists of five convolutional layers that further refine and smooth out the speech signal. 8. 2 Layer Pre-Net: - Another part of the system, labeled as 2 Layer Pre-Net, processes the input text before it reaches the main processing blocks. 9. Mel Spectrogram: - At the top center of the diagram is a visual representation called Mel Spectrogram. This shows the frequency content over time in a logarithmic scale and is used to represent the target output for speech synthesis. 10. WaveNet Samples: - On the right side, there's an input labeled WaveNet Samples, which represents the raw audio samples that are generated by WaveNet. Additional Details: - The diagram includes labels such as Location Sensitive Attention and Linear Projection, indicating specific processing steps. - There is a + symbol between the 2 Layer Pre-Net and the 5 Conv Layer Post-Net, suggesting an addition or concatenation of outputs from these two components before they are fed into the next part of the system. This diagram provides a clear overview of how Tacotron 2 processes input text to generate speech by converting it into audio waveforms. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### This allows us to go from text to speech! - We feed in text, and we get back a wave, with no humans involved past making training data! - State-of-the-art models are getting very, very good! <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### The State of the Art is Advanced, but closed - Current state of the art models from ElevenLabs, OpenAI, Google, and Amazon are all closed and proprietary - If you want the best TTS in the world, it has to happen on somebody else's computer - Details are often not published and considered "trade secrets" - It's not currently possible to teach the state of the art in TTS! - ... and this should disturb us as a society --- ### Neural TTS can be trained using *any* voice - You can build a model from the ground up using any voice you'd like - If all your training data are from a bored Bostonian, you'll end up with a bored Bostonian TTS voice - Yet, we might want different voices... --- > All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Humans know that content and style are different - Speech expresses linguistic 'content' - Phonemes, with ordering, and necessary pitch and timing for comprehension - This is 'all we need' to understand the utterances - There's also 'Style', which gives us lots of other details - 'Speaker' identity - Social components - Emotional content - Plus things like speed, emphasis, pitch 'tunes', sarcasm --- ### Couldn't we just abstract out the 'style' component and apply it to whatever Linguistic content we'd like? - Yes! --- ### Here's a Multi-Speaker Version of TacoTron 2 <img class="r-stretch" src="phonmedia/tts_tacotron2_multispeaker.png" alt="ttstacotron2multispeaker.png The image is a detailed diagram illustrating a multi-speaker Text-to-Speech (TTS) system. The diagram includes various components and their interactions within this system. Components of the Multi-Speaker TTS System: 1. Text Input: - At the bottom left, there's an input labeled Text, which is the starting point for the process. 2. Speaker Embedding: - Below the text input, there are two blocks labeled Speaker Embedding. - These blocks represent the speaker-specific information that will be used to generate speech. 3. Speaker Encoder: - To the right of the Speaker Embedding block is a red box labeled Speaker Encoder. - This component processes the speaker embedding and prepares it for further use in generating speech. 4. CBH-LSTM (Conditional Bi-directional Hierarchical Long Short-Term Memory): - Below the Speaker Encoder, there's another block labeled CBH-LSTM, which is a type of neural network used to process sequences. - This component takes the speaker embedding and text input as inputs. 5. Phone Embedding: - To the right of CBH-LSTM, there’s an orange box labeled “Phone Embedding.” - This block converts the text into phonetic representations (phones) which are used to generate speech sounds. 6. Self Attention: - Below Phone Embedding is a block labeled Self Attention. - Self-attention mechanisms help in understanding the relationships between different parts of the input sequence, which aids in generating more natural-sounding speech. 7. Decoder RNN (Recurrent Neural Network): - To the right of Self Attention, there’s another orange box labeled “Decoder RNN.” - This component takes the phonetic representations and generates a sequence of audio frames that represent the speech sound. 8. Additive Attention: - Below Decoder RNN is an orange block labeled Additive Attention. - This mechanism helps in focusing on specific parts of the input to generate more accurate speech sounds. 9. Concat: - To the right of Additive Attention, there’s a block labeled “Concat.” - This component combines multiple outputs from previous blocks into one sequence for further processing. 10. Pre-Net and Post-NET Blocks (Green Boxes): - At the top left is a green box labeled Pre-Net. - To its right is another green box labeled Post-Net. - These blocks are used to preprocess and post-process the audio signal, respectively. 11. WaveNet: - At the far right of the diagram, there’s a yellow block labeled “WaveNet.” - WaveNet is a type of neural network that generates raw audio samples from the processed outputs. 12. Sigmoid Function and Stop Token: - Below WaveNet are two blocks labeled Sigmoid and Stop Token. - The Sigmoid function is used to normalize the output, while the Stop Token helps in determining when to stop generating speech. Flow of Information: - The process starts with text input. - This text goes through a Speaker Encoder which processes speaker-specific information. - The processed speaker embedding then flows into CBH-LSTM along with phone embeddings. - The outputs from these components are combined and passed through various attention mechanisms (Self Attention, Additive Attention). - These combined outputs go to the Decoder RNN, which generates phonetic representations of speech sounds. - Further processing involves concatenation and additional attention mechanisms before being fed into Pre-NET and Post-NET blocks for final audio processing. Visual Representation: - The diagram uses arrows to indicate the flow of information between different components. - Each block is labeled with its function or name, making it easier to follow the process step by step. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### The results of this are... terrifying - ... and has given rise to 'Deepfake' voices --- ### Neural Styler Transfer <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (TacoTron2) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p>(ElevenLabs) (Credit to Erick Amaro and Mia Khattar!) --- ### Multilingual Examples <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (English) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (French) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Spanish) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Mandarin) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Italian) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Russian) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Japanese) --- ### Prosody is still hard <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### ... but OMG, this thing can do arbitrary speech, in an arbitrary voice - ... and it's never had a tongue, had phonics training, and doesn't actually know anything at all about mouths - Arguably, it doesn't know anything about English - ... although some systems use a language model too - *This is amazing!* - ... but it can model more complexity still --- ### Code Switching It's like sometimes mezclo un poco de español con my English, cuando me siento particularmente spicy, y tengo curiosidad to know cómo la TTS handles it. <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Wow. - Not only can exposure to data allow a deep neural network to learn to map written language into speech in one language - ... but it can do it for two languages - ... at once - ... with clear mixing of the two - **Espicy!** --- # What about Speech Perception? --- ### Speech Perception is *hard* - Speech is flapping bits of meat around in your head and throat while you expel air. - This creates tiny vibrations in the air - **Speech perception is turning the resulting vibrations in the air back into language** --- ### Let's focus on one of the really hard problems --- ## Vowel Perception --- ### What is a vowel? * A vowel is letting the voice resonate in the vocal tract while you move the tongue * If we change the position of the tongue, we change the resonances --- <img class="r-stretch" src="phonmedia/aeiou.png" alt="aeiou.png The image contains a list of letters with some marked with red circles and crosses, indicating they are incorrect choices in what appears to be an educational context, possibly related to phonetics or spelling. The letters shown are: - a (marked with a red circle) - e (marked with a red circle) - i (marked with a red circle) - o (marked with a red circle) - u (marked with a red circle) Following these, there is text that reads and sometimes in black. Next to this phrase, the letter v has been crossed out with a red line and marked with a red cross. To the right of all the letters, there are two large words written in red: - The first word says NEVER!!!! - Below it is another word that reads sometimes but with the letter v crossed out. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### English has lots of vowels /i/ - beet, see, seen, sear, seal /ɪ/ - bit, sit, tin, sill /ɛ/ - bet, set, sent, fair, sell /æ/ - bat, sat, pant, pal /ʌ/ - but, sun, pun, lull (ə in sofa, amount) /əɹ/ - bird, purr, earl, butter, clamor (this is often broken into two vowels!) /ɑ/ - bot, saw, star, paul, pawn, (cot*) /ɔ/ - corn /kɔɹn/, boy /bɔj/ (caught*) /ʊ/ - book, hood, puss /u/ - boot, who’d, loose, lure, loon --- ### Diphthongs, too! /ɔj/ - boy, soy, toy, join, oil, Roy /aj/ - buy, right, try, sigh, die, fire /ej/ - play, bay, may, ray, lay, trail /ow/ - boat, oat, wrote, pope, toll /aw/ - how, now, brown, cow, prow, louse --- <img class="r-stretch big" src="phonmedia/voweltongue.png" alt="voweltongue.png The image is a diagram illustrating the positions of the tongue for different vowel sounds in speech. The diagram shows a side view of the human head with an emphasis on the mouth and throat area. The tongue's position relative to the palate (the roof of the mouth) is highlighted, indicating how it changes shape and position to produce various vowel sounds. The diagram includes labels for four different vowels: i, e, ɛ, and a. Each label corresponds to a specific line that shows where the tongue should be placed in your mouth when pronouncing these vowels. The lines are color-coded, with each vowel having its own distinct color: - i is represented by a red line. - e is shown as a blue line. - ɛ is depicted using a green line. - a is indicated by a purple line. The diagram also includes small dots on the tongue to show where it should be placed for each vowel sound. The i position shows the tongue positioned high in the mouth, close to the palate, which is typical for front and high vowels like /ɪ/. For e, the tongue is shown lower than for i, but still relatively high, closer to the middle of the mouth. The ɛ vowel has a slightly more open position compared to e, with the tongue positioned further back in the mouth. This is characteristic of front and mid vowels like /ɛ/. Finally, the a vowel shows the tongue at its lowest point, close to the throat, which is typical for back and low vowels such as /ɑː/ or /a/. The diagram provides a clear visual guide on how the position of the tongue changes when pronouncing different vowels. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch big" src="phonmedia/voweltongue2.png" alt="voweltongue2.png The image consists of four detailed line drawings that represent different vowel sounds produced by the human vocal tract. Each drawing shows a side view of the mouth and throat area, focusing on the position of the tongue and lips for various vowels. 1. Top Left (i): This diagram illustrates the position of the tongue when producing the vowel sound /i/, which is often used in words like see or she. The tongue is positioned high in the mouth with the tip near the front, and the lips are rounded slightly. 2. Top Right (u): Here, we see the diagram for the vowel sound /u/, as in boot or put. The tongue is placed low back in the mouth, close to the soft palate, while the lips are rounded tightly together. 3. Bottom Left (æ): This drawing shows the position of the tongue and lips when producing the vowel sound /æ/, which can be heard in words like cat or hat. The tongue is relatively low in the mouth with a slight forward movement, and the lips are slightly apart but not rounded. 4. Bottom Right (ɑ): This diagram depicts the position for the vowel sound /ɑ/, often used in words such as father or father. The tongue is positioned low back in the mouth near the soft palate, and the lips are relatively relaxed with a slight parting. Each drawing includes a small dot to indicate where the center of the tongue should be placed within the oral cavity for each vowel sound. The diagrams are simple line drawings without any additional colors or shading, focusing solely on illustrating the vocal tract positions necessary for producing these specific sounds. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### What do vowels sound like? * We talk about vowel quality in terms of "formants" * These are bands of the spectrum where the energy is strongest * The frequencies of these formants are how we distinguish vowels --- <img class="r-stretch big" src="phonmedia/iformants.png" alt="iformants.png The image appears to be a spectrogram, which is a visual representation of the spectrum of frequencies in a sound signal as it varies with time. The horizontal axis represents time, and the vertical axis represents frequency (in Hertz). The intensity of the color at each point indicates the amplitude or volume of the sound at that particular frequency and moment. In this spectrogram: - The y-axis is labeled Hz, which stands for Hertz, a unit of measurement for frequency. It ranges from 0 to approximately 4000 Hz. - The x-axis represents time, though it's not marked with specific intervals or labels indicating the duration in seconds or minutes. The spectrogram shows a sound that has a consistent pattern across different frequencies over time: - There is a band of activity around 2000 Hz and another at approximately 1000 Hz. - The intensity of these bands fluctuates slightly, suggesting variations in volume within those frequency ranges as the sound progresses. There are no people or characters depicted in this image. It focuses solely on the analysis of audio data through a spectrogram format. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch big" src="phonmedia/iformantslabeled.png" alt="iformantslabeled.png The image appears to be a spectrogram of speech, which is a visual representation of the spectrum of frequencies present in a sound over time. This type of graph is commonly used in phonetics and linguistics to analyze the frequency content of sounds. Key Features: 1. X-Axis (Horizontal Line): Represents time. 2. Y-Axis (Vertical Line): Represents frequency, measured in Hertz (Hz). The scale on the left side indicates that frequencies range from 0 Hz at the bottom to approximately 4000 Hz at the top. Spectrogram Details: - Frequency Bands: - There are three labeled bands of interest marked as F1, F2, and F3. These correspond to formant frequencies in speech sounds. - F1 (First Formant): Typically associated with vowels and is lower than F2 and F3. - F2 (Second Formant): Higher frequency band that varies depending on the vowel being produced. - F3 (Third Formant): Even higher in frequency compared to F1 and F2, often used for distinguishing between different vowels. - Intensity Bands: - The spectrogram shows varying intensities of sound energy over time. Darker areas indicate louder sounds or more intense frequencies at that point in time. Time Axis: - The horizontal axis represents the duration of speech. The left side is labeled with a number [1], which might be indicating the start of a specific segment. People and Characters: The image does not contain any people, characters, or text other than what has been described above. It focuses solely on the spectrogram representation of sound frequencies over time. If you need further assistance understanding this type of graph or have additional questions about it, feel free to ask! This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### So, different vowels are basically different formant patterns --- <img class="r-stretch big" src="phonmedia/vowelformants.gif" alt="vowelformants.gif This image displays a scientific chart containing eight individual spectrograms arranged in a grid of two rows and four columns. The background is white, and the graphical elements are black. There are no people or characters in this image. The vertical axis on the far left represents frequency in Hertz (Hz), with markings for 1000, 2000, 3000, and 4000 Hz increasing from bottom to top. The horizontal axis at the very bottom represents time in milliseconds (ms), with markings for 0, 300, and 400 visible. Each of the eight panels displays a spectrogram of a specific vowel sound, identified by an International Phonetic Alphabet (IPA) symbol centered below each graph. In these graphs, darker horizontal bands represent formants, which are concentrations of acoustic energy at specific frequencies. Small black arrows point from the left margin toward these dark bands in most panels to highlight them. Top Row (from left to right): 1. Panel labeled [ i ]: Shows a spectrogram with distinct dark bands. One band is visible near the top around 3000 Hz, another below it around 2500 Hz, and a lower one near 1000 Hz. Arrows point to these frequencies. 2. Panel labeled [ ɪ ]: Similar to the first panel but with slightly different band positions. The upper bands are clustered between 2000 and 3000 Hz. Arrows indicate specific formant locations. 3. Panel labeled [ ɛ ]: Shows bands that are generally lower in frequency than the previous two. There is a cluster of energy around 2000-2500 Hz and a distinct band near 1000 Hz. Arrows point to these features. 4. Panel labeled [ æ ]: This graph shows dark bands distributed across the vertical range, with significant energy visible between 2000 and 3000 Hz and another band lower down around 800-900 Hz. Bottom Row (from left to right): 1. Panel labeled [ a ]: This spectrogram shows three distinct dark bands indicated by arrows on the far left. One is low, near 500 Hz; one is in the middle around 1200 Hz; and one is higher up around 2800 Hz. 2. Panel labeled [ ɔ ]: Shows a pattern with energy concentrated between 1000 and 2500 Hz. Arrows point to bands near 600 Hz, 1200 Hz, and 2300 Hz. 3. Panel labeled [ ʊ ]: This graph shows a concentration of energy lower down on the frequency scale compared to the front vowels. There are arrows pointing to bands around 800 Hz and 1500 Hz. The texture is somewhat grainier than the others. 4. Panel labeled [ u ]: This final panel shows dark bands primarily in the lower half of the graph, below 2000 Hz. On the right side of this specific graph, there is a distinct curved line or hook shape rising from the middle frequency area up towards the top right corner. Arrows point to bands near 500 Hz and 1200 Hz. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> <small>Different American English vowels, as spoken by a male speaker</small> --- ### Speaker Vowel Space Variation * Different speakers produce different resonances, even for the “same” vowels * Vocal tracts can be shorter, longer, wider... --- ### Here's the weird part! - Different speakers have different formants, even for the “same” vowels! * Every person has a different set of basic vowel formant positions * This is called the speaker’s “vowel space” --- ### 'Idealized' Formants <img class="r-stretch" src="phonmedia/ipaformantsgraph.png" alt="ipaformantsgraph.png The image is a scatter plot titled F1 and F2 Values by Vowel. It displays data points representing different vowels on a graph with two axes: F1 (on the horizontal axis) and F2 (on the vertical axis). The values for F1 range from approximately 300 to 4,000 Hz, while the values for F2 range from about 50 to 1,500 Hz. Each data point on the graph is labeled with a vowel symbol. Here are the details of each label: - i: Located in the upper left quadrant. - I: Positioned slightly below and to the right of i. - e: Found near the center-left area. - æ: Situated close to the bottom-center part of the graph. - ε: Placed above and to the left of æ. - ə: Located in the lower middle section, closer to the horizontal axis. - ʊ: Positioned towards the upper right quadrant. - u: Found slightly below and to the right of ʊ. - ʌ: Situated near the bottom-right area. - ɒ: Placed close to the center-right part. - ɑ: Located in the lower middle section, closer to the vertical axis. The graph uses a Cartesian coordinate system with labeled axes for F1 (horizontal) and F2 (vertical). The data points are plotted as black dots. There is no additional text or diagram within the image that provides further context or explanation of the relationship between the vowels and their corresponding F1 and F2 values. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Speaker Average Formants <img class="r-stretch" src="phonmedia/clearspeech_speakeraverages.png" alt="clearspeechspeakeraverages.png The image is a scatter plot titled F1 and F2 Values by Vowel. It displays data points representing different vowels across two axes: F1 (on the horizontal axis) and F2 (on the vertical axis). The plot uses various symbols to represent different vowel sounds, as indicated in the legend on the right side of the image. Here's a detailed description: - Axes: - The x-axis is labeled F1 and ranges from approximately 0 to 5. - The y-axis is labeled F2 and ranges from approximately 0 to 1. - Legend: - The legend on the right side of the plot explains which symbol corresponds to each vowel sound. - A blue circle with a caret (^) represents /a/. - A green circle with an at sign (@) represents /æ/. - A purple circle with a lowercase a (a) represents /ɑː/. - An orange circle with the letter i in it represents /ɪ/. - A red circle with two dots above the i (ih) represents /ɨ̞/ or /ɨ/. - A cyan circle with a lowercase u (u) represents /ʊ/. - Data Points: - Each vowel sound is represented by multiple data points, which are scattered across the F1 and F2 axes. The distribution of these points varies for each vowel type. The plot provides a visual representation of how different vowels can be distinguished based on their F1 and F2 values in speech production. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Moment-to-moment Vowel Variation * Even the same speaker will have variation from moment to moment * We often move our tongues differently, changing the vowel's quality * For many, many reasons * This leads to constant and massive changes in vowel production --- ### Speaker Average Formants <img class="r-stretch" src="phonmedia/clearspeech_speakeraverages.png" alt="clearspeechspeakeraverages.png The image is a scatter plot titled F1 and F2 Values by Vowel. It displays data points representing different vowels across two axes: F1 (on the horizontal axis) and F2 (on the vertical axis). The plot uses various symbols to represent different vowel sounds, as indicated in the legend on the right side of the image. Here's a detailed description: - Axes: - The x-axis is labeled F1 and ranges from approximately 0 to 5. - The y-axis is labeled F2 and ranges from approximately 0 to 1. - Legend: - The legend on the right side of the plot explains which symbol corresponds to each vowel sound. - A blue circle with a caret (^) represents /a/. - A green circle with an at sign (@) represents /æ/. - A purple circle with a lowercase a (a) represents /ɑː/. - An orange circle with the letter i in it represents /ɪ/. - A red circle with two dots above the i (ih) represents /ɨ̞/ or /ɨ/. - A cyan circle with a lowercase u (u) represents /ʊ/. - Data Points: - Each vowel sound is represented by multiple data points, which are scattered across the F1 and F2 axes. The distribution of these points varies for each vowel type. The plot provides a visual representation of how different vowels can be distinguished based on their F1 and F2 values in speech production. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Individual Token Formants <img class="r-stretch" src="phonmedia/clearspeech_alltokens.png" alt="clearspeechalltokens.png The image is a scatter plot titled Vowels from 8 clear-speech speakers. It displays data points representing different vowels produced by eight speakers of English. The x-axis represents F2 (fundamental frequency) in Hertz, ranging from approximately 350 to 1400 Hz. The y-axis represents F1 (first formant frequency), which ranges from about 200 to 800 Hz. Each data point is color-coded and marked with a symbol that corresponds to the type of vowel it represents: - Red circles represent @ - Green triangles represent @ - Blue squares represent u - Yellow diamonds represent a - Cyan pentagons represent e - Light blue plus signs represent ei - Brown stars represent au - Dark red diamonds represent ou - Purple hexagons represent i - Orange asterisks represent oi The plot includes 2802 tokens, all values are in Hertz. The data points are scattered across the graph, with each vowel type grouped together but spread out based on their respective F1 and F2 values. There is no additional context or background information provided about the speakers or the specific phonetic analysis being conducted. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Individual Token Formants <img class="r-stretch" src="phonmedia/clearspeech_alltokensellipses.png" alt="clearspeechalltokensellipses.png The image is a scatter plot titled Vowels from 8 clear-speech speakers. It displays data points representing different vowels spoken by eight individuals, with each vowel represented by a specific color and symbol. The x-axis of the graph is labeled as F2 (Frequency 2) in Hertz (Hz), ranging from approximately 500 Hz to 1400 Hz. The y-axis is labeled as F1 (Frequency 1) also in Hz, ranging from about 3500 Hz to 800 Hz. The data points are color-coded and marked with symbols that correspond to different vowels: - Red circles represent the vowel @. - Green triangles represent the vowel @h. - Blue squares represent the vowel a. - Yellow diamonds represent the vowel u. - Light blue pentagons represent the vowel a. - Orange plus signs represent the vowel ei. - Brown squares represent the vowel au. - Purple diamonds represent the vowel ou. - Pink hexagons represent the vowel iu. - Magenta stars represent the vowel oi. The graph shows that each vowel has its own distinct cluster of data points, indicating different frequency patterns for F1 and F2. The clusters are spread across the graph, with some overlap but clear differentiation between vowels based on their color and symbol representation. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch" src="humorimg/trainwreck.png" alt="trainwreck.png The image depicts a railway accident scene with several damaged train cars lying on their sides across multiple tracks. The train cars are in various states of disrepair; some have been bent and twisted, while others appear more intact but still show signs of impact. In the foreground, there is a green freight car that has derailed and is resting on its side over the tracks. Its roof is crushed, and it appears to be leaking or damaged at the top. Nearby, another train car with a brownish-green coloration lies on its side as well, with visible damage to its structure. Further back in the image, there are more train cars that have also derailed and are scattered across the tracks. Some of these cars appear to be intact but leaning at odd angles due to the collision or derailment. Several people can be seen near the scene of the accident. They seem to be inspecting the damage or possibly assessing the situation for safety reasons. The individuals vary in appearance, with some wearing darker clothing and others lighter attire. Their postures suggest they are engaged in conversation or observation. The tracks themselves appear to be part of a larger railway system, as there is another set of tracks running parallel to the damaged ones on the right side of the image. The ground around the tracks consists of gravel and dirt, typical for railway infrastructure. In the background, some greenery can be seen, indicating that this accident has occurred in an area with vegetation nearby. There are also some structures visible at a distance, possibly part of the railway station or related facilities. The sky is clear, suggesting it might have been taken during daytime under fair weather conditions. The overall scene conveys a sense of aftermath and investigation following a significant railway incident. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Every person you've ever talked with has had different vowel formant patterns * ... and yet, we understand each other, somehow --- <img class="r-stretch big" src="img/magic.jpg" alt="magic.jpg The image displays a cartoon illustration set against a plain white background. In the bottom left corner, a hand and forearm reach upward and slightly to the right. The figure appears to be wearing formal attire, indicated by a dark brown or black suit sleeve and a white cuff at the wrist. The hand is gripping the brim of a tall top hat. The top hat is tilted diagonally toward the upper left corner. It is decorated with vertical stripes in various colors, including purple, yellow, light blue-green, and dark blue. At the very top center of the hat, there is a small black circle or detail. Emerging from the opening of the tilted hat is a cascade of five-pointed stars that flow toward the right side of the image, simulating a magic trick. These stars are scattered in an arc pattern. They come in several bright colors: yellow, magenta pink, light blue, dark blue, green, and purple. Each star has a thick outline, primarily in blue or black. There is no text present in the illustration. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Humans are able to cope with this easily, somehow - ... but what about computers? --- ## Automatic Speech Recognition --- ### ASR builds mappings from audio to text - We feed the system lots of text, and lots of corresponding audio - It learns the patterns of sound associated with a given text - Some use language models to give better predictions --- ### Vintage ASR used to require explicit speaker adaptation - Around the turn of the century, ASR software required personalization and 'training' - Setup began with "Read these texts aloud" - It would then process for a little while as it 'customized' to your voice - The model *simply wouldn't work* without this level of customization --- ### Then, Artificial Neural Networks arrived, and everything changed (again) <img class="r-stretch" src="dalle/cuteneuralnetwork.jpg" alt="cuteneuralnetwork.jpg This image features a colorful, anthropomorphic character that resembles a friendly virus or cell set against a glowing, abstract background. The character is the central focus and appears to be walking or dancing towards the viewer. The character has a round, bulbous body covered in a texture that looks like a digital mesh or network of glowing white lines connecting small dots. This mesh creates a gradient of colors across its body, shifting from cool blues and teals on the left side to warm oranges and yellows on the right. Radiating outward from the main body are approximately twelve distinct spikes or antennae. Each spike ends in a shiny, spherical orb. These orbs vary in color, including metallic gold, bright orange, deep purple, and teal green. The character has a face with two very large, round eyes positioned side-by-side near the top of its body. The eyes are dark brown with thick black outlines and white highlights that suggest they are shiny or reflecting light. Below the eyes is a simple, curved black line forming a gentle smile. At the bottom of the character's body are two short, stubby legs that look like rounded feet. The left leg (viewer's left) is primarily orange and brown with a metallic sheen, while the right leg is blue and purple. They appear to be in mid-stride, suggesting movement. The background is soft and out of focus, filled with bokeh effects—circular blurs of light in shades of pink, white, yellow, and blue. Behind the character, there are faint, thin white lines connecting dots in a geometric pattern, resembling a constellation or a neural network map, which extends across the entire backdrop. There is no text or specific diagrams present in the image. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Whisper is OpenAI's Neural ASR Tool <img class="r-stretch" src="phonmedia/asr_whisper.png" alt="asrwhisper.png The image is a detailed diagram illustrating the process of sequence-to-sequence learning using transformer models in natural language processing (NLP). The diagram is divided into two main sections: the left side represents the encoder part of the model, and the right side shows the decoder part. Encoder Section: - Input: The input to the encoder section is a log-Mel spectrogram. This is represented at the bottom-left corner with a visual depiction showing frequency bands over time. - Transformer Encoder Blocks: These blocks are responsible for processing the input sequence. There are three such blocks, each containing two components: - MLP (Multi-Layer Perceptron): A neural network layer that processes information in multiple layers. - Self Attention: This mechanism allows different parts of the input to interact with each other based on their relevance and importance within the sequence. It is represented by a blue box labeled self attention. - Sinusoidal Positional Encoding: This is added to the input at the bottom-left corner, which helps the model understand the position of elements in the sequence. - Cross Attention: This mechanism allows information from the encoder blocks to be used during decoding. It connects with the decoder section through a connection labeled cross attention. Decoder Section: - Transformer Decoder Blocks: These blocks are responsible for generating output sequences based on the encoded input. There are three such blocks, each containing two components similar to those in the encoder: - MLP (Multi-Layer Perceptron). - Self Attention: This mechanism allows different parts of the generated sequence to interact with each other. - Learned Positional Encoding: This is added at the bottom-right corner and helps the model understand the position of elements in the output sequence. Output: - The final output consists of tokens in a multitask training format. These tokens are labeled as SOT, EN, TRANSCRIBE, followed by numerical values (e.g., 0.0, 1.0) and words like The quick brown. - The process involves predicting the next token based on the current input sequence. Text: - At the top of the diagram, there is a title: Sequence-to-sequence learning. - On the right side, above the decoder blocks, there are labels indicating different stages or components such as next-token prediction, cross attention, and self attention. This detailed breakdown should help you understand the structure and function of sequence-to-sequence learning using transformer models. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### ... but it is wildly effective - It works relatively quickly - On relatively low-end hardware - ... and most amazing of all... --- ### Whisper can get human-like performance in speech transcription* <img class="r-stretch" src="phonmedia/asr_whispervshumans.png" alt="asrwhispervshumans.png The image is a bar chart that compares word error rates for different methods of transcription: ASR (Automatic Speech Recognition) and human transcription with computer assistance. The x-axis represents various transcription methods labeled as Whisper, A through I. The y-axis shows the Word Error Rate in percentage. Each method has two bars representing its performance under ASR and human transcription with computer assistance. The bars are color-coded for clarity: - ASR: - Whisper: Blue - Others (A to I): Different colors (orange, green, red, purple, pink, gray, yellow, cyan) - Human Transcription with Computer Assistance: - Whisper: Light blue - Others (A to I): Different colors (light orange, light green, light red, light purple, light pink, light gray, light yellow, light cyan) Each bar has a box plot at the bottom showing the distribution of error rates. The median value is indicated by a horizontal line inside each box. The numbers above some bars represent specific values: - Whisper: 8.81 (ASR), 9.66 (human transcription with computer assistance) - A: 9.74 (ASR), 10.9 (human transcription with computer assistance) - B: 9.74 (ASR), 12.2 (human transcription with computer assistance) - C: 8.65 (ASR), 8.14 (human transcription with computer assistance) - D: 8.65 (ASR), 8.14 (human transcription with computer assistance) - E: 7.61 (ASR), 8.14 (human transcription with computer assistance) - F: 8.65 (ASR), 8.14 (human transcription with computer assistance) - G: 8.65 (ASR), 8.14 (human transcription with computer assistance) - H: 8.96 (ASR), 10.5 (human transcription with computer assistance) - I: 10.5 (ASR), 12.2 (human transcription with computer assistance) The chart is designed to compare the performance of different transcription methods in terms of word error rates, highlighting how human transcription with computer assistance generally performs better than ASR for most methods except Whisper and B. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Wow. - These ASR tools just 'listened' to a bunch of audio with texts - They built representations of speech - They combined it with some knowledge of how text usually looks - ... and suddenly, it approaches human ability in speech perception* --- ### Wait, what were those asterisks? - About that.... --- ### How many of you have had great experiences with speech-to-text? - How many think it's OK? - How many think it works terribly? --- ### These tools are great at recognizing speech *for the dialects that they were trained on* - ... but they're substantially weaker at adapting to different dialects - Many people are working on that - ... including your very own Oishani Bandopadhyay! - **So, humans still win!** --- ### Hooray! <img class="r-stretch" src="img/morpheus.jpg" alt="morpheus.jpg The image shows a medium shot of a man sitting in a large red chair. The man is Laurence Fishburne, recognizable as the character Morpheus from The Matrix. He has a shaved head and is wearing dark, opaque sunglasses that hide his eyes completely. His expression is serious and neutral, with his mouth slightly parted. He is dressed in a distinctive black coat made of a material with a textured, reptile-skin pattern, likely leather or faux leather. The coat features a very high, stiff collar that rises up around his neck. The surface of the coat reflects light, indicating a smooth but textured finish. The chair he sits in is large and upholstered in red material, possibly leather. It has a high back with tufted stitching visible near the top center. On the left side of the chair's backrest (from the viewer's perspective), there is a noticeable patch of damage where the red surface is peeling or worn away, exposing a lighter, yellowish layer underneath. The background is dark and dimly lit. Behind the man, there are indistinct shapes that suggest a room with dark wood paneling or heavy drapes. To the far left edge of the frame, a blurry, rectangular object hangs on the wall, possibly a painting or tapestry, but it is too out of focus to identify clearly. There is no text or diagrams in the image. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- Yet, there's still one incredible fact... --- ### Neural networks can be as good as humans at speech - ... without tongues, ears, grammatical knowledge, or human brains - Just like LLMs are the second thing *ever* which can do human language - *All it takes it lots of data and the right architecture* - Take LIGN 168 to understand more about speech tools and follow the progress! --- ### We're still figuring out what this means for Language - ... but one thing is for sure ... --- ### It's a fascinating time to study the Cognitive Science of Language! - ... and to have a Linguistics double major - (Sorry, couldn't resist) --- <huge>Thank you!</huge> --- ### Bonus Content: Formants are enough for speech perception in humans --- ### Let's listen to some sounds <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <br> ### Now let's play all three at once! <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <br> ### Does this help? <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p>