# Neural Speech Recognition ### Will Styler - LIGN 168 --- ### So, now we know what Neural Networks are - We understand their pros, their cons, and their various types - **How can they be applied for ASR?** --- ### Today is just a sampler platter - There are hundreds of ways to do this task - We're going to talk about a few interesting approaches - As always, you can dive deeper if you'd like --- ### Today's Plan - Spectral and CNN-based Approaches - The Alignment Problem - Sound-to-Vector Models and Wav2Vec 2 - Whisper - Neural ASR is boring --- ## Spectral and CNN-based Approaches --- ### So, we know Neural Networks win at ASR - ... but how do we give acoustical data to these models? --- ### Spectral Representations as Input - "Let's give the model a spectrogram directly" - Often something like a logged mel spectrogram - Sometimes something like a mel cepstrogram - Sometimes a png file of a spectrogram itself - Please don't --- ### Once we have a grid containing useful acoustic data... - Treat it like any other image! - Convolutional Neural Networks excel at finding patterns in images - Spectrogram reading is just finding patterns in images - Neural Network training will ensure you find the right patterns --- ### You can go from spectrogram to phones! <img class="r-stretch" src="img/nn_conv_fullpath.jpg" alt="nnconvfullpath.jpg This image is a schematic diagram illustrating the architecture and data flow of a Convolutional Neural Network (CNN), which is a type of machine learning model used for image processing. The diagram reads from left to right, starting with an input image and ending with a classification result. On the far left is the input: a color photograph of a harbor scene. In the photo, there is a white motorboat with red trim docked near a grassy bank on the left side. Behind it, further in the water, is another boat that appears to be a sailboat. The sky is blue with some clouds. A dashed black rectangle outlines a small section of the grass and water in the foreground, indicating a specific area being analyzed by the network. To the right of the photo are several stages of processing represented by stacks of square outlines: 1. Convolution + ReLU: Above the first stack of three large squares is the text Convolution followed by + ReLU in red letters. Dashed lines connect the small highlighted box from the original photo to a corresponding small square on this map, illustrating how the network scans parts of the image. 2. Pooling: Next is a section labeled Pooling. Here, there are fewer and smaller squares than before. Dashed lines show that groups of pixels from the previous layer are being combined or downsampled here. 3. Convolution + ReLU: The process repeats with another stack of squares labeled Convolution and + ReLU (in red). These squares represent further feature extraction. 4. Pooling: Another Pooling stage follows, reducing the dimensions further into a smaller set of squares. Following these stages, the diagram shows two long, thin strips made of many small squares. Above these are labels reading Fully Connected. Dashed lines connect the previous pooling layer to these strips, indicating that the 2D image data is being flattened and connected to neurons for decision-making. At the far right end of the diagram is the final output stage labeled Output Predictions. Next to a small stack of squares representing the final nodes, there is a list of categories with numerical values in parentheses: - Dog (0) - Cat (0) - Boat (1) - Bird (0) This indicates that the network has analyzed the image and determined it contains a boat (score of 1) but not dogs, cats, or birds (scores of 0). Below this list is a mathematical formula: E total equals summation of one-half times (target minus output) squared. This represents the calculation of error or loss in the model. At the very bottom of the image, two blue brackets group the stages together. The bracket under the first four layers (the convolution and pooling steps) is labeled Feature Extraction from Image. The bracket under the fully connected layers and the final output is labeled Classification. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Now you just need to match those phones to words - ... which raises.... --- ## The Alignment Problem --- ### Note that 'alignment' has many meanings in ML - "Is this system going to do what I want it to do, for the reasons I want it to do it?" - "Are these two things synchronized, ordered, and time-aligned properly?" - This is the problem we have today! --- ### Some tasks have one-to-one input/output alignment - "Is this picture a rabbit or a cat?" - "Would this Instagram user buy diet pills, based on their follow list?" - "Is this email likely an attempt to purchase chemical weapons?" --- ### Other tasks have ambiguous alignment - "Detect motion in this noisy video feed from the warehouse" - "Identify all characters in this cursive sentence" - "Take this music and generate a score for it" - **ASR!** --- ### Alignment is hard in ASR - I have an input waveform which describes a sequence of phones - e.g. /lɪŋgwɪstɪks/ - I have a set of output characters I'd like from that - e.g. "linguistics" - You'll likely need to process the sound frame-by-frame --- ### Alignment for a given frame is hard - Not all frames contain a character - Some characters take more than one frame to complete - A long /i/ vowel might be several frames long - Some characters are repeated - 'Mississippi' doesn't involve multiple /s/ or /p/ - **Any ASR approach needs to address this!** --- ### Connectionist Temporal Classification (CTC) is one way - Let's assume something allows us to assign every frame to a likely category label (e.g. phone) - Create a list of possible output labels (e.g. 'cat', 'penguin', 'access') - If output repeats characters place a blank ('-') into the label between the repeated items ('ac-ces-s') - "If a frame label repeats across multiple frames, it's probably the same chunk" - "If an output label has a blank between, it's probably just one chunk" --- ### CTC Collapsing <img class="r-stretch" src="phonmedia/asr_ctc.png" alt="asrctc.png The image is a visual representation explaining how CTC (Connectionist Temporal Classification) collapsing works in the context of speech recognition systems. It includes an audio waveform at the top and text below it that illustrates the process step-by-step. Audio Waveform: - The top part of the image shows an audio waveform, which represents sound over time. - The waveform is divided into segments, each corresponding to a specific phoneme or sound unit in speech. Textual Explanation: The text below the waveform explains the CTC collapsing process: 1. Predict a sequence of tokens: - The first step involves predicting a sequence of tokens (which could be letters, words, or other units) from the audio input. - In this example, the predicted sequence is w o r r e l l d !. 2. Merge repeats and drop ε: - The next step merges repeated characters in the token sequence to form a more coherent output. - Here, 'o' and 'r' are merged into one character each, resulting in w o r e l l d !. - The symbol 'ε' (epsilon) is dropped as it represents an empty state or placeholder that doesn't contribute meaningfully. 3. Final output: - After merging repeats and dropping unnecessary characters, the final output is world!. - This is the intended word that was spoken in the audio input. Source: - The image includes a source link at the bottom: https://alanfangblog.com/2021/08/29/ASR-CTC/ Characters: There are no characters or people depicted in this image. It is purely a diagram explaining the CTC collapsing process used in speech recognition systems. This explanation should help you understand the content of the image without relying on visual elements. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### CTC just finds the most likely path through the options - Sort of like decoding an HMM using the Viterbi algorithm - "What's the output label(s) for this sequence that best fit the data?" - Now, you can match a series of frame labels, to the most likely output labels - **The probability of the correct choice being made is also a good loss function!** --- ### CTC Decoding <img class="r-stretch" src="phonmedia/asr_ctctotal.jpg" alt="asrctctotal.jpg The image is a detailed diagram explaining how audio recognition works using a combination of a CTC (Connectionist Temporal Classification) network and a language model. The process starts with raw audio data on the far left side of the image, which appears as a series of horizontal lines representing sound waves. Next to the raw audio, there's a section labeled CTC NETWORK, showing a grid-like structure with green circles arranged in rows and columns. This network processes the raw audio input into a sequence that can be interpreted by the language model. Following this is the CTC OUTPUT section, which displays a matrix filled with red squares and green squares. The red squares represent the highest probability of certain outputs at each time step during the CTC process, while the green squares indicate lower probabilities or less likely outcomes. The diagram then moves to the right to show the DECODING phase, where the output from the CTC network is translated into a sequence of words. This section shows a series of circles connected by lines, representing the decoding process that converts the numerical outputs into text. Finally, on the far right side, there's a LANGUAGE MODEL section with a circular chart labeled HELLO WORLD. The chart lists various words along with their probabilities: WORLD has a probability of 0.87, followed by HALT at 0.06, and other words like WORD, HAVE, WILL, and WALL have much lower probabilities. At the bottom right corner of the image, there's text that reads: TRANSCRIPT: HELLO WORLD. This indicates what was recognized from the audio input as a result of the process described in the diagram. The source of this information is cited at the very bottom left with the URL: https://medium.com/@riki/ctc-networks-and-language-models-prefill-beam-search-explained-c11dfec23306. The image provides a clear step-by-step breakdown of how an audio recognition system works, from raw audio to text transcription. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### We can unite the output with a language model <img class="r-stretch" src="diagrams/asr_architecture.jpg" alt="asrarchitecture.jpg This image displays a flowchart diagram illustrating a system architecture, likely for speech recognition or natural language processing. The chart is organized into two main vertical columns that eventually merge in the center. The boxes are color-coded: light green for inputs, light purple/lavender for processing steps, and light grey/white for predictions and final outputs. On the left side, a vertical sequence begins at the top with a light green box labeled Spoken Query. A black arrow points downward to a light purple box labeled Signal Processing (ADC, VAD, Denoise). From there, an arrow leads down to another light purple box reading Acoustical Feature Extraction. The flow continues downward via an arrow to a third light purple box labeled Acoustical Model. Finally, an arrow points from this model to a light grey box at the bottom of this column labeled Acoustical Predictions. On the right side, there is a parallel vertical sequence. It begins at the top with a light green box labeled Text Corpus. An arrow points down to a light purple box containing the text LIGN 165/167. Below that, an arrow leads to another light purple box labeled Language Model. From this model, an arrow points down to a light grey box labeled Language Model Predictions. In the center of the diagram, the two columns converge. An arrow extends from the right side of the Acoustical Predictions box and meets with an arrow extending from the left side of the Language Model Predictions box. Both arrows point into a central light purple box labeled Hypothesis Search. From this central Hypothesis Search box, a solid black arrow points downward to the final element at the bottom center: a light grey box labeled Output Text. Additionally, there is a dashed curved line indicating a feedback loop or connection. This line originates near the right side of the Hypothesis Search box and curves upward to point into the left side of the Language Model box on the right column. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### ... and we figure out the most likely candidates - Either choose the most likely candidate over and over given both acoustic and language model data - "Greedy" decoding - Or you do [Beam Search](https://en.wikipedia.org/wiki/Beam_search) to identify multiple candidates - You limit the number of candidates to consider at once - Then, choose the sequence with the highest overall probability --- ### Beam Search <img class="r-stretch" src="phonmedia/asr_beam_search.gif" alt="asrbeamsearch.gif The user wants a detailed description of an image showing a tree diagram. 1. Analyze the Image: Type: It's a black and white line drawing of a hierarchical structure, specifically a tree graph or data structure. Structure: It has levels (root at top, children below). Nodes: The nodes are circles containing numbers. Some are empty (the root), some have single digits, some have double digits. Edges: Lines connect the parent nodes to their child nodes. 2. Breakdown by Level/Node: Root Node (Level 0): Top center. It's an empty circle. No number inside. Children of Root (Level 1): There are four children connected directly to the root. From left to right: Node with 1 Node with 5 Node with 7 Node with 4 Children of Node 1: Two children below it. Left child: Node with 12 (double digit). Right child: Node with 8. Children of Node 5: Three children below it. Left child: Node with 9. Middle child: Node with 8. Right child: Node with 7 (Wait, let me look closer at the connections). Actually, looking at the lines from node 5: One goes to 9. One goes to 8. Wait, is that a line from 5 to 7? No, let's trace carefully. Node 5 connects to Node 9 (left-ish), Node 8 (middle-ish). And there is a line going down-right towards... wait, let me re-evaluate the connections for node 5 and 7. Let's look at the cluster under 5 and 7. Under 5: There is a line to 9. There is a line to 8. Wait, looking at the layout... Let's trace from the top down again. Root -> 1, 5, 7, 4. Correct. From 1: Line to 12 (left), Line to 8 (right). Correct. From 5: Line to 9 (left). Line to 8 (middle). Wait, is there a third line? Let's look at the node 7 below the root. Ah, let's look at the connections between the second and third levels carefully. Node 5 connects to Node 9. Node 5 also seems to connect to Node 8 (the one directly below it). Wait, let's look at the node labeled 7 under the root. It has children. Let's re-examine the whole structure. It looks like a generic tree diagram often used in computer science examples. Let's try tracing from bottom up or grouping by parent. Parent 1: Children are 12 and 8. Parent 5: Children appear to be 9, 8 (the one below it), and... wait. Let's look at the lines crossing. There is a line from 5 going down-left to 9. There is a line from 5 going straight down to 8. Is there a connection to the node labeled 7 below? No, that looks like it belongs to parent 7. Parent 7: Children appear to be... wait, let's look at the node labeled 7 under the root. It has a line going down-left to... actually, looking closely at the junction near 9, 8, 7. Let's trace the lines from the second level nodes (1, 5, 7, 4). Node 1: Connects to 12 and 8. Node 5: Connects to 9. And connects to 8 (the one below it). Wait, looking at the line from 5... it goes down to 9. Another line goes down to 8. Is there a third? Let's look at the node labeled 7 under the root. It connects to... wait, is that a 7 or part of the previous group? Let's look at the labels again. Top: Empty circle. Row 2: 1, 5, 7, 4. Row 3 (under 1): 12, 8. Row 3 (under 5/7 area): There is a node 9, a node 8, and a node 7. The line from 5 goes to 9. The line from 5 also seems to go to the node labeled 8 below it? No, let's look at the vertical alignment. Under 5 is an empty space then a node 8? No, under 5 is a node 9 and a node 8. Let's trace the line from 7 (the parent). It goes to a node labeled 7 below it? And a node labeled 5? Let's try a different approach: Identify every node and its connections. Root: Top center circle. No number. Level 1 Nodes (Left to Right): Node A: Number 1. Node B: Number 5. Node C: Number 7. Node D: Number 4. Children of Node A (1): Left child: Node with number 12. Right child: Node with number 8. Children of Node B (5): It has a line going down-left to Node with number 9. It has a line going straight down to Node with number 8 (the one below the 9). Wait, let's look at the node labeled 8 under the root. No, that's under the space between 5 and 7. Let's look really closely at the connections around the middle. There is a node 9. Above it is a line coming from 5. To the right of 9 is a node 8. Above it is a vertical line coming from... where? It looks like it comes from 5. To the right of that 8 is a node 7. Above it is a line coming from 7 (the parent). Wait, let's look at the node labeled 7 in Level 1. It has a line going down-left to... actually, looking at the diagram, there is a node labeled 7 below the root, slightly right of center. Let's call this Parent Node C. Parent Node C (7) connects to: A node labeled 7 (below it). A node labeled 5 (to its right). Let's restart the tracing, very carefully. Root: Top circle. Branches from Root: 1. Left branch to Node 1. 2. Middle-left branch to Node 5. 3. Middle-right branch to Node 7. 4. Right branch to Node 4. Sub-branches from Node 1: Line down-left to Node 12. Line straight down to Node 8. Sub-branches from Node 5: Line down-left to Node 9. Line straight down to Node 8 (the one below the 9). Wait, looking at the node labeled 8 under the root... no, that's not right. Let's look at the cluster: 9, 8, 7. The node 9 is connected to 5. The node 8 (below the 9) is connected to... actually, looking at the line, it seems to come from 5. Wait, let's look at the node labeled 7 under the root. It connects to a node labeled 7 below it and a node labeled 5 to its right. Let's try tracing from the bottom up. This is often easier for trees. Bottom Row (Leaves): 4, 6, 6, 9, 2, 8, 7. Wait, let's count the circles in the lowest visible row. Far left: 4. Connected to... 9? Yes, line goes up-right to 9. Next: 6. Connected to... 9? Yes, line goes up-left to 9. Next: 6. Connected to... 8 (the one above it)? Yes. Next: 9. Connected to... 8 (the one above it)? Yes. Wait, let's look at the node labeled 8 in the middle row. There is a node labeled 8 below the root area. Let's call this Middle Node E. It connects down to 9. Next: 2. Connected to... 7 (the one above it)? Yes. Next: 8. Connected to... 7 (the one above it)? Yes. Next: 7. Connected to... 7 (the one above it)? Yes. Let's refine the structure based on this bottom-up analysis. Level 3 (Bottom-most visible nodes): Node 4 (far left). Parent is 9. Node 6 (left-center). Parent is 9. Node 6 (center-left). Parent is... let's trace up. It goes to the node labeled 8 which is below the root area? No, that doesn't fit. Let's look at the node labeled 9 in Level 2. It has children: 4, 6. And another line going down to... 6? Yes, a vertical line goes from 9 down to a 6. Let's look at the node labeled 8 below the root area (let's call it Node X). It is connected to parent 5? Or parent 7? Let's trace the lines from Level 1 again. Parent 5: Connects to Node 9. Connects to Node 8 (Node X). Parent 7: Connects to Node 7 (Node Y). Connects to Node 5 (Node Z). Let's try one more pass, strictly left-to-right. Root: Top circle. Level 1 Nodes: Left: Circle with 1. Center-Left: Circle with 5. Center-Right: Circle with 7. Right: Circle with 4. Children of Node 1: Left child: Circle with 12. Right child: Circle with 8. Children of Node 5: It has a line going down-left to a circle with 9. It has a line going straight down to a circle with 8 (let's call this Middle-Child-1). Wait, looking at the diagram, there is a node labeled 8 directly below the space between 5 and 7. Let's assume it belongs to 5. Actually, let's look at the node labeled 9. It has children: 4, 6, and another 6. Wait, looking at the lines from 9: One goes left-down to 4. One goes right-down to 6. One goes straight down to 6. So Node 9 is a parent of three nodes: 4, 6, 6. Children of Node 7 (the one in Level 1): It has a line going down-left to a circle with 7 (let's call this Middle-Child-2). Wait, looking at the diagram... There is a node labeled 8 below the root. Let's trace its parent. The line goes up-right towards 5. So Node 8 is a child of 5. There is a node labeled 7 below the root (to the right of the previous 8). Let's trace its parent. The line goes up-left towards 7 (Level 1). So this 7 is a child of Level-1 Node 7. Wait, let's look at the node labeled 5 to the right of that 7. It connects to... nothing above? No, it must connect to something. The line goes up-left towards 7 (Level 1). So this 5 is a child of Level-1 Node 7. Children of Node 4 (Level 1): Left child: Circle with 6. Right child: Circle with 7. Let's re-evaluate the middle section. It's confusing. Let's look at the node labeled 9 again. Parent: Node 5. (Line goes up-right to 5). Children: Left: Node 4. Middle: Node 6. Right: Node 6. Wait, looking at the bottom row... 4, 6, 6. Yes. Let's look at the node labeled 8 (the one below the root area). Parent: Node 5? Or Node 7? The line goes up-right to Node 5. So it is a child of 5. Children: One vertical line down to Node 9. Wait, no. Let's look at the node labeled 8 in the middle row (below 5/7). It has a line going straight down to Node 9? No, that doesn't make sense if 9 is already a child of 5. Let's look at the lines again. From Node 5: Line to 9. Line to 8 (the one below it). From Node 7 (Level 1): Line to 7 (below it). Line to 5 (to its right). Let's try to map the connections strictly. Root connects to: 1, 5, 7, 4. Node 1 connects to: 12, 8. Node 5 connects to: 9, 8 (the one below it). Wait, looking at the node labeled 8 below the root... is that a child of 5? Yes. And does it have children? The line from that 8 goes down to... 9? No, that's a different 9. Let's look at the bottom row again: 4, 6, 6, 9, 2, 8, 7. Wait, I missed some nodes. Let's count all circles. Top: 1 (empty). Row 2: 4 circles (1, 5, 7, 4). Total = 5. Row 3: Under 1 -> 12, 8. Under 5/7 area -> 9, 8, 7, 5? No. Let's look at the cluster under 5 and 7. There is a node 9. Parent seems to be 5. There is a node 8 (below 9). Parent seems to be... actually, looking at the line, it goes up to 5? Or does it go up to 7? It looks like it's under 5. There is a node 7 (to the right of the previous 8). Parent seems to be 7 (Level 1). There is a node 5 (to the right of that 7). Parent seems to be 7 (Level 1). Let's look at the bottom-most row again. Under Node 9: Nodes 4, 6. And another line going down to... 6? Yes. So Node 9 has children 4, 6, 6. Under the This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> (Source: BogdanShevchenko - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=128835919) --- ### Now we have a working ASR model! - We get predictions from the acoustics by feeding - We match those to word representations using CTC - We can unite this with a language model, evaluating probabilities using beam search - This can turn a sequence of phones into a transcript! --- ### That's amazing! - ... but can't we make this simpler? --- ## Sound-to-Vector Models and Wav2Vec2 --- ### It would be nice not to need to generate spectral represenations - What if we could turn the waveform into a vector directly, and just go end-to-end? - This leads to approaches like... --- ### Wav2Vec - Developed by Facebook's AI team - Takes in a waveform directly, and outputs higher quality features for use in ASR - **Feature Encoding:** Uses CNNs to create 'latent representations' every 10ms or so - These aren't exactly segments, they're just 'things the network notices' - **Context Network:** Combines the features to capture adjacency effects --- ### Wav2Vec is trained in a *self supervised* way - We're going to take away ('mask') some of the representations coming out of the model - Now, we make the algorithm predict what's missing - The loss function involves correctly choosing the missing representations, among a random set sampled from elsewhere - **This allows it to learn without as much labeled data** --- ### Wav2Vec 2.0 - Same idea, but the encoded features get quantized to a set of specific 'tokens' - Tokens are kind of like phones, but not 1-to-1 - We get a sequence out of the waveform - The context network can now be a transformer - This has all the benefits, plus easy fine-tuning for new languages or tasks - **You can map these tokens to whatever representation you'd like** - Phones/Diphones/Triphones - IPA characters - Even direct to orthography! --- ### Wav2Vec2 <img class="r-stretch" src="phonmedia/asr_wav2vec2.png" alt="asrwav2vec2.png The image is a detailed diagram illustrating a neural network architecture used for speech recognition tasks, specifically focusing on the WaveNet model version 2.0 as described in a research paper referenced at the bottom of the image. Key Components: 1. Raw Waveform (x): - At the very bottom of the diagram, there is a visual representation of an audio waveform labeled raw waveform x. This represents the original sound signal that has not been processed yet. 2. Latent Speech Representations (z): - Above the raw waveform, there are several blue blocks labeled with the letter 'z'. These represent latent speech representations, which are intermediate features extracted from the raw audio data by a series of convolutional neural networks (CNNs). 3. Quantized Representations (q): - Further up in the diagram, there is another set of blue blocks labeled quantized representations q. These quantized representations are derived from the latent speech representations and serve as further processed features. 4. Context Representations (c): - At the topmost part of the diagram, there are orange blocks labeled with 'c'. These represent context representations that capture information about the surrounding audio context to improve the model's understanding of the spoken content. 5. Transformer: - The central yellow block is labeled Transformer. This component processes the latent speech and quantized representations together using a transformer architecture, which allows for parallel processing and attention mechanisms across different parts of the input sequence. 6. Masked Transformer: - Below the main transformer block, there's another similar block labeled Masked Transformer. This masked version is used to compute contrastive loss, which helps in training the model by comparing representations from different segments of the audio. 7. Contrastive Loss (L): - The arrows pointing towards the masked transformer indicate that it computes a loss function 'L', which is likely used for training purposes. This loss function compares and contrasts features extracted at different points within the input sequence to improve the model's ability to distinguish between similar sounds. 8. Source Information: - At the bottom of the diagram, there’s a reference to the source: WaveNet 2.0 Paper with a link provided for further reading (https://arxiv.org/abs/2006.11477). People in the Image: There are no people depicted in this image; it is purely a technical diagram of a neural network architecture. This detailed description should help you understand the structure and function of the model without relying on visual elements that might be inaccessible to someone who cannot see the image. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Wav2Vec2 can go from sound to orthography! - This is absolutely wild, given how bad our writing system is! - It also doesn't need CTC, as that's a part of the transformer's core competencies - These quantized units raise the possibility of a language-independent ASR system! - The 'Universal Transcriber' --- ### Wav2Vec2 replaces the entire acoustical pipeline <img class="r-stretch" src="diagrams/asr_architecture.jpg" alt="asrarchitecture.jpg This image displays a flowchart diagram illustrating a system architecture, likely for speech recognition or natural language processing. The chart is organized into two main vertical columns that eventually merge in the center. The boxes are color-coded: light green for inputs, light purple/lavender for processing steps, and light grey/white for predictions and final outputs. On the left side, a vertical sequence begins at the top with a light green box labeled Spoken Query. A black arrow points downward to a light purple box labeled Signal Processing (ADC, VAD, Denoise). From there, an arrow leads down to another light purple box reading Acoustical Feature Extraction. The flow continues downward via an arrow to a third light purple box labeled Acoustical Model. Finally, an arrow points from this model to a light grey box at the bottom of this column labeled Acoustical Predictions. On the right side, there is a parallel vertical sequence. It begins at the top with a light green box labeled Text Corpus. An arrow points down to a light purple box containing the text LIGN 165/167. Below that, an arrow leads to another light purple box labeled Language Model. From this model, an arrow points down to a light grey box labeled Language Model Predictions. In the center of the diagram, the two columns converge. An arrow extends from the right side of the Acoustical Predictions box and meets with an arrow extending from the left side of the Language Model Predictions box. Both arrows point into a central light purple box labeled Hypothesis Search. From this central Hypothesis Search box, a solid black arrow points downward to the final element at the bottom center: a light grey box labeled Output Text. Additionally, there is a dashed curved line indicating a feedback loop or connection. This line originates near the right side of the Hypothesis Search box and curves upward to point into the left side of the Language Model box on the right column. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### To 'Decode' Wav2Vec2 data into words You combine the token-representation with a separate language model to find hypotheses - Again, you can use probabilities and/or beam search to find the best approach - Some people are trying to make this *fully* end to end, building the language model into the Wav2Vec Model - We'll see an example of this shortly! --- ### This is a working ASR system! - Wav2Vec2 goes from acoustics to quantized tokens - Those tokens can be mapped directly to linguistic units - Linguistic units plus a language model give probabilities for outputs - Output probabilities can be beam-searched, to arrive at the best transcription - **We've gone from Waves to Words in just two steps!** - ... and they're both very boring steps --- ### You can also build on this system! - You can take an existing model, and fine tune with a bit of data for another language - So, we use a model trained on all speech, and then fine tune on Tira - This is great for low-resource languages - [Char Siu](https://github.com/lingjzhu/charsiu) turns Wav2Vec2 into a forced aligner - "Find me the exact temporal boundaries of these segments" - **Open Models make the world better!** --- ### This is a very common setup now - It's not often revealed how commercial ASR systems work - Secret Sauce abounds! - We should assume many systems are using a similar architecture under the hood! - ... but at least one system is different! --- ## Whisper --- ### Whisper is an ASR Model from OpenAI - It does transcription - It also does time-aligned transcription (e.g. for video captioning) - It does some multilingual ASR and translation too! - This is a neat trick! - It also can be used for language identification! - It is shockingly good! --- ### Whisper performs nearly as well as human transcribers <img class="r-stretch" src="phonmedia/asr_whispervshumans.png" alt="asrwhispervshumans.png The image is a bar chart that compares word error rates for different methods of transcription: ASR (Automatic Speech Recognition) and human transcription with computer assistance. The x-axis represents various transcription methods labeled as Whisper, A through I. The y-axis shows the Word Error Rate in percentage. Each method has two bars representing its performance under ASR and human transcription with computer assistance. The bars are color-coded for clarity: - ASR: - Whisper: Blue - Others (A to I): Different colors (orange, green, red, purple, pink, gray, yellow, cyan) - Human Transcription with Computer Assistance: - Whisper: Light blue - Others (A to I): Different colors (light orange, light green, light red, light purple, light pink, light gray, light yellow, light cyan) Each bar has a box plot at the bottom showing the distribution of error rates. The median value is indicated by a horizontal line inside each box. The numbers above some bars represent specific values: - Whisper: 8.81 (ASR), 9.66 (human transcription with computer assistance) - A: 9.74 (ASR), 10.9 (human transcription with computer assistance) - B: 9.74 (ASR), 12.2 (human transcription with computer assistance) - C: 8.65 (ASR), 8.14 (human transcription with computer assistance) - D: 8.65 (ASR), 8.14 (human transcription with computer assistance) - E: 7.61 (ASR), 8.14 (human transcription with computer assistance) - F: 8.65 (ASR), 8.14 (human transcription with computer assistance) - G: 8.65 (ASR), 8.14 (human transcription with computer assistance) - H: 8.96 (ASR), 10.5 (human transcription with computer assistance) - I: 10.5 (ASR), 12.2 (human transcription with computer assistance) The chart is designed to compare the performance of different transcription methods in terms of word error rates, highlighting how human transcription with computer assistance generally performs better than ASR for most methods except Whisper and B. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### You can actually use it!! - It is free and open - Unlike anything OpenAI does these days - You can use and download the models for free - <https://github.com/openai/whisper> - It's been tuned to run (slowly) even without a GPU! - This allows fully local transcription!! --- ### Whisper uses a hybrid of CNNs and Transformers <img class="r-stretch" src="phonmedia/asr_whisper.png" alt="asrwhisper.png The image is a detailed diagram illustrating the process of sequence-to-sequence learning using transformer models in natural language processing (NLP). The diagram is divided into two main sections: the left side represents the encoder part of the model, and the right side shows the decoder part. Encoder Section: - Input: The input to the encoder section is a log-Mel spectrogram. This is represented at the bottom-left corner with a visual depiction showing frequency bands over time. - Transformer Encoder Blocks: These blocks are responsible for processing the input sequence. There are three such blocks, each containing two components: - MLP (Multi-Layer Perceptron): A neural network layer that processes information in multiple layers. - Self Attention: This mechanism allows different parts of the input to interact with each other based on their relevance and importance within the sequence. It is represented by a blue box labeled self attention. - Sinusoidal Positional Encoding: This is added to the input at the bottom-left corner, which helps the model understand the position of elements in the sequence. - Cross Attention: This mechanism allows information from the encoder blocks to be used during decoding. It connects with the decoder section through a connection labeled cross attention. Decoder Section: - Transformer Decoder Blocks: These blocks are responsible for generating output sequences based on the encoded input. There are three such blocks, each containing two components similar to those in the encoder: - MLP (Multi-Layer Perceptron). - Self Attention: This mechanism allows different parts of the generated sequence to interact with each other. - Learned Positional Encoding: This is added at the bottom-right corner and helps the model understand the position of elements in the output sequence. Output: - The final output consists of tokens in a multitask training format. These tokens are labeled as SOT, EN, TRANSCRIBE, followed by numerical values (e.g., 0.0, 1.0) and words like The quick brown. - The process involves predicting the next token based on the current input sequence. Text: - At the top of the diagram, there is a title: Sequence-to-sequence learning. - On the right side, above the decoder blocks, there are labels indicating different stages or components such as next-token prediction, cross attention, and self attention. This detailed breakdown should help you understand the structure and function of sequence-to-sequence learning using transformer models. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Whisper embeds the language model into the acoustic model! - There is no separate hypothesis step and no separate language model! --- ### Classical ASR Architecture <img class="r-stretch" src="diagrams/asr_architecture.jpg" alt="asrarchitecture.jpg This image displays a flowchart diagram illustrating a system architecture, likely for speech recognition or natural language processing. The chart is organized into two main vertical columns that eventually merge in the center. The boxes are color-coded: light green for inputs, light purple/lavender for processing steps, and light grey/white for predictions and final outputs. On the left side, a vertical sequence begins at the top with a light green box labeled Spoken Query. A black arrow points downward to a light purple box labeled Signal Processing (ADC, VAD, Denoise). From there, an arrow leads down to another light purple box reading Acoustical Feature Extraction. The flow continues downward via an arrow to a third light purple box labeled Acoustical Model. Finally, an arrow points from this model to a light grey box at the bottom of this column labeled Acoustical Predictions. On the right side, there is a parallel vertical sequence. It begins at the top with a light green box labeled Text Corpus. An arrow points down to a light purple box containing the text LIGN 165/167. Below that, an arrow leads to another light purple box labeled Language Model. From this model, an arrow points down to a light grey box labeled Language Model Predictions. In the center of the diagram, the two columns converge. An arrow extends from the right side of the Acoustical Predictions box and meets with an arrow extending from the left side of the Language Model Predictions box. Both arrows point into a central light purple box labeled Hypothesis Search. From this central Hypothesis Search box, a solid black arrow points downward to the final element at the bottom center: a light grey box labeled Output Text. Additionally, there is a dashed curved line indicating a feedback loop or connection. This line originates near the right side of the Hypothesis Search box and curves upward to point into the left side of the Language Model box on the right column. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Whisper's End-to-End Architecture <img class="r-stretch" src="diagrams/asr_architecture_endtoend.jpg" alt="asrarchitectureendtoend.jpg The image displays a vertical flowchart set against a plain white background. The chart illustrates a five-step process, moving from top to bottom, connected by black arrows pointing downwards. Each step is contained within a rounded rectangular box with a colored background and black text. At the very top of the chart is a light green box containing the text Spoken Query. An arrow points down from this first box to a larger, light purple box labeled Signal Processing (ADC, VAD, Denoise). The text inside is bolded. Following another downward arrow is a second light purple box of similar size, which reads Acoustical Feature Extraction in bold black text. The flow continues down via an arrow to a third light purple box containing the text Joint Text/Acoustics Model, also in bold. Finally, at the bottom of the chart, an arrow points to a light gray box. This final box contains the text Output Text written in a monospaced, typewriter-style font. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Whisper offers a number of models <img class="r-stretch" src="phonmedia/asr_whisper_layers.png" alt="asrwhisperlayers.png The image is a table that provides architectural details of different models within the Whisper model family. The table is titled Table 1: Architecture details of the Whisper model family. It lists five models: Tiny, Base, Small, Medium, and Large. Each row in the table contains information about one model: - Model: Lists the name of each model. - Layers: Indicates the number of layers in the model. The numbers range from 4 for the Tiny model to 32 for the Large model. - Width: Represents the width or size of the hidden layer, measured in units (likely neurons). The values increase as you move down the table: starting at 384 for the Tiny model and ending at 1280 for the Large model. - Heads: Refers to the number of attention heads used by each model. This number also increases from 6 for the Tiny model to 20 for the Large model. - Parameters: Indicates the total number of parameters in the model, measured in millions (M). The values range from 39 million for the Tiny model to 1550 million for the Large model. The table provides a clear comparison between these models based on their architectural components and computational complexity. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Whisper works (mostly) great! - It's probably the right choice for transcribing files at the moment! - ... it's also the last neural ASR architecture we're going to look at this quarter! --- ## Neural ASR is boring lately --- ### This is a rant - Many people have lovely careers, and the tools are exhilaratingly powerful - ... but... --- ### Neural ASR has largely abandoned linguistics - "Feed in waves and text to a transformer, cook for two weeks, then have a model" - There is no linguistic nuance - There's not even transcription anymore! --- ### More parameters == More Better - Thus, more memory and energy cost generally wins - This privileges large companies with large resources - This focuses development on wealthy languages and groups --- ### ASR is approaching 'solved' for high-resource people - For the mainest-stream speakers of American English, ASR is amazing - ASR performance for me is at a place I didn't expect to see in my lifetime - There is room for improvement, to be sure, but it's not a lot of room - Most of the improvement is in brittleness with other dialects and variation --- ### The most interesting work in ASR right now is for low resource languages - How do we get *great* ASR working for a language with low amounts of data/money/hardware? - Imagine your language not having voice-to-type? - How can we use ASR to more quickly generate, study, and clean language data? - How can we use ASR to enable field work and linguistic inquiry more effectively - **How can ASR help people who the tech industry doesn't care enough to help?** --- ### For information on this, talk to Mark Simmons! - He's in the trenches with these ideas right now! - ... and we'll hear from him next time! --- ### Wrapping up - Feeding spectral information into CNNs is a great way to extract features - You need to solve the alignment problem, either with CTC or Transformers - Wav2Vec2 offers a path straight from audio to intermediate representations - ... which can be fed into a language model - Whisper is free, great, and completely end-to-end! --- <huge>Thank you!</huge>