# Linguistic Problems with Statistic Solutions Will Styler <http://savethevowels.org/talks/colloq_ucriverside_2021.html> --- ### Today's Plan - What is Linguistics, and why? - Statistics in linguistics - Phonetics and Coarticulation - A Coarticulatory Measurement Success Story - A Coarticulatory Measurement Failure - Why am I here? --- # What is Linguistics, and why? --- ### Linguistics is the study of Language - What is this thing I'm doing right now with my flapping bits of meat around in my head and you then understanding my thoughts? - How can we describe what languages are doing? - How can we understand the differences and similarities among them? - What does language tell us about cognition and culture? --- ### Linguists study languages to understand Language - Many linguists speak lots of languages, but some don't! - We're interested in the whole enterprise, and study it scientifically --- ### We break Linguistics into subfields - "How does talking and understanding speech work?" - Phonetics - "How do units of sound or gesture change when we combine them?" - Phonology - "How do we build words?" - Morphology - "How do we combine words into sentences?" - Syntax - "How do we understand meaning in language, both generally and in context?" - Semantics and Pragmatics - "How does this less-well-known language work?" - Lg. Documentation - ... and many more! --- ### Linguistics is an increasingly experimental discipline - Some folks still work in armchairs - ... or in the homes and worlds of language experts - Theory is now often supported by quantitative or experimental data - Especially where the patterns are small, variable, or difficult to ferret out --- ### Almost every type of linguistic research has data to analyze - Text data (e.g. large corpora) - Survey data (e.g. responses, free text) - Experimental data (e.g. eye tracking, reaction time, accuracy) - Neural data (e.g. EEG, fMRI, PET, MEG) - Imaging data (e.g. video, ultrasound) - Spatial data (e.g. GIS info, 3D spatial movement tracking) --- # Statistics in Linguistics --- ## The State of Linguistic Statistics --- ### Most linguists take some basic statistics classes - "Statistics for Psychology Graduate Students" - This is often the minimum requirement - Increasingly more sophisticated classes are available - "Probabilistic Methods in Linguistics" (an intro to Bayesian stats in our department) - "Analyzing time series data using Generalized Additive Models" at the Linguistic Institute --- ### There are dedicated resources for statistics for Linguists > [Baayen, R. H. (2008). Analyzing Linguistic Data: A practical introduction to statistics using R. Cambridge University Press.](https://www.cambridge.org/us/academic/subjects/languages-linguistics/grammar-and-syntax/analyzing-linguistic-data-practical-introduction-statistics-using-r) > [Winter, Bodo (2020). Statistics for Linguists: An Introduction Using R. Routledge.](https://www.routledge.com/Statistics-for-Linguists-An-Introduction-Using-R/Winter/p/book/9781138056091) - ... alongside an increasing corpus of tutorials from statistically focused linguists --- ### There are complex analyses occurring in our field - Some specializations (e.g. neurolinguistics) require advanced models to function - ... but often have overfit to those models - Some linguists are statistical thought-leaders and have strong expertise - Bodo Winter, Harald Baayen, Jacolien van Rij, Martjin Wieling, and more - Some statistically saavy moonlight in linguistics (to varying degrees of success) - "I'm a physicist with a neat model so I now understand how language works..." --- ### But the average linguist is still using relatively unsophisticated models - T-Tests and Chi-Square are being phased out in publication - ANOVA and basic linear models are probably still the mode - There's lots of recent movement towards Linear Mixed Effects Regression - Speaker, word, and ordering differences are nicely handled by random effects - Reviewers are starting to demand mixed models where relevant - ... but mixed models are right at the edge of many linguists' understanding - This has led to a saying... --- ## "Giving Linear Mixed Models to Linguists is like giving shotguns to toddlers" --- ### There's a danger in escalating complexity without escalating education - We need more nuanced methods for more nuanced questions - ... but it's hard to find the right path ahead - We need new statistical methods for new experimental methods - ... but we seldom have the time or know-how to develop them --- ### We're going to look at these ideas in practice - ... by examining two case studies from phonetics --- # Coarticulation in Phonetics --- ### I'm a phonetician - My focus is on understanding exactly what's happening in the mouth when we talk - "What are you doing inside your body to produce this word?" - What do the 'speech gestures' of the tongue and mouth look like? - "How are listeners able parse or reconstruct that to understand that you've produced this word" - ... and we're going to focus on some phonetic questions today --- ### Speech isn't cleanly separable - We write letters one after the other, but letters are lies - We're more concerned with tongue gestures, which aren't separable - Speech sounds are **not** beads on a string - We often begin moving our articulators towards the next gesture before we've finished the current one - ... and the last sound can often have an influence on the current one - This overlap is called **coarticulation** - A nice example: 'car key' --- ### Coarticulation is easier when speaking - "Car key" is changing the articulation of one sound to better 'match' the next - Air starts flowing out the nose in words like 'bend' before we actually make the /n/ sound where it's supposed to - We might raise the tongue tip 'L' in 'bulk' earlier, both for efficiency and because... --- ### Coarticulation is helpful for perception too - It provides redundancy in signaling speech contrasts - It provides information about upcoming sounds *before they arrive* - It can help to reconstruct 'missing' sounds --- ### Coarticulation is important - We want to know how people produce it - ... to be able to model speech - We want to know how people perceive it - ... to be able to model listening - So, let's look at two specific instances of it --- # A Coarticulatory Measurement Success Story --- ### Nasal Coarticulation - /n/ is a 'nasal' sound, with airflow from the nose - This is accomplished by lowering the 'velum' <img class="r-stretch" src="phonmedia/sag_alveolar.jpg" alt="sagalveolar.jpg The image is a simple line drawing of a side profile of a human head and neck. The lines are smooth and continuous, forming the outline of the face and hair. The hair appears to be short and slightly wavy, with some strands falling over the forehead. There are no distinct facial features such as eyes, nose, or mouth depicted in this minimalist style. The drawing is monochromatic, using only black lines on a white background. There are no colors, textures, or additional elements present in the image. The simplicity of the line work suggests it might be an abstract representation rather than a detailed portrait. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."><img class="r-stretch" src="phonmedia/sag_nasal.jpg" alt="sagnasal.jpg The image is a simple line drawing of a side profile of a human head and neck. The lines are smooth and continuous, forming the outline of the face and hair. The hair appears short and is drawn with a few curved lines at the top, suggesting it might be styled in a way that frames the face. The facial features include an open mouth, which could indicate speaking or singing, and a nose with a small tip. The eyes are not detailed but are represented by two curved lines above the nose, giving the impression of closed eyelids. The ears are depicted as simple shapes on either side of the head, close to where the hair ends. The neck is drawn with a few wavy lines that suggest movement or perhaps a slight bend in the neck. There are no other elements such as text, numbers, or additional figures present in this image. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <huge>bend</huge><br> * **...but there's more to it than the letters show us!** * In the word "bend", we start nasal airflow before the nasal /n/, *during the vowel* --- ### This is audible and useful to us - Is this 'bob' or 'bomb'? <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> - **We use can use coarticulation to tell what the upcoming word will be more quickly!** --- ### We can measure nasal coarticulation by measuring airflow from the mouth and nose - This is called 'pneumotachography' <img class="r-stretch" src="phonmedia/tools_airflowcu.jpg" alt="toolsairflowcu.jpg The image shows a person sitting at a desk working on a computer. The individual is wearing glasses and a blue shirt. They are holding a small microphone close to their mouth with one hand while using the other hand to operate a white mouse that is connected to an Apple iMac desktop computer. The screen of the computer displays various audio waveforms, suggesting that they might be editing or analyzing sound files. The desk has several items on it: there's a keyboard in front of the person and what appears to be some electronic equipment with knobs and buttons next to the monitor. There is also a small white box near the bottom right corner of the desk. The background includes a window, which allows natural light into the room, and part of another piece of furniture or wall can be seen on the left side. The person seems focused on their task, indicating that they are engaged in some form of audio work, possibly related to sound editing or analysis. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Airflow measurement gives us curves - Oral and nasal flow in mL/sec - Sampled (here) at 50 points through the vowel --- ### The word 'bed' has no nasal airflow <img class="r-stretch" src="phonmedia/airflow_bed.png" alt="airflowbed.png The image is a line graph that displays data related to Average Oral and Nasal Flow across all tokens of bed for p03. The x-axis represents Normalized Time, ranging from -25 to 50, while the y-axis measures flow in milliliters per second (mL/Sec), with values ranging from -5 to 20. There are two lines on the graph: one red line and one green line. The red line represents oral flow, which is positive when above zero and negative below it. The green line represents nasal flow, which remains consistently at or near zero throughout the entire time range shown in the graph. The shaded area between these two lines indicates a confidence interval for the data points, suggesting variability around the average values. There are no people depicted in this image; instead, it is purely a graphical representation of data related to speech or breathing patterns. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### The word 'bend' is more complicated <img class="r-stretch" src="phonmedia/airflow_bend.png" alt="airflowbend.png The image is a line graph that displays data related to Average Oral and Nasal Flow across all tokens of bend for p03. The x-axis represents Normalized Time, ranging from -25 to 50, which likely indicates the time in milliseconds normalized relative to some event or point of interest. The y-axis represents Flow (in mL/sec), indicating the flow rate measured in milliliters per second. There are two lines on the graph: one red and one green. Both lines represent different types of flow—oral and nasal, respectively. - The red line represents oral flow. - The green line represents nasal flow. The graph shows that both flows start at zero around normalized time -25 and then increase to a peak value for the oral flow (red) near 0 on the x-axis. After reaching this peak, the oral flow decreases again but remains above the baseline until it drops sharply towards the end of the graph. The nasal flow (green line), in contrast, shows a more gradual increase from zero and then plateaus around normalized time -25 before decreasing slightly. The shaded areas between these lines represent the variability or range of measurements for each type of flow across all tokens of bend for p03. These shaded regions are darker at their peaks to indicate higher variability during those times, suggesting that there is more fluctuation in both oral and nasal flows around normalized time 0. There are no people depicted in this image; it is purely a data visualization. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### The /b/ has no nasal flow <img class="r-stretch" src="phonmedia/airflow_bend_annot_b.png" alt="airflowbendannotb.png The image is a graph that displays data related to Average Oral and Nasal Flow across all tokens of bend for p03. The x-axis represents Normalized Time, which ranges from -25 to 50. The y-axis represents Flow (in mL/Sec) ranging from -5 to 20. The graph includes several lines representing different data points or averages: - A red line with a shaded area around it, indicating variability. - A green line with another shaded area around it, also showing variability. - A gray line that appears to represent the average flow over time for Bend. There is a highlighted section in blue on the left side of the graph. This section has text overlaid: This is the 'B'. The blue highlight seems to indicate where the data for the letter B is concentrated. The shaded areas around each line suggest that there is some variability or range within the flow measurements, which are represented by the lines themselves. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### The /n/ has lots of nasal flow and little oral flow <img class="r-stretch" src="phonmedia/airflow_bend_annot_n.png" alt="airflowbendannotn.png The image is a graph that shows data related to Average Oral and Nasal Flow across all tokens of bend for p03. The x-axis represents Normalized Time, which ranges from -25 to 50. The y-axis represents Flow (in mL/sec) ranging from -5 to 20. The graph has three lines representing different data series: 1. A red line, which appears to represent nasal flow. 2. A green line, which seems to represent oral flow. 3. A light pink area around the red and green lines, indicating a range or standard deviation of the measurements. On the right side of the graph, there is an inset box with a zoomed-in view of part of the data. This inset highlights the N sound in more detail. The inset also shows the same three lines (red for nasal flow, green for oral flow) and includes a light blue shaded area around them. The text at the top right corner of the graph states: This is the 'N'. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### The vowel in the middle shows coarticulation <img class="r-stretch" src="phonmedia/airflow_bend_annot_coart.png" alt="airflowbendannotcoart.png The image is a graph that displays data related to Average Oral and Nasal Flow across all tokens of bend for p03. The x-axis represents Normalized Time, ranging from -25 to 50, indicating the time in normalized units. The y-axis represents Flow (in mL/sec), with values ranging from -5 to 20. The graph consists of two lines representing different types of flow: - A red line labeled as Oral Flow. - A green line labeled as Nasal Flow. Both lines show fluctuations in the flow over time. The red line for oral flow peaks at around +10 mL/sec and then decreases, while the green line for nasal flow remains relatively low with a peak close to 5 mL/sec. There is also a shaded area between the two lines that represents the variability or range of these flows across different tokens of bend for p03. This shaded area is more prominent during certain time intervals and less so in others, indicating greater variability at specific points in normalized time. A box on the graph highlights a section labeled coarticulation, which appears to be an interval where both oral and nasal flows are relatively high compared to other parts of the graph. The exact duration or range within this coarticulation is not specified by numbers but can be inferred from the visual representation. The graph does not include any people, characters, or additional text beyond what has been described. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Looking at airflow we can see coarticulation directly - Both the *amount* of flow and the *timing* of the flow --- ### Some speakers show only a bit of coarticulation <img class="r-stretch" src="phonmedia/airflow_nasal_lowcoart.jpg" alt="airflownasallowcoart.jpg The image is a line graph that displays data related to Average Nasal Flow across all tokens of CVND for p32. The x-axis represents Normalized Time, which ranges from approximately -15 to 40. The y-axis represents Flow (in mL/sec) and spans from 0 to around 16. The graph features a central line that fluctuates, indicating the average nasal flow over time. There are shaded areas on either side of this central line, representing some form of variability or standard deviation in the data points. The shaded area is darker near the bottom left corner and becomes lighter as it moves to the right, suggesting an increase in variability. There are two vertical lines on the graph: one at approximately -10 normalized time and another at around 25 normalized time. These lines might indicate specific events or reference points within the data being analyzed. The central line starts near zero flow, dips slightly below zero as it moves to the left (negative normalized time), then rises sharply after reaching a minimum point before leveling off again. The shaded area shows that there is more variability in the nasal flow during the initial negative time period and at the end of the graph compared to the middle section. The title Average Nasal Flow across all tokens of CVND for p32 suggests that this data might be related to a specific study or experiment involving nasal flow measurements, possibly in the context of speech recognition (CVND likely stands for Context-Dependent Vowel-Nucleus-Diagnostic). The term p32 could refer to a participant number or some other identifier within the dataset. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Some speakers show only a bit of coarticulation <img class="r-stretch" src="phonmedia/airflow_nasal_lowcoart2.jpg" alt="airflownasallowcoart2.jpg The image is a line graph that displays data related to Average Nasal Flow across all tokens of CVND for p82. The x-axis represents Normalized Time, which ranges from approximately -15 to 35. The y-axis represents the Flow (in mL/sec) and spans from 0 to around 16. The graph shows a smooth curve that starts at a negative value, dips slightly below zero, then rises sharply towards positive values as it moves along the x-axis. There are two vertical lines on the graph: one near -5 on the normalized time axis and another closer to 27. These lines might represent specific points of interest or data markers. The curve is shaded in a light gray area around its central line, indicating variability or error margins associated with the measurements. The central line itself is dark blue, and it appears to be the average flow measurement for all tokens of CVND for p82. There are no people depicted in this image; instead, it focuses on data visualization related to nasal flow measurements. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Some speakers show moderate coarticulation <img class="r-stretch" src="phonmedia/airflow_nasal_midcoart.jpg" alt="airflownasalmidcoart.jpg The image is a line graph that displays data related to Average Nasal Flow across all tokens of CVND for p65. The x-axis represents Normalized Time, which ranges from approximately -10 to around 35. The y-axis measures the flow in milliliters per second (mL/sec), ranging from 0 to about 15. The graph shows a smooth curve that starts at a negative value, dips slightly below zero, and then rises sharply after reaching its lowest point. There is a vertical line on the x-axis around normalized time -2, which might indicate a specific event or reference point in the data being measured. Another vertical line appears near normalized time 30. The curve fluctuates above and below an orange horizontal line that represents a baseline value of approximately 1 mL/sec. The shaded area between the curve and this baseline indicates the range of variation around the average nasal flow for p65 across all tokens in CVND, suggesting variability or error margins associated with the measurements. There are no people depicted in this image; it is purely a data visualization. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Some speakers show massive coarticulation <img class="r-stretch" src="phonmedia/airflow_nasal_highcoart.jpg" alt="airflownasalhighcoart.jpg The image is a line graph that displays data related to Average Nasal Flow across all tokens of CVND for p42. The x-axis represents Normalized Time, which ranges from approximately -15 to 35. The y-axis represents the Flow (in mL/sec) and spans from 0 to around 15. The graph features a central line that fluctuates, indicating the average nasal flow over time. Surrounding this central line are shaded areas on both sides, which represent variability or standard deviation in the data points. These shaded regions extend horizontally across the x-axis, showing how the flow values vary at different normalized times. There is also a yellow horizontal line near the bottom of the graph, which appears to be a baseline indicating a reference value for nasal flow (likely around 1 mL/sec). The graph does not include any people or characters. It focuses solely on presenting data in a visual format. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Some speakers show massive coarticulation <img class="r-stretch" src="phonmedia/airflow_nasal_highcoart2.jpg" alt="airflownasalhighcoart2.jpg The image is a line graph that displays data related to Average Nasal Flow across all tokens of CVND for p56. The x-axis represents Normalized Time, which ranges from approximately -10 to 35. The y-axis represents the flow in milliliters per second (mL/sec), ranging from 0 to around 15. The graph shows a smooth curve that starts at a low value, increases gradually over time, and then experiences a sharp increase near the normalized time of 25 before leveling off again. There are two vertical lines on the x-axis: one at approximately -3 and another at about 27. These lines might indicate specific points or events in the data. The graph includes shaded areas around the central line, which likely represent standard deviations or confidence intervals for the average nasal flow values across different tokens of CVND for p56. The central line is a solid dark blue line that represents the mean nasal flow value over time. There are no people depicted in this image; it focuses solely on data visualization related to nasal flow measurements. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Speakers differ greatly in their *production* of coarticulation - Ranging from 'practically none' to 'it's all nasal' - Inference can be done using splined mixed models, GAMs, and more - Functional data analysis isn't common in Linguistics, but it does happen! --- ### If speakers vary in their production of coarticulation - Do they differ in their *perception* of coarticulation as well? --- ### Measuring the Perception of Coarticulation - Often done using eyetracking - "When does the participant look at the correct image on the screen?" - "Does this person use vowel nasality to choose 'send' over 'said' more quickly?" --- ### Visual World Eyetracking <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Eye Tracking Data - For each trial, 1000 binary points over the course of a second, 'Are they looking at the nasal word?' - 0000000000000001111111111... - Occasionally 00000000000000011111111110000000... - Many, many trials are averaged out to create response curves - "Generally speaking, does this person make a choice earlier in this condition than that one?" --- ### Conditions - "Early Nasalization": Coarticulation begins very early in the vowel - "Late Nasalization": Coarticulation begins later in the vowel - *How early is information about the word made available to listeners?* --- ### Listeners can be compared on the basis of their use of nasality - People who use coarticulation strongly in perception will decide 'send' over 'said' earlier for 'early' nasalization tokens - People who don't use coarticulation in perception will show little distinction between the conditions --- ### Listeners who use coarticulation <img class="r-stretch" src="phonmedia/eyetracking_largeuse1.jpg" alt="eyetrackinglargeuse1.jpg The image is a line graph that illustrates data related to Perception for p92, which seems to be a specific measurement in some context, possibly linguistic or phonetic. The x-axis represents time, ranging from 0 to 1000 units, and the y-axis measures nasality perception on a scale from 0 to 1. There are two lines plotted on this graph: - A blue line labeled early which starts at approximately 0 and gradually increases over time. - A yellow line labeled late, which also begins near 0 but rises more steeply than the early line, reaching higher values by the end of the time range. The y-axis is marked with numerical values: 0.00, 0.25, 0.50, 0.75, and 1.00. The x-axis has intervals at 250, 500, 750, and 1000. In the top left corner of the graph, there is a label that reads nasalization with additional information: PC2 rank: 40, and PC2 Score: -2.47. This suggests that the data being plotted might be related to principal component analysis (PCA) or some other form of statistical analysis. The right side of the graph has a vertical bar labeled voiced, which seems to indicate a category or classification for the data points, possibly denoting whether they are voiced sounds in speech. The blue and yellow lines both reach close to this voiced label by the end of the time range on the x-axis. The overall trend shows that nasality perception increases over time, with the early onset showing a slower rise compared to the late onset which rises more sharply. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Listeners who use coarticulation <img class="r-stretch" src="phonmedia/eyetracking_largeuse2.jpg" alt="eyetrackinglargeuse2.jpg The image is a line graph that illustrates data related to Perception for p96, with specific details about PC2 rank (9) and PC2 Score (1.1). The x-axis represents time in milliseconds, ranging from 0 to approximately 1000. The y-axis measures the level of perception, which is labeled as nasalization on the graph. There are two lines plotted on the graph: - A blue line representing early nasalization. - A yellow line representing late nasalization. The blue line starts at a value close to 0 and gradually increases over time. It reaches its peak around the 750 mark, where it stays relatively stable until about the 900 mark before dropping slightly. The yellow line begins near zero as well but remains flat for most of the graph's duration. It shows no significant increase or decrease in value throughout the entire range shown on the x-axis (from 0 to approximately 1000 milliseconds). On the right side of the graph, there is a shaded area labeled Voiced, indicating that this section represents voiced sounds. The graph provides data points for two different types of nasalization: early and late. The blue line indicates an increase in perception over time for early nasalization, while the yellow line shows no significant change or perception for late nasalization within the given timeframe. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Listeners who largely ignore coarticulation <img class="r-stretch" src="phonmedia/eyetracking_littleuse1.jpg" alt="eyetrackinglittleuse1.jpg The image is a line graph that illustrates the perception of nasalization over time for a specific phonetic feature labeled as p55. The x-axis represents time in milliseconds (ms), ranging from 0 to 1000. The y-axis represents the probability or degree of nasalization, which ranges from 0 to 1. There are two lines on the graph representing different categories of perception: - A blue line labeled early indicates an early perception of nasalization. - A yellow line labeled late indicates a late perception of nasalization. The graph shows that both perceptions start at zero and increase over time. The blue line (early) rises more steeply than the yellow line (late), indicating that the early perception reaches higher values faster but then plateaus, while the late perception continues to rise steadily until it also plateaus around 1000 ms. In the top left corner of the graph, there is a legend explaining what p55 refers to. It states: nasalization (PC2 rank: 42, PC2 Score: -2.95), indicating that this phonetic feature has been analyzed using Principal Component Analysis (PCA), and it ranks as the 42nd most significant component with a score of -2.95. The right side of the graph is labeled voiced, suggesting that the data might be related to voiced sounds in speech, where nasalization can play an important role. There are no people or characters depicted in this image; it focuses solely on the phonetic analysis and perception over time. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Listeners who largely ignore coarticulation <img class="r-stretch" src="phonmedia/eyetracking_littleuse2.jpg" alt="eyetrackinglittleuse2.jpg The image is a line graph that tracks the perception of nasalization over time for an individual labeled as p45. The x-axis represents time in milliseconds (ms), ranging from 0 to 1000. The y-axis measures the level of nasalization, with values starting at 0 and increasing up to approximately 1. There are two lines on the graph: - A blue line representing early nasalization. - A yellow line representing late nasalization. The blue line starts at a value close to zero around the 250 ms mark. It then increases sharply until it reaches its peak, which is slightly above 0.75 and occurs around the 650 ms mark. After this point, there are some fluctuations but the overall trend shows that the level of early nasalization remains relatively high. The yellow line starts at zero for a longer period compared to the blue line, beginning around the 400 ms mark. It then increases gradually and steadily until it reaches its peak value close to 1 around the 850 ms mark. After this point, there are some minor fluctuations but the level of late nasalization remains consistently high. In the top left corner of the graph, there is a legend that explains what each line represents: - Blue line: early nasalization - Yellow line: late nasalization Additionally, in the bottom left corner, there is text that reads nasalization and provides further details about the data being represented. It states: - PC2 rank: 24 - PC2 Score: 0.07 On the right side of the graph, a vertical bar labeled Voiced indicates the time range where voiced sounds are expected to occur. The graph is titled Perception for p45 (PC2 rank: 24, PC2 Score: 0.07). This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### So, now we can measure perception of coarticulation - ... and production - This allowed us to ask one very large question... --- ### Is a listener's production of coarticulation related to their perception of coarticulation? - Put differently, do people who coarticulate early, listen for it early? - *Do people who talk unusually expect others to talk the same way?* - This was tested in [Beddor et al. 2018](https://muse.jhu.edu/article/712563) - <small> Beddor, P.S., Coetzee, A.W., Styler, W., McGowan, K.B., & Boland, J.E. (2018). The time course of individuals’ perception of coarticulatory information is linked to their production: Implications for sound change. _Language_ _94_(4), [doi:10.1353/lan.2018.0071](http://doi.org/10.1353/lan.2018.0071)</small> --- ### This is a surprisingly useful question - It gets at the heart of the gesture vs acoustics debate in speech perception - It tells us about the role of our own productions in guiding our learning of a language - It has massive implications for how languages change over time --- ### But it's really, really unpleasant to test - Correlating a functional airflow curve (with massive variation in values) with the overall trend across a large set of logistic time series from eye tracking trials - We have truly random factors we want to get rid of - Variation in frequency and 'lookability' across words - Our research question involves speaker variation in time-to-look by condition and variation in flow slope and time onset - ... but we want to control for speaker variation in pre-look processing time, absolute differences in airflow volume - We're interested in speaker variation, but the experiment was so complex that we could only collect 42 participants --- # Yikes - We needed help --- ### Help [Kerby Shedden](https://sph.umich.edu/faculty-profiles/shedden-kerby.html) from University of Michigan Department of Statistics <img class="r-stretch" src="people/kerbyshedden.jpg" alt="kerbyshedden.jpg This image is a head-and-shoulders portrait of a middle-aged Caucasian man facing directly forward. He has short, light brown or dark blonde hair that is cut close to the scalp, with a slightly receding hairline at the temples. His skin is fair, and he has blue eyes visible behind his eyewear. He is wearing rectangular, wire-rimmed glasses. His expression is neutral but pleasant, with a very slight, closed-mouth smile. He appears to be looking straight into the camera lens. Regarding his clothing, he is wearing a brown t-shirt. Underneath the brown shirt, there is a visible collar of an olive green undershirt or t-shirt peeking out around the neckline. The background is a solid, neutral grey color that is slightly out of focus, suggesting a studio setting with soft lighting illuminating his face evenly from the front. There are no other people, objects, text, diagrams, numbers, or figures visible in the image. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### We ended up collapsing the airflow data using PCA - This gave us a single quantity representing timing and degree of coarticulation ('PC2') which we could insert into a model of perception - The perception model was run using ``mcmcglmm`` in R, with b-splines to model temporal variation --- ### Turns out that people who produce early coarticulation generally listen for early coarticulation <img class="r-stretch" src="phonmedia/beddor_et_all_fig11_mod.jpg" alt="beddoretallfig11mod.jpg The image consists of two graphs side by side, both titled CVearlyNC on the left and CVlateNC on the right. These titles suggest that the data might be related to a cognitive process involving consonant-vowel sequences (CV) in early and late stages. Graph Details: Left Graph: CVearlyNC - X-axis: Labeled as Time from Stimulus Onset (ms) indicating time in milliseconds after stimulus presentation. - The range of the x-axis is from 200 ms to 1000 ms, with a vertical dashed line at approximately 450 ms. - Y-axis: Labeled as Proportion CVNC Fixations, which measures the proportion of fixations on consonant-vowel-nucleus-consonant (CVNC) sequences over time. - The range of the y-axis is from 0.0 to 1.0, indicating a complete fixation rate. - Curves: - There are three curves in this graph: - A yellow curve labeled High PC2. - A red dashed line labeled Mean PC2. - A blue curve labeled Low PC2. Right Graph: CVlateNC - X-axis: Same as the left graph, labeled Time from Stimulus Onset (ms) with a range of 200 ms to 1000 ms and a vertical dashed line at approximately 450 ms. - Y-axis: Same as the left graph, labeled Proportion CVNC Fixations. - Curves: - There are three curves in this graph as well: - A yellow curve labeled High PC2. - A red dashed line labeled Mean PC2. - A blue curve labeled Low PC2. Observations: 1. Both graphs show a similar pattern where the proportion of CVNC fixations increases over time, reaching a peak and then plateauing. 2. The yellow curves (High PC2) are consistently above the red dashed lines (Mean PC2), which in turn are above the blue curves (Low PC2). 3. In both graphs, there is a noticeable dip around 450 ms on the x-axis. People in the Image: There are no people depicted in this image; it contains only graphical data and labels related to cognitive processes involving consonant-vowel sequences. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> (Adapted from Beddor et al 2018) --- ### Work is ongoing to continue investigating these issues - The perception/production link can be examined in *many* speech phenomena - What if we tested the perception/production link in another domain? - **Let's see if we can correlate tongue movement with perception!** --- # A Coarticulation Measurement Failure --- ### How does the middle part of 'bulk' work? - For some people, there's the vowel in 'buck', then an L sound, with minimal overlap - For others, the L is strongly **coarticulated** with the vowel - And for many, there's just one steady mixed "ul" vowel - This is an interesting question for speech perception, speech recognition, and text-to-speech - **How can we tell who's doing which of these things, quantitatively?** - ... and can we predict who does what? --- ### Ultrasound Imaging - Pulse high-frequency sound waves into the body - Measure the patterns in which they return to image internal structure - The resulting data are black and white image frames showing areas of high and low reflection --- ### Ultrasound Data Acquisition <img class="r-stretch" src="phonmedia/tools_ultrasound.jpg" alt="toolsultrasound.jpg The image shows a person engaged with an imaging device that appears to be used for medical purposes, possibly related to brain imaging given the context of the text on the screen. The individual is wearing large over-ear headphones and seems focused on the monitor in front of them. The monitor displays a 3D image of what looks like a human brain, with various labels such as P1 and P2, which might indicate different parts or perspectives of the brain being examined. There are also some numerical values next to these labels, suggesting measurements or coordinates related to the brain's structure. The person is seated in front of the monitor, which is mounted on a stand with multiple foam pads for stability. The individual appears to be wearing a light-colored top and has dark hair. They seem to have a microphone attached near their mouth, indicating that they might be communicating or providing feedback during this process. To the right side of the image, there's a piece of paper taped to the wall, which likely contains instructions or information related to the procedure being performed on the monitor. The background is plain and light-colored, suggesting an indoor setting, possibly a medical facility or research lab. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Sample Speech Ultrasound file <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### Ultrasound in Speech - Captures the motion of the tongue in (generally) two dimensions - Offers 60+ frames per second time resolution - Ideal for tracking the *relative location* and *contour* of the tongue and --- ### Ultrasound 'Splining' - The machine outputs a series of images (or grayscale matrices) at a fixed sampling rate - We transform images into lists of ordered points representing the tongue shape and location - This is done by the researcher and team directly - ... or using [neural networks](https://arxiv.org/abs/1907.10210) - Or full-frame PCA is used to look for changes (c.f. [Faytak et al. 2020](https://www.journal-labphon.org/article/id/6281/)) - There are other complexities I'm skipping here! --- <section> <img class="r-stretch" src="phonmedia/ultrasound_raw.jpg" alt="ultrasoundraw.jpg The image appears to be a medical ultrasound scan, likely of an internal organ such as the heart or another soft tissue structure within the body. The scan is in grayscale and shows various shades that represent different densities of tissues. In the center of the image, there is a prominent white area that stands out against the darker background. This could indicate a region with higher density, possibly representing blood flow or a specific organ's structure like the heart valves or chambers. Surrounding this central area are varying shades of gray and black, which might represent different layers of tissue or fluid. The overall shape is somewhat irregular but has a general outline that suggests it may be an internal organ or part of one. The edges are not sharply defined, which is typical for ultrasound images due to the nature of how sound waves interact with tissues in the body. There are no visible text labels or diagrams within this image; it focuses solely on the grayscale representation of the scanned area. If you need further clarification about any specific aspect of the scan, feel free to ask! This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> </section> --- <section> <img class="r-stretch" src="phonmedia/ultrasound_splined.jpg" alt="ultrasoundsplined.jpg The image depicts a simple line drawing on a black background. The line is white and appears to be hand-drawn with a marker or pen. It starts from the left side of the image and curves upwards towards the right, ending in an open curve that does not form a complete loop. There are no other elements, text, numbers, or figures present in this image. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> </section> --- ### This splined data gives us details about articulation - How do tongue contours differ between sounds? - "Does the tongue shape differ for 'buck' and 'bulk'?" - How do tongue contours change during sounds? - "At what point does the tongue start moving towards the /l/ gesture in 'bulk'?" - ... or is there no change at all for this speaker? --- ### Does the tongue shape differ for 'buck' vs. 'bulk'? <img class="r-stretch" src="phonmedia/ultrasound_vowel.jpg" alt="ultrasoundvowel.jpg The image depicts a series of curved lines on a grid background. The grid consists of white squares with light gray borders, creating a uniform pattern across the entire image. The lines are drawn in a light blue color and appear to be slightly overlapping each other, forming an arch-like shape that peaks at the center. Each line is straight at both ends but curves upward in the middle, resembling the trajectory of a thrown object or a parabolic path. There are no people or characters present in this image; it focuses solely on these curved lines and the grid they rest upon. The lines do not have any text or numbers associated with them. The overall appearance is simple and geometric, likely intended to represent some form of data visualization or mathematical concept such as a distribution curve or trajectory analysis. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> <img class="r-stretch" src="phonmedia/ultrasound_lateral.jpg" alt="ultrasoundlateral.jpg The image is a graph with a grid background consisting of horizontal and vertical lines that form small squares across the entire area. The graph appears to represent some kind of data distribution or trend over time. There are numerous curved lines plotted on this grid, which vary in color from light gray to blue. These lines seem to be part of a statistical analysis or simulation, as they do not follow any straight path but rather curve and overlap each other, suggesting variability or randomness within the dataset being represented. The lines start at different points along the x-axis (horizontal axis) and extend upwards towards the y-axis (vertical axis). The density of these lines increases in certain areas, indicating a higher concentration of data points there. There are no labels on either the x-axis or the y-axis to provide specific information about what is being measured. Overall, this graph seems to be used for visualizing some form of statistical analysis where multiple trials or measurements have been conducted and their results plotted together to show variability within that dataset. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Comparing Contours is difficult (for us) - Averages and center of gravity can give some clarity - Usually done using Smoothing Spline ANOVA in Linguistics - Occasionally mixed models with B-Splines, Generalized Additive Models (GAM), and Growth Curves <img class="r-stretch" src="phonmedia/ultrasound_vowel.jpg" alt="ultrasoundvowel.jpg The image depicts a series of curved lines on a grid background. The grid consists of white squares with light gray borders, creating a uniform pattern across the entire image. The lines are drawn in a light blue color and appear to be slightly overlapping each other, forming an arch-like shape that peaks at the center. Each line is straight at both ends but curves upward in the middle, resembling the trajectory of a thrown object or a parabolic path. There are no people or characters present in this image; it focuses solely on these curved lines and the grid they rest upon. The lines do not have any text or numbers associated with them. The overall appearance is simple and geometric, likely intended to represent some form of data visualization or mathematical concept such as a distribution curve or trajectory analysis. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> <img class="r-stretch" src="phonmedia/ultrasound_lateral.jpg" alt="ultrasoundlateral.jpg The image is a graph with a grid background consisting of horizontal and vertical lines that form small squares across the entire area. The graph appears to represent some kind of data distribution or trend over time. There are numerous curved lines plotted on this grid, which vary in color from light gray to blue. These lines seem to be part of a statistical analysis or simulation, as they do not follow any straight path but rather curve and overlap each other, suggesting variability or randomness within the dataset being represented. The lines start at different points along the x-axis (horizontal axis) and extend upwards towards the y-axis (vertical axis). The density of these lines increases in certain areas, indicating a higher concentration of data points there. There are no labels on either the x-axis or the y-axis to provide specific information about what is being measured. Overall, this graph seems to be used for visualizing some form of statistical analysis where multiple trials or measurements have been conducted and their results plotted together to show variability within that dataset. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### At what point does the tongue start moving towards the /l/ gesture in 'bulk'? - This is a place where speakers vary - We can look at the time course of the vowel+l portion of the word --- ### Some people show some change later <img class="r-stretch" src="phonmedia/ultrasound_bulk_somechange.jpg" alt="ultrasoundbulksomechange.jpg The image is a three-dimensional graph that illustrates the contour of a tongue over time for a specific nucleus labeled as p19bulk267. The graph has three axes: - The x-axis represents Tongue (Y) which measures the vertical position or height of different points on the tongue. - The y-axis represents Tongue (X), indicating horizontal positions across the tongue's surface. - The z-axis, labeled as Time (within nucleus), shows how these positions change over time. The graph is a grid-like structure with numerous lines forming a mesh that depicts the movement and shape changes of the tongue. The lines are arranged in such a way to show the contour or profile of the tongue at different points along its length and height, as it evolves over time within this nucleus. There are no people depicted in this image; instead, it is purely an abstract representation of data related to the movement and shape changes of a tongue. The graph does not include any text or diagrams that would provide additional context beyond what has been described here. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Some people have massive change early on <img class="r-stretch" src="phonmedia/ultrasound_bulk_bigchange.jpg" alt="ultrasoundbulkbigchange.jpg The image is a three-dimensional graph that represents Tongue contour over time for nucleus of p07bulk212. The graph shows how the shape and position of the tongue change as it moves along the x-axis, which represents Time (offset nucleus). The y-axis represents Tongue (Front →), indicating the front-to-back movement of the tongue. The z-axis represents Tongue (Y), showing the vertical movement or height of the tongue. The graph is a grid-like structure with lines forming a series of peaks and valleys, illustrating the dynamic changes in the shape of the tongue over time as it moves along its length. There are no people depicted in this image; instead, it focuses on the graphical representation of the tongue's movements during speech or articulation processes. The title at the top of the graph provides context for what is being measured and analyzed: Tongue contour over time for nucleus of p07bulk212. This suggests that the data might be related to a specific phonetic study, possibly focusing on the pronunciation of certain sounds in speech. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Some people don't show change at all <img class="r-stretch" src="phonmedia/ultrasound_bulk_nochange.jpg" alt="ultrasoundbulknochange.jpg The image is a three-dimensional graph that illustrates the movement of the tongue over time for a specific nucleus labeled as p04bulk98. The graph has three axes: 1. Tongue (Y) - This axis represents the vertical position of different parts of the tongue. 2. Tongue Front (X) - This axis represents the horizontal position along the front part of the tongue, moving from left to right. 3. Time (within nucleus) - This axis represents time in seconds as it progresses from bottom to top. The graph is a wireframe mesh that shows how the shape and position of the tongue change over this period. The darker areas within the mesh indicate regions where the tongue is more prominent or active, while lighter areas suggest less activity or movement. There are no people depicted in this image; instead, it focuses on the graphical representation of the tongue's movements during a specific vocalization or speech sound production process. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Measuring these changes is very difficult (for us) - Quantifying the degree of change in a 50 point spline which changes contour and position over time - Variably, across speakers - Identifying the *onset* of the contour change in time - Identifying specific types of contour change which are most relevant - Finding 'targeted' vs 'untargeted' change - **There isn't a well-established statistical method for doing this in our field!** --- ### We had hoped to continue down this road - Also to recreate the perception/production study above with these data - **This is a very hard analysis problem, and we weren't up to the task** - *... and we also had trouble finding enough people with changing vowels* --- ### This is a case where our ability to analyze hindered our ability to describe - We're finding more of those all the time! - Which helps answer the question... --- # Why am I here? --- ### Increasingly complex data has pulled us into complex territories - We've moved from single variable correlations into functional data - In many cases, functional data is itself captured in a time series - New methods are arriving for us every day - ... but our questions are generally different enough that existing statistical toolchains don't cleanly apply --- ### Increasingly complex questions require increasingly nuanced analyses - We've moved from presence/absence into time course information - We're now increasingly studying the kinds of variability which conventional models attempt to factor out - Potentially explanatory data is seldom low-dimensional! --- ### Our statistical needs have surpassed our statistical abilities - Grad level Psych Stats has very little to say about comparing 3D meshes of tongue motion by conditions - This poses a massive pedagogical problem! - Reviewers are generally chosen for knowledge of specific linguistic domains, with varying statistical saavy - "Why not just use an ANOVA here?" or "How did you settle on the right number of spline coefficients?" - Keeping up with the statistical state-of-the-art is a full-time job - So us statistically-interested linguists are just toddlers with bigger shotguns --- ### That's why I'm here today - (That and Shuheng's gracious invitation) --- ## Linguists and Statisticians should talk more! --- ### Language is uniquely rewarding as an area of research - You are quite literally always using language - Problems are often interpretable in terms of linguistic experience - It offers a diversity of data types, often in the same experiments - Text data, behavioral experiments, sensor output, imaging data, GIS, and more - Linguistic knowledge is helpful for breaking into Natural Language Processing, and other language-focused data science - Everything I've talked about today has straightforward applications in speech recognition and text-to-speech --- ### Collaborations can be very fruitful! - We often have questions with more nuance than the techniques we know - Basic-to-you statistical approaches may enable advanced-for-us questions! - It's very possible that new techniques in statistics could allow revolutionary questions in our field - Often, new analytical approaches provide new avenues for research - Collaborations can be mutually rewarding and mutually beneficial - Linguists learn Stats, Statisticians learn Language --- ### This is a potential growth area in our field - There is increasing discussion of hiring statisticians in departments and divisions for consulting and collaboration - ... and already, statistical saavy is a common desired trait for new hires - Statisticians who know even basic elements of language will be increasingly valued in industry and life - Natural Language Processing is Statistics and Machine Learning about language --- ### Teamwork can make the dream work - Linguistic work is often held back by relatively basic inference approaches - Increased complexity of data, and increased complexity of questions, both leave ample room for collaboration - New methods in statistics likely have testable uses in language - New questions in linguistics may require new methods in statistics - And people collaborating in this world have a very real chance to make a difference in both fields --- ### Let's talk! - Next time you're looking to branch out, remember that we linguists are here - That we've got amazing data - ... and that at the very least, you may be able to help keep a toddler safe! --- <huge>Thank you!</huge> Questions? <wstyler@ucsd.edu> <http://savethevowels.org/talks/colloq_ucriverside_2021.html> <!-- Linguistic Problems with Statistic Solutions In this talk, my goal is to briefly introduce a statistical audience to some of the particularly interesting types of data, hypotheses, and open questions found in Phonetics, the subfield of Linguistics dedicated to the study of speech and speech perception. We'll discuss the state of the field, and then look at two case studies, the analysis of ultrasound data of tongues, and the production-perception loop in speech and perception. In doing so, we'll discuss some of the myriad difficulties left for linguists, and highlight areas which may prove fertile ground for collaborative statistical research. -->