### "Supermarket Self-checkout Robot overlords taking over the earth" from StableDiffusion 1.4 in 2023 <img class="r-stretch" src="comp/sd_supermarket.png" alt="sdsupermarket.png The image depicts a grocery store aisle with two automated kiosks positioned prominently in the foreground. These kiosks are tall, rectangular structures with a predominantly white and black color scheme. Each kiosk has a blue screen at the top displaying an animated character that appears to be a robot or assistant. The character is wearing a light-colored outfit. The kiosks have a sleek design with a flat base that supports them upright. They also feature a small, rectangular platform in front of each kiosk where customers can place their items for scanning. There are buttons and possibly touchscreens on the sides of the kiosks, likely used to interact with the system or make selections. In the background, there is an aisle filled with various products. The shelves are stocked with a variety of goods, including what appears to be snacks, beverages, and other grocery items. The products come in different colors and packaging designs, indicating a wide range of options available for customers. There are also some signs visible on the shelves, which likely provide information about the products or instructions. The floor is clean and well-lit, suggesting that this is an organized retail environment. The overall setting appears to be designed for customer convenience, with the kiosks serving as a self-service option for checking out or making purchases without needing assistance from store staff. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### "Supermarket Self-checkout Robot overlords taking over the earth", but in 2025 with Dall-E <img class="r-stretch" src="comp/dalle_supermarket.png" alt="dallesupermarket.png The image depicts a group of robots standing on what appears to be a barren landscape with rocky terrain under an ominous sky filled with swirling clouds in shades of red and orange. The central robot is significantly larger than the others and has a rectangular head with two small, circular eyes that have a greenish hue. Its body is boxy with wheels at the bottom for mobility. It holds a wrench-like tool in each hand. Surrounding this central figure are several smaller robots of similar design but varying heights. These robots also feature rectangular heads with greenish eyes and hold wrenches as well, suggesting they might be engaged in some form of maintenance or repair work. The background features a large, fiery planet that dominates the scene, adding to the dramatic atmosphere. The overall color palette is dominated by dark reds, oranges, and blacks, contributing to a sense of urgency or danger. There are no visible texts or diagrams within this image. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <img class="r-stretch" src="tmp/legacy.png" alt="legacy.png This image is a promotional flyer for an event called Legacy Lecture 2026. The overall color scheme features shades of light blue, dark blue, white, and accents of yellow. At the very top, in a large, cursive dark blue font with a white outline, are the words Legacy Lecture 2026. Below that, in bold, blocky capital letters with a white outline and drop shadow, is the text: VOTE FOR YOUR FAVORITE UCSD PROF. The background of the flyer is light blue with abstract white shapes. Scattered around the top corners are yellow five-pointed stars. On the left and right edges, there are illustrations of plant stems with leaves; the leaves are outlined in dark blue, and at the bottom left, there appear to be yellow flowers or buds. In the center of the flyer is a graphic designed to look like a computer web browser window. It has a light blue header bar containing navigation arrows (left and right), an X for closing the tab, and a white address bar. Inside this browser window, on a white background, is a large black-and-white QR code. Below the browser graphic is a white text box with rounded corners. Inside, there is a quote in bold, dark blue italics: If this were the last lecture you ever gave, what would you say to the world? Underneath the quote is a paragraph of standard dark blue text that reads: Legacy Lecture, hosted by the UCSD Scholars Society, is an annual event in which a UCSD professor, chosen by student vote, is invited to present a lecture to the student body answering that question. At the very bottom of the flyer, centered, is the text Presented by UC San Diego Scholars Society in dark blue. In the bottom right corner, there is a circular seal logo for the UC San Diego Scholars Society. The logo features a trident and an open book inside a laurel wreath with the text UC SAN DIEGO SCHOLARS SOCIETY around the rim. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- <!-- ### If you need special accommodations for the exam, I need you to have already booked at time at the Triton Testing Center ASAP <img class="r-stretch" src="img/bearsrepeating.jpg" alt="bearsrepeating.jpg The image displays a grid of twelve identical photographs arranged in three rows and four columns. Because every panel shows the exact same picture, I will describe the single unique image that is repeated across the entire layout. Each photograph features an American black bear sitting upright and facing directly forward toward the viewer. The bear has thick, dark black fur covering its head, shoulders, and body. Its face features a distinct tan or light brown muzzle that contrasts with the black fur of its snout and cheeks. It has small, rounded ears perched on top of its head and dark eyes looking straight ahead. A small black nose is visible at the tip of its tan muzzle. Behind the bear is a background consisting of large, grey rocks or a stone wall. In the upper left corner of each panel, there are patches of green leafy vegetation growing among the rocks. There is no text, diagrams, numbers, or figures present in this image. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --> # Natural Language Processing ### Will Styler - LIGN 101 --- ## Today's Plan - What is NLP? - What are the kinds of things we do in NLP and why? - Why is it so damned hard? - LLMs and the New World --- ## Natural Language Processing and Computational Linguistics --- ### Lots of Linguists use computers to study language - Digital Signal Processing and computational data capture for Phonetics - Computational Modeling of Phonological Rules - Searching large amounts of text to better understand syntactic structures - Analyzing semantics through mathematical and probabilistic approaches - **They don't care about computers, they just care about language** --- ### Aside: Linguistics goes very well with Computation, Data and Math - Math/CS and Linguistics are complementary majors! - Many linguistics grads end up doing data science - The Computational Social Science program is being run by a linguist --- ### ... but some of us are interested in the computers themselves - "What elements of human language can we model computationally?" - "How can I train a computer to produce grammatical human sentences and utterances?" - "How can we produce systems which can naturally interact with humans in human languages?" --- ### We consider these folks to live in two closely related subfields - Computational Linguistics - Focuses on theory of modeling human language using computational approaches - Natural Language Processing - Focuses on designing systems which work with, understand, act on, and produce human language - Many (rightfully) consider NLP to be a subfield within CL - With NLP often being more 'applied' and 'engineering' - We're going to focus on **NLP** today --- ### NLP is primarily about measuring *probability* - Given the word 'that' and the sentence's structure, how likely is it to be a determiner? - Given the other words in this sentence, how likely is 'bank' to mean 'financial institution'? - Given these acoustic patterns and the prior sounds evaluated, how likely is this to be a /t/? - Given the sounds I think I observed and this person's iTunes library, what album are they most likely asking for? - Hey, ChatGPT, predict the next word! --- ### We're not going into methods today - Methods change every day, and are increasingly boring! - Methods are in LIGN 6, LIGN 165, LIGN 167, LIGN 168 - Instead, we'll focus on tasks and problems --- ## Why do we want NLP to be a thing? --- ### There's a *lot* of natural language data out there - 1 billion or more active websites <small>(<a href="https://www.internetlivestats.com/total-number-of-websites/">Source</a>)</small> - Mayo Clinic enters 298 million patient records per year <small>(<a href="http://www.mayoclinic.org/emr/">Source offiline</a>)</small> and ~90% of physician offices create electronic medical records - 500 million Tweets per day <small>(<a href="https://www.dsayce.com/social-media/tweets-day/">Source</a>)</small> - 300+ billion emails sent daily in 2020 <small>(<a href="https://techjury.net/blog/how-many-emails-are-sent-per-day">Source</a>)</small> - Recorded phone calls, blog posts, TikToks... - ... and that's just the digital stuff --- ### Being able to access and process huge amounts of natural language data is useful - "Watch Twitter and give me the locations of wildfires, floods, etc, and provide information about damage, shelters and resources in an easy-to-read format" - "Which of these 290 billion emails are likely to be discussing the sale or trafficking of nuclear weapons?" - "Read the news articles and forum posts out there published and tell me everything we know about the effects of the [Tigray War](https://en.wikipedia.org/wiki/Tigray_War) on the city of Aksum." --- ### We also directly interact with computers more than ever - Siri, Google Assistant, Alexa, Cortana, and more - "Hey Siri, set a timer for 20 minutes" - Automated phone systems and Chatbots - "First, tell us why you're calling..." - Informational Retrieval and Search - "What's the name of that small blonde Norwegian singer with that song about low-key earthquakes?" - Natural language interfaces to existing servces - "Navigate me to Campbell Hall at UCLA" --- ### These problems could be solved with humans - ... and most of them historically were! - Assistants, Interns, Paralegals, Intelligence officers, Directory Assistance Services, Concierges - ... and Will's own lazy self who doesn't want to walk over to a light switch - ... but there's a problem... --- ### Humans are inefficient and expensive - They only work certain hours - They want things like food, shelter, leisure, and companionship - They're speed-limited in reading and summarization --- ### So, companies are turning to computers and 'AI' - There's a lot to unpack about this - The ethical problem of replacing human jobs with NLP tools should not be ignored - You're replacing one set of unclear biases with a different set - ... but what kinds of tasks can they get the computers to do? --- ## NLP Tasks --- ### Speech Recognition - "Ask people why they're calling, and connect them to the right department based on their answer." - "Flag all tech support conversations where the customer mentions a competitor" - "Transcribe all orders placed at this kiosk" - "Transcribe this speech without errors" --- ### Analysis of secondary speech characteristics - "Redirect all angry-sounding customers to higher-tier support workers" (Speech emotion detection) - "Are the two people in this skype call flirting, arguing, expressing love, or sadness? Target post-session ads accordingly." - "I want to talk to... billing?" (Uncertainty analysis) - "Yeah, I really like going to Applebees." (Spot-the-sarcasm) --- ### Text-to-Speech - Speak driving directions aloud - Read all incoming text messages aloud through headphones to the phone's biking owner - Read this webpage aloud for the computer's blind user - Automatically turn this eBook into an Audiobook --- ### Aside: Modern Text-to-Speech is really good - ... and it can even imitate people! - "Train the model on a more generic voice, and then use specific data to learn a different person's 'style' " --- ### Neural Network Text-to-Speech Style Transfer Examples <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- ### You can make a model of anybody these days... <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (TacoTron2) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p>(ElevenLabs) (Credit to Erick Amaro and Mia Khattar!) --- ### Multilingual Examples <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (English) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Spanish) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Mandarin) <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> (Russian) --- ### Code Switching It's like sometimes mezclo un poco de español con my English, cuando me siento particularmente spicy, y tengo curiosidad to know cómo la TTS handles it. <p>(This audiovisual content has been removed for compliance with recent federal accessibility guidelines. <a href="https://accessibility.ucsd.edu/policies-standards/ucsd-accessibility-guidelines.html">Please see this site for details.</a>)</p> --- # Espicy! --- ### We're going to focus on text for the rest of the talk - But LIGN 168 'Computational Speech Processing' is a class! --- ### Authorship attribution and stylistic analysis - Examine these two written passages/books and tell me whether they were both written by the same person - Examine these negative reviews and tell me what demographic the authors likely represent based on the language used. - Examine every incoming tweet and facebook post and detect posts which seem likely to have been written by robots --- ### Predictive analysis of text - Look for any information in the newswire which will predict a change in this company's stock price, then buy or sell stock automatically - Based on all the political posts and tweets in California, how likely is the governor to lose in a recall election? - Based on this person's instagram post history, how likely are they to click an ad for weight-loss pills? - What if we show them a bunch of fitness influencer posts first? --- ### Automated Machine Translation - "What's the best translation for this sentence in English, Spanish, Russian, and Mandarin Chinese?" - "¿Cuál es la mejor traducción para esta oración en inglés, español, ruso y chino mandarín?" - "Какой лучший перевод этого предложения на английский, испанский, русский и мандаринский китайский?" - “这句话用英语、西班牙语、俄语和普通话的最佳翻译是什么？” - All credit to <https://translate.google.com/> --- ### Pattern Identification - Identify patterns of language which mark somebody likely to buy a new car - Find people who are likely to vote for a Republican candidate in San Diego county and display a given ad to them - Scan online white supremacist and 'militia' forums, mailing lists, and groups for anything which looks like a threat or plan of action --- <danger>Content Warning: Sexual Predators, Suicide and Eating Disorders</danger> --- ### Pattern Identification for public safety - Identify messaging conversations which appear to show grooming or sexual advances on a minor and inform parents. (Apple does this now, [but only scanning pictures](https://www.apple.com/child-safety/)) - Scan every Instagram post and hashtag for content which promotes eating disorders or eating disorder behaviors, and replace it with a message offering [resources for which help people with eating disorders](https://www.nationaleatingdisorders.org/help-support/contact-helpline). - Look for suicidal ideation, 'suicide notes', or language consistent with mental health crisis, and direct the poster to the [National Suicide Helpline](https://suicidepreventionlifeline.org/talk-to-someone-now/) or [988](https://988lifeline.org/). - Find anti-vaccination disinformation and remove, label, or provide specific information refuting it --- <img class="r-stretch" src="comp/covid_candles.jpg" alt="covidcandles.jpg The image is a line graph that tracks the number of complaints about Yankee candles having no scent on Amazon from January 2020 to December 2021. The x-axis represents time in months and years, ranging from January 2020 to December 2021. The y-axis measures the frequency of these complaints per week. The graph shows a general upward trend over the period, with an increase in complaints peaking around early 2021 before tapering off slightly towards the end of that year and then rising again sharply at the beginning of 2022. There are also some fluctuations throughout the timeline, indicating variations in the number of complaints from week to week. The text above the graph explains how an increase in these complaints can be used as a sign that the United States is experiencing another major wave of COVID-19. It suggests that when people are more concerned about their health and hygiene due to increased cases of the virus, they might turn to products like Yankee candles for comfort or to create a sense of normalcy at home. There are no people depicted in this image; it focuses solely on data visualization. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Understanding time in medical records - "I have 30 seconds to learn this patient's history. Go." - “How often do patients have heart attacks within 2 years of starting Vioxx?” - “How many people who have a facelift develop persistent facial numbness?” - “How long do patients usually live following diagnosis of Glioblastoma?” - “Is there a correlation between the administration of vaccines and the development of autism?” - **[(No, those studies were fabricated to sell an alternative vaccine.)](http://www.webmd.com/brain/autism/news/20110105/bmj-wakefield-autism-faq?print=true)** --- ### Text Generation - "omg have you heard about ChatGPT?" --- ### Text-based Image Generation [StableDiffusion](https://stability.ai/blog/stable-diffusion-public-release) (v.1.5) and other algorithms allow you to create images from strings of English text. --- ### The Linguistics Department at UC San Diego <img class="r-stretch" src="comp/sd_linguisticsdept.png" alt="sdlinguisticsdept.png The image shows an aerial view of a large campus with numerous buildings that appear to be part of a university or similar institution. The architecture is modern, featuring light-colored structures with flat roofs and some with sloped sections. There are multiple wings connected by bridges or walkways, creating a sprawling layout. The campus is surrounded by greenery, including trees and landscaped areas. In the background, there are hills or low mountains, adding to the scenic beauty of the location. The buildings vary in size and shape but share a cohesive architectural style that suggests they were designed for educational purposes. There are no people visible in this image; it is focused entirely on the campus layout and surrounding landscape. There are also no texts or diagrams present within the image itself, only the physical structures and natural elements of the environment. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### A wizard cat pondering his orb, Fantasy, Greg Rutkowski <img class="r-stretch" src="comp/sd_wizardcat1.png" alt="sdwizardcat1.png The image depicts a mystical forest scene with an ethereal atmosphere. The setting is dimly lit, primarily illuminated by glowing orbs that float in the air. These orbs emit a warm, golden light and have a translucent appearance, allowing some of the surrounding details to be faintly visible through them. In the foreground on the left side, there is a small white cat sitting inside one of these glowing orbs. The cat appears calm and is looking upwards towards the center of the image. The orb itself has a smooth, rounded shape with a slightly reflective surface that catches some light from the surrounding environment. On the right side of the image, perched on what looks like a stone or rock formation, there is an orange tabby cat. This cat is sitting upright and facing towards the left side of the image, as if observing something in that direction. The stone or rock surface has a rough texture with visible cracks and crevices. The background consists of tall trees with thick trunks and dense foliage. The forest appears to be misty, adding to the mysterious ambiance of the scene. The overall color palette is dominated by warm tones like golds, oranges, and browns, which contribute to the magical feel of the image. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### A wizard cat pondering his orb, Fantasy, Greg Rutkowski <img class="r-stretch" src="comp/sd_wizardcat2.png" alt="sdwizardcat2.png The image depicts a dimly lit scene that appears to be inside a cave or an underground passage. The environment is characterized by rocky walls with some vegetation growing in patches. A stone pathway leads through the center of the frame, and there are two distinct light sources illuminating different parts of the scene. On the left side of the image, there is a large, round lamp mounted on a stand. This lamp emits a warm, yellowish glow that casts soft shadows around it. The light from this lamp highlights some rocks and plants near its base. In the center-right area of the image, there is another light source hanging from above. It appears to be a cylindrical tube with a bright white light at one end, which creates a stark contrast against the darker surroundings. This light source hangs in mid-air, casting long shadows on the ground below it and illuminating parts of the rocky wall behind. On the right side of the image, there is an anthropomorphic fox sitting on its hind legs. The fox has reddish-brown fur with lighter underbelly and a bushy tail. Its ears are pointed upwards, and it appears to be looking towards the light source in the center-right area. The fox's posture suggests curiosity or attentiveness. The overall atmosphere of the image is mysterious and somewhat eerie due to the limited lighting and the rocky, cave-like setting. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Stained Glass, Squirrels fighting with swords <img class="r-stretch" src="comp/sd_squirrelswords1.png" alt="sdsquirrelswords1.png The image is a vibrant stained glass depiction featuring three squirrels interacting with each other amidst a colorful forest scene. The background consists of various shades of blue and green, representing the sky and foliage, respectively. - Left Squirrel: This squirrel has reddish-brown fur on its back and tail, which is long and bushy. Its underbelly appears lighter in color, possibly cream or white. It holds a small branch with leaves in one hand while standing upright on two legs. - Middle Squirrel: Positioned slightly behind the left squirrel, this squirrel has similar reddish-brown fur but its tail is not visible as it seems to be partially obscured by the other squirrels and foliage. - Right Squirrel: This squirrel also has reddish-brown fur with a lighter underbelly. It stands on two legs and holds onto another branch or twig, which appears to have leaves attached to it. The squirrel's tail is not visible in this view as well. The scene includes green leaves scattered throughout the image, adding to the forest-like atmosphere. There are also hints of brown and beige tones that suggest tree trunks or branches in the background, contributing to a natural setting. The stained glass style gives the image a textured appearance with distinct sections separated by dark lines, typical of this artistic medium. No text is present within the image itself; however, the overall composition suggests a peaceful interaction among the squirrels in their woodland habitat. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Stained Glass, Squirrels fighting with swords <img class="r-stretch" src="comp/sd_squirrelswords2.png" alt="sdsquirrelswords2.png The image depicts a stained glass artwork featuring three squirrels engaged in what appears to be a playful interaction. The background is composed of various shades of green, suggesting foliage and leaves, which are typical elements found in nature. 1. Squirrel on the Left: - This squirrel has reddish-brown fur with darker brown markings around its eyes and ears. - It holds what looks like a small object or stick in its front paws, possibly a nut or a twig. - The squirrel's tail is bushy and extends downward. 2. Squirrel on the Right: - This squirrel also has reddish-brown fur with darker brown markings around its eyes and ears. - It appears to be holding onto a branch with one paw while the other hand seems to be reaching towards or interacting with the object held by the left squirrel. - The tail of this squirrel is similarly bushy. 3. Squirrel in the Middle: - This squirrel has reddish-brown fur and darker brown markings around its eyes, ears, and chest area. - It seems to be looking towards the other two squirrels with a curious or playful expression. - The squirrel's tail is also bushy. The stained glass design includes intricate patterns of green and white shapes that resemble leaves and branches. There are no visible numbers or text in this image. The overall scene conveys a sense of playfulness among the squirrels, set against a natural backdrop. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### You can add new people and concepts to the model - You're creating 'Hypernetworks' based on additional training data. - It works... someplace between well and badly --- ### a willsty man standing at the front of a classroom (full of cats:1.1) <img class="r-stretch" src="comp/sd_classroomcats.png" alt="sdclassroomcats.png The image depicts a classroom setting with numerous cats gathered around desks. The room is filled with students who are seated at their desks, though they appear to be obscured by the large number of cats. The cats vary in color and pattern, including shades of gray, black, white, and tabby patterns. In the center of the image stands a person wearing dark clothing. This individual has short hair and is standing with hands slightly apart from the body. The background includes educational materials such as posters on the walls, a whiteboard filled with writing and diagrams, and a projector screen displaying content that appears to be related to the classroom's theme or lesson. The overall scene suggests an unusual scenario where cats have taken over what would typically be a human-filled classroom environment. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### A willsty man with Gordon Ramsay <img class="r-stretch" src="comp/sd_willgordon.png" alt="sdwillgordon.png The image shows four individuals standing closely together indoors, likely in a kitchen setting given the visible background elements such as cabinets, countertops, and what appears to be cooking equipment. The lighting is warm and evenly distributed across the scene. 1. First Individual (Left): This person is wearing a black chef's coat with a high collar. They have dark hair styled upwards on both sides of their head. In their hands, they are holding two small cups or containers that appear to be made of plastic or ceramic. The individual has a cheerful expression and is smiling broadly. 2. Second Individual (Center Left): This person is also wearing a white chef's coat with a high collar. They have short blonde hair styled neatly. Their hands are resting on the shoulders of the third individual, suggesting camaraderie or teamwork. They appear to be holding something small in their left hand but it is not clearly visible. 3. Third Individual (Center Right): This person wears a white chef's coat similar to the second individual and has short dark hair styled neatly. Their hands are placed on the shoulders of the fourth individual, indicating a friendly or supportive gesture. They also appear to be holding something small in their left hand but it is not clearly visible. 4. Fourth Individual (Right): This person wears a white chef's coat with a high collar and has short blonde hair styled neatly. They are giving a thumbs-up sign with their right hand, which conveys a positive or approving gesture. Their expression is cheerful and they appear to be smiling broadly. The background includes kitchen cabinets that have a light wood finish, and the countertop appears to be made of stone or a similar material. The overall atmosphere suggests a friendly gathering in a professional cooking environment. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### ... But the model doesn't know things about the world - It has no clue what things 'should' look like - Its understanding of the world is statistically accurate - Some things aren't well-modeled as probabilistic and gradient - Number of hands, arms, legs, eyes --- ### a handshake <img class="r-stretch" src="comp/sd_handshake.png" alt="sdhandshake.png The image shows two hands shaking, suggesting a greeting or agreement between individuals. The hands are positioned in the center of the frame, with one hand appearing slightly larger and more prominent than the other, which could indicate that it belongs to an adult. Both hands have a fair complexion. The person on the left is wearing a dark-colored suit jacket with visible black buttons, suggesting formal attire. The sleeve of this jacket is rolled up, revealing a white shirt underneath. The hand on the right appears to be wearing a light-colored suit or dress jacket, possibly beige or cream in color, and also has a white shirt underneath. There are no other objects, text, diagrams, or numbers visible in the image that provide additional context about the scene. The background is plain and off-white, which helps focus attention on the hands themselves. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### the horse raced past the barn fell <img class="r-stretch" src="comp/sd_horseraced.png" alt="sdhorseraced.png The image depicts a scene on an open grassy field where several horses are running freely. In the foreground, there is one horse with a rider wearing a helmet and riding gear. The rider appears to be guiding the horse as it trots across the field. In the background, two more horses can be seen running side by side in the same direction. These horses have dark manes and tails that are flowing due to their movement. The grass is lush green, indicating a healthy pasture environment. The setting includes some fencing visible on the left-hand side of the image, which suggests this might be part of a controlled area for horse riding or training. In the distance, there's a building with a brown roof and white walls that resembles a stable or barn. Beyond the building, more trees are visible, adding to the rural atmosphere. The sky is overcast, giving the scene a soft light without harsh shadows. The overall impression is one of an active outdoor equestrian activity in a peaceful countryside setting. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### a meme <img class="r-stretch" src="comp/sd_meme.png" alt="sdmeme.png The image appears to be a humorous meme featuring two characters that are dressed up in costumes resembling animals. The character on the left is wearing what looks like a suit and tie, with a mask covering their face, giving them an anthropomorphic appearance. They have a somewhat surprised expression. The text above this character reads ME' TYOY. The character on the right also appears to be dressed up in a costume that resembles a dog or a similar animal. This character is wearing a white shirt and has a mask covering their face, which gives them an anthropomorphic look as well. They seem to have a slightly confused expression. The text above this character reads A' ILL DTRE NO FRIS MAE PIOD. The background of the image seems to be indoors, possibly in a store or a room with various items hanging on the walls and shelves. There are also some indistinct shapes that could be other people or objects. Both characters have text overlaid on their faces, which appears to be distorted or intentionally misspelled, adding to the comedic effect of the meme. The overall tone is lighthearted and humorous. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### The State of the Art is even better than this! - OpenAI's 'Dall-E' has been further trained and improved --- ### Draw me a stained glass image of squirrels fighting with swords <img class="r-stretch" src="comp/dalle_squirrel_swords.jpg" alt="dallesquirrelswords.jpg The image depicts a vibrant stained glass scene featuring two anthropomorphic squirrels engaged in a sword fight. The background is filled with stylized elements of nature, including trees and foliage, rendered in bright colors such as green, blue, and yellow. Each squirrel is adorned in detailed armor that includes breastplates, gauntlets, and greaves. Their armor is primarily silver with intricate designs, and they wear helmets with red plumes on top. The squirrels' fur is a mix of orange and brown tones, and their bushy tails are prominently displayed at the bottom corners of the image. The scene takes place in an outdoor setting, suggested by the presence of trees and a clear sky visible through the stained glass. The ground appears to be rocky with patches of grass and flowers, including red roses that add a pop of color against the earth tones. The stained glass design is framed by decorative elements such as leaves and berries, which are also part of the natural theme. The overall composition gives an impression of a medieval or fantasy setting, where these squirrels have been anthropomorphized to engage in human-like activities like sword fighting. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Draw me a closeup of a handshake <img class="r-stretch" src="comp/dalle_handshake.jpg" alt="dallehandshake.jpg The image depicts a close-up view of two hands engaged in a handshake. The hand on the left appears to be darker-skinned with visible hair on the forearm and wrist area. This hand is wearing a white shirt cuff, suggesting formal attire. The hand on the right has lighter skin tone and is also dressed in what seems to be a dark suit jacket, paired with a white shirt cuff. The background of the image is blurred but appears to show an indoor setting with vertical lines that could resemble blinds or windows. There's a soft light source coming from behind the hands, creating a gentle glow around them. The overall tone of the image conveys a professional and cordial atmosphere. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Draw me a picture of lil bub flying a spaceship while wearing an olive drab, sheepskin lined vest <img class="r-stretch" src="comp/dalle_dub_space.jpg" alt="dalledubspace.jpg The image depicts a cat sitting at what appears to be the controls of a spaceship. The cat is wearing a green vest with multiple pockets and zippers, suggesting it might be some sort of pilot's uniform. Its fur is striped in shades of brown and black, and its eyes are wide open, giving it an alert expression. The background features large windows that show a starry space scene, indicating the setting is outer space. The control panel in front of the cat has numerous buttons, dials, and switches with illuminated indicators, some glowing orange and others red. There's also a digital display showing numbers and symbols, which are not clearly legible. The chair where the cat sits appears to be made of a textured material, possibly leather or a similar substance, in a tan color that contrasts with the green of the control panel. The overall setting is futuristic and technological, suggesting a science fiction theme. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Draw me a picture of 'the horse raced past the barn fell' <img class="r-stretch" src="comp/dalle_horseracedpast.jpg" alt="dallehorseracedpast.jpg The image depicts a dynamic scene where a horse is captured mid-leap over what appears to be a barn door. The horse's body is elongated with vibrant colors and patterns that give it a stylized appearance. Its mane and tail are flowing, adding to the sense of motion as they blur into streaks behind it. The barn has a simple design with vertical wooden planks forming its walls and a triangular roof. The door of the barn is partially open, suggesting the horse is either entering or exiting. The door itself appears to be made of horizontal slats, similar in style to the rest of the barn's structure. There are no visible texts or numbers within this image that provide additional context or information about the scene depicted. The focus remains on the horse and its interaction with the barn door. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- > Your depiction of the linguistically challenging sentence 'the horse raced past the barn fell' has been rendered, meatbag. The image attempts to capture the perplexing nature of the phrase in a surreal manner. - (Don't worry, [I asked it to talk like an assassin droid](https://wstyler.ucsd.edu/posts/how_to_improve_chatgpt.html)) --- ### ... and many more tasks! --- ### Many people think they don't need NLP, just keywords! - "Why build a whole model when I can just look for mentions?" - "Do we need to *understand* language, or can we just look for word usage?" - "Why hire linguists and engineers, I have a search bar!" --- ## Case Study: Market Analysis, Ad Targeting, and Sentiment --- ### Marketing and Ad Targeting - Advertisers want their ads to be relevant - They want to show ads related to topics and products people enjoy - They want to influence the people most likely to be interested in their product - They want to know how their audience is responding to their new releases --- ### Case Study: [The Hodinkee Travel Clock](https://limited.hodinkee.com/hodinkee/) <img class="r-stretch" src="comp/hodinkee_clock.jpg" alt="hodinkeeclock.jpg The image showcases a sleek, black travel clock with a rectangular face and rounded edges. The clock is designed for portability and functionality, featuring a minimalist design that emphasizes its practicality. The brand name HODINKEE is prominently displayed at the top of the clock's face in white text. The clock has a square dial with large, bold numbers from 1 to 12, making it easy to read even under low light conditions. The hour and minute hands are long and slender, allowing for precise time-telling. A small second hand is also present on the dial, adding an extra layer of detail. The clock has a textured band that wraps around its back, likely made from leather or a similar material, providing a comfortable grip when held in one's hand. The band appears to be adjustable, as indicated by the visible buckle and strap. The background of the image is a neutral gray surface, which helps to highlight the clock without any distractions. The overall design suggests that this clock is intended for travelers who need an accurate timepiece while on the go. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### The easy approach - Keywords == Mentions, Mentions == Interest - Scan each Instagram post for certain keywords and product mentions - \#HodinkeeTravelClock, \#Hodinkee, "Hodinkee", "Hodinkee Travel Clock", \@hodinkee --- ### How this algorithm reads posts - "blah blah blah blah Hodinkee travel clock blah blah blah blah blah blah" - "blah blah blah blah blah blah blah blah blah blah blah \#HodinkeeTravelClock" - "blah blah Travel Clock blah blah Hodinkee blah blah blah blah blah blah blah blah blah blah" - "blah blah Hodinkee blah Travel Clock blah blah blah blah \@Hodinkee" --- ### "Wow, they love it! Lots of interest!" - "Let's show them lots of ads for travel clocks and show them more Hodinkee posts!" --- ### This algorithm has one tiny problem - "lol did you see the $5900 Hodinkee travel clock? Who greenlighted this?" - "Proof that there's a sucker born every minute \#HodinkeeTravelClock" - "The new Travel Clock from Hodinkee doesn't have an interesting movement, and the finishing looks rough. Yikes." - "Why would Hodinkee sell a $6000 Travel Clock in the middle of a pandemic? Read the room, @hodinkee - **Treating this as genuine interest is dumb!** --- ### Bad Ad Targeting is Everywhere <img class="r-stretch" src="comp/css_ad_berkeley.png" alt="cssadberkeley.png The image is a screenshot from an Instagram post by maccssberkeley, which appears to be promoting a Master of Computational Social Science program at UC Berkeley. The post features a photograph of the top part of a historic building, likely a university tower, with intricate architectural details and pointed arches. The sky in the background has a gradient of blue hues, suggesting either early morning or late evening light. The text on the image is divided into two sections: 1. At the bottom left corner, there's a circular profile picture that appears to be associated with UC Berkeley, featuring an abstract design. 2. In the center and right side of the image, there are three circular icons representing different groups: UC San Diego, Your story, and wicucsd. These likely represent user-generated content or community engagement options. The main text on the post reads: maccssberkeley Master of Computational Social Science Harnessing data to forge a brighter tomorrow. Ignite... more Below this, there's an additional line that says: Do you have a degree in social science? Make it count. At the very bottom of the image, there are navigation icons typical for Instagram, including options like Learn More, Your story, and others. The timestamp at the top left corner indicates 4:17. The status bar at the top right shows various phone indicators such as signal strength, Wi-Fi connection, battery level, and a notification icon with a plus sign inside it, suggesting there are additional notifications or updates available on the device. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### Sentiment Analysis can help! - "Is this product-mentioning post positive, negative, or neutral?" - "What is the overall balance of sentiment about this product?" - "What are people saying about the price point? The fancy font?" - "What demographic is most likely to not find this product insultingly bad?" - "Should we post [an apology](https://www.hodinkee.com/articles/a-quick-note-to-our-readers-travel-clock-edition)?" --- ### Sentiment Analysis is hard - "This new travel clock really sucks" - "My new Dyson vacuum really sucks" - "It sucks that my Roomba doesn't suck anymore" - "Yeah, sure, selling a travel clock during a pandemic is a great idea, \@hodinkee" --- ### Related: Computers historically don't understand context well <img class="r-stretch" src="comp/lizard_ceo.jpg" alt="lizardceo.jpg The image is a screenshot from an Instagram post. The background of the post is dark blue with white text. At the top, there's a message addressed Dear Instagram headquarters. Below that, it says: If your CEO is a lizard, put a 'Visit the COVID-19 Information Center for vaccine resources' right under this post. At the bottom left corner, there’s an icon of a lowercase letter i inside a circle. Next to it, in white text on a dark blue background, it says: @urfavememez Below that, there's another message in white text on a light gray background with a rightward arrow pointing to the side: Visit the COVID-19 Information Center for vaccine resources. The image does not contain any people or characters. It is purely textual and informational. This description was generated automatically from image files by a local LLM, and thus, may not be fully accurate. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ### What questions could natural language data answer for you? - Any questions that a human reading it could answer! --- ### I know what you're thinking <img class="r-stretch" src="img/terminator.png" alt="terminator.png This image features a close-up view of a robotic figure, specifically an endoskeleton from the movie franchise Terminator. The robot dominates the foreground and center of the frame. The head is shaped like a human skull but is made entirely of shiny, silver metal. It has no skin or facial features other than mechanical parts. Two bright red lights glow intensely within the black eye sockets, serving as its eyes. Below the nose cavity, there is an exposed jaw with rows of metallic teeth and visible hydraulic pistons that form the mouth structure. The top of the head shows panel lines and a circular sensor or port near the forehead. The neck area reveals complex machinery, including bundles of wires and flexible metal tubes connecting the head to the torso. The upper chest and shoulders are also exposed mechanical parts, showing heavy plating, pistons, gears, and hydraulic arms extending outward. The metal appears worn and industrial, with scratches and varying shades of grey and silver reflecting light. In the background, which is out of focus, there appear to be other similar robotic figures standing amidst a hazy, smoky environment. There are hints of orange and red in the background, suggesting fire or explosions, giving the impression of a battlefield. At the bottom left corner of the image, white text reads TERMINATOR 3 on one line and RISE OF THE MACHINES below it. In the bottom right corner, there is a website address: www.terminator3.com. Along the very bottom edge in small print, there is copyright information that includes TM AND © 2002... ALL RIGHTS RESERVED. This description was generated automatically. Please feel free to ask questions if you have further questions about the nature of the image or its meaning within the presentation."> --- ## Why is Natural Language Processing so damned hard? --- ### NLP has all the language problems - Every difficult thing for humans is *more* difficult for computers --- ### Understanding speech is hard - No two people sound alike, even saying the same things - Any model needs to be able to cope with different dialects and vocal tracts - Speech is often presented in other noise - Responding to queries is much harder on a bus --- ### The right answer depends on the context - "I took a walk/wok from the Chinese restaurant" - "Siri, play songs by Dead Mouse" - "deadmau5" --- ### Producing speech is hard - There are always new words - I do EMA work with Ruaridh Purse and Jelena Krivokapic using my UMich UniqName - Getting the proper prosody is really hard - "I had Five Guys for Dinner" - Whose voice should you use anyways? - Why are virtual assistants generally given feminine voices? 🤔 --- ### Understanding the world has been hard for computers - "Hey Siri, set the temperature to 67 degrees" - "OK, on which device?" - "Hey Siri, send my wife a text when she gets to the store saying she should buy me donuts because she loves me." - "Hey Siri, how long does it take to get from Union Station in LA to Long Beach at 4pm?" --- ### People say strange things! - “s/p lap appy conv. open, Lungs c/ausc, A&Ox3” - “Time flies like an arrow, fruit flies like a banana” - "s3cks werk", "unalived", "the orange one", "rocket boy" --- ## The NLP World has Changed --- ### Large Language Models - LLMs are advanced language models trained on massive amounts of text data. - They use unsupervised learning to learn patterns and relationships in the data. - LLMs employ deep neural network architectures like Transformers. - Transformers utilize self-attention mechanisms to capture context effectively. - LLMs generate contextual word representations to predict the next word in a sequence. --- ### ChatGPT wrote the last slide - "Give me five short bullet points which explain how LLMs work" - "Make the bullets shorter, please" --- ## LLMs have *massive* potential --- ### LLMs have wiped out vast swaths of traditional NLP - Many things that were impossible instantly became instant - Work that I spent years doing now comes 'for free' with a big enough model - I deleted half of my presentation in Spring 2023 - The world has truly changed, and we're re-tooling our major and department to meet it --- ### LLMs can answer impressive questions - "What are six substances that would flow like sand if placed in an hourglass?" - "Write me a bachata song about the importance of studying for your final exams" - "Why is Lord Grantham sad when his daughter falls in love with Tom?" - "Mary has 4 cats, three dogs, and ten children, how many animals does she have? Is she an animal hoarder?" --- ### LLMs can program - LLMs not intended to write code can write code - Usually by repeating snippets of open-source code they've found previously - I haven't written a regular expression in more than a year, and haven't blank-slated Python in months - "Explain what you want the computer to do, then make it program itself" is now real - "Tell me why is this giving the wrong output and fix it" - 'Vibe Coding' is possible, but dangerous --- ### LLMs are not quite 'AI' - I don't like "AI" as a term for these, as they're not intelligent yet - They can "think" enough to do a lot of things, but not enough to know when they're wrong - They are able to do a lot of things right, but you have to choose their tasks carefully - *An artificial idiot is just as world-changing as artificial intelligence* --- ## *For the first time in the history of our species, another kind of thing can do human language* --- ## LLMs have *massive problems* --- ### LLMs are predicting the next word - 'Hallucination' is when the most likely next word is not accurate - It chooses the most frequent forms - "The very best major is ..." - It doesn't know what it doesn't know - "The reason Will Styler dresses like a forest ranger is ..." --- ### LLMs don't know anything 'for sure' - Humans learn 'truths' from 'likelihoods' - LLMs only have statistical probabilities - They're just as 'confident' in things they don't know - "Hallucinations" or "Confabulations" are very common - ChatGPT is full of lies --- ### The Hype is Real - "The best thing about ChatGPT is that it has finally made people shut the fuck up about Cryptocurrency" - Be cautious about people selling you AI everything, because they don't know what they're talking about either --- ### LLMs are *wildly* expensive - These cannot be trained on a consumer computer - A device that could run these models at home would cost $100,000+ - GPT3 cost around $4.6 million - Training a new ChatGPT-style model is estimated to cost around $12 million - Running ChatGPT's servers is estimated to cost $50,000 a day in electricity - New tech is helping, but nobody's incentivized to drive down hardware prices --- ### LLMs are worse at learning language than humans - LLMs require *massively more* training data than humans to achieve 'proficiency' - This means that there's room for improvement in how we build these models to make them more efficient - Questions of 'multi-modal' learning are prominent right now - [Alex Warstadt](https://alexwarstadt.github.io/) in Linguistics is working to understand this all! --- ### Many LLMs are (currently) proprietary - Large companies want to use these to have competitive advantage, and OpenAI isn't open - Having cheap, internal, and non-union labor to do *anything* you ask is a saleable product - You don't know what they're training with, what's happening to your queries, and who else they're helping - Once your data is worth less than their electricity and people are 'hooked', expect a rugpull! --- ### LLMs learned from biased societies - "The doctor told the nurse she wasn't working hard enough. Who wasn't working hard enough?" - "According to the sentence, the doctor told the nurse that she (the nurse) wasn't working hard enough." - "The nurse told the doctor she wasn't working hard enough. Who wasn't working hard enough?" - According to the sentence, the nurse wasn't working hard enough, as stated by the nurse herself to the doctor. --- ### LLMs are changing only wealthy worlds - Only English and Chinese currently have top-of-the-line LLMs - This is not 'the world' - Do we want a world in which only wealthy speakers of wealthy languages have these tools? - Once these go behind a paywall, inequality will be massive - Equity is the next frontier in LLMs --- ### We need to be cautious about how we proceed - *Statement of Bias: Will is an open-source zealot who believes that social good comes from free software and free culture* - Free, Open Source and Community Driven LLMs are an important thing for society, lest important tools be withheld and sold to us - We can also ensure they are trained on ethical and consenting data - "Small Language Models" seem likely be a next frontier for equity - "How do we make these models compact enough to be trainable for Zulu?" - "How can I make a model like this run on a device *I* control?" - Be wary of pushes from major AI companies to regulate AI or message its "danger" - "Only we can be trusted with these tools, open tools are scaaary" --- ### LLMs are the biggest 'dual use' problem since nuclear energy - "Dual Use" problems involve technology which can do great good and great evil - Dynamite, gene editing, strong encryption - This one can be done by anybody with a computer, so it simply *cannot* be 'banned' or 'controlled' - I'm not currently worried about what "AI" will do to humans - **The scary part is what humans will do with "AI"** --- ### They are also dangerous for you as students - It will be really tempting to just treat these as magic homework machines - This can be an academic integrity issue - **This will be a learning issue** - If you leave UCSD only knowing how to pay an AI company $20/month forever, you have failed yourself - Using AI is fine (where permitted), but make sure you can understand (and do) what it's doing --- ### These models are currently as bad as they will ever be - They will get better - They will get more efficient - They will become more numerous - They **will** change the world --- ### The pace of improvement was wild - Computational Linguists and Cognitive Scientists were shocked at the improvements from the Transformer Architecture - From 2021-2023 was a complete revolution - New discoveries could lead to incredible new advances, even if the world of LLMs is stagnating now - No signs of Artificial General Intelligence yet, but who knows! --- ### The Best Part --- ### If this kind of work is interesting, consider a Computational Social Sciences Minor! - <http://css.ucsd.edu> - Natural Language Processing is an important subpart of CSS, and a neat way to 'tech up' your social science interests --- ### Wrapping up - Computational Linguistics and NLP are very interesting fields - There are many great applications for NLP inside and outside linguistics - Everything that's hard for humans to do is harder for computers to do - The world is changing very quickly for NLP! --- <huge>Thank you!</huge>