Will Styler's Homepage
Will Styler

Associate Teaching Professor of Linguistics at UC San Diego

Director of UCSD's Computational Social Science Program

A Reference for Course Design

Will Styler - Revised Fall 2022

Note: This document is evolving and living, and although it’s targeted towards linguists, it may be valuable to others. Check back for updates, and email for typoes. Thanks to the LIGN 287 Students in Fall 2022, for whom this resource was designed, for their assistance and additions to the work here.

Learning Outcomes

This is a brief listing of some types of Learning Outcomes. Not exhaustive nor principled, just a synthesis of existing ontologies.

Knowledge/Fact Outcomes

Pieces of information about the world, about concepts, or about relationships between concepts. Examples could include:

Algorithmic/Analytical Outcomes

Knowledge of processes, methods, or algorithms relevant to a field of study. Examples could include:

Motor Outcomes

Knowledge of how to perform specific physical actions required to succeed in the field. This generally involves interaction with something in the physical realm. Examples could include:

Metacognitive or Attitude Outcomes

Changes in a student’s attitude or understanding of their own thinking towards material in the course. Examples could include:

Emotional Outcomes

Changes in a student’s emotional state regarding the material in the course.

Value-based Outcomes

Changes in a student’s values or moral approach to the world coming about from the course.

Threshold Concepts

Definition

‘Threshold Concept’ is a term originally from Jan Meyer and Ray Land in 2003. Here’s the Publication. Here’s another great resource.

Threshold concepts are generally considered to have the following properties:

Strategies for Addressing Threshold Concepts

Learning Modalities

What are different ways to teach material? What are their pros and cons?

Lectures

Lecturer or lecturers talk to students in large groups with majority of interaction being one-way.

Sections and Labs

Smaller groups of students (e.g. 25-40) in a dedicated TA-led session, with activities, interactive Q&A, or discussions.

Task Group Work

‘Let’s have everybody get into groups and complete this activity’. ‘Flipped classrooms’ often fit into this category.

Discussion-based Classes

Each day has a topic (or reading, theme), instructor introduces it and gives background, then people (generally in the 10-40 range) discuss, with instructor acting as a facilitator.

Seminar Model

Instructor loosely facilitates, or assigns students to facilitate, letting the group go where it pleases.

Guided Exercises/Workshops

Imagine conference workshops, generally involving a skill/algorithm, or guided activities found in a language classroom

Field Trip

Same as above, but offsite, in-situ

Independent Work

Students come to a place and work, and the instructor/TA floats and answers questions/offers help

Forced Independent Research

Homework or papers requires students to do the learning and report back. Group projects can fall into this category too.

Demonstrations

Instructor shows a task or process, students watch

Learning Communities

Everybody collaborates, some limited facilitation

Critique-based Classes

Students do work outside class, and class time is used to discuss and critique that work.

Course Scheduling

Generalized Course Scheduling Concerns

Course schedule matters. The structure of your course reflects structure in the learning, and should help both you and your students to organize information. A well scheduled course will…

Ways to think about Schedule Interdependence in courses

Most courses have some elements of the schedule which depend on others (that is, they are necessarily ordered with respect to one another). Each of these has pros and cons, and each class is someplace in the gradient here. I’m also presenting some TV shows which follow a similar pattern, as a sort of analogy.

Unordered, ‘Grab Bag’ Schedules

There is no dictated order for the course, and any week could be scheduled any week. Common for colloquium series or student research presentation courses.

This is like a TV show with strictly independent episodes and plotlines. Watching them out of order doesn’t matter, you miss nothing, and you can’t tell if you missed an episode. Think ‘The Simpsons’, or many episodic wildlife documentaries

Clustered Schedules

Imagine a course where weeks 1-3 cover Topic A, weeks 4-6 cover Topic B, weeks 7-9 cover Topic C. Within these topics there is strong internal order, but it’s not crucial whether Topic A comes before Topic B or vice versa. This works well with giving major concepts ‘a week’ in the schedule.

This is like a TV show with largely independent episodes, but some longer narrative arcs spanning groups of episodes, or a given season. If you’re watching out of order, you’ll be a bit confused in a few places or might have some minor ‘spoilers’, but it’s still mostly understandable and enjoyable. Think about the subplots within ‘The Office’, or perhaps different seasons of ‘Great British Bake Off’ or other reality shows

Strictly Cumulative Schedules

In this sort of course, every session depends on knowledge from prior sessions. Classes have a necessary order relative to one-another, which could not be reversed.

This is a TV show with a strong narrative arc, where missing an episode makes subsequent episodes very hard to understand. “Previously on…” is a necessity, and ‘spoilers’ are a very real thing. Most major drama shows are this way.

Course Scheduling Approaches

Your course can have many relationships to scheduling and building a greater narrative arc. Once you’ve figured out your course’s ordering type, you can choose any of the below ideas to organize your course (or your clusters). This is, again, not a complete ontology, just ones that I’ve encountered, and each of these could structure a whole course, or work within a cluster within a course

Genealogical Approach

In this structure, you organize information in terms of how it was developed in the field, either as a network of interrelated theories, or as a timeline. “First, this theory happened, which is week 1. Then, we’ll talk about this theory, which was a reaction to the first theory. From there, we developed…”.

Ontological Approach

Your structure itself provides a characterization of the types, natures, and relationships between the concepts present. “Our theme this quarter is theories in speech perception, and we’ll first look at motor-based theories for the first few weeks, then acoustic theories, then direct realist theories, then second language acquisition focused theories.” Or “Let’s look at the different ways that fear plays a role in human psychology, spending each week on a different kind of fear”

Metaphorical Approach

Your course uses a central, meaningful metaphor or analogy to structure the content. This could be something like ‘moving from smallest units of language (phones) up to the largest units (discourse)’, a language acquisition course which starts with the youngest ages and looks at progressively older kids through the term, or a class on the Silk Road which talks about participating countries from East to West during the quarter.

Step by Step Approach

For some particularly algorithmic tasks, there is truly a step-by-step way that the problem must be approached. For natural language processing, you might first think about corpus selection, then text formats, data cleaning, then tokenization, then language model construction, training, testing, and tweaking. This is a very plausible course structure, in and of itself.

Tick-tock Approach

Within the course, there’s a consistent, repeating structure and rhythm to how sessions are composed. Perhaps Monday and Wednesday are lecture based, and Friday’s session is an in-class data analysis. Or on Tuesdays, you talk about theory, and on Thursdays, talk about the theory’s application. The course for which this was originally written uses a tick-tock structure, with Tuesdays being workshopping of the work done over the weekend, and Thursdays introducing new material and concepts to be implemented over the weekend.

Storytelling Approach

Your structure is determined creatively, with the intent to produce a final course layout which is maximally engaging, entertaining, or ‘interesting’, even if the reasoning for a given ordering isn’t clear outside the instructor’s mind, and even if it may not be optimally efficient.

“Before and After” Approach

Your course is structured around a major paradigm shift, whether theoretical, analytical, or methodological. First, you do, analyze, or contemplate things ‘the old way’, then introduce the ‘new’, and proceed to re-cover the previously covered content through the lens of the new approach. The goal is for the students to experience the disruption, improvement, and change directly and intuitively. Think “Syntax before and after Chomsky”, “Rome before and after Julius Caesar”, or “Generative vs. Usage approaches to Phonology”

Spiraling approach

A variant of ‘before and after’, this structure repeatedly returns to the same set of questions, or conducts the same set of analyses, each time in a different context. The individual elements of changing context could be different theories or analytical frameworks, different language families, different cultures, each of which is subject to the same battery of examinations. Combined with opening and closing summaries, this is very powerful.

“Learn the thing, then do the thing”

In this structure, the first portion of the class is spent with detailed learning of a method, analysis, approach, or tool, and the second portion is spent almost exclusively on application, with a strong applied, hands-on, or lab-focused component. (Credit to Ben Lang for a nice title for this category)

Inheriting External Structures

Your course takes its ordering directly from an external structure outside of your control. This could be mirroring the required class ordering for your major, this could be mirroring a textbook, or integrating with another class or lecture series. This can also happen when many guest speakers are invited with schedules out of your control

Grading

Generalized Grading Concerns

Here are a few things to contemplate as you consider grading policies:

Grading Schemes

There are many ways to calculate grades in a course. Here are a few common ones, named as sensible to me:

Qualitative Grading

Instructors simply say ‘This is your grade’ based on some rough metrics or subjective impression of work and effort.

Single Assignment Grading

The quarter’s grade is based solely on a final exam, paper, or project. No other work is graded, or if it is, no other work counts.

Cumulative point-based Grading

Students earn points throughout the term on assignments, and the course grade is based entirely on the number of points earned out of the hypothetical maximum number. Each point of each assessment contributes equally to the final grade, and ‘weighting’ is done entirely by assignment point totals.

Percentage-based Grading

Every assignment is graded and only percentages are saved. Final grades are calculated as an average across percentages (within categories, or overall).

Distribution-based or ‘Curve’ Grading

Students are given a quantitative score (using one of the prior methods). At the end of the term, students are assigned a final grade which is derived by formula from the distribution of student grades. For example, percentiles could be mapped onto letter grades (such that only students in the 90%+ percentile are awarded ‘A’), or the top N students get A+, then the next N students get A, and so forth.

Contract-Based or Completion Grading

Students are expected to put in consistent effort, and they log their efforts. If students do this, and any checkpoint assignments, they are guaranteed a given grade. Points ‘lost’ or grade reductions are based instead on failure to complete work or attend.

Self Grading

Students are given rubrics or keys and proceed to complete and then grade their own work. The instructor then spot-checks some or all of these grades, with the goal being to identify students who are either being overly harsh to themselves, or are taking advantage by assigning a grade that’s higher than the rubric would allow.

Skills-based Grading

Instructors provide students with a complete list of ‘skills’ (applied learning outcomes) at the start of the quarter. Assignments are graded by marking which skills each student has demonstrated. Student grades are based on how many of those skills they’ve demonstrated by the end of the quarter.

Assignment Design

When designing assignments for your class, here are some basic ideas to consider:

Question Types

Below are some of the common question types you’ll find in many assignments. I’ve annotated them with letters corresponding to their type in Bloom’s Taxonomy Revisted. C = Creating, E = Evaluating, An = Analyzing, Ap = Applying, U = Understanding, R = Remembering.

‘Fill in the Blank’

Questions which ask students to give a string in response to a prompt, where there is a set of fixed, correct answers, any of which are expected to get full credit. Variations include paragraphs with a set of blanks.

Matching

Questions which provide a bank of answers, and ask students to associate answers with their correct question. Variants can include unused answers, multiple questions matching to the same question, or ‘fill in the blank’ with an answer key, where each is mapped to the proper place.

Multiple Choice

Single questions with a bank of possible answers. Variations include ‘Choose all that apply’, or the addition of ‘None of these’.

True/False

Single assertions which are evaluated by the student as being ‘True’ or ‘False’.

Short Answer (Open Ended)

Students are asked to provide a short, 1-2 sentence answer to a question, but there are many answers possible for full credit.

Longer Answer (Open Ended)

Students are asked to give, as a part of a greater assignment, a longer response, analysis, or creation, which is graded as a writing assignment, but is not ‘the whole assignment’. Variants include giving a ‘question bank’, where students are provided a list of 5-10 questions, one of which will be on the exam.

‘Show your work’ Questions

Students are asked to solve a problem or complete an analysis, but are forced to describe every step of the process, rather than simply submitting the answer. This is common in math classrooms, but also valuable as an anti-cheating method for other analytical disciplines.

Metacognitive Questions

Metacognitive questions ask directly about a student’s thought process regarding the material. These don’t always cleanly fit into Bloom’s Taxonomy, but often end up in ‘analyze’ and ‘evaluate’. Some example metacognitive questions are below:

Piecemeal Analysis Questions

Although having one ‘big’ question for an analysis can be valuable, it is considerably higher-stakes than having a question with many subparts, which break a larger algorithmic task into smaller components. This is, perhaps, more a way of asking a ‘show your work’ question, but has the advantage of making partial credit much easier to assign, and making ‘just ask a friend for the answer’ feel less workable. It is also valuable for guiding students through the process, although runs the risk of students struggling to generalize without this formatting.

Writing Assignments

Writing assignments generally involve little machinery, and simply prompt the students to share their written thoughts. This could come in the form of an essay based on course material, a full paper with independent research, position/reaction papers based on the material or discussions recently covered, and many more.

These assignments can address all levels of Bloom’s Taxonomy, as well as help students hone their argumentation skills. For more details on creating writing assignments, here’s a great page from University of Minnesota to that end.