LIGN 6 Final Project Writeup Rubrics

Will Styler, Revised Winter 2021

For a successful final project, you will be graded on the following dimensions, each contributing in varying degrees to your final grade. They are presented roughly in order of importance, and the four ‘Levels’ of mastery can be thought of as corresponding roughly to ‘A+’, ‘B’, ‘C-’, and ‘F’. Your overall grade for the project will be based roughly on your overall level across the categories, adjusting for other factors (unclarity, missing sections, exceptional or low effort, etc).

Project Proposal

Your project proposal should be around one single spaced page, plus or minus, providing and discussing the following information:

• Your name
• Who you’re working with
• What you’re planning to do in your final project
  • If you’re planning to do Option 1:
    • What’s your system designed to do?
    • What are some sample queries/commands? What answers should the system give?
    • What are a few of the complexities that make this different from what existing assistants do?
    • What do you think will be ‘the hard part(s)’ for your system?
  • If you’re planning to do Option 2:
    • What task do you want to work on?
    • What tool are you interested in using?
    • What kind of experience do you have with similar tools or languages? (e.g. explain why this is possible)
    • What kind of data do you want to use to test your tool?
  • If you’re planning to do Option 3:
    • What do you have planned?
    • What is the final product?
• Why is this topic interesting to you and your group members?

Grading Rubric

A Masterful proposal will…

• Discuss all of the elements above, providing concrete examples and deliverables
• Provide clear discussion, particularly in option 2 projects, of how you plan to implement this
• Lay out the steps required to complete this project in a clear outline
• Show evidence of careful consideration of the problems and difficulties involved.
• Show clear understanding of the challenges, particularly for option 2, and discuss the how the student(s) can meet them

An Acceptable proposal will…

• Discuss all of the elements above, but with fewer examples or direct deliverable items
• Provide discussion of how you plan to implement the project, but with some areas of vagueness or large missing steps
• Focus more on deliverables than the process of getting there, with little evidence of planning
• Feel a bit more rushed, or like a rough gesture in the direction, rather than a clear set of plans
• Show understanding of the challenges involved, but doesn’t show how the student(s) are equipped to meet them

A Novice proposal will…

• Discuss most or all of the elements above, but perhaps omitting some examples.
• Provide only vague information about the planned direction
• Promise an outcome with no discussion of the process
• Feel last-minute, under-planned, or show little effort.
• Be substantially shorter than the one page target length

A Way Off proposal will…

• Fail to discuss many or all of the elements above.
• Describe only a vague plan, with little differentiating information
• Give no details about the planned process
• Feel like little effort was spent in creating the proposal
• Be substantially shorter than the one page target length

Project Option 1: Design a System

Scope of Writeup

• Masterful: Discusses, indirectly or directly, the ‘Things to consider’ for all elements in the Virtual Assistant Interaction process guide in sufficient depth to demonstrate understanding of the difficulties (or simplicities) of their particular system, or, when a given step doesn’t apply, explain why. If additional steps are required, they are described in detail, and the ethical concerns in the process guide are addressed as well.
• Acceptable: Discusses all of the elements in the process, but in some areas shows lack of depth, missing aspects, failure to capture or discuss important complexities faced by their particular system.
• Novice: Fails to discuss some of the elements of the interaction process (without justification of why that step doesn’t apply), or shows repeated failures of understanding of the process and complexities.
• Way Off: Writeup focuses exclusively on one or two elements of the process without prior approval, or does not demonstrate understanding of the many intermediate steps in the NLP chain.

Demonstration of Knowledge

• Masterful: Student paper shows considerable and nuanced knowledge and attention to the material from the course, making regular reference to concepts from class and homeworks, fully engaging with the material taught in class, and potentially, goes beyond it into independent research. Additionally, the student shows some realistic understanding of the ease or difficulty of designing the system as described.
• Acceptable: The paper shows attention and engagement with the class materials, but occasionally neglects some important facts or discussions from the course, or occasionally shows confusion with the nature, functioning or limitations of some of the NLP processes discussed.
• Novice: The paper makes occasional reference to the concepts discussed in class, but repeatedly shows failures of understanding of the concepts discussed or their applicability to the system in design.
• Way Off: Writeup largely fails to demonstrate attention to or knowledge of the material from the course, instead focusing on non-NLP related elements or giving explanations so superficial as to be available even to those who haven’t taken LIGN 6.

Richness

• Masterful: Project describes a system with considerable interactional depth, discusses 5-10 different commands, involving back-and-forth interaction with the human, creation of a discourse context, and complexity of commands at or beyond current commercial offerings (e.g. Siri or Alexa). The writeup
• Acceptable: Project describes a system with some richness, but which primarily processes single complex commands and gives output, without considerable interaction or discourse. Or, the student focuses exclusively on one or two commands, without prior approval.
• Novice: Student describes a simple call-and-response system with low NLP complexity, where pre-specified commands are responded to with canned phrases or actions. (e.g. “Alexa, what is your name?” “My name is Alexa, thanks for asking.”)
• Way Off: The system described makes only limited use of natural language processing, or the system’s principal described complexities are outside of the NLP world (e.g. for a pizza ordering system, the bulk of the paper is spent discussing the robotic creation of the physical pizzas).

Formatting and Length

There is not a strong length requirement for this paper. I suspect you’ll land around 8-10 single spaced pages to do this assignment well, but I’ve seen papers which were excellent in 5 pages, and papers which were pretty awful even twenty pages deep. Note that papers submitted by groups will be expected to have a greater scope and detail than individual papers, so a 8 page paper from five people will be considered ‘suspiciously short’, but it’d be fine from a single person. Additionally, feel free to use APA or MLA formatting as a baseline, particularly for citations, and you should use hierarchical formatting (e.g. labeled sections, subsections, subsubsections), but I will not be grading on deviations from an arbitrary set of formatting laws, because we both have better things to do with our lives.

• Masterful: The paper is sufficiently long to demonstrate knowledge, richness of the system, and show proper scope, but without dragging or feeling like words are being added for the sake of talking. Formatting is reasonable, readable, and enhances the text. Citations, where given, are reasonably formatted according to an accepted standard, and contain the information needed to be followable.
• Acceptable: Paper length is sufficient, although perhaps a bit too brief or too wordy. Formatting is acceptable, if occasionally distracting or adding difficulties.
• Novice: Paper is so short as to appear rushed or ‘last-minute’ (or so long as to feel that the student is just ‘filling pages’ to distract from lack of knowledge, and/or the formatting is distracting, problematic, or hurtful to the argumentation.
• Way Off: The paper is typeset using ‘Comic Sans MS’ or ‘Papyrus’ fonts, is presented without formatting or sectioning, citations are missing crucial information, or is so short as to be completely unable to address the necessary scope.

Structure and Organization

Please structure your paper with sections, subsections, and subsubsections (where needed), to make things easier both in reference and readability. Don’t worry about ‘transitions’, just as long as the organization is OK.

• Masterful: The paper is well organized, with the introduction discussing the structure of the paper to guide the reader. Sections are numbered and cross-referenced throughout the paper where needed. Transitions between sections are clear, and the sections make sense. (Hint: The Interaction Process guide is a pretty reasonable outline!)
• Acceptable: Structure of the paper is acceptable, if somewhat difficult to understand or poorly explained. The reader has a rough understanding what’s going on, even if it’s never explained.
• Novice: The paper’s structure is counterintuitive and not understandable, with the structure (or lack thereof) hurting the argumentation considerably.
• Way Off: The paper has no discernable structure nor organization.

Language and Argumentation

• Masterful: Student language use is consistently clear and understandable, with a reasonable level of formality for academic writing. Additionally, the paper was clearly proofread, with typoes and English language grammar issues relatively rare and not considerably affecting the argumentation.
• Acceptable: Student language use is largely understandable, but in places unclear, or with a sufficient number of typoes or English language grammar issues as to start to hinder my understanding of the argumentation.
• Novice: The argumentation is regularly unclear, and my ability to understand the content of the paper is considerably hindered by typoes or English language grammar issues.
• Way Off: The paper is not written in English, is largely incomprehensible, or shows evidence of automatic machine translation.

Academic Integrity and Citation

Note that plagiarism or other academic integrity issues will result in an automatic ‘0’ on the paper.

• Masterful: All direct quotes or external references are cited, with the source given clearly in a ‘Works Cited’ section, in the format of your choice (APA is a fine default) including the necessary information to track down the resource. All other words are your own.
• Acceptable: All direct quotes or external references are cited, but the citations do not include enough information for the reader to locate the original resource.
• Novice: All direct quotes or external references are cited, but the citations do not include additional information.
• Way Off: Sources are not cited, or there is evidence of academic dishonesty. Be careful to cite all your sources and quotes (outside of the class material).

Project Option 2: Implement some NLP

Scope of Writeup

• Masterful: Discusses, indirectly or directly, all ten elements of a successful project option 2 writeup described in the course syllabus guide in sufficient depth to demonstrate their engagement with that particular system, or, when a given step doesn’t apply, explain why. If additional steps are required, they are described in detail.
• Acceptable: Discusses all of the elements, but in some areas shows lack of depth, missing elements, failure to capture or discuss important complexities faced by their particular system.
• Novice: Fails to discuss some of the elements of the interaction process (without justification of why that step doesn’t apply), or shows repeated failures of understanding of the process and complexities.
• Way Off: Writeup focuses exclusively on one or two elements of the process without prior approval, or does not demonstrate understanding of the many intermediate steps in the NLP chain.

Demonstration of knowledge

• Masterful: The student has successfully installed the tool locally, and has demonstrated the use of the tool on real natural language data.
• Acceptable: The student shows that they’ve managed to make the tool work, and demonstrates learning in that process.
• Novice: The student has not managed to successfully implement the tool, but presents deep discussion as to the difficulties of this above and beyond ‘computers are hard’, engaging with the NLP aspects of the process.
• Way Off: The student fails to install the tool without discussion, or demonstrates the ability to install or implement the tool, but does not engage with the NLP elements of it, going little further than ‘Hello World’. Or, student has used an online implementation of this tool, e.g. a web interface.

Richness

• Masterful: Student paper shows considerable and nuanced engagement with natural language processing and the tool(s) of their choice, showing that they’ve not just ‘managed to install it’, but have begun to demonstrate skill and facility with it, and have applied it in ways that are revealing and demonstrative for NLP. Student also provides nuanced discussions of the failings and stengths of the system, making concrete reference to their work.
• Acceptable: The student has implemented the tool, but without discussion of the implications for NLP and for the implementation of this tool in production settings. Discussions of the failings and stengths of the system are general, or fail to engage in consistent depth.
• Novice: Results from the tool’s run are presented with major problems, or without discussion and analysis.
• Way Off: The tool is not implemented, or results are only on a test file.

Formatting and Length

The rubric here is the same as for Option 1, but the expected length will be shorter (5-7 single spaced pages) because more of the work is happening ‘outside the paper’.

Structure and Organization

Option 2 projects, in particular, lend themselves well to a section-by-section write-up, with large chunks of code/output interspersed. The rubric for this portion is the same as for option 1.

Language and Argumentation

The rubric here is identical to that for Option 1.

Academic Integrity and Citation

The rubric here is identical to that for Option 1.

Option 3: Build-Your-Own

With a build-your-own project, it’s difficult to describe the exact expectations, as they vary from paper to paper. However, you can always email me to get clarifications if you’re unsure what I’m looking for. These do tend to be a bit longer, closer to 10-15 pages single spaced, particularly if they’re more research-based papers (e.g. “The history of TTS”). However, the ‘Formatting and Length’, ‘Structure and Organization’, ‘Language and Argumentation’, and ‘Academic Integrity and Citation’ guidelines will apply here as for projects 1-2.