Advanced Natural Language Processing / Spring 2025


Advanced natural language processing is an introductory graduate-level course on natural language processing aimed at students who are interested in doing cutting-edge research in the field. In it, we describe fundamental tasks in natural language processing as well as methods to solve these tasks. The course focuses on modern methods using neural networks, and covers the basic modeling, learning, and inference algorithms required therefore. The class culminates in a project in which students attempt to reimplement and improve upon a research paper in a topic of their choosing.


Course Details

Instructor

Sean Welleck


Teaching Assistants

Darsh Agrawal

Hugo Contant

Alex Fang

Akshita Gupta

Trisha Sarkar

Manan Sharma

Sanidhya Vijayvargiya

Logistics

  • Class times: TR 3:30pm - 4:50pm
  • Room: TEP 1403
  • Course identifier: LTI 11-711
  • Office hours: See Piazza

Grading

  • The assignments will be given a grade of A+ (100), A (96), A- (92), B+ (88), B (85), B- (82), or below.
  • The final grades will be determined based on the weighted average of the quizzes, assignments, and project. Cutoffs for final grades will be approximately 97+ A+, 93+ A, 90+ A-, 87+ B+, 83+ B, 80+ B-, etc., although we reserve some flexibility to change these thresholds slightly.
  • Quizzes: Worth 20% of the grade. Your lowest 3 quiz grades will be dropped.
  • Assignments: There will be 4 assignments (the final one being the project), worth respectively 15%, 15%, 20%, 30% of the grade.

Course description

The course covers key algorithmic foundations and applications of advanced natural language processing.

There are no hard pre-requisites for the course, but programming experience in Python and knowledge of probability and linear algebra are expected. It will be helpful if you have used neural networks previously.

Acknowledgements. This semester's course is adapted from Advanced NLP Fall 2024, designed and taught by Graham Neubig. The course structure (e.g., grading, course description, class format, assignments, poster presentation) is from Advanced NLP Fall 2024. Many lectures are adapted from Advanced NLP Fall 2024; please refer to individual slides.

Class format

Lectures: For each class there will be:
  • Reading: Most classes will have associated reading material that we recommend you read before the class to familiarize yourself with the topic.
  • Lecture and Discussion: There will be a lecture and discussion regarding the class material. This will be recorded and posted online for those who cannot make the in-person class.
  • Code/Data Walkthrough: Some classes will involve looking through code or data.
  • Quiz: There will be a quiz covering the reading material and/or lecture material that you can fill out on Canvas. The quiz will be released by the end of the day of the class and will be due at the end of the following day.
Recitation Sections: additional recitation sections offer hands-on introductions to practical tools. These take place during the office hours of the TA leading the recitation. For those unable to attend in person, recordings and a hybrid meeting link are provided.

Questions and Discussion: Ideally in class or through piazza so we can share information with the class, but emailing the TA mailing list and coming to office hours are also encouraged.

Schedule

Assignments

The aim of the assignment and project is to build basic understanding and advanced implementation skills needed to build cutting-edge systems or do cutting-edge research using neural networks for NLP, culminating with a project that demonstrates these abilities through a project.

Read all the instructions on this page carefully
You are responsible for reading these instructions and following them carefully. If you do not, you may be marked down as a result.

Assignment Policies

Working in Teams:

There are 4 assignments in the class. Assignment 1 must be done individually, while Assignments 2, 3, and 4 must be done in teams of 2-3 (individual submissions will not be accepted for these assignments). If you are having trouble finding a group, the instructor and TAs will help you find one after the first initial survey.

Submission Information:

To submit your assignment you must submit via Canvas a zip file containing:

  • your code: This should be in a directory “code” in the top directory unless specified otherwise.
  • system outputs (assignments 1 and 2): The format will be specified separately for each assignment.
  • a report (assignments 2, 3 and 4, optional for assignment 1): This should be named “report.pdf” in the top directory. This is for assignments 2, 3 and 4, and can be up to 7 pages for assignments 2 and 3 and 9 pages for assignment 4. References are not included in the page count, and it is OK to submit appendices that include supplementary information such as hyperparameter settings or additional output examples, although there is no guarantee that the TAs will read them. Submissions that exceed the page count will be penalized one third grade for each page over (e.g., A to A- or A- to B+). You may also submit report.pdf for assignment 1 if you have any interesting information to convey to the TAs, for example, if you did anything interesting above and beyond the minimal requirements.
  • a link to a GitHub repository containing your code (assignments 2, 3 and 4): This should be a single line file “github.txt” in the top directory. Your GitHub repository must be viewable to the TAs in charge of the assignment by the submission deadline. If your repository is private, make it accessible to the TAs by the submission deadline. If your repository is not visible to the TAs, your assignment will not be considered complete, so if you are worried, please submit well in advance of the deadline so we can confirm the submission is visible. We use this repository to check contributions of all team members.

Late Day Policy:

In case there are unforeseen circumstances that don’t let you turn in your assignment on time, 5 late days total for assignments 2 and 3 will be allowed. Note that other than these late days, we will not be making exceptions and extending deadlines except for health reasons, so please try to be frugal with your late days and use them only if necessary. Assignments that are late beyond the allowed late days will be graded down one third-grade per day late (e.g., A to A- for one day, and A to B+ for two days).

Plagiarism/Code Reuse Policy:

All assignments are expected to be conducted under the CMU policy for academic integrity. All rules here apply and violations will be subject to penalty including zero credit on the assignment, failing the course, or other disciplinary measures. In particular, in your implementation:

  • Code or pseudo-code provided by the TAs or instructor may be used freely without restriction.
  • For assignment 2, you may not just re-use an existing implementation written by someone else. The implementation should basically be your own.
  • Code written by other students in the class cannot be used (except, obviously, you can share code within your group for assignments 2, 3, and 4).
  • If you are doing a similar project for a graded class at CMU (including independent studies or directed research), you must declare so on your report, and note which parts of the project are for 11-711, and which parts are for the other class. Consult with the TA mailing list if you are unsure.

Consulting w/ Instructors/TAs:

For assignments and projects, you are free to consult as much as you want, any time you want with the instructors and TAs. That is what we’re here for, and in no way is this considered cheating. In fact, if you don’t have much experience with NLP previously, it will be helpful to liberally consult with the instructors and TAs to learn about how to do the implementation and finish the assignments. So please do so.

Because this is a project-based course, we assume that many of the students taking the course will be interested in turning their assignments or project into research papers. In this case, if you have received useful advice from the instructor or TAs that made the project significantly better, consider inviting them to be co-authors on the paper. Of course, you do not need to do so just because the paper is a result of the class, only if you feel that their advice or help made a contribution.

Details of Each Assignment

  1. Assignment 1: To be released
  2. Assignment 2: To be released
  3. Assignment 3: To be released
  4. Assignment 4: To be released

Poster Presentation

Time/Location

  • Time: TBD
  • Location: TBD
We will announce which teams will be presenting on which days during the course. If you have a major, immovable conflict that will prevent your team from presenting on one day please contact us via piazza and we will try to make accommodations.

Goals and Grading

The intention of the poster is several-fold:
  • That you share your preliminary results with the TAs and instructor so we can give feedback to make any last adjustments to improve your final project report.
  • That you can see the other projects in the class to learn from them and get any ideas that may improve your final project report.
  • That you can practice explaining the work that you did.
The poster is graded for attendance, so you need to show up with a poster or will be graded down on the project. However, it is basically pass/fail, so basically if you show up with a poster you will not be graded down. That being said, putting the important information on the poster will help you get better feedback.

What information should be included in a poster? It should be mostly:
  • What is the problem you’re solving
  • What is your method for solving that problem
  • What are the results
There is not a set format for creating a poster, but if you would like some guidance, I would suggest creating three columns, where the left one describes “1”, the middle one describes “2”, and the right one describes “3”. The middle one can be a bit wider.

Poster Printing

If you are a member of SCS, we suggest that you use SCS poster printing. If you are not a member of SCS, you can send your PDF to the TAs no less than 5 days before your presentation, and we will print it for you.