A main component of the course is a semester-long, research-oriented project on using AI techniques to address a systems problem. You will work as a team of 2-3 students. The project should identify a clear problem, motivate why that problem is important and why existing solutions are insufficient, propose an idea to address the problem, implement a prototype, and demonstrate the idea’s efficacy through experiments and comparisons with existing solutions. In the end, you will write up a project report about your work, just like the research papers you read in the class.
A project generally takes one of three forms:
Whichever form you choose, the project is expected to produce new insights and artifacts in this space.
Please use the USENIX conference paper template to format your project proposal and report.
A great outcome is that the project will be suitable for publication at a major systems or MLSys venue (OSDI, SOSP, EuroSys, NSDI, ASPLOS, USENIX ATC, MLSys). Additional work after the course is often needed to reach publication quality, but the course project should establish the core idea, a working prototype, and a credible initial evaluation.
The minimum expectations for the project are: (1) it should be interesting! (2) it should produce a software artifact. In other words, even if the project does not yield a real publication, it should have intellectual depth and something that the readers can learn from. For (2), the resulting software artifact should be usable.
A survey of existing literature will not be suitable for the course project. Simply fine-tuning an off-the-shelf model on a systems dataset, or applying a standard model to a standard benchmark without a systems insight, is also not sufficient. The systems contribution must be real.
Learning-based systems introduce several evaluation pitfalls. Plan for them early and use the following guidance:
I will provide a list of suggested topics, but you are encouraged to come up with your own!
I will regularly meet with each team to help brainstorm ideas, refine the project scope, and provide guidance. I will always be available for project discussions.
| Date | Milestone |
|---|---|
| Thu 09/10 | Team formation |
| Thu 09/24 | Proposal due |
| Tue 09/29 | Proposal pitches in class |
| Thu 10/22 | Project hack day; Progress #1 report due |
| Thu 11/05 | Project progress presentations in class |
| Tue 11/24 | Progress #2 report due |
| Tue 12/08 | Project hack day |
| Thu 12/10 | Final presentations; slides due 24 hours before |
| Fri 12/18 | Final report and source code due |
Note these dates are tentative and will be confirmed on the Schedule page.
In the proposal, briefly describe the following elements:
The proposal should be around 2 pages and include your group members’ names and emails. The proposal only needs to be submitted by one of the group members. Each team will also give a short proposal pitch in class.
Submit two progress reports, each around 3 pages. Each report should describe the current status of your project, the solution you have built so far, initial results, new challenges, and any adjustments to your approach or timeline. The second report should explain what has changed since the first report, address feedback, summarize evaluation progress, and identify the remaining work and risks. Each team will also give a short mid-semester progress presentation in class between the two reports.
The final report should look like a research paper. It motivates the importance of the problem with concrete evidence or data points. It explains why existing solutions do not solve the problem well. Then it describes the insights and ideas of your work, the challenges you needed to address, how you addressed them, why your idea is better, and what your system design and implementation are. Then it shows the experiment results, interprets the results, describes the observations and implications, and concludes whether or how the results demonstrate your solution’s superiority over the state-of-the-art solution. Lastly, include a related work section to compare and contrast with your solution.
There are no hard restrictions on the page limit for the final report, because papers do not need to be long to be good! But your final report must comprehensively cover all the points mentioned above. The rule of thumb is that when it does so, the report is around 6 to 10 pages.
Each project team will be given a slot to give a conference-style talk in class to showcase your ideas and results. A live demo of the system you developed would be a nice element of the presentation. We might host a poster session in Tishman Hall if conditions allow. Presentation slides are due 24 hours before the scheduled presentation.
You will be required to turn in your source code along with a detailed README and scripts for grading. The software artifact should be runnable by the instructor. For learning-based components, also include your training and evaluation scripts, model checkpoints (or a script that reproduces them), and a pointer to the datasets or traces you used.
Each student is expected to make substantial and clearly identifiable technical contributions. At each major milestone, you will submit an individual contribution summary describing what you personally did. Significant imbalance in contribution will result in different grades among members of the same team.