Project Instructions

Come back to these instructions from time to time since we’ll update them as questions arise.

Back to the Project - 2024 Spring (AI5308/AI4005)

Goal

Your goal is to build a compelling ML-powered application. Users, who are not part of your team, should be able to interact with your application. You’ll be giving a live demo of your application at the end of the semester. The demos will be evaluated by diverse participants including your colleagues. The milestones and reports will be evaluated by course staff.

You are not required to build a market-ready product, e.g. it doesn’t have to be able to serve 1000s of requests at the same time or have a flawless user interface. However, we do expect a functioning prototype that captures the essence of the problem you set out to solve. For example, if you want to build an application to detect household amenities in an image, users should be able to upload a photo and get back a list of objects in that image.

Process

In general, you can follow the four-step process.

Step #1: Brainstorm

Brainstorm an application that you’d like to build. You’re free to pick absolutely anything as long as at least one of its features is powered by machine learning models.

You can make your project novel by adapting a popular idea for a niche market. For example, image classification is a common task but how about using image classification to detect whether a house plant needs more water?

You can also adapt a common problem for a new medium, such as a Chrome extension, a VSCode extension, a Spotify app, or a Discord/Slack bot.

Step #2: MVP - Minimum Viable Product

Once you know what you want to build, figure out how you’ll get to an MVP. That means:

  • starting with a very, very simple model (which can be a bunch of if/else or a logistic regression model) hooking everything up end-to-end to have the application up and running
  • validating that your data is useful
  • validating that your pipeline is correct and debuggable
  • validating that the application you want to build is feasible. Establish feasibility early.

Tips:

  • Don’t be afraid of drawing out diagrams to help with designing your system and user interface.
  • Prototype and build out parts of the pipeline, use mock-ups and/or dummy methods if you need to.
  • Don’t obsess over any part of the pipeline yet, but focus on your system as a whole.

Step #3: Iterate

Once you have a working MVP, iterate on it until you get a better application. Some ideas for iteration:

  • more data
  • better data
  • more data preprocessing
  • more features
  • different model architectures
  • different loss functions
  • hyperparameter tuning
  • ensemble

One quick rule of thumb is 20% of time on MVP, 80% of time in iteration.

Step #4: Polish

Spend time thinking about how to improve the user’s experience. This will be important to your demo.

Design doesn’t mean aesthetics here (though an aesthetically pleasing interface doesn’t hurt), but you do want to think about whether your application’s design serves its intended purpose well.

Aspects to consider

While you’re brainstorming, make sure to plan ahead. You need to know what the application requires. Here are a few things to consider.

  1. Objectives
    • What is the problem that your app will solve?
    • If a company was to use this application, what would be their ML objectives and business objectives?
  2. Users
    • Who will be the users of your application?
    • How are users going to interact with your application?
    • What’s necessary for a good user experience, e.g. latency?
  3. Interface
    • We accept all interface solutions:
      • Standalone apps (includes mobile apps)
      • Websites
      • Apps that can be run via platforms like Chrome extensions, Facebook Messenger, Discord, Slack, Shopify apps, even Twitter bots users can tweet at to get a response
      • APIs like OpenAI API, HuggingFace API Inference
      • CLI tools
    • Creative and polished interfaces will get bonus points.
  4. Data
    • Where would you get your data from?
    • How much data would you need?
    • Is there anything publicly available or do you need to build your own dataset?
  5. Modeling
    • What types of models/architectures will you be using for this application? Which ones would you start with?
    • You’re free to train a model from scratch or use a pretrained model.
    • We accept models from logistic regression to complex neural networks.
  6. Evaluation
    • How would you evaluate your model performance, both during training and inference?
    • How would you evaluate whether your application satisfies its objectives?
  7. Inference
    • Will you be doing online prediction or batch prediction or a combination of both?
    • Will the inference run on-device or through a server?
    • Can you run inference on CPUs or an edge device or do you need GPUs?
  8. Compute
    • How much compute do you need to develop this application as a market-ready product?
      • To train your model (if you train your own model): do you need GPUs/TPUs or just CPUs? How many machines? For how long?
      • To serve your model: can your models make predictions on CPUs?
    • How much compute do you need to develop this application for this project?
    • Feel free to reference machine cost on AWS/GCP to get an estimate of compute cost.
  9. MVP
    • What would the MVP be?
    • How difficult is it to get there?
  10. Pre-mortems
    • What are the risky aspects of the project?
      • e.g. not enough data, limited compute resources, not knowing how to implement an interface, network latency, inference latency, etc.
    • If your team fails to build the application you want, what do you think might have caused the failure?
    • What are the limitations of your application?
    • What are the potential biases of your application?

Tips

  • Start early. You’re going to need time to ideate, build, and polish your application. Start early so you can iterate.
  • Start simple. If you’re starting with a deep neural network, you’re probably doing it wrong.
  • Figure out how to evaluate your model in production. It’s surprisingly hard when you don’t have ground truth labels.
  • Get data right early. Getting the data right is non-trivial.
  • Get an MVP early. It’s better to have a simple end-to-end working version of your application than to perfect one part of the pipeline. Don’t waste time obsessing over things like model quality early on.
  • Avoid the model performance trap. The goal is not to train the best possible model. The goal is to build a practical application to address a compelling use case.
  • Understand your application’s limitations.
  • Put yourself in the shoes of your users. You’ll want to think carefully about how a user interacts with your application. If your model takes 30 seconds to make a prediction for an app that has high user interactivity, it’s not going to make for a good demo.

Milestones, deliverables, and grading

There are five milestones – team-matching, a proposal, an MVP, a demo, and a final report.

Team Matching

  • Deliverables: You should have a team by week 3. If you don’t have a team by the end of Sunday, please let the course staff know and we’ll try our best to match you with a team. You’re also welcome to use the Search for Teammates! thread on Ed to post and discuss ideas with classmates.

Proposal (PDF)

  • Deliverables: Up to 3 page PDF-formatted project proposal outlining your preliminary project idea.
  • Grading: 3 out of 45 points

MVP (Video)

  • Deliverables: Up to 7 minute long video containing 3-page slide pitch + Demo of an MVP
  • Grading: 9 out of 45 points

Demo day

End-of-semester Report

  • Delivetables: Up to 2500 words long, free-formatted report summarizing your project work and learning.
  • Details will be be announced
  • Grading: 12 out of 45 points
  • The last 4/45 points will be spared for peer review

FAQs

  1. What does ML-powered mean? Your application must have a predictive component i.e. it must run some ML model underneath that makes predictions.

  2. Can I use starter-code/off-the-shelf models/datasets? Yes, as long as you credit them appropriately. Keep in mind that we want to see what you build, so make sure you’re sufficiently ambitious.

  3. Do I have to use any particular framework (e.g. Pytorch) or tool? No, there are really no limitations.

  4. Do I need to learn web or mobile development? You’re free to build your application as an API, a Streamlit/Dash app, command-line tool etc. Whatever you like, as long as the interface for users is a good fit for the problem.

  5. Can I use a project at work for this? Yes. Clearly clarify the contributions you made for this class, and let us know you’re doing this in your proposal.

  6. Can I do this project in conjunction with another course’s project? Yes. Clearly clarify the contributions you made for this class, and let us know you’re doing this in your proposal.

  7. When is the deadline for forming teams? Your team will be reported in the proposal. You will not be able to change teams after the proposal, unless an exception is made with course staff approval.

  8. Must all teams consist of four people? Mostly. Exceptions require course staff approval. Larger teams will be expected to put in more work, while smaller teams will not be given any extra leniency.

Resources

This list of resources will be continually updated as we find more good resources for you, and based upon student requests/project areas of focus. If you have questions about resources, and the list below doesn’t contain what you’re looking for, reach out and ask us! We’ll find you what you need and update the list below accordingly.

Clips

To get familiar with ML web application development, you can first refer to short clips such as:

Tutorials

Other projects

While implementing your idea, please think about the concepts we learned during the class (e.g., Better model design, sustainability, scalability, cold-start problem, interpretability, data shift, fairness, data imbalance, etc)