Applied Machine Learning — Course Project

Building and Shipping an End-to-End Machine Learning System

Objective

You will take a real problem from a domain you care about (healthcare, finance, education, agriculture, retail, transport, energy, public services…) and carry it through the full industry ML workflow covered in this module:

• Identify a problem where a real decision changes because of your model.
• Select and secure a dataset that can actually answer it.
• Follow the workflow from business understanding through data preparation, feature engineering, modelling, evaluation and error analysis.
• Track every experiment with a proper tool (MLflow, Aim or Weights & Biases) instead of a notebook full of printed scores.
• Package and deploy the trained pipeline behind a simple API or interface, and write down how you would monitor it.

Keep it small and finished. A well-framed logistic regression, tracked in MLflow and served by a working FastAPI endpoint, scores higher than an ambitious model that only exists inside a notebook.

What is graded is the reasoning, not the score. An honest AUC of 0.68 with a clear decision and a measured baseline beats a 0.95 produced by a leaked column, every time. If your first model looks excellent, your first job is to find out why.

The Workflow You Will Follow

The project mirrors the module, session by session. Each stage produces something you hand in.

Session 10 walks through this entire sequence on one dataset. Use it as your template.

Step 1 — Identify the Problem

Start from a decision, not from a dataset or an algorithm. Before choosing data, you must be able to fill in these four lines:

1. The decision: "At _ (decision point), _ (who) decides whether to ____."
2. The owner: the person or role whose work changes because of your model.
3. The action: what they will do differently with a prediction in hand.
4. The cost of being wrong: what a false positive costs, and what a false negative costs.

If you cannot name the person whose work changes, pick a different problem. "Understanding the drivers of X" is a report, not a decision.

Also ask session 1's question honestly: is this an ML problem at all? If a simple rule, a lookup or a dashboard would do the job, say so — and then either reframe or choose another problem.

Project types

The module covers supervised learning on tabular data in depth, so that is the default.

Deep learning is allowed if it beats a well-tuned classical baseline on your data and you can justify its cost. On tabular data of this size it usually will not, and it earns no marks by itself.

Sample project ideas by domain

Step 2 — Select a Dataset

Dataset requirements

Where to find data

General repositories

Domain-specific

Nepal and regional data

Workplace or self-collected data is encouraged — it usually produces the most interesting projects. Workplace data requires written permission and must be anonymised; never commit personal data to a public repository. If you scrape, respect robots.txt and the site's terms of service, and rate-limit your requests.

Three data rules

1. Check the licence and cite the source. If it forbids redistribution, link to it instead of committing the file.
2. Freeze your data. Download once, record its MD5 hash, and work from that copy. If you pull from an API, cache the response to CSV. A dataset that changes underneath you makes every number in your report irreproducible.
3. Keep the repository small. Commit files under ~50 MB. For anything larger, put download instructions in data/README.md and git-ignore data/raw/. (DVC is a welcome optional extra.)

Step 3 — Write the Project Charter (Business Understanding)

Before touching a model, write session 1's one-page charter. Every later stage refers back to it: choosing a metric fills in a charter field; detecting drift tests a charter assumption.

Success and kill criteria are written before modelling and are not changed afterwards to fit the result. If you change them, say so and say why.

Step 4 — Data Understanding, Preparation and Features

• Point-in-time audit of every column: when is it written, relative to the decision point? Drop anything written afterwards. Be especially suspicious of status, final_, resolution_, closed_ and any single column that is suspiciously predictive.
• Missingness: understand why values are missing before filling them. Missingness itself is often a feature.
• Distributions, outliers, duplicates, class balance — with written insights, not just plots.
• A justified split: random, stratified, grouped or time-based — and why that one matches how the model will be used.
• All preprocessing inside a scikit-learn Pipeline / ColumnTransformer, fitted only on training data. If preprocessing is fitted before the split, the evaluation is invalid.
• Engineered features built from information available at prediction time, implemented in src/ so training and serving use the same code.

Step 5 — Baselines, Modelling and Evaluation

Baselines first. You must report them, and you must try to beat them. At minimum:

1. A constant (majority class or mean).
2. A rule (the existing business practice, or the most obvious one-line heuristic).
3. A simple model (logistic or linear regression on a few obvious features).

Then:

• Compare at least two model families (e.g. regularised linear model vs. gradient boosting / random forest) with cross-validation, reporting the spread across folds, not just the mean.
• Tune hyperparameters inside cross-validation (GridSearchCV, RandomizedSearchCV or Optuna).
• Choose the metric for the decision. Accuracy on imbalanced data is not acceptable. For classification, pick an operating point (threshold) using cost or capacity — e.g. "the team can call 200 patients a day".
• Touch the held-out test set once, at the end.
• Error analysis by slice: where is the model wrong, and for whom? Check performance across meaningful groups (region, gender, age band, product type…) for both accuracy and fairness.
• Optional but valued: explainability (permutation importance, SHAP) and probability calibration.

Step 6 — Experiment Tracking

Every training run that appears in your report must be logged in an experiment tracker. "It's somewhere in the notebook" is not acceptable.

Minimum requirements for every run:

Also required:

• Baselines logged as runs too, so the comparison lives in the tracker.
• The final model registered or tagged (e.g. MLflow Model Registry, or a final tag) so it can be reloaded by run ID.
• Screenshots of the run comparison view in your report.

import mlflow

mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("my-project")

with mlflow.start_run(run_name="gradient-boosting-v2"):
    mlflow.log_params({"model": "HistGradientBoosting", "learning_rate": 0.05, "seed": 42})
    mlflow.log_metrics({"cv_pr_auc_mean": 0.41, "cv_pr_auc_std": 0.02})
    mlflow.sklearn.log_model(pipeline, "model")

Step 7 — Packaging and Basic Deployment

The artifact

Save the entire pipeline (preprocessing + model) with joblib, plus a metadata JSON beside it, as in session 9: model name and version, training date, number of training rows, data hash, feature lists, threshold, library versions and a one-line description of what the prediction means. Always check the round trip: a reloaded model must give the same predictions.

The served model (required)

Build one of the following that loads the saved artifact once at startup:

Doing both (Streamlit calling your FastAPI) is welcome but not required.

One rule that matters: never copy preprocessing code from a notebook into the app. Import it from src/, or load it inside the saved pipeline. Otherwise the two drift apart and the app predicts nonsense.

Containerisation (required)

A Dockerfile that builds and runs your service from a clean clone. docker-compose.yml is recommended if you run more than one service (e.g. API + UI, or API + MLflow).

Public deployment (bonus)

Deploy to a free host and put the URL in your README — e.g. Hugging Face Spaces, Streamlit Community Cloud, Render or Railway.

Step 8 — Monitoring Plan

Half a page, in the format of session 8's runbook. Labels usually arrive late, so monitor inputs as well as outcomes.

State when you would retrain, and what would make you roll back.

Core Requirements

Project Structure

your-project/
├── README.md                 # recommendation, problem, data, results, how to run
├── CHARTER.md                # the one-page project charter
├── MONITORING.md             # the monitoring runbook
├── requirements.txt          # pinned versions
├── Dockerfile
├── docker-compose.yml        # recommended
├── data/
│   ├── README.md             # source, licence, download date, MD5 hash
│   ├── raw/                  # original data (git-ignored if large)
│   └── processed/            # cleaned data used for training
├── notebooks/
│   ├── 01_data_audit_eda.ipynb
│   ├── 02_baselines_modelling.ipynb
│   └── 03_evaluation_error_analysis.ipynb
├── src/
│   ├── config.py             # paths, seeds, feature lists, threshold
│   ├── data.py               # loading, cleaning, splitting
│   ├── features.py           # preprocessing + feature pipeline
│   ├── train.py              # trains, logs to the tracker, saves the artifact
│   ├── evaluate.py           # test-set and slice evaluation
│   └── predict.py            # loads the artifact and predicts
├── models/
│   ├── model.joblib          # the full pipeline
│   └── model.json            # metadata
├── app/
│   ├── api.py                # FastAPI service
│   └── ui.py                 # Streamlit / Gradio (if used)
└── tests/                    # optional: pytest for features and the API

Notebooks are for exploring and explaining; src/ holds everything the app depends on. Add mlruns/, mlflow.db, .aim/ and wandb/ to .gitignore.

You may use a standard template instead — say which in your README: Cookiecutter Data Science, Kedro, or your own layout, as long as it separates data, exploration, source code, models and the app.

What to Submit

Submit a single GitHub repository link containing:

Final Report Structure (4–6 pages)

1. Recommendation — one paragraph, in words a head of department would use: deploy, do not deploy, or what it would take.
2. Problem and charter — the decision, its owner, success and kill criteria.
3. Data — source, licence, size, point-in-time audit, leakage checks, split choice.
4. Preparation and features — key insights and the decisions you made, with reasons.
5. Modelling — baselines, models compared, tuning, with fold spread.
6. Evaluation and error analysis — metric at the operating point, cost/capacity reasoning, slices where the model fails.
7. Experiment tracking — how runs were organised, screenshots, how to recover the final model.
8. Deployment — architecture, how to run it, screenshots or sample request/response.
9. Monitoring — summary of the runbook.
10. Limitations, ethics and next steps — what you would do with more time or data; who could be harmed by errors; note any use of AI assistants and how.

Evaluation Criteria

Bonus (up to 5%): a public deployment URL; unit tests (pytest) for features and the API; a CI workflow (GitHub Actions) that runs tests and builds the Docker image; model explainability in the app; data versioning with DVC; a drift report (e.g. Evidently) on a simulated "future" slice of your data.

Common Ways Projects Lose Marks

• No named decision. The single biggest one.
• No baseline, or a baseline that is assumed rather than measured.
• A leaked column — and a celebrated score that came from it.
• Accuracy on imbalanced data.
• Preprocessing fitted before the split. Use a pipeline and it cannot happen.
• Results in the report that cannot be found in the tracker.
• An app that re-implements preprocessing instead of loading the pipeline.
• "Works on my machine" — the Docker image or requirements.txt fails from a clean clone.
• A notebook that does not run after restarting the kernel.
• Burying the recommendation at the end of the report.

Recommended Tools

Getting Started

# Repository and environment
git init my-project && cd my-project
python -m venv .venv && source .venv/bin/activate
pip install pandas numpy scikit-learn matplotlib seaborn joblib mlflow fastapi uvicorn streamlit
pip freeze > requirements.txt

# Train and log runs
python src/train.py
mlflow ui --backend-store-uri sqlite:///mlflow.db     # open http://127.0.0.1:5000

# Serve the model
uvicorn app.api:app --reload                          # open http://127.0.0.1:8000/docs
streamlit run app/ui.py                               # open http://localhost:8501

# Containerised
docker compose up --build

Frequently Asked

My model is barely better than the baseline. Then that is your finding. Report it clearly, show what you tried, and say what data would change the answer. This is a pass-worthy and often excellent project.

Do I have to use MLflow? No — Aim or Weights & Biases are equally acceptable. What is required is that every reported number is traceable to a logged run.

Does "deployment" mean it has to be public? No. A Dockerised service that runs locally from a clean clone meets the requirement. A public URL is a bonus.

How long should the notebook be? Long enough to follow the reasoning, short enough that someone reads it all. Session 10's Part A is about the right length.

Academic Integrity

You may use libraries, documentation and AI tools as references, but you must be able to explain every line of your code and every modelling decision in the presentation. Cite your data sources, and state in your report whether you used an AI assistant and how.