Brief
Weather apps say "40% chance of rain". That number is only useful if it's honest: on days the app says 40%, it should rain about 40% of the time. In this project you build a rain-tomorrow forecaster that outputs a probability, train it with gradient descent you write yourself, and then check that its probabilities can be trusted.
The data is the "Rain in Australia" dataset on Kaggle (file weatherAUS.csv). It has about ten years of daily
observations from dozens of Australian weather stations, taken from the Australian Bureau of Meteorology, with a
RainTomorrow column to predict. Several columns, such as Sunshine and Evaporation, are missing for many stations, so
handling missing data is part of the job.
Logistic regression is linear regression with a squashing step. It computes and turns it into a probability with the sigmoid, . You train it by minimising cross-entropy, also called log-loss:
The gradient is short: . There is no closed-form answer as there was for linear regression, so you walk downhill with gradient descent, , where is the learning rate.
DATA · a public weather history
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Your test log-loss is within 1% of sklearn LogisticRegression trained on the same features with the same regularisation. Record it with `python tools/mlarena.py record BL-03 "<your log-loss>"`.
- Your gradient matches a finite-difference check to a relative error under 1e-5.
- The README has a reliability diagram for the test set and the expected calibration error for both your model and sklearn's.
Side quests
Terms in plain words
- Logistic regression
- A linear model whose output goes through the sigmoid function, so it predicts a probability between 0 and 1.
- Sigmoid
- The function 1 / (1 + e^(-z)), which squashes any number into the range 0 to 1.
- Cross-entropy (log-loss)
- The average of minus the log of the probability the model gave to what actually happened. Confident wrong answers cost the most.
- Gradient descent
- Repeatedly nudging the weights a small step against the slope of the loss until it stops going down.
- Calibration
- A model is calibrated if, of all the days it says 30% rain, about 30% actually get rain.