← All Base Form ML projects
BL-03NOT STARTEDBASE FORM · CLASSICAL MLMLPYTHON

Rain-tomorrow forecaster

Predict rain from today's weather with logistic regression trained by your own gradient descent, and check its probabilities are honest.

EST.5 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
log-loss; calibration
GOAL · log-loss within 1%
sklearn LogisticRegression—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Weather apps say "40% chance of rain". That number is only useful if it's honest: on days the app says 40%, it should rain about 40% of the time. In this project you build a rain-tomorrow forecaster that outputs a probability, train it with gradient descent you write yourself, and then check that its probabilities can be trusted.

The data is the "Rain in Australia" dataset on Kaggle (file weatherAUS.csv). It has about ten years of daily observations from dozens of Australian weather stations, taken from the Australian Bureau of Meteorology, with a RainTomorrow column to predict. Several columns, such as Sunshine and Evaporation, are missing for many stations, so handling missing data is part of the job.

Logistic regression is linear regression with a squashing step. It computes z=w⊤x+bz = w^\top x + b and turns it into a probability with the sigmoid, p=σ(z)=11+e−zp = \sigma(z) = \frac{1}{1 + e^{-z}}. You train it by minimising cross-entropy, also called log-loss:

L=−1n∑i[yilog⁡pi+(1−yi)log⁡(1−pi)].L = -\frac{1}{n} \sum_i \big[ y_i \log p_i + (1 - y_i) \log(1 - p_i) \big].

The gradient is short: ∇wL=1nX⊤(p−y)\nabla_w L = \frac{1}{n} X^\top (p - y). There is no closed-form answer as there was for linear regression, so you walk downhill with gradient descent, w←w−η∇wLw \leftarrow w - \eta \nabla_w L, where η\eta is the learning rate.

EXPECTEDFig. 1
00.51 00.51 PREDICTED P(RAIN) OBSERVED RAIN RATE perfect
A reliability diagram. Points on the diagonal mean the predicted probability matches how often it rained. Values are illustrative. Why it's here: Objective 6 asks for this plot. Use it to read your own: points below the line mean the model is overconfident about rain.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

DATA · a public weather history

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Your test log-loss is within 1% of sklearn LogisticRegression trained on the same features with the same regularisation. Record it with `python tools/mlarena.py record BL-03 "<your log-loss>"`.
  2. Your gradient matches a finite-difference check to a relative error under 1e-5.
  3. The README has a reliability diagram for the test set and the expected calibration error for both your model and sklearn's.

Side quests

Compare against the baseline "it will rain tomorrow if it rained today" on log-loss and on accuracy, and explain why the two metrics rank it differently.
Train a random-forest model from sklearn, plot its reliability diagram, and fix it with sklearn's CalibratedClassifierCV.

Terms in plain words

Logistic regression
A linear model whose output goes through the sigmoid function, so it predicts a probability between 0 and 1.
Sigmoid
The function 1 / (1 + e^(-z)), which squashes any number into the range 0 to 1.
Cross-entropy (log-loss)
The average of minus the log of the probability the model gave to what actually happened. Confident wrong answers cost the most.
Gradient descent
Repeatedly nudging the weights a small step against the slope of the loss until it stops going down.
Calibration
A model is calibrated if, of all the days it says 30% rain, about 30% actually get rain.