Tree Models

POLS 3220: How to Predict the Future

Today’s Agenda

• Last time, we introduced linear models as a machine learning tool.

  • The key idea was to make predictions based on a linear combination of predictor variables.

• Today, we’ll discuss an approach that combines variables in a nonlinear fashion, called classification and regression trees (CART).

  • It looks exactly like the probability trees we worked with in the first half of the semester.

  • Except we let computer to build them for us!

Motivating Problem

• Can we predict a movie’s Rotten Tomatoes rating?

• Predictor Variables: Content Rating, Genre, Release Date, Distributor, Runtime, Number of Audience Reviews

Outside View

If we had no other information, what would we predict based solely on historical base rates?

. . .

Rating	Total	%
	3,259	18.4
	6,844	38.7
	7,565	42.8

. . .

We can do better!

Classification Trees

• A classification tree works by recursively partitioning the dataset (i.e. repeatedly splitting the data into smaller subsets).

. . .

Classification Trees

With each partition, your prediction becomes less biased.

. . .

Criterion, Searchlight, or A24

Rating	Total	%
	118	37.2
	135	42.6
	64	20.2

Other Distributors

Rating	Total	%
	3,105	18.4
	6,396	37.9
	7,357	43.6

Classification Trees

With each partition, your prediction becomes less biased.

. . .

Audience Count < 10,000

Rating	Total	%
	1,293	12.0
	5,052	46.9
	4,427	41.1

Audience Count >= 10,000

Rating	Total	%
	1,811	30.8
	1,193	20.3
	2,870	48.9

Choosing Splits

• What is the “best” way to partition the data?

• Up to now, we’ve been choosing our splits based purely on vibes.

• A more principled way is to partition the data based on how much information it reveals.

Information Theory: Review

Recall the definitions of information and entropy.

• Information is how surprised we are when we learn an outcome. Equal to \(-log(p)\).

• Entropy is “expected surprise”: how much surprise we experience on average when we learn the outcome.

• If we could perfectly predict the Rotten Tomatoes rating a of movie, then entropy would be zero. (We would never be surprised).

• Therefore, our optimal strategy is to seek out information that reduces entropy as much as possible.

Information Theory: Review

Rating	Total	p
	3,259	0.184
	6,844	0.387
	7,565	0.428

Information Theory: Review

Rating	Total	p	-log(p)
	3,259	0.184	2.44
	6,844	0.387	1.37
	7,565	0.428	1.22

. . .

\(\text{Entropy} = -\sum p\times log(p)=\) 1.5 bits.

Information Gain

• Each time you partition the dataset, it reduces entropy.

• This is because you become more certain about your prediction, therefore less likely to be surprised!

• The information gain from a partition equals how much it reduces entropy.

Information Gain

Audience Count < 9,000

Rating	Total	%
	1,328	12.1
	5,161	47.1
	4,462	40.7

Entropy: 1.41 bits

Audience Count >= 9,000

Rating	Total	%
	1,894	31.5
	1,219	20.3
	2,899	48.2

Entropy: 1.5 bits

Classification Trees

• Trying to find the best partition by hand would be tremendously tedious.

• But computers are great at it.

• When creating a classification tree, the computer will check thousands of possible partitions, see how much it reduces entropy, and pick the best one.

Classification Trees

How Complex Should The Tree Get?

Bias-Variance Tradeoff

• It’s our old friend, the bias-variance tradeoff!

• Outside View (zero partitions) is biased.

• Inside View (lots of partitions) has higher variance, because your predictions are based on less data.

• Sweet spot is somewhere in the middle.

Bias-Variance Tradeoff

• In machine learning, this tradeoff is called overfitting vs. underfitting.

. . .

Next Time

• Next time, we’ll talk about how to hit that “sweet spot” between overfitting and underfitting.

• And we’ll show how you can harness the wisdom of crowds with machine learning models.

• Spoiler alert: we’re going to take a bunch of different classification trees and ask them to “vote” on the best prediction.

• The resulting machine learning model is called random forest.