Run/Pass Prediction with a Random Forest

September 2023

Down, distance, score, and clock tell a defense a lot about whether a run or a pass is coming. I trained a random forest on three seasons of play-by-play and tested it on every 2023 play to measure exactly how much.

Why a random forest

Play-calling is full of interactions. 3rd & 2 means something very different when a team is up 14 with three minutes left than when it's down 4 in the second quarter. A random forest handles this without hand-built interaction terms: it trains hundreds of decision trees on random subsets of the plays and features, then averages their votes. It captures non-linear thresholds (like "it's 3rd and more than 2") on its own, and it outputs a probability rather than just a label.

The previous version of this project modeled Clemson's play-calling with college data. This one uses every NFL team, trains on 100,082 designed runs and dropbacks from 2020–2022, and tests on all 33,836 from 2023. Testing on a later season, instead of a random split, is the honest version: the model only knows what a team could have known going into that season.

The tendencies the model learns

Figure 1: League pass rate by down and distance, 2020–2023. Scrambles and sacks count as pass calls.

The jump from 3rd & 1–2 (41% pass) to 3rd & 3–4 (85%) is the sharpest edge on the grid, and exactly the kind of threshold a tree-based model is good at finding. Early downs are where play-callers mix it up: 1st & 10 is close to a coin flip.

Features

The base model sees only the game situation: down, distance, field position, quarter, time left in the half and game, score differential, goal-to-go, home/away, and two derived flags: an "obvious pass" flag (3rd & 7+, or down 10+ in the last ten minutes) and a two-minute-drill flag. A second model adds two things a defense can see before the snap: shotgun formation and no-huddle.

from sklearn.ensemble import RandomForestClassifier

plays["obvious_pass"] = (((plays.down == 3) & (plays.ydstogo >= 7)) |
                         ((plays.score_differential <= -10) & (plays.game_seconds_remaining <= 600))).astype(int)
plays["two_minute"] = (plays.half_seconds_remaining <= 120).astype(int)
plays["is_home"] = (plays.posteam == plays.home_team).astype(int)

SITUATION = ["down", "ydstogo", "yardline_100", "qtr", "half_seconds_remaining",
             "game_seconds_remaining", "score_differential", "goal_to_go",
             "obvious_pass", "two_minute", "is_home"]
PRESNAP = SITUATION + ["shotgun", "no_huddle"]

train, test = plays[plays.season < 2023], plays[plays.season == 2023]
rf = RandomForestClassifier(n_estimators=300, max_depth=12, min_samples_leaf=20,
                            n_jobs=-1, random_state=42)
rf.fit(train[PRESNAP], train.is_pass)
p_pass = rf.predict_proba(test[PRESNAP])[:, 1]
accuracy = ((p_pass >= 0.5) == test.is_pass).mean()   # 0.736

Results

Figure 2: Accuracy on every 2023 run/pass play. Gray bars are benchmarks. nflfastR's xpass is a published pass-probability model built on similar inputs.

The situational random forest (70.3%) lands essentially on top of nflfastR's xpass model (70.6%). That's a useful sanity check: two independently built models with similar inputs reach the same ceiling, which suggests it's the limit of what game state can tell you.

Passes are easier to call than runs. On the situational model, pass precision and recall are both about 0.76, against about 0.62 for runs. Offenses run on predictable downs, but they also throw on downs that look like running downs, which is the point of play-action.

Situational model, 2023	Precision	Recall	F1	Plays
Run	0.62	0.63	0.62	13,141
Pass	0.76	0.75	0.76	20,695

What the model actually uses

Permutation importance shuffles one feature at a time on the 2023 data and measures how much accuracy drops. It's a more honest measure than the built-in "impurity" importance, which tends to overrate continuous features like the clock.

Figure 3: Permutation importance (accuracy lost when a feature is shuffled). Full pre-snap model, 20,000 sampled 2023 plays, 5 repeats.

Shotgun dominates because it partly encodes the play call. Teams line up in shotgun on 3rd & long because they're passing. The situational features overlap heavily (down, distance, and the "obvious pass" flag all describe the same thing), so shuffling any one of them costs little while the others cover for it. Taken together they're worth the full 9 points over baseline.

Can you trust the probabilities?

Figure 4: Calibration: predicted vs. actual pass rate, 2023.

Because the probabilities are reasonably well calibrated, they're more useful than the yes/no prediction. Here's the model on a few situations:

Figure 5: Predicted pass probability in example situations. Full pre-snap model. Under center unless the label says shotgun; 3rd-down scenarios assume shotgun.

Which teams were most predictable?

Running the same league-wide model on each 2023 offense shows which teams followed league tendencies and which broke from them.

Figure 6: Model accuracy by offense, 2023. what the model expected. A negative gap means the team ran more than expected.

The least predictable offenses mostly share one trait: they ran more than the situation called for. Atlanta threw 9.6 points less often than the model expected, Indianapolis 8.3 points, and Philadelphia 7.3 points. Philadelphia's quarterback run game, including designed Jalen Hurts runs and the "tush push" sneak, is a likely part of its gap. A league-wide model can't know that a specific team has a specialty play. The obvious fix is to tell it which team is on offense, which the next section tests.

Other models, and does knowing the team help?

Two natural questions follow. First, is the random forest the right tool, or would a simpler or a stronger model do as well? Second, since teams clearly differ, does telling the model which team is on offense improve it? I tested both on the same 2023 holdout, with the same pre-snap features.

For team information I tried two versions. One adds a one-hot column per team, so the model can learn each franchise's tendencies directly. The other adds a single number: the team's pass rate over expected from the previous season, a measure of how pass-happy its play-caller is once situation is accounted for. That second version can generalize to coaching changes, because it updates each year.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier

# Last season's pass rate over expected, joined onto each play
proe = (plays.groupby(["season", "posteam"])
             .apply(lambda g: g["pass"].mean() - g.xpass.mean())
             .rename("prior_proe").reset_index())
proe["season"] += 1                                   # 2022's value describes the 2023 team
train = train.merge(proe, on=["season", "posteam"], how="left")
test = test.merge(proe, on=["season", "posteam"], how="left")

logit = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
xgb = XGBClassifier(n_estimators=400, max_depth=6, learning_rate=0.05,
                    subsample=0.8, colsample_bytree=0.8)
rf_proe = RandomForestClassifier(n_estimators=300, max_depth=12, min_samples_leaf=20)

for model, cols in [(logit, PRESNAP), (xgb, PRESNAP), (rf_proe, PRESNAP + ["prior_proe"])]:
    model.fit(train[cols], train.is_pass)
    acc = ((model.predict_proba(test[cols])[:, 1] >= 0.5) == test.is_pass).mean()

Figure 7: Accuracy on every 2023 run/pass play by model and feature set. The axis starts at 70% to make the small differences visible.

Model family matters a little. Logistic regression reaches 72.1%. It can't easily represent thresholds like "3rd and more than 2" without hand-built features. The random forest gets 73.6%, and XGBoost, which builds trees one after another to correct the previous trees' mistakes, does best at 74.2%, with the best AUC and Brier score too.

Team information didn't help at all. The one-hot team columns made the model worse (73.1%): it learned each franchise's 2020–22 habits, and those didn't carry over to 2023 coaching staffs. Last season's PROE was neutral for the random forest (73.5% vs. 73.6%) and slightly hurt XGBoost. Even Atlanta, which had the second-most run-heavy prior season in the league (−13.7% in 2022, behind only Chicago) and was the most run-heavy team in 2023, gained only 0.6 points of accuracy from it. Team tendencies are real, but they're small next to down, distance, and formation, and they shift every time a coordinator changes.

Limitations

No personnel or formation detail beyond shotgun/no-huddle. Knowing whether it's 11 or 22 personnel would very likely add accuracy.
Scrambles and sacks are labeled as passes (they were pass calls), but RPOs are labeled by what happened, not what was called.
Team effects were tested with team identity and last season's PROE only. A within-season version, updated week by week, would track coaching changes faster.

Data and code

Play-by-play from nflverse, regular seasons only, limited to plays whose type is run or pass (excluding kneels, spikes, and plays wiped out by penalty). The 2019 season is used only to compute the prior-season PROE for 2020 teams. Every number and chart comes from football-analysis/play_prediction.py.

← Back to all posts