My Fall-Detection Model Scored 94% — And It Was Lying to Me

The 94% That Wasn't

I still remember the terminal output: 94.3% accuracy. I checked the confusion matrix twice. The number went into my README and onto my CV. It was also wrong — not by a rounding error, but by 25 full points. The honest figure was 69%.

No one caught it for me. It only surfaced because I decided to build tools that could verify my own results. Fixing it taught me more about production ML than building the original system ever did. If you evaluate models on sequential data — video, audio, wearable sensors, time series — there's a decent chance the same bug is sitting in your pipeline right now.

This is a post-mortem of a data leakage bug that looked completely ordinary, and of the evaluation rebuild that followed. For a broader look at how AI-assisted engineering is reshaping the MVP workflow, see our analysis of Go as the unexpected MVP language for AI-assisted software engineering.

The Leak: Random Splits on Dependent Samples

The system was a real-time fall detector for assistive monitoring. A webcam runs through MediaPipe Pose to extract skeletal landmarks. From those landmarks I compute biomechanical features — torso angle, normalized hip and shoulder heights, bounding-box geometry. A Random Forest classifies every frame, and a smoothing window over recent predictions decides whether to raise the alarm. Everything runs on CPU. No video ever leaves the device.

I trained on the Le2i fall detection dataset and used the standard, innocent-looking line:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

The problem: this split is random over individual frames, and frames from the same video are not independent. Two frames captured 80 milliseconds apart look almost identical — same person, same pose, same lighting. When you shuffle all frames together and hold out 20%, nearly every test frame has a close twin in the training set.

So the test doesn't measure generalization to unseen people and rooms. It measures memorization, and a Random Forest can memorize almost anything. This is data leakage in its most ordinary form: the samples aren't independent, so the random split hands the test set the answers in advance.

The fix is simple to state: split by the thing that makes your samples dependent. For video, split by video.

from sklearn.model_selection import GroupKFold

# Group by video_id so no recording appears on both sides of a fold
gkf = GroupKFold(n_splits=5)
for train_idx, test_idx in gkf.split(X, y, groups=video_ids):
    model.fit(X[train_idx], y[train_idx])
    score = model.score(X[test_idx], y[test_idx])

The grouped score came back at 69.1%. One fold, with the same code split honestly, landed at 31%. That's the real number.

The Labels Were Worse Than the Split

Once I audited the split, I started auditing everything else. My training labels came from folder names — if a video sat in falls/, every frame was labelled fall. But a staged fall lasts one to two seconds; the rest of the clip is ordinary activity. The dataset actually provides frame-level annotations marking the exact start and end of every fall, so I cross-referenced them. The result was embarrassing:

• 76% of frames carrying a fall label showed the person just walking around normally.
• Only 6% captured an actual fall.
• The remaining 18% showed someone already lying on the floor afterwards.
• One 78-second video contributed 44% of the entire positive class, and 1.5 seconds of it was a fall.

My labels had silently changed the task. The model solved the easier problem they described, not the one I meant to set. A leaky evaluation can't tell you that your labels are wrong, because it only measures memorization — and a model can memorize wrong labels just as easily as right ones.

Rebuilding from Ground Truth

I rebuilt the training set from the annotations:

• A frame is positive if it sits inside the annotated fall window or the two seconds of lying that follow it.
• Everything before the fall is negative.
• Ambiguous frames late after a fall (person may still be lying down or getting up, no ground truth) were excluded, not guessed.

I also used the whole dataset — 186 of 190 videos, instead of my original 28. Two scenes had no annotation files at all, so I watched 59 videos frame by frame and hand-annotated 29 fall windows myself, with a safety buffer around my estimated boundaries so my own imprecision couldn't contaminate the negative class.

Measure What the User Experiences

A caregiver never sees a frame. They either get an alert or they don't. So the evaluation now simulates the exact production alarm logic — a voting window over per-frame probabilities — on held-out videos, and reports event-level results:

Every miss came down to the same cause: the pose estimator lost the person mid-fall — occlusion behind furniture, motion blur, or footage too low-res to track. If the pose estimator can't see the fall, the classifier never gets its turn. You only find that out by evaluating whole events.

The Trade-off a Single Metric Hides

Honest evaluation surfaced a decision I never noticed I was making:

Neither model is wrong. In elderly care you probably want an alarm whenever someone is on the floor. If the goal is to distinguish a fall from a lie-down, the motion model wins. The behavior is controlled by the label policy, not the architecture — and a single accuracy score would have hidden the entire trade-off. I ship both and document the difference.

For a look at how feature-flag tooling is evolving alongside this kind of production ML work, see Vercel Flags GA: a deep dive into feature flag management.

The Checklist

If you take one thing from my 25 wasted percentage points, take the first item:

1. Group your splits by whatever makes samples dependent — video, patient, subject, session, device. If switching to a random split makes your score jump, the higher number is the liar.
2. Compare your labels with real annotations instead of trusting folder names. Then look inside your positive class and see what it actually contains.
3. Measure whole events, the way a user would: how many incidents it caught, how quickly, and how often it went off when nothing happened.
4. Keep a held-out behavioral test for the failure mode that scares you most. Mine is a deliberate lie-down that must not raise an alarm.
5. Publish the correction. A corrected system people can believe is worth more than an impressive one they can't.

I only recorded the 55-second live demo — a real fall, pipeline live, cloud dashboard turning red, alert acknowledged — because I finally trusted the system behind it. I would never have dared to film the 94.3% version.

Limitations and Caveats

• The Le2i dataset is staged falls in home-like rooms, not real emergencies. Real-world distribution shift will be larger than anything measured here.
• The 96% detection rate is on held-out videos from the same dataset. Cross-dataset generalization is untested.
• Pose estimation fails on occlusion, motion blur, and low resolution — this is the dominant remaining failure mode, and it's a perception problem, not a classifier problem.
• A single-metric headline is never enough for safety-adjacent systems. Report event-level metrics, latency, and false-alarm rates together.

Next Steps

If you're building anything similar, the priority list is: (1) fix your splits, (2) fix your labels, (3) move to event-level evaluation, (4) then — and only then — tune the model. Most of the gains hide in steps 1–3. My next round of work is on perception, not classifier tuning.

Source: My Fall-Detection Model Scored 94%, and It Was Lying to Me — aggregate metrics only, dataset used with attribution to Charfi et al. (2013).