I spent most of my career building data systems at enterprise scale. Big warehouses, big pipelines, models trained on tables that other people had already cleaned before they ever reached me. I did not realize how much that setup was cushioning me until I strapped a sensor to a tennis racquet and tried to make it think.
The idea was simple on paper. Put a small motion sensor on the racquet, capture the swing, and tell the player what shot they just hit and how well they hit it. In real time, on their phone, on the court. I figured the machine learning would be the hard part.
The machine learning was the easy part. Here is what actually humbled me.
Before the lessons, here is the whole system end-to-end, because every hard problem lives somewhere in this pipeline:
- Racquet IMU (accel + gyro): the raw motion stream, most of which is not a swing.
- Phone app, over BLE: receives the stream, and drops it more often than you would like.
- Swing detection and segmentation: find where a swing starts and ends inside constant noise. This was half the problem.
- Feature windows: a short window around the hit, turned into a handful of cheap, explainable features.
- XGBoost classifier: the smallest decision in the whole system.
- On-device inference, under 100 ms: no server round trip, or the feedback is about the wrong shot.
- Shot type and quality feedback: delivered before the next ball.
Real-World Data Does Not Arrive in Rows
In the enterprise world, by the time data reaches your model, it has been through ingestion, validation, and a dozen transformations. It arrives in rows. It arrives late, sure, but it arrives clean enough to reason about.
Sensor data arrives as a firehose of motion, most of which is not the thing you care about. A player walking to the baseline, bouncing the ball, adjusting their grip, and swinging all produce motion. My model did not get a labeled “here comes a forehand” flag. It got a continuous stream, and the first real problem was not classification; it was detecting that a swing had even happened, and where it started and ended, inside noise that never stopped.
That is why the smallest, last line of the whole system is the model call, and everything before it is the actual work:
# The Part that actually mattered: turning a raw motion stream into
# features, and only classifying a window once a swing was detected.
def on_imu_sample (sample, buffer):
buffer.append(sample) # continous accel + gyro stream
if not swing_detected(buffer): # segment before you classify
return None
window = buffer.extract_swing_window() # the ~300ms around the hit
features = [
window.peak_accel(),
window.rotation_energy(),
window.swing_duration(),
window.impact_sharpness(),
# dozens more, most of which I threw away after testing
]
return classifier.predict(features) # the smallest line in the whole system.
I had spent years treating segmentation as somebody else’s job. Out here it was the whole first half of the problem.
The Two-Handed Backhand That Broke My Model
Let me make this concrete, because it is the moment the project stopped being an experiment.
My early model looked great. It looked great because I had built and tested it mostly on my own swings and a handful of friends who hit like me. Then a junior player with a two-handed backhand tried it. The sensor watched her drive a clean two-hander, and my model called it a forehand. Not once. Every single time. I stood there refreshing the screen as if the app were the thing that was broken, watching a textbook backhand come back labeled forehand, forehand, forehand.
It was not a bug in the code. It was a bug in my imagination. A two-handed backhand has a rotation signature my training data barely contained, so the model did the only thing it could, which was map the unfamiliar into the nearest familiar thing it knew. She was not an edge case. She was a completely normal tennis player who happened to fall outside the distribution I had accidentally defined as everyone. It took me longer than I want to admit to accept that the model was right about what it knew and had simply never been taught what she was doing. That one afternoon did not just rewrite how I thought about my dataset. It sent me back to build a real one.
Your Labels Are Noisier Than You Think
So, I ran a real study, around 50 players, to stop building for people who hit like me. That sounds modest until you try to organize 50 humans hitting instrumented shots and then label every one of them correctly.
Players mislabel their own shots. A slice gets called a flat drive. Two shot types blur at the boundary. The ground truth you build on is itself a noisy estimate, and pretending otherwise is how you end up with a model that scores beautifully on your data and falls apart on the next stranger. I stopped trusting any accuracy number that came from data collected the same way, by the same people, on the same day.
The number I trusted was the held-out one: players the model had never seen, hitting naturally, under the same messy conditions the product would face. It sat well below the score I got on the study crowd, and that gap, not the flattering number, was the real state of the model.
The exact score mattered less than the gap. Performance dropped the moment I moved from familiar swings to unseen players, and that gap told me more about the model than the best study-set accuracy ever could.
Latency Is the Product
Here is where enterprise instincts actively hurt me. In a data platform, a prediction that takes two seconds is fine. Nobody notices. On a court, a player finishes a swing and wants to know about it before they set up for the next ball. If the feedback lands two seconds late it is feedback about the wrong shot, and the product is dead.
So, the model had to run on the device, in under 100 milliseconds, without a round trip to a server. That constraint reached backward into every earlier decision. It capped how big the model could be, which is part of why gradient-boosted trees beat anything heavier here. It shaped the features, because some are too expensive to compute in the window I had. And getting it onto the device meant converting and quantizing the model, then checking, carefully, that the quantized version still agreed with the one I trained. “Accurate” and “accurate after it has been squeezed to fit on a phone” are two different claims.
The practical version was deliberately boring: a short swing window from the accelerometer and gyroscope, time-domain features like peak acceleration, rotation energy, duration, and impact sharpness, and a lightweight classifier that could survive conversion for mobile inference. I learned to prefer features I could explain and compute cheaply over features that looked clever in a notebook but collapsed once they had to run between two actual tennis shots.
At the edge, the model is the cheapest thing you build. Everything that matters is in getting clean signal to it, fast enough to be worth having.
The Court Is a Hostile Production Environment
The last thing enterprise ML let me forget is that the world is hostile. My pipeline at work ran in a data center. It did not care about temperature, connectivity, or a user holding the device slightly wrong.
The sensor had to work outdoors, in glare, over a Bluetooth connection that dropped whenever a player turned their body between the racquet and the phone in their pocket. I lost swings mid rally more than once, which meant the app had to treat missing data as normal rather than as an error. Every one of those conditions is a distribution I did not train on. Field testing was not a QA phase at the end. It was where I found out which of my assumptions were real.
What This Changed About My Enterprise ML Work
I went in thinking my enterprise experience would make the ML easy. Instead, the ML made me better at the enterprise work. Building at the edge, with noisy data and no platform to hide behind, forced me to internalize the things that are true everywhere and easy to ignore when the infrastructure is padding your fall.
That is the lesson I carried back: production ML is not about finding the right algorithm. It is about respecting the gap between a clean training set and a messy human context, then building the system so that gap does not surprise you after launch.
What I’d Do Differently Next Time
- Build the dataset around variation first, not after the model fails.
- Treat segmentation as a core ML problem, not preprocessing.
- Track held-out-user performance from day one.
- Optimize for field latency early instead of after training.
- Test in the environment where the product will actually live.
The biggest lesson was not that edge ML is harder than enterprise ML. It was that the mess shows up sooner. On the court, there is no warehouse, no batch job, and no analyst quietly fixing the data before the model sees it. A tennis racquet taught me that. No data platform ever had to.