What continues to pique my interest regarding football, numbers and front office theory
Consider it a bottle episode. Just some stuff I've been thinking about lately
This is more of an old school blog post. Maybe you’re wondering what sort of soccer stuff I’m thinking about during this absence of recurring and/or frequent additions to the “Absolute Unit” and “Theory of Soccer” series. I wonder about this at least. So much, so that I thought I’d put some of it down into a post. Consider this a fun self Q&A where the individual questions aren’t answered while the main overarching question is “what still excites you about thinking about soccer?”
Scouting/Recruitment
Off the ball stuff
How much of a player’s contribution to goal scoring is his off-ball output (his positioning and his movement) as opposed to what he does when he has the ball?
How much of this off-ball output is an inherent ability - the sort of thing you scout for, and how much can be quickly taught by your coaching staff? How much of it is luck?
How much of a player’s contribution to [his team not conceding goals] is his off-ball output (his positioning and his movement) while in attack?
If the answers to any of the above questions are “a significant amount”, then how do you go about recruiting players (with limited resources)? It’s one thing to have the tracking data (and to be able to measure this directly), but let’s say you just have event data and/or scouts watching wyscout… how do you cast a wide net for offball skills that are evidenced by things that are not as easily or as directly recorded? (or are they?)
If part of the deal with baseball and “moneyball” was that in general MLB front offices understood that to score you had to get players on base and then move them around the bases to get home, and they knew it was hits (and homeruns) and walks that accomplished this, and they’d been tracking these things for over a century but that they hadn’t exactly figured out how much to spend on one thing relative to the others (they were spending too much on hits and not enough on walks), I suppose I wonder do we have a similar situation in soccer? In general, front offices understand that a team has to move the ball into their opponents penalty area more than the opponent moves the ball into their penalty area, and let’s say they intuitively know that this involves off-ball movement and on-ball decision making and execution, but do they understand how much to pay for off ball contribution relative to the on-ball stuff? It’s obviously not so simple, because in baseball acquired walks, hits, and homeruns stack somewhat agnostically atop one another the way uh.. money does, but in soccer ball progression needs all of the stuff, a combination of all of the stuff to work well (with room for varying flavors/team styles). Even still, I wonder if the on-ball stuff is overpriced relative to movement simply because it exists in easily digestible scouting playlists and event data.
Transition phase stuff
How much of a player’s contribution to both goal scoring and [his team not conceding goals] occurs in the split seconds before and after turnovers of possession (in both directions)?
Is the above contribution (transition immediacy?) an inherent ability - the sort of thing you scout for, or can it be taught quickly by your coaching staff? How much of it is luck?
Along similar lines as the off-ball questions, and piggybacking off of the Theory of Soccer posts, the real question here is whether there’s something critical to goal scoring and goal prevention that’s not easily inventoried in the data, or compiled in the every-touch playlists, and therefore it is underpriced in the market.
On Regression
Soccer analytics has been harping on regression to the mean for a decade now, specifically the regression of goal scoring rates to some underlying mean goal scoring rate like the avg xG/shot. But as Michael Mauboussin points out in The Success Equation, regression is just something that happens when two things are not perfectly correlated. Past and future goal scoring rates are hardly the only things that regress to some sort of mean in football. What does the full list of regressors look like (it’s basically everything), and how would we rank them in terms of the exuberance with which they revert to the mean.
Not just G to xG, but xG to a team’s long term underlying ability to create chances (likely closer to a league average). Not just passing completion percentage to xPass %, but xPass % to some average figure. Not just saves to xSaves, but xSaves to some more stable xSave rate. Also, the frequency of different types of gamestates experienced by a team, numbers of possessions, transition phases, the list goes on surely.
Soccer analytics has contributed much to guarding against the sort of cognitive biases that make us go “oh this team’s banging in lots of goals, certainly it will continue” but it’s probably also offered up a slightly misleading anchoring metric which isn’t as stable as we like to think it is — the shots themselves and their estimated xG values. It’s easy to think of these stats as “hard” and “real” because of how much more predictive they are than goal scoring rates, but they continue to be “not very predictive” in absolute terms partially because there’s tons of regression and noise in those tallies also.
I’m tossing around this idea in my head which cuts against a lot of my politics around event data. I’ve been fairly harsh in the past on what I frame as the problem associated with using event data solely to judge a player (or even wyscout clips which are basically just event data with a little more context). I’ve criticized this idea that we could measure how good a player is based on what he does having found himself somehow in a situation where he has the ball. The italics here are me somewhat sarcastically stressing that soccer is more than just what you do when you find yourself with the ball obviously, because it’s also what you do without the ball, it’s how you make yourself available to receive the ball, it’s the runs you make to disorganize the back line and to create the possibility that you might get the chance to do something having received the ball in a good spot, or that you might help your teammate do something cool with the ball. You guys remember all the ink I spilled on this. But…
When we zoom out for a minute, it is… somewhat true that stuff just kind of happens to you in soccer. Because you don’t control what the other team does. And despite your positioning and your anticipation, and your touch you don’t control what weird movements the ball will take no matter how hard you concentrate to try to control where it goes next (in defence or attack). While you might influence the degree or frequency with which your opponent makes critical mistakes, you do not control it. So with that in mind, I think the question I’ve been pondering is how much of what happens in a game (what a team experiences) is stuff that a team can reliably repeat with some consistency because of its talent, tactics etc, and how much is a bunch of situations that are either under represented or over represented compared to some long-term average of the kind of situations a team finds itself in a game where it does not substantively influence this frequency.
The example that comes to mind is Brendan Aaronson’s goal for Leeds against Chelsea early in the season. Chelsea blunders during buildup and Aaronson pick pockets the keeper in the goal mouth and taps the ball in (a “chance” if you want to define a shot as a chance worth probably 80-90% of a goal - I can’t actually remember the xG value). Traditional xG analysis for the purpose of analyzing team strength would probably focus in on the fact that the shot was very likely to be scored and that Leeds created a big chance and teams that create big chances consistently are more likely to win etc. If Aaronson had failed to score somehow, that would’ve been the unlucky 10-20% of the time, but since he scored — they’re running about 10-20% hot. xG-based analysis of Leeds would say, “look at Leeds creating high quality shots, a very healthy attribute for a team’s long term results. Teams that have high xG are good.” A possession value model would look at the moment Aaronson takes the ball off the keeper and value that at something a little lower (I don’t know the number but let’s say 50%). After he wins the ball in Chelsea’s box, there’s a really good chance he’s going to get a shot off (but it’s not 100%), and given how close he is and the condition of having just won the ball, a pretty good chance the shot will go in (let’s carry forward that 80-90% idea), and those things together make for a really good chance at a goal before he’s even thought about shooting. Possession value-based analysis of Leeds would include that 50% goal probability possession moment and say, “look here at Leeds creating nice and juicy goal probabilities (not shots, but some other kind of unit) through their press, a very healthy attribute for a team’s long term results.” And of course, either of these analyses based only on a single game’s worth of shots, or a single game’s worth of possession values is vulnerable to small sample sizes. We already know this. This is well caveated and publicized in the soccer analytics spaces. The reason behind this is just more of the same: small sample sizes and mean regression. But even at the 10 game mark, a team experiencing this 80-90% Chelsea goalkeeper fuckup situation to their benefit just one time can skew their aggregate numbers to being too biased to be a good predictor of future results.
When we use data it’s second nature to anchor our insights into some _underlying_ number.. to say not this over here which is ephemeral, but THAT OVER THERE which is solid, stable, persistent. The underlying numbers! And I guess my question is: where actually is the firm footing in soccer? It’s not in the scorelines because finishing is noisy as hell, and not really in the shots (xG) either because those are still pretty noisy except in the long run when we’re all dead. Other discrete possession moments like “chances to shoot?” I don’t think that’s the ticket either. Even over the course of a season, all of the different teams experience incredibly favorable or incredibly adverse goal probability moments at rates which are neither a) at league average rates nor b) due to anything we can identify as particularly causal (i.e. they also don’t represent anything particularly solid as relating directly to that team vs all other teams).
For the purpose of forecasting future team results (and then as a knock-on impact, the concept of sanitizing a player’s historical performances to remove noise), should we just be simulating some season-long randomized collection of team “experiences” that aren’t necessarily causally linked to their style of play or to the talent of their players, or anything else? And then maybe add to this simulated portfolio of experiences, the rates at which teams consistently handle those different states better than other teams and with varying error bars surrounding the different types of experiences and the execution rates attached to each? Does this help or lose all meaning when all is said and done?
At the end of the day, “home team, yes or no” does a pretty good job predicting individual match results, and “who has the highest payroll” does a pretty good job predicting overall season results — it just doesn’t help us understand how soccer works or how to build a team with limited resources. Speaking of, total payroll…
The Moral Hazard of the Sporting Director
At this point there’s virtually a consensus in the soccer analytics and “Moneyball” adjacent analyst community that the primary determinant of team performance in soccer is the skill/ability of the players on the field (one team’s ability relative to that of the others). And it follows that by contrast the manager’s contribution to team results is a great deal less important than the players’ collective contribution to team results. And yet, the manager does have at least one responsibility which has a non-trivial differentiating impact on the team results. He picks the players that make it on the field. The manager evaluates the total player skill/ability available to him and in theory, out of that population he tries to pick the best combination of players to optimize the team performance. This means that while the front office is trying to maximize the team’s performance by improving the primary determinant of team performance in soccer (the total skill/ability of the players on the field), the manager is the final gate through which this process must pass. It is possible for the front office to build a roster of a certain potential team strength but for the team’s ultimate performances to fall short of this by way of the manager keeping the roster’s biggest potential contributors on the bench. And it’s possible for the front office to have misjudged those players and for the manager to be accurately selecting the best players too.
But quickly, on managers. It’s not that a manager is just picking the team. He has a host of duties he’s constantly performing… basically 24/7. He’s a leader, he’s a high level strategist, he’s a tactical planner, a motivator, he’s a trainer, he’s a teacher, a coach, he’s a developer, he’s a listener, he’s (potentially) a recruiter, he’s a peacemaker, he’s an instigator… all of that.
But consider that a striker is responsible for a host of activities too, yet a decade of soccer analytics work has suggested there’s one or two that are outsized in relation to the rest when it comes to impacting performance/results: namely the finding of high probability scoring chances through off-ball movement, anticipation, vision and the ability to get shots off when these opportunities arise. In the same way that data analysis has shown finishing ability is real but just not that important relative to this other stuff, all of those activities I listed off that a coach does are real too. But if we subscribe to the idea that player ability accounts for the vast majority of the performance, then a lot of those tasks/abilities are not differentiating factors. Instead, the two activities the manager performs with the most significant impact on team performance (and over the long term, results), are the ones that directly impact the player ability on the field.
These are:
Picking the players that play in actual games from a population of players that were signed by the front office
Improving the ability of these players through teaching
Again, people may disagree on this, notably coaches. But if we’re following along with these sort of broad analytics teachings, we’re operating from a front office theory perspective of “in order to achieve your sporting goals, you have to maximize the player ability in the squad, and then you employ a coach who also does this by 1) improving the player ability in the squad through teaching, and 2) maximizing the player ability during matches by selecting the best players in the best positions during the games themselves.” And there’s definitely a moral hazard here which we might explore.
The front office is likely confident in its processes and in its appraisals of the players it has evaluated, scouted, and recruited. It likely firmly believes that the squad it has put together is of a certain level of player ability to meet its goals. It also likely has strong beliefs about who the best of those players are (i.e. which players the manager should select in actual matches and where). In fact I would posit that it must have inclinations here or there’s a gap in the process. But the front office having such beliefs presents some problems. Here are two scenarios:
If the team is underperforming its objectives and the coach is picking the players on the team who the sporting director believes should play, then something must be causing this underperformance. It’s either:
a) the Sporting Director’s assessments of the players were incorrect (i.e. he is to blame), or
b) the Sporting Director’s assessments of the players were correct, but the manager is somehow actively destroying team performance via his coaching performance (broadly defined). The trouble here is that there’s not really a theoretical framework in the above by which you can land at this answer if you’re the Sporting Director. If you subscribe to prevailing analytics wisdoms, your main theory of managers is they 1) pick the players, and 2) improve them via teaching (and the other stuff even if they aren’t great at it, largely washes out), so if the team is actively underperforming your assessment of the individual players, you have to wrestle with this question of maybe you’re wrong. Maybe it’s the players you recruited. Or you’re wrong about the manager’s impact being mostly in squad selection and teaching and there’s some other way he’s significantly hurting the team. In either case, you’re going to want to get rid of the manager, right? But that’s tricky because by your own account, it’s more likely that performances are the results of the players, and … gulp .. you recruited the players!
If the team is underperforming and the coach isn’t playing the players the sporting director believes to be the best, then in theory, the coach is actively hurting the team’s production through poor team selection and he should stop doing that or the Sporting Director should find a new manager. Of course there is an issue here, which is that in theory the manager is the one in the best position to determine who the best players are for a given match because he trains them daily for months on end. The information he’s using to gauge who is going to help the team win is in theory “fresher” and possibly more robust. The Sporting Director could have been wrong in his assessment of the players identified by his recruitment process. The coach might be not so subtly saying “find me better players”. And this is tough, because what’s a Sporting Director going to do? Fire himself? Alternatively, the coach could be wrong about his assessment of players, in which case if he doesn’t budge in changing his team selections, he’s gotta go. This is the story the Sporting Director is more inclined to believe based on his view of things. In theory the Sporting Director should either:
a) remove the manager and find a new manger who will agree with the Sporting Director’s assessment of the players, or
b) remove the players who the manager is selecting ahead of the other players the Sporting Director believes should be playing, such that the manager now has no choice but to play the players the Sporting Director believes to be the best.
That first choice feels more coherent and direct but also a bit uncomfortable doesn’t it? Like, it reads a little authoritarian. The second choice is complicated: it’s indirect and sort of clever, but are you really assured that the manager won’t just decide to play whoever your new replacement signing is over the player you believe should be starting? This approach avoids some of the more nuclear risks involved in swapping managers when their impact on performance is modest, but are we even certain it will solve the problem?
A secret third thing?
The other option is a simple enough concept. The way the Sporting Director determined which players to recruit and sign involved some sort of process (if it didn’t then other organizational changes are required). Long time readers know more or less where my cards lie here — the majority of this newsletter/blog is a meditation on what this process might look like. Importantly, a key attribute of this process is that it is a cycle. I summarized this cycle in one such post on this site as:
Setting competitive objectives (e.g. team goal difference)
Authorizing operating budgets that align with these competitive objectives (#1)
Projecting the marginal goal contributions (rates and opportunities) of current and prospective players
Allocating the authorized operating budgets (#2) to current and prospective players and executing on transfer strategies to build a roster that aligns the team’s projected goal difference (sum of #3) with the team’s competitive objectives (#1)
Revising the projected marginal goal contributions (#3) of current and prospective players as time passes, matches are played, and results occur (new information)
Estimating the value of current contracts based on the relationship between actual budget allocations (#4) and the revised projected marginal goal contributions above (#5)
Making adjustments to the roster as marginal goal contribution projections (#5) diverge from organizational competitive goals (#1) by paying transfer fees or receiving transfer fees (or some combination of the two) based on the values estimated above (#6), and re-allocating payroll and transfer costs (#4), so as to realign projected goal difference (#5) with competitive objectives (#1)
The key here is really #5 “Revising your projections based on new information.” This piece is important to an ongoing recruitment cycle because .. stuff happens. You make a projection, an estimation, and then real life happens. You wouldn’t expect a coach to make an assessment of a player on day 1 and then never change it, so obviously any tendency of a front office to say “we did our homework, we got our guy, the guy is good” without actively monitoring and challenging previous conclusions is poor. But what this monitoring, this re-assessment process looks like relative to the initial recruitment evaluation is a bit of a puzzle probably. Because almost by design the initial recruitment evaluation includes a larger population of data, whether that data be modelled statistical data, or watching a shit load of film. It’s supposed to be “robust” and if it’s not, it should be, or the player you signed better be prettttty cheap on the payroll. But an ongoing reassessment of the projected goal difference a player can add to your team over his remaining contract (this is what we care about) based on new match information is necessarily based on a more limited data set (again, broadly defined) because there are only so many new matches to add to the previous data. Or at least, it is if you are only using a similar data set as to the one you used to scout him in the first place.
Should 6 matches at your club really cause you to change your mind about the 3 or 4 seasons you based the recruiting decision on?
The thing is, you have another source of data now you didn’t have before (if you’ll recognize it). You have proprietary in-house information because you have a player who while he may only play matches once a week (or once every few weeks), trains every day. Some of this new data you have comes from the coach and the coaching staff perhaps: their assessment of his skills and development as time passes after his signing. It’s tough to rely on this specific data point however, because the problem we’re trying to solve here is what to do when you disagree with this information as manifest in the manager’s start/sit decisions. There’s also just the problem of this being an “inside view” to borrow a concept from Kanehman’s Thinking Fast and Slow.
Luckily, that’s not the only data you have access to (or could have access to). There’s nothing (but resource constraints I suppose) stopping you from recording objective statistical data and film from training and imbedding these into your projection update process for wholly separate purposes than those for which it might be used by the coaching staff. I don’t know exactly how to square this with your previous recruitment cycle data set, how to train the data for modelling purposes etc, but there’s something there that you can (and should) consider to try to stay on top of player assessments, projections of their contribution to goal difference over the remaining contract term etc, and then to act accordingly.
What’s the alternative? Just be incredibly stubborn about the decisions you make twice a year? Or incredibly acquiescent to the coaching staff? I guess you could.