Among the many critiques of Neal Huntington in his tenure as the Pirates GM, one of the more frequently cited is his inability to draft and develop legitimate baseball talent. A quick search online will find you any number of articles critiquing Huntington ?s drafts, some quantitatively, others qualitatively. Even TPOP ?s own Kevin Creagh has written in Pirates Guide 2018 and 2019 critiquing Huntington ?s drafts and inability to find impact players through the draft.
While many of these analyses are thorough and worth reading in their own right, they fall short of a full accounting of the draft; often only focusing on the visible benefits, the players that make the Major Leagues, while ignoring the hidden costs of drafting players who fail to make the Majors.
What ?s needed is a systematic way to analyze the MLB Draft that both penalizes a drafting team for failing to generate Major League talent, as much as it gives credit for creating quality talent. To create this framework I built a dataset of the first five rounds of draft picks since 2008, all the drafts in Neal Huntington ?s tenure as the Pirates ? GM.
The first step is to determine how likely a given pick is to make the major leagues. We can estimate this probability fairly easily by looking at the fraction of players that make it to the majors by draft pick number. Since we want to avoid underestimating this by including too many players who are still developing, we ?ll limit this analysis to the 2008-2015 drafts; in theory allowing for enough time for the players drafted in the later years the time to at least make a September call-up. Additionally we ?ll limit the players we look at to only signed players, this avoids the ?Mark Appel Case ? where a team drafts a player, fails to sign them, then that player goes on to never make the majors, while the team receives compensatory picks which are worth analyzing.
Below is the probability of a player making the major leagues by pick number; the black horizontal line represents the 50/50 mark.
While illustrative of the intuitive fact that nearly all earlier picks have a higher likelihood of making the MLB, the data is pretty noisy, in large part because we have a sample of 7-8 players to pick from, leading to a highly variable result.
A fix for this small sample error is to simply bin together approximately similar picks. For instance if we assume picks 1, 2, and 3 are all of relatively similar quality, a fair assessment based on the graph above, and the same with picks 4-6, 7-9, and so on through picks 148-150, then we can increase the sample that we ?re estimating this probability from, while still maintaining much of the signal we care about. Essentially we ?re ?binning ? our draft picks into 3 player groups and making the educated guess that they ?re all relatively equal.
Below we get a much smoother looking graph and a better overall estimate of the probability that players selected in a given ?draft bin ? will make the major leagues.
Again, more clearly: the earlier the pick the higher the likelihood that they will ever play on a major league field.
Alternatively we could just assume that players taken in the first 30 picks (approximately one round) are better than those taken in the next 30 and so on, effectively making ?30 pick bins ?. The results are represented below; we ?ll keep it in our back pocket for later. One note is that the x-axis is listed as ?Approximate Round ? meaning every 30 picks, this is due to the MLB ?s shifting of the draft design along with varying numbers of compensatory picks every year, so ?Round 1 ? is picks 1-30, 2 is 31-60 etc.
The problem with looking at simply the probability of draft picks making the majors is that it doesn ?t factor in the outcomes of these players once they make the majors. For instance, what if the first round produces more stable major league talent, but subsequent rounds have more tendency of producing higher impact players?
To do this, I brought in each draft pick ?s respective fWAR numbers and averaged them over the player ?s first 6 years of play. This allows us to compare players with differing amounts of service time during their rookie contracts, or the time a team knows they can control a draft pick. We ?ll use the 3 pick bin method again to leverage the larger sample size.
Here we see that in our first bin, picks 1-3, had an average annual fWAR of 2.35 wins. This was more than a full win better than the next highest, Bin 3 (picks 7-9) which averaged 1.30 fWAR per year.
However, when you look at the graph of the probability that a draft pick makes the majors from above, we can see that picks in the first bin only have about a 78% probability of making the majors, while Bin 3 picks have a 95% chance of getting to the majors. What this means is that we ?re not accounting for the fact that there are more first bin draft picks that aren ?t making the majors relative to their third bin counterparts. In other words, these fWAR expectations are valid only for players who actually make the major leagues, but if we want to analyze overall draft performance we need to factor in the players that never make the majors.
Making this adjustment is ultimately somewhat arbitrary. Since these players never actually make the major leagues we have no idea what their value would have been. We can reasonably assume that teams are rational and would prefer to play replacement level players over worse-than-replacement level players, all else being equal. Since these players never made the majors, by that logic they ?d have been below replacement level, had they played; in other words they ?d be worth negative WAR.
The only question that remains is how much? For this I looked at Fangraph ?s leaderboards for all batters between 2008 and 2015 and found the lower bound of WAR to be somewhere around -2.5 wins. Since this is the best estimate I could come up with, I assigned it to all drafted and signed players who did not make the major leagues between 2008 and 2015.
This adjustment is necessary for two reasons. Reason 1) if a team selects a player that ultimately can ?t make it to the major leagues, they ?re expending resources in the form of coaching, etc. that they therefore aren ?t focusing on players with higher ceilings. Reason 2) if a team that selects a player that can ?t make the majors, they ?re ostensibly missing out on a player that can make the majors, thus spending the draft pick to no team benefit.
Below is the expected WAR by binned pick, with the adjustment for players who failed to make the Majors.
This graph is rather bleak. In essence the draft, on average, does not produce positive major league value beyond the 15th pick. While obviously there is upside for particular players, overall fans should not expect drafted players to either make the major leagues or be productive upon making the majors. This should provide a better, if not depressing, baseline for those fans who follow the draft: the average draft pick is not that valuable.
To highlight the negative average value point, we can again look at the draft using our ?approximate round ? framework from before.
What is clear is that no round provides positive value for teams, on average. This is interesting because even though, as we saw, first round picks are the only group with a better-than-50/50-chance of making the majors, on average they still fail to produce positive value for the teams that draft them.
In spite of this negative value, these two graphs provide us a framework to analyze each front office ?s ability to select talent since 2008 relative to the average drafting team, since all teams face this negative value prospect. If we simply take the draft pick ?s actual annual WAR (assigning players who didn ?t make it -2.5) and subtract out the expected WAR based on their 3-pick Bin or Approximate Round drafted depending on which way we want to run the analysis, we can see if a player worked out above or below their expectation. The WAR above the Binned expectation will be denoted BEW for ?Binned Excess WAR ? and the WAR above Round expectation, REW for ?Round Excess WAR ?.
Individually, it ?s difficult to determine whether the team did well to identify a specific player ?s ability or were just lucky; however, if we sum these BEWs and REWs over the course of a draft, or several drafts, we can get some estimate on an organization ?s ability to systematically identify quality major league talent.
Doing so gives us the table below of the top half of the league in Binned Excess WAR and Round Excess WAR.
|1||St. Louis Cardinals||24.3|
|2||San Francisco Giants||18.3|
|6||Los Angeles Angels||11.1|
|7||San Diego Padres||9.7|
|8||Boston Red Sox||7.6|
|9||Los Angeles Dodgers||4.6|
|12||New York Mets||1.5|
|13||New York Yankees||1.4|
|15||Chicago White Sox||-1.2|
|1||St. Louis Cardinals||22.2|
|3||San Francisco Giants||13.6|
|7||Los Angeles Angels||5.6|
|9||Boston Red Sox||5.1|
|10||San Diego Padres||4.7|
|11||New York Mets||4.4|
|14||Chicago White Sox||3.7|
|15||Los Angeles Dodgers||1.1|
Frustratingly, though ultimately unsurprisingly, the St. Louis Cardinals are far and away the best team at identifying and developing drafted talent. The Pirates, on the other hand, are the definition of average, posting just 0.2 and 3.8 WAR per year above their Binned and Round expectations respectively.
There are examples of small market teams being competitive without drafting effectively: the Rays ? -18.1 BEW and -21.7 REW, are 29th and 30th in the league respectively, as well as the Milwaukee Brewers with -12.4 BEW and -15.7 REW, ranking 27th and 28th. These do, however, seem to be more the exception rather than the rule.
It is also worth noting that this isn ?t a complete analysis, just a framework for continued analysis; however, this article does provide some sense of teams ? abilities to assess talent in the most impactful rounds of the draft. In order to have a complete draft analysis, the complete set of draft picks since 2008 should be compared using the BEW and REW methodology to determine the organization best at MLB drafting.
While not a complete analysis, it is clear that the Pirates are rather mediocre at drafting, at least through the first 5 rounds of the draft. While sitting in the top half of the league, they are situated behind divisional opponents in the Cardinals and Cubs, limiting the potential for winning the division. If the Pirates want to be competitive while just being mediocre at drafting, they need to be effective in the other aspects of team building, like knowing when to separate with prospects at their peak value, getting undervalued prospects when trading major leaguers, and finding positive value in international free agency.
Mediocrity in drafting is not a death knell for a team ?s competitive hopes, but it does reduce the number of scenarios in which they can become competitive.