What makes a good goalie? There is the obvious answer of “someone who stops the ball a lot”, but we all know that! When we dig down, what really, truly makes a good goalie good?
Trevor Tierney answered this question for us from a training and advice perspective, and you can read that post in our “Classics” archive, but if you’re talking about rating good goalies, it’s a totally different conversation. Check out Trevor’s post if you’re looking for ways to improve, but check out the below interview about RATING goalies… with math. This could get weird!
The Goal Guardian National Championship is an intriguing event to me. It takes goalies from all over the country (the world even!) and tries to compare them head to head, using metrics, to determine who is truly the best keeper in the land. I spoke with the big brain behind it all, Ginny Capicchioni, to try to learn more. As I said, it’s a truly interesting concept!
What Makes A Good Goalie?
Ok Ginny, you guys have set up a numeric “index”, which will measure goalies in an objective manner, using a set course of drills and skill tests. Let’s start with the basics. What was the inspiration for creating this index?
The GRC was created for two reasons. The first was for the goaltenders themselves. We wanted to establish a set of benchmarks that measure the standard competencies for a goaltender. Ranked/indexed goalies are better able to see their current strengths and weaknesses and develop an action plan for improvement.
The second was for college coaches. The GRC is a tool for the identification and recruitment of goaltenders in the most objective and quantifiable way possible.
So what’s in a number? Or more specifically, what’s in your index number? Metrics are so important nowadays. So how did you guys come up with your numeric process? How does the index judge a goalie? Is this a MoneyBall scenario where you are finding what’s truly important? How is that achieved?
The Guardian Rating and Index are both based on a given goaltender’s performance at the annual Guardian National Championship/combine. At the combine we run a number of stations for each of the four key (4) components that we have determined make up a functional goaltender:
Points for each station are weighted and then referenced against a sliding scale of one (1) to seven (7). The scale is used to determine a goalie’s readiness for college play. For example, a rating of seven relates to a goalie being classified as college ready, a goalie with a three (3) rating has functionality in some areas but not others, and a goalie ranking of one (1) relates to a beginner/intermediate goalie.
The Moneyball scenario used in Major League Baseball was based upon the concept that the effectiveness of a player can be quantified with data. The GRC breaks down the core competencies of a solid goaltender through data and analysis.
Since it’s just a test and not a game, could a really good goalie “fail” this test? Could a subpar goalie “succeed”?
The GRC index is based on the testing done over a series of days at the annual Guardian National Championship. To the extent that someone was having a bad few days, then yes, a strong goaltender could possibly score lower than expected in this test similar to a great goalie performing poorly in a particular game. Conversely, a subpar goaltender could potentially receive a score that reflects a score higher than their current level, but it would likely be in the “middle of the pack”.
Keep in mind that at the 2014 event there were sixteen stations testing specific goalie skills. There are many elements built into the test and underlying stations that would prevent a wildly unlikely outcome. For example, a level 4 goalie would not score a level 7 grade.
How do you weight the tests in terms of importance?
I believe that the test is a comprehensive measurement of functionality for a goalie. It measures the extent of what the goalie can and cannot do. As with all athletes, a goaltender’s success rate in a given game / season is based upon mental capacity and toughness, discipline, as well as environmental issues like life experience/change, functionality of interpersonal relationships with teammates / coaches, etc.
If I were recruiting a goaltender, I would definitely want to know what the goaltender’s GRC rating and ranks are. Upon finding a Level 6 or Level 7 goaltender (in comparison to their current age), I would then try to find out as much as possible about the goalie’s personality and mental toughness.
Does the index adjust for age? Can it help you compare goalies across age or even recruiting class groups?
The test does not account for age because it is based on a standard of functionality, i.e., the stations and shooters are the same for all participants. The high school test measures the projected efficiency of a high level high school goaltender. As such, you would expect that the average freshman would score lower than the average rising junior and this was evident after we analyzed the data. A level three (3) may be a low score for a rising senior but a decent score for a freshman.
We have heard from several college level goaltenders who have asked to participate in the upcoming July 2015 Guardian National. If we decide to accommodate these players, the college test and associated scales will be changed to account for a higher degree of efficiency, in order to yield an accurate index for the older, more experienced age group. We would wind up with a high school goaltender index and a separate (more demanding) college/university level goaltender index.
How do college coaches currently judge goalies? How could the index conceivably change or even improve their approach?
Previously, the most efficient way of finding a goalie has been to go to the best programs, either high school or club, and take those goalies. The thought process is that, at the very least, the goalie has been in a top-notch program and been exposed to great shooters and competition.
There are a lot of holes in this theory, but I do understand the thought process behind it. Prior to the announcement of the GRC, an objective analysis and evaluation available for lacrosse goalies wasn’t available. The GRC allows recruiting coaches to move away from the current recruiting model to move a more fact based recruiting method.
The GRC levels the playing field and allows the goalies from the weaker, moderate, and strong programs, to be compared in an independent and objective manner. As a college coach, if you are set on a goaltender that is ranked as a level 4 and take a look at the GRC and see a goaltender from a weaker program that is ranked as a level 6, that college coaches is doing him/herself a disservice by not taking a second look at the level 6.
Is this something that could be on a nationwide level? Or do the goalies need to come to you?
Currently, goalies participation in the annual Guardian National Championship is the only way to receive a rating and index/level. We are looking to eventually add Guardian Regional Qualifiers; however, there are different factors that we have to take into account in order to have the test administered consistently in multiple locations. We take our goal of accuracy and consistency very seriously.
How long have you been testing this indexing process? Why did it take so long to create a working formula?
The test / index’s formula have been in the works for the past 18 months. In order to make it an accurate and consistent evaluation of each participant there were many components that needed to be taken into account. Each testing station had its own variables that needed to be worked through.
After reviewing the results, we created a weighted formula. To test the validity of the formula, we analyzed and compared the results of 35-40 goalies from our training program that competed in the Championship. As we already knew their strengths and weaknesses we were able to compare them to their results on the test to ensure an accurate result.
Goalie play has often been judged subjectively, how is this index purely objective? Or does it still take some subjective tweaking here and there?
Rather than saying our testing is purely objective, I would say that our testing is as objective as possible at this time. I believe that no test can be purely objective as human beings play lacrosse and with humans come error.
For example, the reactionary portion of the testing involves 35 shooters per gender running five different stations with different shooting requirements. These stations have the highest risk of error for the index as the level of shooting that the goaltenders face throughout the testing must be as consistent as possible. Thus we are very conscious of the administration of this part of the test. The 2015 test will be even tighter in this area to ensure ongoing accuracy.
The uniform comparison of skill sets under identical conditions is one of the core objectives of the GRC testing. The current subjective nature of goalie comparison without the GRC involves the reviewing of live or taped game play to account for skill level. The “conditions” that the goalie is under – the offense against them, their defense in front of them, the location/difficulty level of the shots, varies tremendously making the current evaluation of goaltenders extremely subjective.
In addition, the observer sees the goalie ONLY in situations that this particular game yields. If the game doesn’t produce a second/rebounded shot on the goalie, are they able to save it? And if so, how consistently can they make these saves? Are they able to ball handle at full speed and throw accurately?
Who helped design the process and create the testing? Have college coaches shown any interest?
My business partners, the Guardian Selection Board members, and various consultants in the lacrosse community all provided a tremendous amount of support and insight. Our CFO, Charlie Hillock, was instrumental in working with me translating the raw data and connecting it with the competencies of a goaltender.
I’ve since reviewed the GRC in depth with several College Coaches and each thought it was a “game-changer” and a “revolutionary” new tool to assist them in the identification and recruitment of goalies and will also enable them to make their program’s goalies more efficient.
I have to say that this is a really interesting concept. There is clearly still value in a keeper who can simply stop the ball, but this test could help define the position a lot more. It could also yield interesting results when current high school players have completed their college careers… but we’ll have to wait for that!