by Robert G. Atkins
A sound Program Evaluation System (PES) marshals the data, software, people, and decision-making processes that enable fast, well-informed, and broadly supported program decisions. A PES brings structure and a common vocabulary to program evaluation. Importantly, it also allows institutions to increase enrollment by investing in the right programs while reducing costs.
For example, we believe a healthy PES includes market data on student demand, employer needs, competitive intensity, and degree level. Most of our clients find this self-evident, which leads to a new and consistent vocabulary for discussing and evaluating the market for an academic program.
First, all the data should be for the actual market or markets served by the institution, not states, metropolitan areas, or other artificial boundaries.
Student demand should include current indicators, such as Google search volume and enrollment data, and more detailed historical data from IPEDS. Employer needs should include occupation and wage data from the Bureau of Labor Statistics, job postings, and career path data from the American Community Survey. Each data source should be understood – including its flaws – by the faculty and administrators involved in the evaluation process. Competitive intensity should not just identify competing schools but reveal whether the market is saturated. For example, is the Google competition index too high? Are competitors’ completions growing or shrinking?
The PES should reveal the economics of each program, including revenue, cost, and instructional margin (before overheads). It should benchmark costs across programs and include benchmarks for the same programs at other institutions. Interestingly, this data often ends the use of rules of thumb that dictate the closure of small programs. While some small programs lose money, many others are contribution positive – if cut, revenue would fall faster than cost, leaving the school worse off.
The PES must also be easy for experienced users and approachable for relative novices. In addition to raw data (e.g., the number of job postings), each metric should be compared to other programs using percentiles (much like a grade curve), which makes it much easier to interpret the information.
However, a PES is more than data and software. It defines the process, participants, and schedule for program decisions and their implementation. The process is data-informed, fast (two days), transparent, and strengthens campus relationships.
Today, declining enrollment, aggressive competition, and limited funds doom most ill-informed academic program decisions. When program proposals cover different topics using unique, unvetted data, it is challenging for decision-makers to evaluate them. A PES can enable a higher-education institution to gather disparate data to evaluate and enhance its program portfolio. Several of the largest institutions now use a PES to guide their program decisions and spur growth.
Robert G. Atkins
Robert G. Atkins is CEO and founder of Gray Associates.
