There aren't many who would argue that a college diploma isn't worth pursuing. And, while I would never suggest that college is for every one, I also wouldn't ever suggest that a diploma isn't worth earning. Even so, there has to be a better way to measure ROI of the large industrial higher education complex than the largely faith-based assumption that we think "it's worth it."
In short, this post is an attempt to simultaneously hack the ROI for EDU and outline a barrier to it. In particular, given the modern technology and social media platforms available these days, we aught to be able to measure ROI by harnessing an array of APIs (possibly with LinkedIn, government data sets, etc..) to assess the output of our graduates.
Because the main line of business in the EDU space is teaching, and the main outcome is learning, the true measure of worth for a college or university has to be some evaluative measure of the alumni application of that learning over time. In other words, the worth of a college has to be obviated by & directly tied to what our alumni do with the education obtained (for better or worse). We can use available API to scour LinkedIn for a whole set of alums from any one college or university along a set of threaded variables to measure that, when combined, result in a score that reflects a return on the investment balanced with the overall costs to educate the individual. What should be the equation that comprises that algorithm is the question at hand?
The title for this post requires a translation for those unfamiliar with the acronyms. ROI stands for Return on Investment, and by EDU, I mean to discuss the higher education enterprise know as college and universities across the land. PRJs stand for Peer Review Journal Articles, long considered the coin of the realm in the higher education space, and PhDs stand for doctors of philosophy in whatever disciplinary stripe. API stands for application programming interface. I am using 4 as a surrogate for "for," to ensure that the headline is Tweet-able and minimizes characters so folks who use the platform have some characters remaining of the 140 to say something else about it.
Since about the mid-1900s, and in particular after the WWII GI Bill, there has been a grand, largely uncontested and widely held assumption that attending college or university was "worth it." That, if one spent four years with her/his nose to the grind stone, the outcome would be better employment, and society as a whole would benefit from the further education of individuals. The trouble is, the ROI on the investment in higher education is largely subjective, and very difficult to measure. Frankly, we don't have a very good way to measure the output of college and university operations, much like we don't have a very good way to measure high quality teaching.
Here's where the identification of the barrier to this hack comes into play. Ever since the McCarthy-era witchhunts, the emphasis on tenure as a means of protecting academic freedom has become a mainstay among the faculty ranks as something to protect with all the might available. Unfortunately, this deeply institutionalized practice, once begun as a means to protect faculty members who might have been saying & researching what some would consider heretical things from the onslaught of attacks from outside the academy and preserve their jobs, has produced the only real de facto management tool available to those who "supervise faculty." The tenure decision. Basically, after seven or so years at an institution, a faculty member comes up for a tenure decision, this is the only real time that she or he is evaluated on all the worthy metrics that comprise the faculty main line of work (teaching, research & service).
Moreover, the product mainly measured and with the most weight in this decision is that which is most easily measured, the Peer Review Journal (PRJ) article, no matter how obscure the topic, journal or small the audience. It's as if the act and art of teaching is a secondary thought in the business of a university let alone, the meager valuation placed on service to the community (either micro local at the college or department level or macro level in the community at large). My suggestion here is that we are distracted by what can be measured, and place too much emphasis on the PRJ, and this emphasis on counting PRJs and in many cases numbers of citations of these PRJs is misguided at best.
The PRJ, might be best left alone and the ROI of the PRJ measured separately from what really matters in the assessment of our institutional ROI. Minimally, it should be relegated to a smallish portion of the whole ROI algorithm. Yet, because it is an engrained cultural aspect of the tenure process, the PRJ will be a barrier to this new method for computing ROI.
With the advent of social media, and LinkedIn in particular, we have the capacity to coagulate the massive quantities of data around what our alumni accomplish as a means of measuring the quality of their learning, and in the end, the return on our investment (which is both research and knowledge transfer by teaching). Think of it this way. If we believe in the products of our operation, might we not best measure our output by the positive accomplishments and good that our alumni do in the world?
Here's where I need some help. If you believe that a measure of the accomplishments of your alumni is the single best indicator of the return on the investments of the work of a college or university, we now have a clearly defined task at hand. What should the variables be that fit into the measurement of ROI? Job title and position? Works of art accomplished? Plays produced? Social problems solved? Innovations made? Patents gained?
The reason I'm leaning toward some sort of API connectivity with LinkedIn is two fold. 1) I regularly quiz my alums who return and report that they have received a promotion and the like and I ask them - "did you update your LinkedIn profile or tell your parents first?" Almost 100% of the time, they slap their foreheads and say, "Oh, no. I updated my profile first!" Thus, LinkedIn has become the single best source to discover what your alums are currently doing professionally. 2) Because LinkedIn is growing exponentially as a place to put your living CV, more and more alums are joining and putting up their profiles. We could do surveys of our alumni. However, those rarely obtain high response rates & are costly. I work for a University that was founded in 1899. There are a large number of alumni. As we sift through the massive amount of data, LinkedIn is the location that we can find the most up-to-date information about our graduates.
We can also harness other available data sets and get a quality rubric that figures into the ROI algorithm. For example, we could compose salary estimations based on national jobs data, and identify other data sets from a variety of sources to mash up the data. This algorithm could produce a number that is calculated in real time, balanced against the investments made (salary of faculty and staff, rents, costs for technology, and all those items that go into the grand expense of running a college or university). Each school that uses this metric could post their scores in real time (like a Klout Score), and those looking for a best case institution to attend could have an objective measure to test whether their own experience may measure up.
Given that this whole hack & barrier is a relatively new idea, please post some ideas about what might comprise the equation in a comment to this post.
The practical impact of broad adoption of such a measurement would be, for the first time ever, an available, objective measure of ROI for any particular school. Instead of the annual rankings produced by magazines trying to sell you a subscription, or the success or lack thereof of the quasi-pro sports teams broadcast across the nation, prospective students can tell which school might have the highest ROI (and may even be able to refine by major).
We might find the ROI evaluations for schools that produced the folks that gave us Enron or Credit Default Swaps come out markedly different than those schools that produce Nobel Laureates, or folks who invent solutions to the climate change problems, or a new way to generate power that eliminates our dependency on fossil fuels.
There are many challenges to this hack. First and foremost is what would we include in the algorithm to comprise the numerator and denominator that will produce the score? Also, how would you differentiate & evaluate the output of students in the fine arts versus those in the business schools? Could it be that some one producing the next Mona Lisa would be more valuable than some one promoted to Chief Technical Officer of a smallish company? What about the person who produces the next Great American Novel? Would she or he be weighed as heavily as the person who gains a patent for some new technological marvel?
Much of what our alumni accomplish may have only value for the individual, or small subsets of individuals. Should we weigh ROI more heavily with people who are affecting broader populations or the outcome of the application of their learning has a global impact? What about the person who dedicates their lives to advancing and raising a fully functional family, and the offspring of those families go do great things? How many generations do we include in the loop of this ROI equations?
Also, it's going to be very difficult to surmount the barrier and ease by which we are currently able to measure high quality participation by those who are the engine of the operation - the faculty members. It's very easy to measure PRJs and Citations. Student evaluations of courses are often figured into the tenure decision as well, but are easily dismissible as only filed by disgruntled students who didn't earn the grade they expected.
Lastly, how are we to balance & figure in negative accomplishments - say, you are the school that taught Ken Lay or Bernie Madoff - as a part of the ROI equation? Are the institutions that gave these men their diplomas partly responsible for the outcomes of the application of their knowledge?
I'm not quite sure what the next steps might be for this hack. Perhaps I need to connect with some one at LinkedIn to explore the possibilities, and answer the question as to if this is even feasible. I'm pretty sure that with the massive data analytic tools we have at hand, we should be able to crunch this kind of data, but I could be way off base.
Perhaps this hack is just the first step. But, the technology that exists to mine vast data sets out there is readily available. Perhaps some bright hacker is already hunkered down on this task and can outline the best way forward at this time.
Perhaps this hack is just the first step. But, the technology that exists to mine vast data sets out there is readily available. Perhaps some bright hacker is already hunkered down on this task and can outline the best way forward at this time.
Near as I know, I'm the only one who has been thinking through this idea. I take full responsiblity for the content. Now that it's free, lets see where the conversation take this. One of the primary sets of thinking that this idea rests, is a prior hack of mine - The Trustworthiness Stool. Also, you may get a flavor of my prior post - Education needs to grab it's swagger.
Funny, because I cross posted this article to the Higher Education LinkedIn group, I'm getting comments over there as well. I just responded to a person over there the following that I think addresses some of the concerns raised by John & Keith.
"The "Show off your best," has worked for a great long while, but aren't those folks the outliers? What can an ordinary person expect from submersion into the four or so years at a particular school and expect to gain? There will always be outliers that will have massive success despite intervention by formal, industrial style schooling. Think Bill Gates, Steve Jobs, and Mark Zuckerberg, all of whom dropped out of school to become captains of their element.
My question relates more to what's the ROI from the perspective of the college or university, less about the ROI for the individual? In other words, think of our operations like venture capitalist firms, but instead of the primary outcome being money and profit, what is it that can be considered our product/outcome? I'd like to think of our institutions as venture capitalists of the mind and the soul. The infrastructure and the intellectual capital we use to seed these minds and souls have to have some measurable output besides the intrinsic understanding that we "do good" for lots of individuals.
Perhaps the fact that it's messy and massive makes this project perfect for the new millennium. It's through the massive analysis of data self reported by alumni of various schools on LinkedIn and other social media platforms, the leavening of big data analysis will even the playing field to create a meaningful measure of ROI for the particular school that mitigates for all the outliers. A school can still highlight the exceptions to the rule, but for the vast majority of those looking at particular schools will find a meaningful report of ROI to go more to the heart of what they can expect from being a part of this venture called college. After all, not every one can be a Malcolm Gladwell or Steve Jobs. What's the median performance expectations after I've been indoctrinated by this college or university? "
Further to Keith and John's thoughtful comments:
Perhaps the colleges and universities have much to gain because if they really are to prove to the world that they are worth seeding with capital to support their ventures, they aught to be able to articulate the value delivered. I understand why some of the elite schools would be reluctant to have their own scores posted as it may become readily apparent that "state U," has a much higher value and ROI, particularly given the escalating costs associated with attending such top-brand-name schools.
I like transparency, as a principle & I'm a huge fan of execution of transparent systems (harder to do in practice than in theory). So, perhaps this isn't something the schools themselves will do. Unlike the US News and World Report rankings that depend heavily on the self-reported data (making those annual rankings very suspect), this algorithm is an open sourced application that anyone can download to their mobile device, punch in a college and ask the question - what's the ROI. In real time the app would scour LinkedIn and several other dynamic data sources where alumni self-report, or data is objectively (as possible) collected, and it spits out the number. Not unlike a consumer reports assessment of the newest Tesla auto, you get a ranking in real time of the colleges you are interested in. The application would have zero affiliation with any institution, and take no advertisements from them, and they would report a number for any school you want to run the numbers. Set the knowledge free, and let the end user figure out if it's valuable. I'm betting you see a shift in the way people select their schools, but also in the way alumni report data, that is, if they care about the schools from which they have sprung.
Thanks for the comments Keith & John. Some day, all three of us should have a google hang out to discuss. Might be fun to record and post as a comment as well - and knowing both of you, I'll bet you would enjoy "meeting" one another.
- Log in to post comments
This is an important and yet very challenging task.
My concerns about measurement of a student's future success, however we choose to define it, is that it will invariably be the result of a wide range of factors - well beyond the quality of their higher education experience. And I suspect that higher education actually plays a less important role than other factors, such as the student's parents (e.g. reading at an early age).
To make any sort of ROI meaningful, then, it would need to include a lifetime of factors - many of which would be difficult to quantify.
Also, I agree with John Matthew's suggestion (comment), that there is little incentive for institutions to participate. And, I would add, the schools with the most to lose by publishing ROI scores are those that are ranked highest (given the perverse logic behind the current ranking systems). Transparent ROI will very likely make clear that the more expensive degrees are adding less value.
- Log in to post comments
"Each school that uses this metric could post their scores in real time"
I like the vision of attenuating the PRJ against other metrics (though it surely reflects some degree of value), and I see the value for potential consumers of EDU, I'm struggling to find the incentive for the institutions themselves. What's in it for the university? Why allocate resources to assess and post the results?
- Log in to post comments
You need to register in order to submit a comment.