Any one on the tenure ratchet knows that the lag time between writing a high quality, empirical journal article and it's actual print publication date can be irritating at the least, and downright frustrating & infuriating most of the time; particularly if one's tenure decision hangs in the balance. On the opposite end, by the time the consumer - that would be you, the reader - is first exposed to the work in it's respectable publication its two or more years old (and even older if you count the time it takes to conduct the studies, and get it through the peer review process). There is a danger in applying "new" knowledge that may be trumped my even "newer" studies executed well after the present study has been concluded an finally published.

The solution here is to un-tether the peer review process from the archaic machinery of the old-wold publication houses, and unleash the power of the internet to beta test research articles in real time. That is, by publishing your work as it unfolds, you expose your work to a larger audience (of peers and non-peers as well), and get instantaneous feedback that may lead to an overall improved effort, and hopefully better knowledge.

Problem

The lag time for new knowledge to penetrate to the population is about two years average, and more most of the time
Gate keeping by the journal review process limits the amount of new and inventive research available to the public
The existing tenure rules and publication requirements make information and knowledge isomorphic (in other words, you get only fractional improvements in what we know rather than frame-breaking solutions being invented all the time)
The publication industry is highly dependent upon an all volunteer corps of expert reviewers and a steady stream of authors willing to subjugate themselves to the process
widening the circle of involvement is a big challenge

Solution

Much like open sourced software organizations build out and beta test applications, we can build out and beta test new journal articles before they are submitted - or may even remove the need for the articles to be submitted to a "journal," and in that way get the knowledge closer to the end user in a swifter fashion.The idea here is to unleash the power of the internet to beta test journal articles much like software development organizations beta test applications. Once your research gets to a point where you have publishable results, you write a brief, post it on the internet, and like this Mix exchange, you invite people to make comments. In this way you end up harnessing the power of a global village rather than restrict review to only the handful of carefully selected gate keepers we know to be the peer reviewers. As feedback coagulates around the most salient areas, you can discern which elements need of your work need be retained, tweaked or junked.

Practical Impact

The effect of opening up the peer reviewed journal to beta testing may lead to the ultimate demise of the traditional Peer Reviewed Journal (PRJ) process. Unbuckling those engaged from this time consuming process may produce substantial amounts of slack in the system to provide more time for more inventive research and get us to speedier solutions to long standing empirical problems. Effectively, the solution should move the development of knowledge many steps closer to the end user, and encourage practical application of the new knowledge to determine if it survives the acid test of applicability.

First Steps

Well, I believe Gandhi said it best - the first step for us in making this change real is "being the change we want to see in the world." Beyond that, I have no idea how to make this solution work in the real world as the barriers to it are enormous and institutionalized in the very fabric of the academic profession. What do you think would be the best way to set this hack on a pathway to being realized?

Credits

I don't think I'm the first one who had this idea, but it came to be fully realized while I was having coffee yesterday with one of our alumni - JJ Panzer, from Cohort 13. Together we can flesh out the mechanisms by which we can make this happen. Clearly, the technology is there to make it happen. When iPad and "thread books" become the means of knowledge delivery in the next few years, academics will have to change or we risk being leapfrogged by our students and becoming obsolete sages on the stage, hopeful for an audience that never comes. And as George Kuh always used to tell us in our classrooms while I was at IU, "There's no one stopping the students from NOT coming."

You need to register in order to submit a comment.

Sort By: Latest Oldest Likes Replies

October 14, 2010 at 12:13pm

Suzuki Cady

This is just the kind of thoughtful questioning and bold engagement more people need to develop, certainly not just in this field!

October 14, 2010 at 1:33pm

Ellen Weber

Well stated, Aaron, and thanks. What an interesting fix to a broken system! Thanks for your thoughtful reflections here.

Recently a good friend, who writes such articles as I too have enjoyed doing, told me a shocking reality.

He said statistics indicated that peer reviewed articles get an average of 4 readers per work. Yikes! Is it possible that statistics (of less and less interest in articles that command a very narrow readership) drives change - such as the kind you suggest here? An interesting question ...

October 14, 2010 at 5:56pm

Dan Oestreich

The other upside is the potential for re-engineering funding. increasingly, as I understand it (and I this is mostly hearsay from scientist friends) proposals are evaluated positively to the degree they can guarantee the results in advance rather than truly open a door to a unique exploration. That is, a proposal is more likely to be funded when the research has, in effect, already been done. Perhaps in some way the reinvention you are proposing could also be a step toward renewing faith in more open-ended forms of research.

October 15, 2010 at 11:23am

P. Singh

Aaron,

It must have taken courage to write something so anti-establishment. There is great truth in the problem as defined. You may like to consider a course correction in the solution.

The need is feedback from Beta sites. I have yet to see a guinea pig volunteer on the net or anywhere. This has to be part of a business deal where the volunteer gets a share of the possible success and the frontiersman gets the proof point. Such business deals have to be built in advance. It would be too expensive to negotiate them on a case by case basis. So a facilitating organization is needed to foster early adopters per suitable terms and conditions. All that an intrepid innovator need do is approach the facilitating organization, register, obtain the list of early adopters, decide who is a likely target and convince the target to take a call. The terms will already be on the table. It is the communication that will decide. The biggest gain will be interacting with the likely customer from day one per established terms.

If only the developing countries could organize themselves on the above lines they would swiftly catch up with the developed world. It is so much cheaper to experiment in the developing countries. They have the infrastructure and the time and the ambition.

I must acknowledge I got this thought while reading Raj Kumar's story.

Regards,

P.Singh

October 26, 2010 at 10:28am

Aaron Anderson

Dear All,

Something must be broken with my feed as I just now realized that some folks have commented on this hack. Typically, it shoots me an email every time some one comments. Thank you for taking the time to review my idea, and Singh, I think if those in developing countries were able to make this a reality, they could quite literally leapfrog those in the supposed developed world, and teach us a thing or two along the way.

The other problem that this solution may remedy the age old Catch 22 for emerging and new scholars – how do you get one of those coveted faculty positions if you don’t have any publications when you can’t get any publications because you don’t have a faculty position. In other words, it’s very difficult to get an established research agenda without obtaining first a faculty gig, but one cannot get a faculty gig without first generating a research agenda. Which comes first? By beta testing your work, perhaps you can have many people – even those unaffiliated with research operations – posting their findings. In this way, if you have sound methodology, you could conduct your own research and not require the heavy burden of infrastructure required to become a part of the establishment.

December 12, 2011 at 8:24pm

Brett Kokot

Thanks for sharing your idea with me Aaron, and I'm on board so far, I think this would be great!

I think we can go a bit farther with this though. Your method only seems inclusive of potential new research and articles, I think a critical component could be increasing the ability to get access to the information that already exists.

One of the reasons students like google is because it's easy to use. Even the most user friendly of scholarly article search engines (EBSCO host) is fairly difficult to use, especially if you aren't exactly sure what you are looking for. Another problem is that most scholarly journal search engines like this require an access fee, which students usually have for free from their university library, but lose access to once they graduate and are part of the workforce. So I believe some sort of new search structure would be optimal.

Another great feature would be to encourage authors to provide summaries of their published works. Many people writing for these journals are also teachers, so encourage them to use that part of their job description. Academic papers can be very intimidating to those who aren't experts in the field in which the paper is written. Writing an executive summary even well after your work has been published would dramatically increase the readability and access to the information.

I'm not sure if you've seen this before, but it seems very closely related to your idea.

http://www.ted.com/talks/lang/en/michael_nielsen_open_science_now.html

It's talking about science and math, but it can easily be applied to other areas as well.

Good work so far, but don't stop here! Keep going.

December 13, 2011 at 10:13am

Aaron Anderson

Brett,

Thanks for taking the time to pop over and leave the comment. I'm listening to Michael Nielsen's talk right now. Seems like a great approach. In my classroom, I'm not accepting papers any more as deliverables for any writing assignments. We use sites.google.com to build out web locations that contain useful content which is shareable to a wide audience depending on how comfortable one is with spreading their ideas. Minimally, this expands the typical audience from two - the instructor and the students, to the whole class, and if the student wishes to share her or his portfolio, as broad an audience as possible.

The students aren't always keen on sharing their work, as they are still learning. One would think that faculty members, and all who conduct research as a part of the knowledge discovery and generation industry, might be more apt and willing to compose content for general public consumption, but there is the troubling aspect of our capitalistic society that foils this plan somewhat - knowledge and information are currency, and there is a tendency to want to keep it proprietary for one's own personal gain (either wealth generation, fame generation, expertize creation, and in the minds of many faculty, enable them for tenure and promotion).

Until we get to a true open-sorce mindset where the fundamental belief is that we all grow exponentially by the universal distribution and sharing of knowledge, we may not get to a place where what's known is shared, let alone digested and interpreted for the masses. Think of it this way. Why do we have a copyright and patenting system that is a part of the US legal infrastructure?

There still is hope, as people realize that fame is fleeting, wealth can be temporary (and measured in other ways besides widgets sold and profits taken), and the individual wins when we all win: then, perhaps, we will see the full power of the socially networked, knowledge driven economy harnessed well. Even then, we still have to wrestle with the age old question of the stark disparity that is the digital divide which places those who don't have the wherewithal to own the technology to access the information at a serious disadvantage...but I digress.

Personally, I'm of the mind that content rules, and applicability is the true measure of worth. And there is no way to know if your content is valuable until you set it free and see if it sticks.