The Deliberatorium is a software tool designed to help organizations better harvest the knowledge and incorporate the perspectives of their members to identify solutions for complex problems, avoiding the dysfunctional behaviors (such as noise, disorganized content, and polarization) that other social media often produce when applied to challenging topics.

Problem

Decision-making in organizations rarely fully harvests the collective intelligence of its members, even for very high-stakes strategic decisions where such contributions can make the difference between disaster and success. Social media technologies (e.g. email, web forums, chat rooms, blogs, wikis, and idea forums) have the potential to address this critical gap but, while they have enabled an explosion in how much communities can weigh in on topics they care about, they often create more heat than light when applied to complex, controversial problems:

Scattered content: The content created by existing social media is typically highly scattered, so it's hard to find all the different contributions on a topic of interest. This also fosters unsystematic coverage, since users are often unable to quickly identify which areas are well-covered, and which need more attention.

Low signal-to-noise ratio. Social media content is notorious for being highly redundant, with potentially important contributions often drowned-out by repeated postings made by a small fraction of the community.

Non-collaborativeness. Social media systems often elicit many relatively small standalone contributions, rather than a smaller number of more deeply-considered ideas, because collaborative refinement is not inherently supported.

Polarization: Users of social media systems often self-assemble into groups that share the same opinions, so they see only a subset of the issues, ideas, and arguments potentially relevant to a problem. This tends to lead people to take on more extreme, but not more broadly informed, versions of the opinions they already had.

Dysfunctional argumentation: Existing social media systems do not inherently encourage or enforce any standards concerning what constitutes valid argumentation, so postings are often bias- rather than evidence- or logic-based.

Disorganized content: Existing social media generally create very disorganized content, so it's time-consuming to find what has been said on any topic of interest. This fosters unsystematic coverage, since users are often unable to quickly identify which areas aren't yet well-covered and need more attention.
Low signal-to-noise ratio. Social media content is notorious for producing highly redundant content, so enormous effort is typically required to "harvest" this wisdom to inform better, more broadly-supported decisions.
Quantity rather than Depth. Social media systems often elicit many relatively small contributions, rather than a smaller number of more deeply-considered ideas, because collaborative refinement is not inherently supported.
Polarization: Users of social media systems often self-assemble into groups that share the same opinions, so they see only a subset of the issues, ideas, and arguments potentially relevant to a problem. People thus tend to take on more extreme, but not more broadly informed, versions of the opinions they already had.

Here's just two examples (see this blog post for more): (1) Intel ran a web forum on organizational health that elicited 1000 posts from 300 participants. A post-discussion analysis team invested over 160 person-hours to summarize these contributions (at 10 minutes a post, probably longer than it took to write many of them in the first place). The team found that there was lots of redundancy, little genuine debate, and few actionable ideas, so that in the end many of the ideas they reported came from the analysis team members themselves, rather than the forum. (2) Google used their "moderator" system to collect ideas for which charitable projects the company should fund (project10tothe100). The company had to recruit 3,000 employees to filter and consolidate the 150,000 ideas they received in a process that put them 9 months behind their original schedule. The vast majority of these ideas were minor variants of simple suggestions (e.g. support public transport, make government more transparent, and so on). Surely that vast amount of effort could have been used to compose a smaller number of more deeply-considered ideas, rather than many shallow ones.

This introduces a quandary. We want as broad participation as possible, since that increases the chance of finding a truly transformative idea, but current social media technologies often capture only a fraction of the collective wisdom of a community, when applied at scale, and harvesting this wisdom can be an expensive proposition.

Solution

The Deliberatorium is a software system, developed at the MIT Center for Collective Intelligence over the last five years, designed to transcend these limitations and realize the enormous potential social media has for enabling better organizational decision-making. The approach is simple. Members of a community are asked to make their contributions in the form of a deliberation map, a tree-structured network of posts each representing a single issue (question to be answered), idea (possible answer for a question), or argument (pro or con for an idea or other argument):

A screenshot from the Deliberatorium. Each line in the left pane represents a single issue, idea, or argument. Each such post can have it's contents viewed, edited, discussed, and rated in the right pane.

{C}

{C}

{C}

{C}

{C}

The community follows a simple process to ensure that the deliberation map is as useful as possible:

Every author starts by "unbundle-ing" their contributions into individual issues, ideas, and arguments, and adding their unique points to the relevant part of the map. A key tenet is the "live and let live" rule: if you disagree with an idea or argument, you shouldn't change the post to undermine it, but should rather create new posts that present the strongest ideas and (counter-)arguments you can muster, so all contributions can compete on an even basis.

Moderators help ensure that these guidelines are followed. New posts can, at first, only be viewed by moderators. When a moderator verifies that a post follows the deliberation map guidelines, they can be viewed, edited, commented, and rated by the full community. If a post doesn’t yet meet the guidelines, the moderator leaves comments explaining what needs to be done to fix them. Moderators play a modest “honest broker” role: their job is not to evaluate or change the content of a post, but simply to help authors ensure that the content is as accessible as possible to the community at large. Moderators can check each other's work, as they do in such systems as Wikipedia and Slashdot, to ensure they are doing a good job.This process is supported by such software capabilities as open editing (any user can check and improve posts), watchlists (which automatically notify users of changes to posts they have registered interest in) and version histories (to allow users to roll-back a post to a previous version if it has been "damaged" by an edit). The system also provides a powerful set of attention allocation metrics to assess how well each part of the deliberation is going, and thereby to help community members focus their efforts where they can do the most good.

Deliberation maps have many advantages over conventional social media. Every unique point appears just once, radically improving the signal-to-noise ratio. All posts appear under the posts they logically refer to, so all content on a given question is co-located, making it easy to find what has and has not been said on any topic, fostering more complete coverage, and counteracting polarization by putting all competing ideas and arguments right next to each other. Careful critical thinking is encouraged, because users are implicitly encouraged to express the evidence and logic favoring the ideas they prefer, and the community can rate each element of an argument individually. Users, finally, can collaboratively refine proposed solution ideas. One user can, for example, propose an idea, a second raise an issue concerning how some aspect of that idea can be implemented, and a third propose possible resolutions for that issue. The value of an argument map can extend far beyond the problem instance it was initially generated for: it represents an entire design space of possible solutions that can be readily harvested, refined and re-combined when similar problems arise at other times and places.

To help make this more concrete, here's a short video of a fictional, but illustrative, use scenario:

I picked a simple example for the video that I thought would be accessible to most people, but deliberation mapping can of course be applied to all sorts of decision problems, including major "management processes" such as strategic planning, budgeting/resource allocation, risk management, product development, and so on.

The current user interface, while functional, obviously has an "old world" feel. This reflects my limitations as a programmer, as well as a strategic calculation. I've noticed that many hugely successful web 2.0 systems (such as email, news groups, Wikipedia, and Facebook) became that way despite having initially quite rudimentary user interfaces. The key success factor often seems to be the compelling-ness of the idea, rather than the polish of the implementation. I've focused, therefore, on rapidly exploring new ideas, rather than polishing old ones. Going forward, my hope is that the most powerful ideas explored in this work will be taken up by commercial vendors and incorporated in their products.

Practical Impact

We've tried out the Deliberatorium in a range of business, government, educational, and other contexts. Our experience to date is that it can make a real difference. Our first experiment involved translating a web forum on carbon offsetting (hosted on planeta.com on May 2008) into a deliberation map. The original 13 page discussion, filled with the digressions and repetitions that typify web forums, translated into the following eight item deliberation map:

The carbon offsetting web forum summary map

This was a startling illustration of the potential of deliberation maps for harvesting a community's collective knowledge in a way that is qualitatively more useful than conventional social media.

Our first large-scale evaluation was at the University of Naples, where 220 masters students in the information engineering program were asked to weigh in, over a period of three weeks, on what use Italy should make of bio-fuels. All told, the students contributed nearly 2000 posts, creating a map that was judged by content experts to represent a remarkably comprehensive and well-organized review of the key issues and options around bio-fuel adoption, exploring everything from technology and policy issues to environmental, economic and socio-political impacts.

A small portion of the Naples bio-fuel deliberation map

We found that, initially, about 2/3rds of user posts were structured correctly as originally created, and this increased to about 85% by the end of the deliberation. Also, the remaining posts almost always just required some simple fix e.g. people would call it an "idea" when it should have been a "pro" (see this paper for details). Overall, we were encouraged by how well people could use the deliberation map structure. It made it possible to support the user community with the part-time support of just two moderators. We were hard-pressed to imagine any other approach that would allow over 200 authors to write what was in effect a substantial book on a complex subject, in a couple weeks, with no one in charge.

Our first business-centric evaluation was conducted with Intel Corporation on the question of how "open computing" (i.e. where users are given greater access to computing tools and data) should be used in the company. Contributions were purely voluntary. A single moderator was able to support the discussion with very little effort. The end result (see below) was that Intel received a substantive and well-organized overview of important issues in this space from 73 contributors, including many from outside the company, at close to zero cost.

Map generated for the Intel deliberation on open computing

We've also conducted evaluations with the US Bureau of Land Management and the University of Zurich, among others, and we've learned that, when compared to conventional social media, it can help you get better content on how to solve complex problems, at lower cost, small voices can be heard and, perhaps most importantly, it's much easier to find the "good stuff".

First Steps

Interested in trying out the Deliberatorium to support decision-making in your organization? The system is publically accessible at http://deliberatorium.mit.edu/ You can look at some existing maps, play around in a "sandbox" to see what authoring is like, and even start up your own deliberation map if you like (contact me if you want to do the latter). If you want to run the system on your own servers, the source code is in the public domain (again, contact me for that).

Our experience from previous evaluations suggests that something like the following process is a good way to go:

1. Embed the tool into a management process so that it's part of the "workflow" and taken seriously (for instance, make it a part of the strategic planning process). Your organization's members are much more likely to try a new approach if it is part of a conversation that matters.

2. As an initial test of the tool's effectiveness, conduct a "controlled experiment":

Select a specific theme/question that needs to be answered as part of the strategic planning process (e.g., what will be the likely impact of emerging market growth to our business model? how will the competitive environment be changed as a result of a stronger role of states and regulation?)
Set up two groups that are comparable e.g. in terms of their size, diversity, and skill sets. In one group, deploy the Deliberatorium to surface perspectives on the strategic planning question; in the other group, run the process in the traditional form.
Compare the results in terms of (i) quality/depth of the discussions and their output; (II) level of engagement/satisfaction in the process by the users, and (iii) organizational effort needed to run and harvest the deliberation.

3. If the results for the experimental group are better than those of the control group, further build out the tool and supporting processes, and deploy more broadly at the next opportunity.

Documents

guidelines.pdf

Videos

You need to register in order to submit a comment.

Sort By: Latest Oldest Likes Replies

February 28, 2012 at 12:37pm

Sean Schofield

Hi Mark,

A few have the raised the point of "user search cost" being a barrier to usage. Presumably, this cost is positively correlated with the quantity of content. More stuff = more stuff to process in order to find where each piece fits best.

If the presumption is true (and we assume people care about respecting the best fit rule - probably safe to assume if there is a visible norm), then this is incrementally more threatening as the number of "deliberators" increases - which as I understand it, is what you ideally want to do.

Assuming this hasn't been solved...

Random thoughts:

1) Could some kind of text analysis tool be used suggest where to place an author's submission? (I.e., after writing a contributor the author hits a kind of "place me" button and it offers probable matches)

2) Could there be a kind of "fast pick" system, where as the user types their contribution a live filter narrows down possible locations based on key words or phrasing.

3) Could a choice driven system narrow down the "box" the contribution is likely to do in. I.e., the deliberatorium provides a decision tree and poses yes/no questions to the author, allowing the author to quickly filter out the non-relevant branches.

4) More of a stretch...if there is a natural "law" about the nature of progression (i.e., pattern of complexity) of a deliberatorium, then you could try to predict where the post is most likely to belong to in combination with any of the above.

Thanks for reading,

Sean

March 5, 2012 at 4:03pm

Mark Klein

Sean,

Thanks for your comment, some great ideas that I intend to explore more. Here are some more detailed responses.

Yes, that is a key issue, though the search cost should, in theory, only grow as a logarithmic function of the size of the argument map, which helps.

I think that makes a lot of sense. I've tried that out a couple years back, but I wasn't happy with the retrieval accuracy and completeness at that point, so I disabled it until I could implement a better approach. Currently, I'm thinking of using a variant of latent semantic indexing that takes advantage of the argument maps' tree structure to help improve the hit rate.

2) Could there be a kind of "fast pick" system, where as the user types their contribution a live filter narrows down possible locations based on key words or phrasing.

Interesting idea, like the google live search thing.

That's also interesting! But let's say the system, instead of following a binary decision tree, provides a list of the words whose selection would, at each point, most rapidly narrow down the range of possible matches, based on what you've already entered. Then you might not even have to type a lot of the time, you could just select words from the list suggested by the system.

I think that would help as well. In an argument map, people will often add posts in a top-down manner, attaching new posts under the posts they've just recently created or edited or rated or viewed. We can use that insight to refine the list of suggestions that the system makes.

Again, thanks for the suggestions, appreciated.

Mark

September 14, 2011 at 10:19am

Mark Klein

Michael Harris (Michael.J.Harris@Intel.com) made an interesting point I wanted to share with the MIX community:

"Some research suggests independent brainstorming followed by collaborative sharing produces much greater diversity and quality of ideas than the method we typically use at Intel which is immediate, collaborative brainstorming.

The limitations of typical team brainstorming include free riding, evaluation apprehension, productivity blocking and conformity effects. I would think these same limitations could apply to online settings in which all participants have immediate access to all content.

Given the primary objective is innovation (with redundancy reduction as a necessary but secondary objective), what strategies might be applied to the deliberatorium to combat these types of limitations? For example, could you require participants to make a contribution before accessing others’ content and organizing their submission?"

September 14, 2011 at 10:20am

Mark Klein

Here was my response:

That's a great question. My understanding is that the problems you describe apply mainly to *face-to-face* brainstorming sessions. In such meetings, only one person can talk at a time, and people typically self-censor to deal with that bottleneck, e.g. based on their perception of the power dynamics in the room. I think that these problems are *much* less of a factor in asynchronous online brainstorming, since everyone can make contributions in parallel without taking away airtime from anyone else. That really seems to take the lid off. In fact, the problem often becomes "too much" participation, rather than too little. The judicious use of anonymity can also help with self-censoring. The Deliberatorium, for example, displays the ratings for posts but not *who* made those ratings, making it more comfortable for people to tag less-promising posts and thereby separate the wheat from the chaff. Some researchers have also made a case for allowing users to use pseudonyms for their account names, there are arguments back and forth on that.

In any case, I think it's a great idea to ask that people brainstorm independently before entering stuff into the deliberation map, it can probably only help. I'm going to make a note of that idea for future reference, thanks.

Mark

September 5, 2011 at 1:34pm

Jerome Gayerie

I was pleased to collaborate with Mark Klein last year, submitting some ideas to him to improve the deliberatorium. And for managing our exchanges, we used.... the deliberatorium itself. So the deliberatorium is not only useful for large groups, as in Naples or Intel but also for very small groups (two persons here).
This is a great tool to create content, store, organize and make decisions. It is based on the simple and powerful principle of issue-maps, enriched with many useful features (comments, search, tracking, rating, version management, etc ...), all in an effective user interface.
Of course the tool is not everything and the methodology (guidelines) is as important as the tool itself. Again, guidelines are common sense and efficient. All this allows to get easily a result content that is both detailed and clean. Particularly because the content is separated from the comments that improve the content (unlike a forum where content and comments are mixed).

By the way, guidelines could also be a subject of deliberation, open to users. This would enrich the guidelines with good practices and different ways to use the tool.

I do not know if it was experimented before, but the deliberatorium may be useful for supporting real meetings (in the same room or conference call): people discussing while the person in charge of organizing the debate, minutes the content in the deliberatorium. Of course, here, the frequency is limited by the fact that only one person speaks (and one person write) at the same time. But on the other hand, the clarification of ambiguous points would be faster than during an online asynchronous exchange. For larger groups, one could even imagine sub-groups working concurrently in a first step. A second step would then be necessary to merge the whole.

September 5, 2011 at 1:43pm

Mark Klein

Jerome,

Thanks for the comments! Deliberation mapping has in fact been used to support synchronous meetings. It can work quite well if you have someone who is mentally agile enough to translate a discussion into a map in real time. I think another good option is the idea of interleaving asynchronous building of the deliberation map with synchronous discussions about the map. You can get breadth of input from the parallel asynchronous contribution phase, and also benefit from the consensus and shared understanding development that can emerge from synchronous discussions. I've used that approach frequently myself, most recently in a project with Siemens.

Mark

September 5, 2011 at 12:21pm

Hajo Reijers

In my view, the deliberatorium packages powerful concepts in an attractive and practical form for collaborative problem solving. While the current interest in sharing content and engaging in discussions over the Internet is tremendous, we need yet to experience the true impact that cooperation between independent and knowledgeable individuals can make. Through its emphasis on condensing the discourse and, in this way, making it accessible and meaningful to all discussants, I think the deliberatorium has the potential to create that.

Hajo

September 5, 2011 at 7:08am

Cristiano Ferri Faria

Deliberatorium is a really interesting project. I have been following it last months as well as Mark Klein’s research progresses. I also work in a large-scale deliberation project, called e-Democracia (www.edemocracia.gov.br). Created in the Brazilian House of Representatives, it’s an official website whereby people can share their opinions, critics and suggestions to legal texts with representatives.
The major challenges we have here in e-Democracia is these same very well-defined (by Mark) problems in large-scale platforms for debate: Disorganized content, low signal-to-noise ratio and quantity rather than depth.
And the solution, with Deliberatorium, in which participants are invited to set their opinion, argument or idea in an argumentation map, is great and functional! In my view, there is a just challenge in this kind of system and I know Mark’s aware of it.
It demands certain initial effort from the participant to read and search in the argumentation map where his point could be inserted. This effort may discourage participants who want to contribute but haven’t great technical knowledge about the subject, so it will be slightly costly to search where his contribution should be included. Moreover, there is a kind of participants who just intend to present a testimony but an argumentation map may seem not appropriate or open to this kind of contribution. One possible solution for that is just let people to participate the way they prefer, so the moderator will have to work harder to organize it.
But I’m sure Deliberatorium is dealing well with these challenges. In my opinion, this is one of the best solutions in this field by far.
Cheers Mark

Cristiano Ferri Faria
Brazilian House of Representatives’ e-Democracia Project
@cristianofaria
@edemocracia

September 4, 2011 at 1:25pm

David Price

Among the excellent set of finalists, the two with which I'm most familiar – and admire greatly – are the Deliberative Corporation and MIT's Deliberatorium.

Both are grounded in a systematic form of deep and wide listening, informed structured dialogue, mutual understanding, and pragmatic negotiation geared towards effective action: and both point towards a more open, agile, rigorous and transparent system of civic and corporate governance.

Both seek to build on social media’s strength as a serendipity catalyst by giving participants the means to interact mindfully, to surface difficult questions, to learn collaboratively, creatively and cumulatively from each other, and to take responsibility and hold each other accountable for the decisions arising from their deliberations.

The deliberative work that both entail is demanding and inspiring – as the best work should be – and in an age of eddying noise and complexity, of financial and political uncertainty, and of intense pressure on resources, this work has never been more important.

Both would be worthy winners of the Management 2.0 Challenge.

September 3, 2011 at 10:32am

Mark Klein

I got the video done and have added it to the post - hope you find it helpful.

Mark

September 2, 2011 at 1:37pm

ana bazzan

I heard about the Deliberatorium during a talk given by Mark Klein as an invited speaker in an AI conference. I immediately though that a) we could use it at our department to reach some consensus regarding more or less contentious issues, and b) this could be a nice application to combine with other AI techniques in order to try to ease the role of the mediator.

So far we are selecting a topic to discuss using the Deliberatorium and I have also recommended the site / the work to a PhD candidate. He is thinking about combining this tool with recommender systems in order to guide new users. I am confident that this will be a nice work and I hope we can work together with Mark in this direction.

Ana Bazzan

September 2, 2011 at 12:37pm

Robert Laubacher

For MIT's Climate CoLab project http://climatecolab.org/, I've had occasion to use output from the Deliberatorium.

As part of Mark's research, he and a colleague oversaw the development in the Deliberatorium of an argument map that outlined the range of possible approaches for addressing climate change.

Mark sent me a link to this argument map, and I found it to be quite impressive. It nicely captured the broad range of ideas that I had seen expressed, albeit in scattered and disorganized form, in the many books, articles, and web sites I'd been reading on climate change.

This map has become a valuable resource to our team as we think about new ways to organize activities on the Climate CoLab in coming months.

And we have implemented a somewhat simplified version of the Deliberatorium in the Climate CoLab, which allows members of our community to comment on and add to mini-argument maps that address specific issues in the climate change debate.

For an example, see http://climatecolab.org/web/guest/plans/-/plans/contestId/4#plans=subview:issues

September 1, 2011 at 5:17pm

Adriana S. Vivacqua

The Deliberatorium is a real step towards collaborative deliberation. I particularly like how the IBIS structure fits into the picture, providing an organization for ideas.

Breaking ideas into smaller elements is key to making individuals think about their posts and structure them correctly. I wonder if users will have difficulty breaking their ideas into the three elements proposed, and what to do if these elements are linked together? Do you know if users have problems with the structure and rules at first? It should take them some time to properly learn what and how to send their ideas and opinions.

I also worry that, with many posts, the tree might become too large, and cumbersome to read through, making it hard to find the appropriate location for a post. Even though you claim it won't, when I think of a service such as twitter, I can't help but imagine what an argument tree would look like (especially if some tweets had to be broken into 2 or 3!) This may become a problem for users. Perhaps some sort of search functionality would help?

All in all, a well rounded and thought out initiative, which will probably benefit many organizations!

September 3, 2011 at 10:21am

Mark Klein

Adriana,

Thanks for your thoughts. In our Naples evaluation (our largest to date), we found that, initially, about 2/3rds of user posts were structured correctly as originally created, and this increased to about 85% by the end of the deliberation. Also, the remaining posts almost always just required some simple fix e.g. people would call it an "idea" when it should have been a "pro" (see this paper for details). So, you're right, not everyone got it perfect at first, many people initially treated the deliberation map as if it was a web forum, but overall we were encouraged by how well people could use the deliberation map structure.

You're also right that complex topics can lead to big maps. The Deliberatorium does include a search capability to help people quickly find where to place their contributions.

Mark

September 1, 2011 at 12:01pm

Ali GURKAN

What I liked most about Deliberatorium the possibility to use it a wide range of areas with minor modification. Although its use should not limited to management purposes, it has one (but a "big" one) advantage over other advantages: it's cheap. There is no regular business trips necessary to gather people together. Well, one may also add that according to his own will. Even so, that shows the flexibility of the platform.
At another presentation I remember seeing the idea of using a random populations for rating. That would work here as well for preventing "friendly" ratings.
Another thing might be limiting posts to a certain number of characters like on twitter. Eliminating endless user comments right from the beginning would also decrease redundancy as people - I assume - are more likely to read brief comments. That's also somehow the idea behind argumentation: Disintegrate the parts of a whole as an idea, a pro or a con argument.

September 3, 2011 at 10:30am

Mark Klein

Ali,

Thanks for your note. I agree, having concise posts for each issue, idea, and argument should help make it easier to quickly get a sense of the key points in a deliberation map. Currently, we achieve this by having length limits for the titles of posts, allowing people to make the post descriptions as long as they need. But maybe it will make sense to also have length limits in the descriptions, something to think about.

The idea of randomly assigning people to rate posts to avoid "gaming" is also an interesting one, thanks.

Mark

August 31, 2011 at 3:06am

Mark Klein

I'd like to thank the MIX judges and community for many helpful comments on how to improve the hack. I made the following changes in response:

added examples and links to flesh out the problems current web 2.0 systems have with supporting complex deliberations
added a short video with a fictional but illustrative use scenario, so people can see the tool in play
discussed how to deal with potential challenges to using this approach, especially the critical challenge of working at scale

Mark

August 29, 2011 at 7:28am

Daniel Bassill

It seems that this could add value to community organizing and on-going problem solving if the facilitation team were sustained for many years. Have you done any experiments with using this as a project management tool? For instance, if you put a set of blueprints for building a hotel on line, you could set up a template for discussions starting with the initial design and financing work through the first steps of laying the foundation through the final step of opening the doors to new customers.

If another group of people wanted to build a similar hotel in a different places they could follow that template and even learn from the work done in earlier projects. I have used concept maps to describe strategies that would result in more and better non-school tutor/mentor programs in inner city neighborhoods. http://tinyurl.com/TMC-4-Part-Strategy

If discussion nodes could be attached to each of the steps of these maps, the facilitation could be a form of project management intended to get more people involved, informed and active in achieving the goals of the project.

Instead of starting a discussion with a blank sheet of paper we’re starting with a goal in mind and working backwards to fill in what we and others already know. As we do that we’re working forward to know more and act on what we know to make volunteer-based tutor/mentor programs available in more places, based on the strategy that is outlined.

Is this possible? Is it already being done this way?

September 3, 2011 at 10:45am

Mark Klein

Daniel,

Thanks for your comments, sorry to take so long to reply. I do think that a deliberation map for one project could add a lot of value to people doing later projects with similar goals. They could quickly see what options people had explored in the past, as well as what the pros and cons were. I haven't personally run this kind of multi-year scenario, but Jeff Conklin at Cognexus Institute observed (see this paper) that a major electric utility, over a period of years, frequently consulted maps from previous deliberations when facing similar challenges.

Mark

August 29, 2011 at 7:27am

Catherine Spence

Intel IT has been pleased to work with Mark Klein from the MIT Center for Collective Intelligence on a series of investigations aimed at exploring the concept of collaborative argumentation.

Our projects validated the potential of the combination of social media and argumentation. Social media enables the hosting of large scale discussions with a vast number of diverse users over the internet. Argumentation captures the logical structure of debates and encourages evidence-based, critical thinking with systematic coverage of topics.

One of the biggest benefits of Deliberatorium was the ease of generating the argument map. The “moderate-as-you-go” approach promoted structured problem solving during the ideation and saved time during the post-processing of content. The compact format was useful to reduce complexity and highlight key content and contributors. As compared with threaded conversations organized by time in other tools such as web jams and wikis, the argument map saved time and effort synthesizing the results. Further, the argument map provided a useful artifact for later data mining and as a historical record. Since we often go back and periodically reevaluate our position on a variety of topics, the argument map can be used as a starting point without recreating the same content on complex issues.

In terms of real world corporate application, we already experienced some technology transfer by influencing Intel’s internal ideation tool to include basic argumentation capabilities. These enhancements were incorporated in a successful pilot, which then led to a production deployment. Outside of Intel, we have debriefed our commercial social computing provider on the collaborative argumentation research and results in order to inspire future products which could eventually be consumed at Intel.

Congratulations to Mark on his excellent work!

July 14, 2011 at 1:57pm

Kartik Subbarao

I think that the Deliberatorium provides a valuable toolset that helps in managing large-scale decision-making discussions. I'd suggest the following to further increase the reach and capability of this approach:

+ Modularize and open-source the code so that it can be deployed alongside existing Wiki/collaborative systems. That way the user interface can be further refined and take advantage of other features. For example, if a social networking engine already has captured tags/reputations on people, these can inform the node ratings.
+ Partner with others in this space -- for example, I think it'd be intriguing to see Quora.com run some of their questions as Deliberatoriums, given the high quality of participants that they have attracted in various topic areas

While running a Deliberatorium on a given subject, I also think it might be useful to run a traditional threaded discussion in parallel, so that people can easily meta-comment in a freeform manner on the process itself, as the discussion is unfolding (i.e. I think that there are too many subtopics under this heading, we should break it up further, etc). This could be as simple as the "Discussion" tab in a Wikipedia page.

I think Deliberatoriums could be launched on a lot of consumer-friendly topics as well -- what type of car to buy, how to allocate a stock portfolio, etc. Here's where partnering with an existing knowledge-based service like Quora.com, Wikipedia, Yahoo Answers, etc would be particularly helpful.

Good luck to you guys on the MIX M-Prize!

July 14, 2011 at 9:39pm

Mark Klein

Kartik,

Thanks for the great suggestions, I'll give them some thought.

Mark

July 7, 2011 at 10:06am

John Doe

I would be curious of the high level view of this component in a holistic collective knowledge platform.

This seems more like an experiment fit for and limited to academic practice (or an innovative R&D group at a corporation) when in isolation. Would I be correct to assume this is indeed a component and theory being developed as one part of a larger effort to develop collective intelligence platforms at MIT?

It seems to me that manual argument mapping through a proprietary system is not the best approach, considering the user education scenario. It is very easy to begin this exercise and maintain a high level of quality for small groups of highly educated and motivated people. But given an implementation of a wider audience, the demand on the moderators go up, the quality of content goes down and the receptiveness to yet another process usually hinders if not thwarts the effort.

Efforts like Vulcan's Evri, Halo, Microsoft's PowerSet and other such companies have proven that you can get the logic of an argument through NLP methods. This allows you to build truth tables and argument maps of large bodies of text. These methods are so accurate, they can "learn" from the text books and pass an AP exam. Another more well publicized version of the same technology is IBM's Watson.

While utilizing the social platforms with only what they offer, I believe you are indeed correct in saying that they have the downfalls listed in challenges. But to approach the issue by building another platform is a bit of an uphill battle. If we truly would like to unlock the potential of collective knowledge, I think the NLP route is a much more viable solution. This solution takes care of all of the downfalls and can even link out of dialogue logic and reason to the current issue at hand. It also allows the group to take non-user generated content from the ages and take that information into consideration.

Can you let me know if you've found this not to be the case and what benefits this hack provides over the NLP/collective knowledge linguistics method?

It is an interesting piece of work and fits well into the larger picture of collective knowledge linguistics and understanding. MIT seems like it is always nourishing the latest and greatest ideas. Bravo!

July 7, 2011 at 8:38pm

Mark Klein

Anthony,

Thanks for your comment.

Yes, I do see the Deliberatorium as part of a larger effort to enable large-scale collective intelligence. The Deliberatorium is aimed at helping communities rapidly and effectively define a space of possible solutions for a complex problem. I have a separate project (see http://cci.mit.edu/klein/negotiation.html) aimed at helping decision-makers rapidly search such spaces for pareto-optimal (win-win) solutions.

My goal, however, is to make this approach useful for large diverse user communities, not just for small groups of researchers or the like. In the evaluations I've done to date, the participants were a pretty mixed bunch, with as many as 200 working together. My hypothesis is that the cost-benefit ratio for the Deliberatorium approach will actually improve with the number of participants. The number of potential moderators, for example, should scale linearly with the size of the community (typically a few percent of the authors in a social media community are interested in being "super-users"), but the number of posts to certify should increase a lot slower than that, as people increasingly find that their points are already captured in the map and they don't need to create new posts. So the per-moderator burden should decrease as the community scales. My goal is to test this hypothesis by running deliberations of increasing scale.

My understanding is that NLP technology is not yet capable of understanding text well enough to summarize the key issues, ideas, and arguments on a complex topic. Watson, for example, doesn't summarize anything, it relies on finding an existing snippet of internet content that answers a simple focused question. When creating a deliberation map, most of the questions are unknown at the start and are generated as part of the deliberation process. Evri and Powerset are also I believe used for search, rather than summarization. Could you point me to documented examples of software generating something like deliberation maps?

Even if/when NLP technology overcomes this hurdle, deliberation maps are still potentially useful because they help make deliberations more complete. In a deliberation map, it's easy to see where there are gaps (e.g. when an idea has few arguments attached to it) and we can direct the communities' attention to filling those gaps in.

Incidentally, I am now looking into whether we can use crowdsourcing (e.g. based on Mechanical Turk) to moderate or even build deliberation maps. At least in the short term, this seems much more viable to me than an AI-based solution.

Regards,

Mark

July 8, 2011 at 10:18am

John Doe

Nice, thank you for the links. You are working on some very interesting topics.

It will be interesting to see how it plays out and the usage once a large scale pilot is done. I hope we will have visibility into the usage statistics. I will be quite interested to see who should break up with who and why and how it is moderate once this hits Hollywood.

As I mentioned, Project Halo is a good example and has some good links (check out SILK). Also, I believe Evri demo'ed this type of functionality at an MIT event here in Seattle. I believe the query was "Who did Google acquire?" and it brought up fairly accurate results.

You are correct. The industry is misusing the IP and applying new technology to solve an old problem. I guess they didn't catch Clayton Christensen's work? Vulcan has been doing some great work in this area, although I'm not sure they found the proper way to bring this to commercialization on a conglomerate level scale. Their consumer products have been entertaining though.

Crowd sourcing is a great approach. Bravo on your work. I'm glad to have been able to discuss a little bit of what's going on with the top industry's top innovators.

July 8, 2011 at 10:48am

John Doe

Here is a quick overview of the process that we used to build a limited system at a past venture I was involved with:

Get the Stanford NLP processor.

Create an application that pulls 10 Ks from SEC or patent information off of USPTO.

Create a process that uses the Stanford NLP processor to tag the content.

Break up the content based upon different proximities (0 for word, 1 for section of sentence, 2 for sentence, 3 for paragraph, 4 for page, 5 for work of literature, 6 for corpus, et cetera).

Begin inserting the works in the following fashion (through a parsing process):

Word

WordID

Word

Part of Speech

POSID

POS

Word_POS

ContextID

POSID

WordID

Section

SectionID

Footprint (Contains delimited strings of the ContextIDs)

NextSectionID

NextSectionProximity

Once you have this, you can begin to build query footprints. For example, if you want to see who Microsoft's potential competitors may be, you can query for something like the following:

Microsoft [Proper Noun type POS] competes [verb type POS, proximity of 2 for query type] _______ [A loose pivot on a proper noun with a proximity of 2 from competes]

This is the first abstraction of the system, which unlocks a bit of I believe what you are saying. Instead of a statically generate deliberation map, you have a dynamic result based off of your large data set. It is of course over simplified, but the next level of abstraction would be doing synonym analysis for another pivot type (include all results for synonyms as well). Then you have footprint identification, entity extraction and other technologies that compliment this approach quite well. I believe this is how Powerset enabled the "In what year was Albert Einstein born?" type of scenario where it would give you your precise answer. From what I understand, this would be very similar to sending a query of, "What are the possible economic models that could be applied to providing a perceived incentive for reducing carbon emissions?"

I know there are some issues with it right now, but it seems like that is where the future is headed. It was interesting to see this approach to unlocking large data, versus the NLP type approach. I'm no expert, but perhaps some food for thought?

July 1, 2011 at 11:26am

David Sims

Hi Mark,

Great hack, thanks for submitting it. This is an critical area for those of us (almost everyone, I guess?) who work in teams, distributed around the globe, and figuring out how to move forward through online channels. I had a couple of questions on aspects of this that seem critical in distinguishing it from a listserv or Wiki, things you've touched on here but I'm curious to learn a little more.

I'm curious about the process behind Argument Mapping, and related, the role of the moderator. Seems like a lot depends on how the argument gets framed. Are there guidelines for how a whole raft of comments (like the one you took on with planeta.com) get categorized? How much does that rely on the moderator's judgment, or do you use anything like a tag cloud to group content under headings?

Also, as an editor, I'm wondering how you "export" the learnings or main ideas from this kind of environment in practice. In the implementations you've tried, does the moderator create a report or powerpoint presentation highlighting the arguments? Or is there something more automatic that can come out of this.

Again, neat idea. Would be interesting to see a screencast of it in play.

-- Dave

July 3, 2011 at 2:06pm

Mark Klein

Dave,

Thanks for your comment.

Framing: The framing of a deliberation (i.e. the issues in the deliberation map) can of course have a substantial impact on where the deliberation goes. We generally recommend that the customers for a deliberation start by defining, when they can, a top-level issue "skeleton" following guidelines detailed in the system help page. Open-ended issues (e.g. "how can we resolve climate change?") are, for example, usually better than yes-no questions (e.g. "should we use technique X to resolve climate change?").

Moderation: The system includes a search function to help authors find the right place to put their posts and, in our experience to date, they typically do a good job of that, without the need for moderator input. We did find, in our largest evaluation, that some of the map branches became pretty "bushy" (i.e. some posts had a lot of children under them); for those situations, we recommend that the customer, with a moderator's help, add new issues and ideas to cluster the children posts, so each post in the map has no more than about 7 children under it (the rough size of human short-term memory).

Exporting: the system allows users to write "guided tours", i.e. narratives made up of text interspersed with links to posts in the map. Each guided tour is thus a different summarization of the results of a deliberation. One could develop algorithms to *automatically* generate such tours (e.g. by doing a breadth-first traversal of the top-rated ideas in a map), though I think that people will generally create more compelling narratives than simple algorithms like that.

I'll look into creating a screencast, thanks for the suggestion.

Mark

July 8, 2011 at 10:57am

David Rader

Guided tours through a deliberation is an excellent idea. Would be very helpful as a teaching aid -- especially for social sciences. Can see applications to literature; maybe even music and visual arts.

Visual annotations would be helpful such as colors or line thicknesses to show which branches have recent activity.

Would be interesting to run controlled experiments on the quality of discussions given different structures. For example, on the deliberation on responses to climate change, one structure might focus on actors (national governments, local/city governments, large (mutliple-geography) businesses, small (local geography) businesses and individuals) versus a product structure (industrial goods, consumer goods, food, services, etc.). A panel could evaluate the breadth, specificity, usefulness, etc of the ideas generated in the two structures.

Are there publically-accessible implementations?

September 3, 2011 at 10:59am

Mark Klein

David,

Thanks for your comments, sorry to take so long to reply, I spend the last two months traveling on three different continents.

The Deliberatorium does include a "highlighting" capability that visually highlights posts in the map according to criteria you select e.g. it can highlight the branches which have been most active, which are the most controversial, the most highly rated, and so on. You can also use the search function for the same purpose, e.g. you can find all the posts that have been active in the last few days. I'm sure there's a lot more that can be done with that, though.

We did ran an evaluation comparing the quality of deliberations conducted, with the same topic and matched user populations, for a web forum vs the Deliberatorium. That data is still being analyzed: it's a big task, complicated by the fact the evaluation took place (at the University of Zurich) in German!

The Deliberatorium is publically accessible at http://deliberatorium.mit.edu/, and the source code is in the public domain. You can look at some existing maps, play around in a "sandbox" to see what authoring is like, and even start up your own deliberation map if you like (contact me if you want to do the latter).

Mark