2007/05/21
Sermo: Web 2.0 in a "Closed" Community
One of the more interesting talks I've seen lately at a professional meeting was one given yesterday by Alex Frost, a VP for a company called Sermo, at the first full day of the annual meeting of the Council of Science Editors. Briefly, Sermo has built a terrific online-community platform for physicians that enables discussion of current techniques -- essentially a "what would you prescribe in this situation" kind of virtual water cooler that addresses the demise of traditional meatspace social networking (e.g., the old Tuesday afternoon golf game) at which clinical problems would be discussed and hashed out. Excellent user interface, and seemingly all the right choices with respect to functionality. And the business model was interesting, too -- basically, the physicians participate for free, and the revenues come from outside concerns (e.g., the drug companies) looking to tap into the front-line knowledge being exchanged on the site. The talk was roughly twice as long as a typical CSE presentation, but just seemed to get more and more interesting.
What really grabbed me about this, though, was that although this application is Web 2.0 to the core -- both from the point of view of the AJAX-y implementation and the whole approach to social networking, content weighting based on user votes, etc. -- it kind of turns the Web 2.0 "openness" mantra on its head. Sermo is very much a closed user community; indeed, the entire system is premised on an automated credentialing system that verifies the qualifications of people who join and freezes out anyone else. (For this reason, Sermo is right now limited to U.S. physicians, as the problems with extending the automated credentialing to folks in other countries are a bit too knotty.) Indeed, Sermo's viability would appear to depend on this kind of strict attention to credentialing, as well as user anonymity, which is strictly enforced, and the site's ability to remain independent from the drug companies and others with a vested interest. (Independent in terms of content creation, at least; as I mentioned, the drug companies and other external users do pay Sermo to get access to the crowd wisdom embodied in Sermo's actual postings.)
Obviously, there's nothing in Web 2.0, strictly speaking, that rules out a closed community, and indeed, Sermo seems to be built on a classic "Architecture of Participation". But colloquially, I think many of us tend to think of large, open, freewheeling communities like del.icio.us and Digg, which require/promote breadth and inclusiveness across the board, when we think of this sort of architecture; Sermo relies heavily on precisely the opposite. It's a "gated community", and needs to be so for its own viability. (On a slightly off-topic note, I also thought that Sermo represents a very useful corrective to the disproportionate influence of the drug companies on physicians -- by providing a "safe space" where physicians can discuss issues among themselves without pharma promotion or blandishments, it seemed, at least to me, to offer a useful vehicle for some frank discussion of things like the appropriateness of some of the off-label uses for drugs that are such a big part of pharma marketing. But I will now get back down from the soapbox.)
There was an awful lot more that was interesting about this talk, and I wish I had time to put it all together. For example, looking at something more germane to scienific publishing sensu strictu, there was a riff on Sermo as a vehicle for post-publication (and even pre-publication) peer review. Because of its emphasis on anonymity and credentialing, a Sermo-like site seems like a better vehicle for such vetting than something like Nature's blog-based open peer review experiment -- which I still believe, as I suggested in this post, was crimped by the fact that the likeliest contributors to such a process were precisely those who had the most to lose by attaching their name to a critical review in an open forum.
Anyway: a very thought-provoking talk from Alex, and one that has me, at least, very excited about the exportability of some of these ideas to other contexts.
What really grabbed me about this, though, was that although this application is Web 2.0 to the core -- both from the point of view of the AJAX-y implementation and the whole approach to social networking, content weighting based on user votes, etc. -- it kind of turns the Web 2.0 "openness" mantra on its head. Sermo is very much a closed user community; indeed, the entire system is premised on an automated credentialing system that verifies the qualifications of people who join and freezes out anyone else. (For this reason, Sermo is right now limited to U.S. physicians, as the problems with extending the automated credentialing to folks in other countries are a bit too knotty.) Indeed, Sermo's viability would appear to depend on this kind of strict attention to credentialing, as well as user anonymity, which is strictly enforced, and the site's ability to remain independent from the drug companies and others with a vested interest. (Independent in terms of content creation, at least; as I mentioned, the drug companies and other external users do pay Sermo to get access to the crowd wisdom embodied in Sermo's actual postings.)
Obviously, there's nothing in Web 2.0, strictly speaking, that rules out a closed community, and indeed, Sermo seems to be built on a classic "Architecture of Participation". But colloquially, I think many of us tend to think of large, open, freewheeling communities like del.icio.us and Digg, which require/promote breadth and inclusiveness across the board, when we think of this sort of architecture; Sermo relies heavily on precisely the opposite. It's a "gated community", and needs to be so for its own viability. (On a slightly off-topic note, I also thought that Sermo represents a very useful corrective to the disproportionate influence of the drug companies on physicians -- by providing a "safe space" where physicians can discuss issues among themselves without pharma promotion or blandishments, it seemed, at least to me, to offer a useful vehicle for some frank discussion of things like the appropriateness of some of the off-label uses for drugs that are such a big part of pharma marketing. But I will now get back down from the soapbox.)
There was an awful lot more that was interesting about this talk, and I wish I had time to put it all together. For example, looking at something more germane to scienific publishing sensu strictu, there was a riff on Sermo as a vehicle for post-publication (and even pre-publication) peer review. Because of its emphasis on anonymity and credentialing, a Sermo-like site seems like a better vehicle for such vetting than something like Nature's blog-based open peer review experiment -- which I still believe, as I suggested in this post, was crimped by the fact that the likeliest contributors to such a process were precisely those who had the most to lose by attaching their name to a critical review in an open forum.
Anyway: a very thought-provoking talk from Alex, and one that has me, at least, very excited about the exportability of some of these ideas to other contexts.
2006/06/14
Nature Open Peer Review Watch
In the kitchens, lunchrooms, and halls of academic departments around the world, even as I write this, postdocs, grad students, and junior faculty are daring each other to be the first to post a critical comment on Nature's new open peer review trial.
In the event you haven't heard -- and if you've managed to dredge down this deep in the Mariana Trench of the blogosphere, you almost certainly have -- Nature, about a week and a half ago, started a three-month experiment in which the journal would for certain papers, in parallel with its normal, anonymous peer review process, put the papers up on a preprint server for very public comments. Basically, it's an instance of the blogging software Movable Type, with comments enabled; commenters would be required to include their name and institution with the comment, and supposedly the comments would be considered along side the reviews received through normal channels in assessing the paper. It's not a completely open forum; Nature's editors will assess the comments received and determine which ones will be posted -- assuming they're not too busy screening out comment spam with titles like "Hot new pix!" and "Cheaper v1agra."
The first two papers were posted on the preprint server last week, both of them in immunology; one of them has since rather mysteriously disappeared -- perhaps the paper was rejected. Today, there are four manuscripts posted -- one of the original two in immunology, plus one in astrophysics, one in structural biology, and one in paleoclimatology. None of them have attracted any comments yet. (There's also an RSS feed if you're itching to keep track of when a new paper is posted; unfortunately, though, it doesn't appear to notify you of new comments to existing papers.)
Don't get me wrong; I think this is a terrific, very clever idea -- blogging the peer review process. But I must admit that I've found myself wondering (1) who would post their paper in such a forum, and (2) who would comment.
On point (1), the FAQ on this experiment includes the following revealing item (answered with admirable forthrightness):
Whether or not the preprint is eventually published in Nature, isn't there a chance of the results being scooped by competitors?Yes.
Even for those who profess to want a more open peer review process from journals practicing anonymous review, I wonder if this process might seem a bit too open.
And with respect to item (2), who is actually going to respond? If I were a young scientist in career-building mode (arguably the most likely person to be interested in a feature like this), I'd be very reluctant to put my name on anything that might come back to haunt me later -- particularly if the author of the manuscript under review were, say, a dyspeptic Nobel laureate with a known vindictive streak (I'm sure there's one or two of them out there). That's an extreme case, of course, but the principle remains. That could just serve to make criticism in these open comments more polite and fruitful than it sometimes is in anonymous review. But it could also select for a preponderance of positive comments and against negative or critical ones.
So it will be very interesting to see what happens when that initial comment is posted -- and who it is that breaks the silence. It's an interesting experiment, and I suspect we'll all learn something from it.
Technorati Tags : Nature, journal, peer, review
2006/06/12
Microsoft Academic Search: Promising
It's become something of an article of faith among the technically savvy and open-source-minded: Google good, Microsoft evil. Google cool, Microsoft kludgy. And certainly, anyone who has nearly put their fist through their monitor after the 1,134th appearance of Clippy, who has gaped in bafflement at Microsoft Bob, or who has spent an hour trying to disable the "improvements" such as real-time "grammar checking," in a given new version of Word, isn't likely to have warm feelings for the Orcs of Redmond. So a comparison between Microsoft's new Academic Search beta and Google Scholar (also officially in "beta," though it's been on the scene, I believe, for nearly two years now) should be a layup for Google, right?
After playing with the MS application (spurred by a demo at a recent trade show), I'm not so sure.
Right now, the MS Academic Search is limited to references in computer science, electrical engineering, and physics, so it's a bit hard to draw direct comparisons of the reliability and comprehensiveness of the results. But a few things stand out with respect to the UIs. The Academic Search UI is very different from Google Scholar's in the search result; the MS application gives you a two-paned view with lots of AJAX skullduggery in the background, so that when you put your mouse over a search result in the left-hand pane, you get the article's complete metadata (including abstract, where available) in the right-hand pane. In that same pane you can view BibTeX and EndNote versions of the citation information. (Though initially I had thought this was rather cluttered, it's all very handy after you've played with it a while.) The feature I'd focus on, though, is the pull-down box that lets you re-sort results by author, date, journal, or conference -- particularly the reverse-chron date sort.
I tried the search string "XML bioinformatics" on both Live Academic Search and Google Scholar. For the relevance-ranked search, both engines returned the same top result -- a review article by Achard et al. from 2001 -- but after that the results diverged sharply. And both, strangely enough, looked like they could be useful, even though I'm not sure that the two relevance-ranked results had any of the top 10 results in common save the first. (It is indeed a big world out there.) Selecting the reverse-chron sort on Academic Search quickly re-sorted the results into a nice list beginning with a 2006 article from Bioessays, and proceeding in well-behaved reverse-chron from there.
The Google Scholar results page, meanwhile, has a toggle in the upper right-hand corner that lets to switch between "All Articles" (the default view) and "Recent Articles." Where would you expect the "Recent Articles" link to take you? A reverse-chron list of the most recent publications relevant to the search string, perhaps?
Me too. But on Google Scholar, the first article you get by selecting the "Recent Articles" toggle is . . . the same 2001 article that was at the top of the "All Articles" list. And you'll find only a single 2006 article on the first page of results (number nine on the page). This is nonsense, of course, but it isn't limited to this particular search string; I've had similar results on Google Scholar with hot-button topics like avian flu, for which it's reasonable to assume that something worthwhile has been published in the current year. And Google's Advanced Search page on Scholar doesn't give you the option of a reverse-chron listing.
So on at least one use case that would seem to be important to research -- "Show me what's recently been published on [insert your topic here]" -- the MS app would seem to beat Google. At least for now.
The MS tool has some clear disadvantages, too: Some of the AJAX witchery leads to unpredictable behavior in the search-results pane, for example, and the design choices in the results don't always seem to have been made with usability as the top priority. Some of the other displays, such as sorting by journal or by author, are awkward to navigate, and the rules underlying the sorts are baffling. And right now the Microsoft engine doesn't do anything with respect to citations, which is of course a core strength of Google Scholar -- though the pleasant chap from Microsoft at the booth told me that's on their things-to-do list (as, of course, is broadening the search universe beyond just EE, physics, and comp sci).
And, of course, there's speed. At the same trade show at which I saw this demo, one of the speakers made the interesting point that Google's big proprietary strength in the marketplace isn't so much search per se, as its ability to leverage and manage a distributed system of servers for incredibly fast results (the "13,900,000 results on Anna Nicole Smith in 0.11 seconds" syndrome). That's going to be hard for anyone to beat.
2006/06/09
Why "Missing Teaspoons"?
OK -- first of all, the name. The concept is borrowed (stolen) from a marvelous little parody by Lim et al. that appeared in the British Medical Journal at the end of 2005, in which three members of an Australian research institute "discreetly numbered" 70 teaspoons in the institute's tearooms and attempted to track them spatially over the ensuing five weeks. This "longitudinal cohort study" resulted in, among other things, a measured teaspoon half-life and a discussion section that related teaspoon fate to the larger construct of the Tragedy of the Commons. Highly recommended. (Don't forget also to read the article's supplemental material and the Rapid Responses, some of which are hilarious.)
