From Wikipedia, the free encyclopedia

Shortcut: WP:EPR

This is a list of peer reviews of Wikipedia content conducted by external agencies. It aims to be both a record of the findings of these reviews and a way to highlight any problems (or indeed positive aspects) that were encountered. While we do have our own internal peer review, those from respected publications are more than welcome and can help to improve our articles. In most cases we will also see an increase in traffic to the articles mentioned, which means that those with poor reviews should be improved to as good a standard as is possible.

Peer reviews are different from general articles about Wikipedia because they give a critique of one or more articles specifically, often noting problems or highlighting their good qualities. Quotes from people that have made comments about Wikipedia as a whole can be found at Wikipedia:Testimonials and Wikipedia:Criticisms.

Articles that have undergone an external peer review are tagged with {{external peer review}}.

Contents 1 Formal 1.1 An empirical examination of Wikipedia's credibility 1.1.1 Findings 1.1.2 Response 1.2 Nature 1.2.1 Findings 1.2.2 Responses 1.2.3 Errors per word comparison 1.2.4 Article status 1.2.5 Tagging all the Nature-reviewed articles with errors a bit of an overreaction? 1.2.6 Revision progress 1.2.7 Britannica's Reply 1.2.8 Nature's Response 1.3 Information Quality Discussions in Wikipedia 1.3.1 Findings 2 Informal 2.1 Shoutwire 2.1.1 Findings 2.1.2 Responses 2.2 The Chronicle of Higher Education 2.2.1 Findings 2.2.2 Responses 2.3 The Independent 2.3.1 Findings 2.3.2 Responses 2.4 Financial Times 2.4.1 Findings 2.5 San Antonio Express-News 2.5.1 Findings 2.6 Mail & Guardian 2.6.1 Findings 2.6.2 Responses 2.7 The Guardian 2.7.1 Findings 2.7.2 Responses 2.8 Roanoke Times 2.8.1 Findings 2.9 St. Petersburg Times 2.9.1 Findings 2.9.2 Responses

[edit] Formal

[edit] An empirical examination of Wikipedia's credibility

• Source: Thomas Chesney (Information Systems, Nottingham University Business School)
• Date: 2006-11-06
• Title: An empirical examination of Wikipedia's credibility
• URLs: http://firstmonday.org/issues/issue11_11/chesney/
  • http://arstechnica.com/news.ars/post/20061127-8296.html
  • http://slashdot.org/article.pl?sid=06/11/28/1622217

[edit] Findings

• "five reported seeing mistakes and one of those five reported spelling mistakes rather than factual errors"

[edit] Response

Email sent asking for a list of the articles with errors (or a list of the errors) if possible, so we can create a to-do list similar to Wikipedia:External_peer_review/Nature_December_2005/Errors. -- Jeandré, 2006-11-29t08:10z

Bleh... I had not noticed this yet, and just sent another. Circeus 23:12, 29 November 2006 (UTC)

Same here. Any replies?-- _{Piotr Konieczny aka Prokonsul Piotrus | talk}  22:20, 16 March 2007 (UTC)

I received an initial reply without a list of articles or errors but the possibility of some more information later; but nothing since then. I suggest that only 1 person, User:Piotrus, does a follow up. -- Jeandré, 2007-03-17t06:07z

I got a reply, which sais, basically, that "About the errors, the questionnaire was completed anonymously so there is no list of the articles that contained the errors, let alone the errors themselves." With that, I think we can archive the study, as with such design it is not really a usable EPR.-- _{Piotr Konieczny aka Prokonsul Piotrus | talk}  17:04, 24 March 2007 (UTC)

[edit] Nature

A complete list of errors with their current status can be found at Wikipedia:External peer review/Nature December 2005/Errors

• Source: Nature
• Date: 14 December 2005
• Title: Internet encyclopedias go head to head
• URLs:
  • "Internet encyclopaedias go head to head"
  • Nature reviewer reports
  • List of articles reviewed
  • Nature blog
  • Nature editorial
• Related news articles:
  • BBC News
  • Business Week
  • Aljazeera.net
  • USA Today
  • Scientific American blog
• other links at Wikinews
• Britannica rebuttal
  • Fatally flawed: refuting the recent study on encyclopedic accuracy by the journal Nature (PDF). Encyclopædia Britannica, Inc. (2006). Retrieved on 2006-03-23.
  • Nature mag cooked Wikipedia study - Britannica hits back at junk science - By Andrew Orlowski in San Francisco
  • James Bone Britannica 'still rules' over web rival in The Times March 25 2006

Nature compared Wikipedia and Britannica science articles and sent them to experts in the field. The number of "factual errors, critical omissions and misleading statements" were recorded.

From their blog:

We're trying to see if we can publish the full list of errors found by our reviewers, or least send them to you (and to Britannica if they want). We'll post an update here as soon as we have a firm answer.

I also received a private email from them in response to a request for more information. I hope they don't mind me posting it below:

In light of the amount of interest, we have decided to make the reviews public as far as possible, although obviously we'll have to edit them to remove the names of the reviewers, any libellous statements etc. The reviewers didn't all respond in the same format, and some of them highlighted points that we didn't consider to be significant errors, so we're also writing up an accompanying document to explain which errors we counted, and how we arrived at all the numbers. We're also asking the reviewers if they mind being identified, so we'll name those who give permission. That's all quite a bit of work, especially with Jim being away, but I hope we can send this to you by the end of next week, as well as putting it up (free) on our own website. Thanks for your patience! (15 December 2005)

Update: The reviewer reports are now available on the Nature web site, in Microsoft Word format. See above URL.

[edit] Findings

Of the 42 articles reviewed, 38 were found to have at least one error – Britannica had 40 articles with at least one error.

The following articles had the highest number of errors:

• 19 errors
  • Dmitry Mendeleev
• 11 errors
  • Cambrian explosion
• 9 errors
  • Paul Dirac
• 7 errors
  • Acheulean industry
  • Neural network
  • Prion
  • Thyroid
• 5 errors
  • Ethanol

The following articles had no errors highlighted:

• Subrahmanyan Chandrasekhar
• Lipid
• Punctuated equilibrium
• Quark

Nature's special report also noted the following:

• "several Nature reviewers" found the Wikipedia article they reviewed to be "poorly structured and confusing" — a criticism that the report notes is common among information scientists;
• unnamed information scientists also "point to other problems with article quality, such as undue prominence given to controversial scientific theories" (see Wikipedia:Guidelines for controversial articles and Wikipedia:Neutral point of view for guidelines related to this problem);
• In Wikipedia's defense, Michael Twidale, an information scientist at the University of Illinois at Urbana-Champaign, told Nature that Wikipedia's strongest suit is the speed at which it can be updated, a factor not considered by the journal's reviewers.

Update: the detailed reviewer reports are now available (see above).

[edit] Responses

• Dmitry Mendeleev has been tagged with {{accuracy}} by BrokenSegue - 20:48, 14 December 2005
  • All of them are now tagged. —Steven G. Johnson 02:31, 15 December 2005 (UTC)
    • And in your zest, you tagged at least one article (Cambrian explosion) that had already been rewritten. Dragons flight 02:48, 15 December 2005 (UTC)
      • I know that it had been rewritten, but we have no way whatsoever to know whether the rewritten version corrects the alleged problems. Unless we get a specific list of the criticisms from Nature, of course, we may have no choice but to remove the tags after giving the article our best look. In the short term, however, it is best to acknowledge that we take seriously reviews by such a reputable publisher. Moreover, realize that a lot of people are going to be visiting the articles that are listed in Nature right now, and it will reflect well on Wikipedia if we have some admission of the potential problems. —Steven G. Johnson 03:05, 15 December 2005 (UTC)
        • Bleh, I don't like having the "disputed" template on an article when A) you don't know what's disputed and B) you don't know if it has already been fixed. Instead I created {{NatureDispute}} and put that on Cambrian explosion, which I think serves your purpose in a friendlier way. Use it on other articles if you wish. Dragons flight 03:41, 15 December 2005 (UTC)
          • They are all retagged now with {{NatureDispute}}, which can itself be edited should anyone want to make a refinement. - Nunh-huh 04:57, 15 December 2005 (UTC)

Should we ask Nature to send us a copy of the errors found? If they have such details, that is. violet/riga (t) 21:57, 14 December 2005 (UTC)

I have emailed Nature and hope they respond by publishing the results. Unfortunately, I suspect they will only be available to paying subscribers to the magazine. — Ambush Commander^(Talk) 21:59, 14 December 2005 (UTC)

That's not a problem. I am a paying subscriber (well, through my university employer), as I'm sure are many other Wikipedians, and would be happy to pass along any errors that they publish. However, I doubt this is the case. I'm more worried that the review was done through the usual refereeing process, which is traditionally protective of the anonymity of the referees—they may be reluctant to simply publish the referee reports, although they might give us a summary. Or they might be persuaded to give them to a Wikipedia representative under some conditions of privacy. —Steven G. Johnson 02:30, 15 December 2005 (UTC)

Well, during the usual refereeing process, the referee reports are sent (anonymously) to the authors of the paper being examined, to allow them to improve their paper (it is got accepted) or to know why it was rejected (otherwise), so this should not be a problem. Schutz 08:24, 15 December 2005 (UTC)

I understand that, but the reports are still considered confidential; the authors do not customarily publicize them. But apparently the Nature editors are dealing with this by "sanitizing" the reports where necessary; see their response above. —Steven G. Johnson 04:50, 16 December 2005 (UTC)

Our four correct articles should be marked somehow. Could these be our first {{stable}} articles? ‣ᓛᖁᑐ 23:41, 14 December 2005 (UTC)

i didn't know there was such a thing as a stable WP article. i thought these articles were to be edited inperpetuity. i just heard about this a few minutes ago on NPR's All Things Considered. anyway, the question i have is simply, how do we know, without examination, that the reviewers were correct in every case? i took a look at the Field effect transistor article and, although i saw something i might question (dunno offhand how a constant current source becomes a voltage amplifier), but no completely glaring error in it. we should (somehow) get a detailed list of the criticism and deal with each one, one at at time. r b-j 22:02, 15 December 2005 (UTC)

I'm glad I'm not the only one who hadn't heard of Template:stable before now. However, note this:

Wales also plans to introduce a 'stable' version of each entry. Once an article reaches a specific quality threshold it will be tagged as stable. Further edits will be made to a separate 'live' version that would replace the stable version when deemed to be a significant improvement. One method for determining that threshold, where users rate article quality, will be trialled early next year.

I think it's an excellent idea. <>< tbc 00:14, 18 December 2005 (UTC)

It's an excellent idea if the selection mechanism works. Take a look at Featured Articles and then Featured Article Candidates to see the current best of the best process in action... ;) --Tsavage 00:05, 21 December 2005 (UTC)

no errors != done (they could have poor grammar or could be incomplete) Broken S 23:44, 14 December 2005 (UTC)

Makes it easier though :-) Ta bu shi da yu 02:13, 15 December 2005 (UTC)

They counted "critical omissions", so at least what we have is fairly complete. All that should be left is checking the structure... should we see what Wikipedia:Peer review thinks? ‣ᓛᖁᑐ 03:36, 15 December 2005 (UTC)

Well, the articles that are accurate should be marked somehow once article validation is turned on, don't you think? Perhaps a "externally peer-reviewed" marker? And yes, send them to Peer review, these should be FAs soon. Titoxd^{(?!? - did you read this?)} 04:17, 15 December 2005 (UTC)

I took a look at the 7 articles with only one or no errors to see if they fit Wikipedia: Good articles, & not all would belong:

• Australopithecus africanus -- had an odd phrase describing the Leaky Familiy as "infamous". I'm not aware that they have committed any crimes.
• Subrahmanyan Chandrasekhar -- little more than a stub article: lots of lists, no account of his life or work, & the section "Quotations" should be moved to Wikiquote.
• Pythagoras' theorem -- a former Featured article, now listed as a Good article. Should it be removed from the GA list until the one error is fixed?

As for the remainder -- Lipid, Bjørn Lomborg, Punctuated equilibrium & Quark -- I'm willing to add them if there are no objections. -- llywrch 20:06, 15 December 2005 (UTC)

IMHO, based on working on it on and off for a while, Lomborg is at present not a "good" article. If the writing quality doesn't get you, there is a lot of basic information, much of it quite readily available elsewhere, still missing. --Tsavage 00:05, 21 December 2005 (UTC)

I like how the BBC are now headlining this "Wikipedia in face off with Encyclopedia Britannica" - somewhat misleading. violet/riga (t) 15:20, 15 December 2005 (UTC)

The headline now reads "Wikipedia survives research test", it's available at http://news.bbc.co.uk/2/hi/technology/4530930.stm Walkerma 15:53, 15 December 2005 (UTC)

• Perhaps some of us who can do good on these errors should volunteer to do a careful fact check on each of the ones marked. It would show our strengths if we could fix the errors within a week or so after they were announced. For the record, I'm happy to go over Hans Bethe, Dmitry Mendeleev, and Andreas Vesalius with a fine-toothed comb. --Fastfission 16:17, 17 December 2005 (UTC)

[edit] Errors per word comparison

Please post below a table of errors/word statistics, based upon the Nature article and the word counts in the corresponding articles, so that we can see a more controlled comparison of error rates. —Steven G. Johnson 02:53, 15 December 2005 (UTC)

If it seems to make sense to calculate and compare the ratio errors/words, it probably makes less sense to compare the ratio omissions/words (1 omission in a 5000 words article might be considered to be more serious than 1 omission in a 1000 words article on the same topic, whereas 1 error in 5000 words is probably better than 1 error in a 1000 words); and it might make even less sense to mix both categories in an attempt to gauge any bias in favour or against Wikipedia (or Britannica) in the Nature experiment. I would therefore be very careful in drawing any conclusions based on the numbers in the table below. --83.180.100.151 20:48, 8 January 2006 (UTC)

Britannica displays the word count for each article, doesn't it? At least that part shouldn't take long. ‣ᓛᖁᑐ 03:29, 15 December 2005 (UTC)

You'd think that not all words are created equal. — Ambush Commander^(Talk) 03:43, 15 December 2005 (UTC)

No doubt, but this seems the quickest reasonable statistic to gather. —Steven G. Johnson

Is this really more controlled? The average WP article has lots of fluff, with no space-pressure to remove same. Errors/omissions-per-article seem a reasonable metric to me. +sj + 07:37, 15 December 2005 (UTC)

Your fluff is another person's interesting tidbits. =) Honestly, it's hard to make a quantitative assessment of information content, but if there is a large difference in article length then it is a hint that apples are not being compared to apples. (Nature claims that the article lengths were comparable, but I'm finding this hard to reconcile in some cases, e.g. West Nile virus (see below) where the WP article has apparently been almost 5 times longer than EB's for a year now.) The size differences seem large enough that I'm inclined to think that the 30% difference between EB and WP in the Nature study is washed out by systematic problems, although that of course depends on the type and severity of the errors. —Steven G. Johnson 08:35, 15 December 2005 (UTC)

Note that, for a fair comparison, we shouldn't include tables of contents, external links, "see also", or references — most Britannica articles do not include these, and the Nature review did not consider referencing quality. Nature refers to "factual errors, omissions or misleading statements", so some of the "errors" listed below may be errors of omission in incomplete articles, rather than factual errors.

Article name	Britannica			Wikipedia
	Word count	Errors	Errors/word	Word count	Errors	Errors/word
Acheulean industry	500	1	0.002	417	7	0.016787
Agent Orange	252	2	0.00793	1270	2	0.0015748
Aldol reaction	130	4	0.030769	660	3	0.0045455
Archimedes' principle	350	2	0.0057143	607	2	0.0032949
Australopithecus africanus	235	1	0.0042553	496	1	0.0020161
Bethe, Hans	658	1	0.0015198	1823	2	0.0010971
Cambrian explosion	519	10	0.019268	702 (13 Dec.)	11	0.0157
Cavity magnetron	394	2	0.0050761	1121	2	0.0017841
Chandrasekhar, Subrahmanyan	365	4	0.010959	417	0	0
CJD	591	2	0.0033841	1373	5	0.0036417
Cloud	641	3	0.0046802	1689	5	0.0029603
Colloid	561	3	0.0053476	896	6	0.0066964
Dirac, Paul	837	10	0.011947	1044	9	0.0086207
Dolly	1334	1	0.00074963	807	4	0.0049566
Epitaxy	178	5	0.028090	235	2	0.0085106
Ethanol *	315	3	0.0095238	2631	5	0.0019004
Field effect transistor	588	3	0.0051020	933	3	0.00322
Haber process	241	1	0.0041494	531	2	0.0037665
Kinetic isotope effect	210	1	0.0047619	569	2	0.0035149
Kin selection	923	3	0.0032503	404	3	0.0074257
Lipid	349	3	0.0085960	676	0	0
Lomborg, Bjorn	518	1	0.0019305	1501	1	0.00066622
Lymphocyte	479	1	0.0020877	351	2	0.0056980
Mayr, Ernst	357	0	0	753	3	0.0039841
Meliaceae	152	1	0.0065789	281	3	0.010676
Mendeleev, Dmitry	1306	8	0.0061256	1134	19	0.016755
Mutation	728	8	0.010989	1557	6	0.0038536
Neural network	557	2	0.0035907	1233	7	0.0056772
Nobel prize	409	4	0.0097800	2052	5	0.0024366
Pheromone	313	3	0.0095847	461	2	0.0043384
Prion	473	3	0.0063425	1583	7	0.0044220
Punctuated equilibrium	943	1	0.0010604	1265	0	0
Pythagoras' theorem *	688	1	0.0014535	1899	1	0.00052659
Quark	1112	5	0.0044964	2060	0	0
Royal Greenwich Observatory	235	3	0.012766	532	5	0.0093985
Royal Society	416	6	0.014423	869	2	0.0023015
Synchrotron	770	2	0.0025974	1590	2	0.0012579
Thyroid	583	4	0.0068611	1459	7	0.0047978
Vesalius, Andreas	930	2	0.0021505	1174	4	0.0034072
West Nile Virus	245	1	0.0040816	1320	5	0.0037879
Wolfram, Stephen	475	2	0.0042105	559	2	0.0035778
Woodward, Robert Burns	873	0	0	2320	3	0.0012931
Total	22733	123		45254	162
Mean	541.26	2.9286	0.0054106	1077.5	3.8571	0.0035798

* - Articles marked as good articles

• Word counts were computed by pasting text from the Firefox browser into the Unix wc program. This gives a slight over-estimate because it counts anything surrounded by whitespace as a "word". Britannica counts were taken from Britannica Online. Wikipedia articles were from 14 December 2005 except where otherwise noted.
• Tables of contents, external links, see also, and reference sections were excluded from word counts.

As a note of caution, since Nature said the lengths of articles they compared were roughly equal, the versions they compared must be different from those whose length we compare here (since our versions are now longer than the EB articles.). This isn't surprising considering the lead time needed to select the articles, send them out for review, gather the reviews, and compile and publish the results. - Nunh-huh 06:43, 15 December 2005 (UTC)

I find that a bit dubious. By that theory, we should be able to date the review by going back to the last time the articles were comparable to the EB articles (which presumably haven't changed much recently). However, I just checked the Vesalius, Andreas article (which is one of the egregious examples where WP is 10 times the word count of EB), and you have to go back to 2002 to get significantly shorter than it is now, which seems unlikely. I find it more likely that the editors simply tossed out any obvious stubs or near-stubs. —Steven G. Johnson 06:55, 15 December 2005 (UTC)

So something is amiss. I somehow doubt Nature compared a 62 word EB article on Robert Burns Woodward to his Wikipedia 2300 word article. (And found no significant omissions in EB!) - Nunh-huh 06:59, 15 December 2005 (UTC)

Oh @#$#@, no it's my fault. I miscounted the EB Vesalius article — it's one of the (very few) EB articles which is spread over multiple pages, and I only counted one page. Similarly with the Woodward article. I'll go back and recheck any others where the imbalance seems to be large. (Update: counts should be corrected now.) —Steven G. Johnson 07:02, 15 December 2005 (UTC)

If there winds up being a significant difference, then the explanation is most likely error or a difference in versions. Nature specifically states "All entries were chosen to be approximately the same length in both encyclopaedias. In a small number of cases some material, such as reference lists, was removed to make the lengths of the entries more similar." [1]. - Nunh-huh 07:24, 15 December 2005 (UTC)

Note that the references, external links, and "see also" sections were excluded from all of my counts. I still find discrepancies I can't explain. Our West Nile virus article, even one year ago, was 1108 words or 4.5 times the word count of the EB article. I've searched around EB, and I can't find any huge alternative article on this virus that they could have used instead. —Steven G. Johnson 07:31, 15 December 2005 (UTC)

Is there an alternative EB website? Are we counting the words in an EB "junior" site? Just a possibility. - Nunh-huh 08:08, 15 December 2005 (UTC)

According to their website (to which my university gives me access), I am searching the full EB. (They also have "student", "concise", and "elementary" versions of EB, but I'm not using those.) I'm not accusing Nature of dishonesty, but I admit I'm mystified. I wouldn't be surprised if they thought a factor of two was "comparable" length, but a factor of almost 5 seems like a lot. —Steven G. Johnson 08:14, 15 December 2005 (UTC)

I've emailed the news editor and asked if they could provide the exact article date/time versions that they sent to those 50 experts. That should be useful for figuring out if the errors have already been fixed, and for comparing the versions they sent and the soon-to-be more accurate ones. It's possible that they didn't think to check their versions, but they probably did. I'll pass on the reply when it comes. --Mr. Billion 23:09, 16 December 2005 (UTC)

No reply. Ah, well. We'll see. --Mr. Billion 19:32, 18 December 2005 (UTC)

From the editors (see supplementary Nature report above):

Each of the reporters that worked on the survey chose 10 to 15 scientific terms that were roughly in their scientific beat – the sorts of things we ourselves would check in an encyclopaedia. We had not looked at any of these entries in either encyclopaedia when we selected them. Then we weeded out the terms that did not have any entry in Britannica (they all appeared in Wikipedia), and any for which the entries were vastly different in length. Sometimes the lengths were balanced by amalgamating two or three Britannica entries into one coherent piece – for example, 'ethanol' was done this way. We felt this represented 'everything Britannica had to say on the subject' – at least, everything we could find by a quick search of Britannica online, exactly the way a user would approach

So, the criterion was not "vastly different" in length, which would allow e.g. a factor of two difference, and maybe even a factor of 5. —Steven G. Johnson 17:35, 22 December 2005 (UTC)

[edit] Article status

None of the articles reviewed have featured article status, and none have undergone our internal peer review process. Two articles, Ethanol and Pythagorean theorem, have good article status with the latter being a former featured article (see here, though there is virtually no discussion there). violet/riga (t) 19:48, 15 December 2005 (UTC)

[edit] Tagging all the Nature-reviewed articles with errors a bit of an overreaction?

Putting a "This article has been identified as possibly containing errors" tag on the front page of ALL articles identified by Nature seems a bit of an overreaction, particularly in the case of articles with one or two "errors". Also, given the lack of adequate information concerning the reviewing criteria, and, most importantly, WHICH REVISION OF EACH ARTICLE WAS REVIEWED, it sets up a bit of an unnecessary witchhunt situation.

• The nature of the errors is not at all clear; according to the Nature report, Only eight serious errors, such as misinterpretations of important concepts, were detected in the pairs of articles reviewed, four from each encyclopaedia, the rest were factual errors, omissions or misleading statements -- it is extremely difficult to decide what might have been considered an error, even a simple typo could result in a "factual error".
• Once again, the revision dates of the reviewed Wikipedia articles are not indicated; without this information, and considering the lead time of weeks even months for magazine articles, it is IMPOSSIBLE to determine whether the "errors" even exist anymore, unless an article has been entirely untouched for a very long time.
• The conclusion, according to Nature, is that Wikipedia comes close to Britannica in terms of the accuracy of its science entries, and the difference in accuracy was not particularly great

Common sense and the findings of Nature's investigation both say that the vast majority of Wikipedia articles will contain some type of error or omission, so why single out these articles? What is this questing for "experts" to validate the work here? (First, from discussion pages, it appears that numerous contributors possess significant practical expertise and academic credentials in many subject areas, so there is already...professional input. Second, the "experts" who "understand" esoteric stuff aren't necessarily the best people to explain 'em, for a number of obvious reasons, or do we want college professors to mark the work?) If Wikipedia is already pretty close to Britannica, why not just get on with things? A Nature tag on the Talk page should be sufficient in most if not all cases. --Tsavage 23:53, 20 December 2005 (UTC)

Your reaction is an overreaction. =)

1. The Nature editors have stated that they will provide us with a specific list of their reviewer comments (see top), at which point we can fix any problems and the warning tags will be removed. Tags for a couple of weeks won't hurt us; quite the opposite.
2. As a purely public relations matter, a lot of people, including journalists, are visiting the articles mentioned in Nature. It will make a good impression to see that the articles already indicate our awareness of potential problems. A lack of any acknowledgement, on the other hand, will suggest that we can't respond quickly or that we are so clueless that we don't take Nature seriously. (The Talk page is effectively invisible to most readers.)
3. Nature is one of the most prestigious scientific journals. We should take their criticism, even vague criticism, very seriously indeed, regardless of PR. Nor are we satisfied with "only" one or two errors, regardless of how we compare to Britannica.
4. I think you're greatly mistaken if you think "experts" who "understand" stuff aren't vitally important in writing and vetting science articles. In my experience, in order to properly explain things at a simple level you have to understand them at a much deeper level or else you will probably make simplifications that are incorrect.

—Steven G. Johnson 06:52, 21 December 2005 (UTC)

Well, I find your reaction to my reaction to "our" reaction, an overreaction... I'm merely contributing my input around the topic.

• I'm here and contributing largely because this is an open content-licensed project that boldly states on its front page: "the free encyclopedia that anyone can edit." You don't even have to register to contribute. That pretty clearly sets up the premise for the whole project. Where do designated experts providing broad critical comments come into the mechanics of the thing, beyond as a matter of peripheral interest?
• I am part of the "we" you refer to, but so is, basically, anyone and everyone. That's Wikipedia. It is all of us... If we/you/they/anyone can view Wikipedia, they can edit it, and therefore they are the "we". So, who are "we" doing PR for? Who are we trying to prove what to?
• The Nature article is interesting, and great for Wikipedia (and bad for Britannica), but it's also more of a catchy news piece than anything "scientific". Nature publishes "peer-reviewed scientific papers", and it has an in-house-written news section; this report is part of the latter. Asking for detailed comment from people recognized as experts in whatever field is a standard journalistic approach. It is not "peer-review" in a rigorous scientific, experimental sense, more like "book review". It is an interesting editorial gambit that can generate lots of press... So, this We should take their criticism, even vague criticism, very seriously indeed is insipid and wanting in pride and backbone (IMHO, of course).
• Yes, of course the best, most elegantly simple explanations tend to come from people who know what they're talking about. But communicating ideas and being expert in working with them are two different things. Every brilliant researcher and theoretician would not make a great teacher. Every brilliant teacher is not a brilliant researcher or theoretician, that is, will not fully understand all of the concepts involved in what he teaches. It takes all types to communicate an idea widely... Giving "all types" the chance is what Wikipedia to me is about. Bowing down to...academia and the generic concept of "experts" is different from respecting those with knowledge who wish to contribute...

I addressed the "we", you addressed me personally. That's my reaction to you trying to make me into your "we"... :) --Tsavage 17:52, 21 December 2005 (UTC)

Tagging articles on the basis of this study seems wholly an overreaction. The table above shows that just about every article they looked at contained some error both in wiki and in EB. So statistically that implies exactly the same tag ought to be applied to every single article in wiki, since they also likely contain the same rate of unidentified errors. So the tags really aren't going to help wiki content. Whether they help wiki PR is another debateable point. Especially tagging these without a better idea of exactly what the problem might be smacks of oversensitivity to fair comment. Sandpiper 02:01, 24 March 2006 (UTC)

[edit] Revision progress

I've posted messages on the talk pages of each article studied, listing what's wrong. -- user:zanimum

• Vesalius, Andreas: The errors have been fixed. GhePeU 19:28, 22 December 2005 (UTC)

Please list corrections (with dates and links) in a neutral fashion at: Wikipedia:External peer review/Nature December 2005/Errors. —Steven G. Johnson 19:47, 22 December 2005 (UTC)

[edit] Britannica's Reply

Britannica has replied to Nature's article and claim the nature article was itself innacurate in a number of ways. It would probably be good for Wikipedians to check the sources Brittanica gives and note needed changes in our articles.

[edit] Nature's Response

Nature has now issued comments on Britannica's rebuttal, and stands by its original article. [2] or [3]

Jim Giles made a presentation at Wikimania on 2006-08-04.

[edit] Information Quality Discussions in Wikipedia

• Source: Besiki Stvilia, Michael B. Twidale, Les Gasser, Linda C. Smith (graduate students at Graduate School of Library and Information Science, University of Illinois)
• Date: Fall 2005
• Title: Information Quality Discussions in Wikipedia
• URLs: http://www.isrl.uiuc.edu/~stvilia/papers/qualWiki.pdf (draft version submitted to CIKM'05)

[edit] Findings

The focus of the paper was on assessing the information quality of Wikipedia:featured articles — when compared to other samples from the project, including featured article removal candidates, pages marked as NPOV disputes, and a selection of random pages. According to the paper, the study showed how seriously the Wikipedia project views issues of article quality. The authors concluded that as a quality standard, the featured article process "is not ideal, but it does seem relatively rigorous." They also noted that the process is not as resource-intensive as other possibilities, such as blind judging.

[edit] Informal

[edit] Shoutwire

• Source: Shoutwire
• Date: March 7, 2007
• Title: The Accuracy of Wikipedia

[edit] Findings

This text is copied from the review, which I feel is fair use for improving our encyclopedia:

• Sudbury, Ontario (Grade: A++):

Sudbury, Ontario is not the most well known city on Earth. I figured that this relatively obscure town would have a paltry Wikipedia entry with very little information, and even less accurate info. I was dead wrong.

The entry for my hometown is absolutely exhaustive. Not a single detail had been overlooked. From history to geography to demographics, the entry could not be more perfect.

• Electro refining (Grade: C-):

Uh oh. Right away I noticed a glaring inaccuracy. Entering “electro-refining” redirected me to “Electrowinning.” Anyone involved in extractive metallurgy can tell you that the two processes are different. Electro-refining should definitely have had it’s own article, since electrowinning pulls the metal out of a solution whereas electro-refining involves a solid anode.

Though they could have gone into more detail, the rest of the article is pretty accurate. That being said, I certainly wouldn’t recommend this article as a source for someone researching either electro-refining or electrowinning.

• Ore dressing (The author was looking for ore milling, Grade: D-):

I searched for “milling”, and found a very general article about milling. Nothing mining-specific was in the article, although a great amount of information on the process of milling was listed. Since I don’t give up that easily, I tried looking up specific milling-related techniques. Nothing for magnetofloatation, but ore dressing finally turned up a shitty little article. The author could have gone into waaaay more detail. This article is inaccurate because of the amount of detail that has been omitted. My conclusion? Worst. Article. Ever. (Emphasis added is mine. Jesse Viviano 20:23, 8 March 2007 (UTC))

• Nickel (Grade: B-)

Quite frankly, I expected to be overwhelmed with detail on this one. After all, nickel is something that most people come into contact with every day. However, the article glazed over some of nickel’s unique history, and that kind of irked me because the story of nickel is actually quite entertaining. They could have gone into more detail with the “Old Nick’s Copper” story, for example. Maybe I’m just anal.

The extractive process was reasonably accurate except for what I can only assume was a typo (substitute “sulfide” for “lateridic” in the first sentence) but no information on milling techniques (some specific to nickel) were present.

Other than that, the article was of acceptable quality. My hometown even got a shout-out.

• Shoutwire (Grade:B+)

There is really only one astonishing inaccuracy in this one, under the “Comparisons to Digg” section. ShoutWire doesn’t filter the word “Digg” at all. As a test, I submitted an article titled “Digg.com – Best Site On Earth” and sure enough it appeared on the submissions page. The rest of it was OK, though. I found the section on editorials to be quite entertaining.

[edit] Responses

[edit] The Chronicle of Higher Education

• Source: The Chronicle of Higher Education
• Date: 27 October 2006
• Title: Can Wikipedia Ever Make the Grade?

[edit] Findings

• Brave New World (Grade: B-): "The entry provides 'a plethora of links' and more information than the typical literary encyclopedia, says Mr. Firchow, but it is flawed by 'the annoying inaccuracies, the glaring omissions, and the inconsistencies.'" Example: "The article says Huxley wrote Brave New World in Britain and was influenced by Yevgeny Zamyatin's dystopic novel We. Actually, Huxley wrote Brave New World primarily in France and said later that he had not known of We at the time"
• African-American Civil Rights Movement (1955-1968) (Grade: C): "While there are no glaring inaccuracies, Mr. McAdam says, the entry does not provide any analytical context as to what led to the civil-rights movement or what happened afterward." Example: "Mr. McAdam notes the insufficiency of links to events before and after this particular 13 years of the movement, not to mention the lack of any broader analysis of what brought about the fight for civil rights, and the movement's political and cultural impacts."
• Flow cytometry (Grade: A): "The main section on flow cytometry was well-done and accurate, ... but as you drop into the subsections, the accuracy drops off. I went in and actually made some corrections to these — I said, 'I can't leave these and let the record stand like this.'" Example: "At one point in the entry, the word "homologous" was misused in talking about blood transfusions."

[edit] Responses

[edit] The Independent

• Source: The Independent
• Date: February 12, 2006
• Title: Wikipedia under the microscope over accuracy
• URL: http://news.independent.co.uk/media/article345103.ece

[edit] Findings

The paper reviewed the following articles:

• Muslim
  • "Yesterday, the entry "Muslim" was changed 12 times. Over the past week, it was changed more than 50 times...these ranged from the puerile...to suggestions which are deliberately offensive...Wikipedia, which constantly checks that changes improve rather than insult, reverted to the previous entry in less than a minute."
• Russian Revolution of 1917
  • "reads like the work of a second-rate undergraduate student... It is a simplistic account"- Orlando Figes, professor of history at Birkbeck College
• Kate Moss
  • "Factually, this is dead accurate, though it is cloaked in po-faced language...it does not mention that she never gives interviews and has never been known to purposefully utter a word in public" - Marcel D'Argy Smith, former editor of 'Cosmopolitan' magazine
• Ann Widdecombe
  • "I think overall that the entry is much better than Dod's parliamentary guide...The references to the 2001 leadership election are categorically wrong...The entry is pretty good though, I would give them 9.5 out of 10." - Ann Widdecombe
• Tony Blair
  • "That is the problem with Wikipedia - most of it is very good and reliable, but it depends on people interested in a subject being able to pontificate... It is opinionated and written from an anti-war point of view" - John Rentoul, biographer of Tony Blair
• In vitro fertilisation
  • "This would undoubtedly serve as a useful introduction for those with little idea about the subject; this entry would actually be more useful to the average inquirer with its links than would anything in the Encyclopedia Britannica...its politics are probably closest to those of Liberal Democrats... I am considerably impressed with the quality of information" - Robert Winston, fertility expert and television presenter
• Philip Larkin
  • "A good and fair account. It sounds approving of Larkin, which is nice, but it is overall a dispassionate account, as one would expect from a dictionary...Though I can see there is an opportunity to whitewash with Wikipedia, the few times I have used it, I have been impressed with it." - Andrew Motion, Poet Laureate
• BBC Radio 1
  • "Accurate, but with an odd conglomeration of facts without a clear idea of what purpose Radio 1 serves or who listens to it. The odd mixture of facts does not tell you about the wider picture." - Simon Garfield
• Punt (boat)
  • "I am impressed by the amount of information on punting; the two key books on punting are mentioned, as are the clubs...I am impressed. It works on the presumption that by and large people will correct things, and I changed one small thing on my own biography." - Sandy Nairne, director of the National Portrait Gallery

[edit] Responses

Most of the articles were tagged with {{High-traffic}}.

• The criticisms of Tony Blair all relate to assertions added over that weekend to the leading paragraph which were never previously present in the article and were quickly reverted (by me, once I had got back from a weekend break). I think it's a pity because it gives a completely misleading impression of the article. David | Talk 15:34, 13 February 2006 (UTC)
  • The comment about him "saying he would only serve 2 terms" was only there for 19 minutes before I removed it! -- Arwel (talk) 23:18, 13 February 2006 (UTC)
    • I think there was another reference which had escaped me which was in the intro and has also been removed (it was a too literal interpretation of the supposed Granita pact). Incidentally the reviewer makes his own howler when he says Lord Liverpool became Prime Minister in 1824. It was actually 1812. David | Talk 09:57, 14 February 2006 (UTC)

• The reviewer of the punt (boat) article mentioned two specific minor criticisms: The reference to 1ft is too narrow; the narrowest punt is 1ft 3 inches. The reason given for racing punters to stand in the middle of the punt [...] is slightly bizarre. Thruston has fixed both and added a section on racing. Telsa ((t)(c)) 08:23, 15 February 2006 (UTC)

[edit] Financial Times

• Source: Financial Times
• Date: December 23, 2005
• Title: Is Wikipedia the business?

[edit] Findings

"To find out how it performs on basic terms in business and economics, we asked Professor Fariba Alamdari, an economist at Cranfield University, to take a look. She says: "I am very impressed. It offers a fairly accurate and comprehensive definition of business terms, with links to other related websites or articles, and sometimes statistics."

Prof Alamdari cites the case of gross domestic product. Wikipedia provides a definition, shows how it is calculated and gives the GDP by country both in nominal terms and based on PPP (purchasing power parity).

She was impressed by the way Wikipedia dealt with such terms as Chapter 11 bankruptcy and return on investment, and those with multiple interpretations such as CRM - which an IT manager would read as Customer Relationship Management but a marketing executive would see as Cause-Related Marketing. Cracks did start to appear, however, with a warning about links to potentially biased commercial sites on the entry for the first interpretation, and no entry at all for the second."

[edit] San Antonio Express-News

• Source: San Antonio Express-News
• Date: December 16, 2005
• Title: David King: Wikipedia accurate for S.A. sports

[edit] Findings

" Well, if you're a sports fan, consider this elsewhere. Among the articles that are substantially correct:

San Antonio Missions: Pretty much on the ball, although describing Henry the Puffy Taco as a "child-friendly mascot themed after local cuisine" seems a little cold and clinical.

San Antonio Texans: The city's one-year adventure into becoming a little bit o' Canada is chronicled in more detail than the Missions' history, which only dates to 1888. It all looks right, too.

San Antonio Spurs: Wow. This reads like NBA.com, which may be why the Baseline Bums aren't included.

Smokey Joe Williams: There's a thorough article about the legendary Negro Leaguer from Seguin, although a link to the San Antonio Black Bronchos, where Williams got his start, is available for the enterprising writer to complete (good luck, by the way).

Ross Youngs: Trust me, this one about our other member of the Baseball Hall of Fame is correct, if too short.

Among those that could use your help:

List of athletes on Wheaties boxes: Even the Wikipedia people admit the list isn't complete, but it is amazing that someone would try to develop one. Among the local notables: Pinkey Whitney, Joel Horlen, Steve Kerr and Dennis Rodman. Among the MIA: David Robinson and Tim Duncan.

And among the missing in action:

San Antonio Tejanos: The one-year experiment in independent baseball, cleverly positioned in San Antonio the year Wolff Stadium opened."

[edit] Mail & Guardian

• Source: Mail & Guardian
• Date: 7 November 2005
• Title: Can you trust Wikipedia?
• URL: http://www.mg.co.za/articlePage.aspx?articleid=255920&area=/insight/insight__national/
• Signpost article: Wikipedia:Wikipedia Signpost/2005-11-14/Article evaluations

[edit] Findings

Reviewed 8 articles relating to South Africa and gave them scores ranging from 2/10 (Media in South Africa) to 10/10 (South Africa national rugby league team and South Africa national rugby union team).

[edit] Responses

Many of the articles were quickly improved and the Mail & Guardian published a second article "Wikipedia springs into action after M&G Online article" which said, in part, "...most of the entries have been edited and improved."

• Elf-friend rewrote the Media in South Africa article.
• Inaccuracies were removed from African National Congress

[edit] The Guardian

• Source: The Guardian
• Date: October 24, 2005
• Title: Can you trust Wikipedia?
• URL: http://www.guardian.co.uk/g2/story/0,,1599116,00.html

[edit] Findings

Experts review 7 articles and rated them on a 0 to 10 scale. Controversially Robert McHenry (former Editor-in-Chief of Britannica) reviewed the encyclopedia article. The scores ranged from 0/10 (Haute couture) to 8/10 (Bob Dylan).

Article name	Mark out of 10
Steve Reich	7
Haute couture	0
Basque people	7
TS Eliot	6
Samuel Pepys	6
Bob Dylan	8
Encyclopedia	5

[edit] Responses

See Wikipedia talk:Response to the Guardian

[edit] Roanoke Times

• Source: Roanoke Times
• Date: July 1, 2005
• Title: Professors give online resource mixed grades

[edit] Findings

Three college professors were asked to evaluate articles in their areas of expertise.

• "Virginia Tech geologist Bob Bodnar looked up the entry on the geologic time scale. 'I found the terminology and ages used to be quite accurate and consistent with the most recent data,' he said."
• "Roanoke College political science professor Bill Hill looked up 19th century politician John Taylor of Caroline County, political philosopher Alexis De Tocqueville and the Federalist Papers. 'In each case, the program responded with information that was accurate and pertinent, if brief,' he wrote. 'I thought the coverage of De Tocqueville, however, was too superficial.'

  (Note: De Tocqueville's entry weighs in at 1,011 words, while Katie Holmes merits 1,893.) Hill said he would discourage his students from using it as a 'serious source,' but noted that he discourages the use of normal encyclopedias as well.

  A feature of the Wikipedia that Hill likes are the links to original sources. It's pretty handy when reading about De Tocqueville to be able to click a link and read the man's own words. But Hill also noted a link that led to a "very brief and shallow essay" about states' rights."

• "Dave West, a retired Virginia Tech biology professor, looked up three biology-related sites. One was good, but two were bad.

  To the average person, the article about 19th century naturalist Fritz Muller would look very detailed and well-researched. But to West, who has written a biography of Muller, it was 'fraught with errors.'

  West listed more than half a dozen of these, including the wrong year of birth and a picture of Muller that's not actually Muller." [West helped to correct the errors.]

[edit] St. Petersburg Times

• Source: St. Petersburg Times
• Date: November 8, 2004
• Title: There's no end to it

[edit] Findings

"The St. Petersburg Times recently asked two University of South Florida professors to read a few Wikipedia articles on topics in their expertise. Chemistry professor Bill Baker said he was surprised at the amount of technical knowledge posted on the site, but said he found several small errors. "The cancer drug Taxol, for example, is not produced by microbial fermentation."

"That bothers me," Baker said of the errors. "I think that even if 99 percent of your facts check out, it is a disservice to promulgate 1 percent inaccuracies."

Professor Philip Levy, an expert on Colonial America, said that "in many respects it's very good," but he, too, had misgivings. He said some articles contained a mishmash of information - the established scholarly knowledge of 15 years ago was mixed in with newer, more controversial theories with little distinction."

[edit] Responses

See Wikipedia talk:Response to the Guardian