Ada Lovelace Day

About The Authors

Suw Charman-Anderson

Suw Charman-Anderson

Suw Charman-Anderson is a social software consultant and writer who specialises in the use of blogs and wikis behind the firewall. With a background in journalism, publishing and web design, Suw is now one of the UK’s best known bloggers, frequently speaking at conferences and seminars.

Her personal blog is Chocolate and Vodka, and yes, she’s married to Kevin.

Email Suw

Kevin Anderson

Kevin Anderson

Kevin Anderson is a freelance journalist and digital strategist with more than a decade of experience with the BBC and the Guardian. He has been a digital journalist since 1996 with experience in radio, television, print and the web. As a journalist, he uses blogs, social networks, Web 2.0 tools and mobile technology to break news, to engage with audiences and tell the story behind the headlines in multiple media and on multiple platforms.

From 2009-2010, he was the digital research editor at The Guardian where he focused on evaluating and adapting digital innovations to support The Guardian’s world-class journalism. He joined The Guardian in September 2006 as their first blogs editor after 8 years with the BBC working across the web, television and radio. He joined the BBC in 1998 to become their first online journalist outside of the UK, working as the Washington correspondent for BBCNews.com.

And, yes, he’s married to Suw.

E-mail Kevin.

Member of the Media 2.0 Workgroup
Dark Blogs Case Study

Case Study 01 - A European Pharmaceutical Group

Find out how a large pharma company uses dark blogs (behind the firewall) to gather and disseminate competitive intelligence material.


free page hit counter



hit counter script


All content © Kevin Anderson and/or Suw Charman

Interview series:
at the FASTforward blog. Amongst them: John Hagel, David Weinberger, JP Rangaswami, Don Tapscott, and many more!

Corante Blog

Monday, July 13th, 2009

The plural of anecdote is not data

Posted by Suw Charman-Anderson

The City, and sections of the media, are getting a touch over-excited by a “research note” written for Morgan Stanley by Matthew Robson, a 15 year old on work experience. The Guardian said:

The US investment bank’s European media analysts asked Matthew Robson, an intern from a London school, to write a report on teenagers’ likes and dislikes, which made the Financial Times’ front page today.

His report, that dismissed Twitter and described online advertising as pointless, proved to be “one of the clearest and most thought-provoking insights we have seen – so we published it”, said Edward Hill-Wood, executive director of Morgan Stanley’s European media team.

“We’ve had dozens and dozens of fund managers, and several CEOs, e-mailing and calling all day.” He said the note had generated five or six times more responses than the team’s usual research.

The research note itself can be read on The Guardian’s site.

I’m going to start by giving Robson the kudos that he deserves. He has written a very well thought out piece which describes the media habits of him and his friends. In no way do I want to criticise a teenager for being thoughtful, engaged and articulate.

But one has to put this research note into context: This is one teen describing his experience. It is not a reliable description of all teens’ attitudes and behaviours, yet both Morgan Stanley and the media seem to be treating it as if Robson has Spoken The One Great Truth. “Twitter is not for teens, Morgan Stanley told by 15-year-old expert” coos The Guardian. “Note by ‘teenage scribbler’ causes sensation” says the FT in astonishment.

Neither Morgan Stanley nor the media seem to be able to tell the difference between anecdote and data. This “research note” is more note than research, and it should not be taken to be representative of all teens. A teenager in a rural setting, or in an inner city estate, or one who feels socially excluded from web culture will have a very different experience than a teen who’s well-connected enough to get himself an internship at Morgan Stanley.

What is worrying about this is not Robson’s note: He’s simply doing what most teens (and most adults) do, which is to extrapolate from his own and his friends’ experience to form generalisations about the world around him. It’s a very human thing to do, but the important thing about businesses like Morgan Stanley, and the journalists who write about them, is that they are supposed to be able to tell the difference between data and generalisations. Yet they don’t seem able to sort the wheat from the chaff. It seems yet another symptom of the group-think in the media and financial sector that led to the Great Recession, rather than an indication that we have learned anything from it.

Sarah Perez on ReadWriteWeb says:

Matthew Robson, a 15-year-old intern at analyst firm Morgan Stanley recently helped compile a report about teenage media habits. Overnight, his findings have become a sensation…which goes to show that people are either obsessed with what “the kids” are into or there’s a distinctive lack of research being done on this demographics’ media use. Robson’s report isn’t even based on any sort of statistical analysis, just good ol’ fashioned teenage honesty. And what was it that he said to cause all this attention? Only that teens aren’t into traditional media (think TV, radio, newspapers) and yet they’re eschewing some new media, too, including sites like Twitter.

Well, research has been done. danah boyd has done some excellent research into the use of the web by teens - it’s her speciality and she’s one of the foremost experts in this area. Her research would help Morgan Stanley understand the teen demographic much more clearly than any single anecdote, however well written, ever can. The fact that they haven’t ever had a clear insight into the teen demographic would seem to imply that their existing researchers and analysts aren’t doing their jobs properly. The information is out there, a lot of it is freely available, and all that remains is for someone to read it and write the report.

This story also feeds into the concept of the ‘digital native’ which, as I’ve blogged before, is a very poor way to talk about a very diverse section of the population. But because this report fits in with widely-held assumptions about teens and technology - not only does it describe ‘digital natives’, it’s written by one too - it’s immediately accepted without query or question. Morgan Stanley and the media both seem to be more interested in having their biases validated than they are in exploring the evidence to see where it leads them. Sadly, it seems that neither have been spending enough time watching CSI and drawing from it key lessons about assumptions, evidence and how to draw conclusions.

If I had been Matthew Robson’s boss at Morgan Stanley, on receiving his report I would have praised him on his good work and then asked him to look for evidence to either support or refute his points. That would have been an interesting exercise for Robson, and would have led to a research note that actually had some research in it. Instead, Morgan Stanley seem to have taken his work as gospel. I wonder why. Perhaps it was because they thought that, as a 15 year old, he’s privy to the inner workings of mysterious teen minds, a High Priest in the Digital Native Mythology?

If I relied on Morgan Stanley for anything, I’d be rather concerned right now regarding their lack of critical thinking.

Email a copy of 'The plural of anecdote is not data' to a friend

EMAIL THIS ENTRY TO A FRIEND



Separate multiple entries with a comma. Maximum 5 entries.



Separate multiple entries with a comma. Maximum 5 entries.





E-Mail Image Verification

Loading ... Loading ...

37 Responses to “The plural of anecdote is not data”

  1. Ernesto Says:

    Thank you very much for saying what many of us thought of the MS “report.”

  2. Benedict Says:

    Actually, what I was thinking was, what a brilliant bit of PR. Get an intern to write what everyone knows, write a press release and you’re all over the press for free.

  3. Mark Trudinger Says:

    Absolutely.

    Matthew Robson’s thoughts are actually not that far away from current research, but the idea of basing corporate decisions based on a sample-size of one is scary.

  4. Chris Edwards Says:

    I read it very much as a story about the reaction of financiers to a report written by a 15 year-old - that they’d clutch at anything to find out what the kids think. That’s certainly the sense of the FT piece. A story about a teen writing ‘with authority’ on media habits isn’t much of a story.

    As far as I can tell, this is an agency piece, probably from PA, given that the same quote about “dozens and dozens of fund managers” turns up in all the daily stories I’ve looked at on this, and they all, apart from the FT, lead on Twitter.

  5. alan p Says:

    Suw, as far as I can see (based on actually owning teenage kids and observing their friends too) he is largely correct for the UK and as Mark above notes he is not that far off current research. A lot of the US stuff you allude to is US only, and well, a bit old. The US digital past is definitley another country.

    Also, he has written clearly and concisely, and has had the courage to identify a few naked emperors that everyone suspected were there anyway (Teen Twitter usage for example) which is why I think it has resonated so far.

    Being put out by Morgan Stanley doesn’t hurt either.

    As far as not trusting Morgan Stanley’s analysis, bearing in mind their inability to predict the cataclysms in their own industry I think this piece is a darn site more accurate ;-)

  6. Charlie Beckett, POLIS Director » Blog Archive » Teenagers: the most dangerous beast in the media ecology Says:

    [...] as Suw Charman-Anderson has written, the picture is more complicated than that. I think it is even more complicated than Suw [...]

  7. Gordon Rae Says:

    I think I may have to start marketing a seminar to investment banks on the subject “How to Generalise”.

    You’ve reminded me that my favourite explanation of the sub-prime mortgage crisis is that it was as though the banks had said “The most weight I’ve ever put on is half a stone in a year, so I’m just going to eat cheesecake for breakfast every day”.

  8. Gus Snarp Says:

    I read this blog and thought it made a lot of sense. Then I read the original memo and thought this blog was too kind. Robson’s note is rubbish, it reads like it was written by a 15 year old, showing no particular insight and all the world assumptions one expects of a teenager (I especially like the assumption that most teenagers are unaffected by most advertising - yes, go on believing that, but I bet you are wearing shoes that had a good ad campaign, buying gaming consoles with good ad campaigns, etc. I thought my generation was immune to advertising too, and was clearly wrong.)

  9. Hugh Says:

    I’m not sure what’s more worrying - Morgan Stanley making a big deal out of a teen’s anecdotal observations of his peers’ media usage; or that any of it is a surprise to anyone in the media business.

  10. Nigel Says:

    To be fair to Morgan Stanley they say in the preamble to the note “Without claiming representation or statistical accuracy” at the start of the sentence containing the lovely “so we published it” line that everyone is quoting.

    The Guardian conveniently omit this preamble from their “full copy” of the note, but it can be found from the Evening Standard’s site: http://i.thisislondon.co.uk/i/pix/2009/07/TEENAGERS1.pdf

  11. Andy Woodworth Says:

    I’d treat it like a teen barometer for the purposes of driving research on the subject. Anecdotal, yes, but worthy of further actual investigation. In addition, there has been no predictions made for future trends (whether they would adopt certain social media or not). I’d be interested to hear about that.

  12. Adam Says:

    @alan p - just devil’s advocating for a moment, surely all that proves is that your kids (and thus, their friends) are in the same social niche as the “report author”, and that two anecodes still doesn’t equal data? :-)

  13. PRoales Says:

    Wall Streets response to your charge that “The plural of anecdote is not data” would be - “See the huge pile of money? - yah I made that taking anecdotal reports just like this one and trading on it - its what I do. Oh yah. Where is your pile of money?”

  14. Robert L. Mitchell Says:

    My 16 year old daughter had just told me over the weekend that she doesn’t Tweet and has no interest in Twitter. Furthermore, she said, the majority of her friends (with the exception of perhaps one or two) don’t use Twitter either.

    So I found Robson’s essay, which is a window into his peer group, quite interesting.

    It is what it is. I don’t think anyone is going to base any big decisions on it. But I’m sure it will spur additional research in the area.

  15. alan p Says:

    @adam - oh, sure - but if Robson and my kids are not near the norm on this one I’d be dumbfounded to know where the “norm” is.

    My reaction was more “yep, most of this is about right - why didn’t these banker types know this?”, and that seems to be a strong theme on the comment boards of the various articles, which actually is corroborative data rather than pure anecdote.

    If I was being provocative (as if…) I’d say this gefuffle is more about a 15 yr old kid nailing this better than all the old media analysts and new media experts put together ;-)

  16. Ben Metcalfe Says:

    I think it’s really hard to generalize “what teenagers do”. It’s like asking the question “what music is popular with teenagers”… you’d get a very varied answer from those into mainstream pop through to the extremes where they’ve discovered rave culture or taken up rock guitar.

    Knowing danah and her research and pairing it with my observations of general tech use in US and my native UK, I would say that it’s entirely likely that danah paints a v North American perspective compared to a British one.

    Sure, the Morgan Stanley report is poor because it’s doesn’t stand up to research quality but I think it’s indicative that some greater British-orientated research would be useful, perhaps by the BBC or IPPR.

  17. Eli Says:

    Is there a UK or European version of the Pew Internet & American Life Project? http://www.pewinternet.org/

  18. Ypulse Essentials: Teen Intern Sparks ‘Sensation’ With Media Report, Zombies Are The New Vampires, Surf Industry Weathers The Storm | Ypulse Says:

    [...] ReadWrite Web for a break down of the findings. Also IT Pro considers teens’ Twitter defection. And Strange Attractor questions the methodology [literally just a teen and his friends] Plus Gawker snarkily says what we [...]

  19. Kevin Anderson Says:

    @PRoales Wall Steet’s big pile of money is mine. I’m an American taxpayer. And as for Morgan Stanley, had the US government not ‘facilitated’ their purchase by Bank of America, they would be on the deadpile with Lehman.

    The fact that this is news to the media explains why one of the few sectors which is underperforming investmnent banks is the media.

  20. Suw Charman-Anderson Says:

    Just got back from dinner (In Prague! Lovely evening!) to find all these comments. Some further thoughts:

    @benedict If one’s best attempt at PR is to make one look like an idiot for acting surprised when a 15 year old states the obvious, well, I wouldn’t want that PR department.

    @mark My post wasn’t so much about the validity or not of Matthew’s comments. He’s described - accurately I’m sure - his and his friends’ experience. But that still doesn’t mean that his experience is descriptive of all teens’. And current research, particularly that done by danah boyd and colleagues, paints a much more nuanced picture than we see in this report. As with many things, the devil is in the detail, and taking sweeping generalisations as gospel is something we should all be wary of.

    @chris If a teen has an authority on being a teen, doesn’t that mean that all teens are authorities? The problem is not that MS want to know what teens think, it’s that they assume that one teen is representative.

    @alan Again, this wasn’t a post about Matthew’s observations. Just as a stopped clock is right twice a day, so a teen’s observations will ring true to anyone inhabiting the same demographic segment. Many of the conclusions of US research are as valid here as they are in the US with regard to the fact that teens are not a homogeneous group and that some of them will not have the kind of access to technology that Matthew has. And again, I give him kudos for good writing, but *his* competency is not the issue and never has been. It’s the reaction to his report that’s the problem.

    @gordon ha ha ha ha! I love that!

    @gus I thought the note was well written given that it was written by a 15 year old. I used to have to read essays written by 18 year olds, and the vast majority of them were not as lucid, clear or well structured as Matthew’s piece. (Hell, half of them couldn’t even spell ‘technology’ — ‘teknowledgey’ anyone?) So I’m not going to criticise him on that. But the point is that he is 15, and his outlook comes with all those limitations that all 15 year olds (and clearly a lot of people older than 15) face. You’re dead right on the advertising thing though. There’s a reason that advertisers bank on ‘pester power’.

    @hugh it’s frankly terrifying.

    @nigel Don’t you think that preamble is full of weasel words? It’s like saying “I don’t want to be mean but…” and then promptly saying something unkind. Declaiming what you’re publishing and then publishing it anyway is equivalent to not declaiming it in the first place. It’s not like they don’t know that everyone’s going to ignore their preamble.

    @andy more research is always good! As Charlie says in his post which links to this, anyone who can predict teen behaviour accurately could become very wealthy, very quickly!

    @adam very true. if one anecdote doesn’t make data, neither does two.

    @proales Wall Street lost their big pile of money, or did the US government’s bailouts pass you by? I suspect Wall Street made such a mess of the economy precisely because of the sort of shoddy thinking that Morgan Stanley are exhibiting now.

    @Robert Undoubtedly some teens don’t use Twitter, but some do (see the comments on the Guardian’s piece). Teens, like adults, tend to move in packs and they go where their friends go. That’s why I’m on Twitter rather than Facebook - it’s where my friends are and where they went first. I wouldn’t extrapolate from even three anecdotes as to the state of play of teen adoption of Twitter or Facebook, particularly not over the long term. Indeed, ReadWriteWeb was writing not very long ago that teens and older students were fleeing Facebook… Which I think goes to prove that it’s all just a bit more complicated.

    @alan What is the norm? Is it what middle class kids do? Where do rural kids sit? Or inner city kids? The thing is, you can use all the common sense in the world to express your idea of ‘the norm’ but until you have hard data to back it up, it doesn’t actually tell us anything. This kerfuffle, in my opinion, is not about a 15 year old’s opinion. His description of his experience is as valid as anyone else’s. It’s about the reaction to his report. And no, he hasn’t nailed it better than “all the … experts put together”, because he hasn’t nailed it better than danah boyd, who’s done actual real statistically valid research into the area.

    @ben British oriented research would be wonderful, yes. I’d very much like to see that happen, not just to examine the use of the internet and technology by teens, but by everyone else too. Ofcom does some, and the Office for National Statistics does some, but no one really seems to address the questions that really need answering.

    @Eli No, there’s no version of Pew here, although I do suggest to the government that we should have one in my report (out soon but you can read most of it here already) on the use of social media by civil society. (See the Carnegie category on the right.)

  21. You Are Creative Says:

    What I don’t understand is this:

    Robson writes

    “Outdoor advertising usually does not trigger a reaction in teenagers, but sometimes they will oppose it (the Benetton baby adverts)”

    The Benetton baby advert appeared in 1991. That would mean that when the ad came out he would have to wait three years until he was born.

  22. Daily Digest for July 13th | DJ.Lazarides Says:

    [...] Shared The plural of anecdote is not data [...]

  23. The Daily Square – Robot Chant Edition | Booksquare Says:

    [...] The plural of anecdote is not dataIn response to the mediastorm surrounding a 15-year old Morgan Stanley intern’s analysis of how teens use media. [...]

  24. Interessantes woanders (2009.07.14) › Immersion I/O Says:

    [...] Strange Attractor » Blog Archive » The plural of anecdote is not data [...]

  25. Morgan Stanley presents teen as consumer media messiah » Stuart Tiffen Says:

    [...] Danah Boyd wrote “Taken Out of Context: American Teen Sociality in Networked Publics” This dissertation documents my 2.5-year ethnographic study of American teens’ engagement with social network sites and the ways in which their participation supported and complicated three practices—self-presentation, peer sociality, and negotiating adult society. via Corante [...]

  26. Lots to understand about teen media use Says:

    [...] Suw Charman-Anderson points out, the most interesting thing is how little seems to be understood about this age group. Every [...]

  27. TechnoLlama » Smells like tweet spirit Says:

    [...] by two other teenagers. In the middle of all the media storm, my favourite opinion has been Suw Charman’s reply, who rightly points out that one anecdotal report does not constitute hard data. As a commenter [...]

  28. How teenagers consume media, apparently - Roo Reynolds Says:

    [...] I think it’s time to get some perspective about a piece of writing that is purely anecdotal. Suw is particularly eloquent on this: “The City, and sections of the media, are getting a touch over-excited by a “research [...]

  29. All Points North » How 31 year olds consume media Says:

    [...] sensible reaction here Share and [...]

  30. How teenagers really consume media / we are social Says:

    [...] as Suw Charman-Anderson notes, the main problem with Robson’s report is that he thinks his experiences are emblematic of [...]

  31. Yishay Mor Says:

    The only data I see here is suggestive that Morgan Stanley execs are a bunch of absent parents.
    You’re right to quote danah boyd, as an example of good science on youth and social media.
    There’s nothing wrong about the Robson report, except that for anyone who has bothered to speak to a teenager recently for more than 5 minutes, its plain common sense.
    Like I said, imagine Morgan Stanley’s shock when they hear that teens have sex.

  32. Too many anecdotes, not enough data - Bokardo Says:

    [...] Charman-Anderson wrote this in reference to a story published last week about a 15-year-old intern at Morgan Stanley who wrote a report on [...]

  33. Weekly links for 29th July | USiT Says:

    [...] The kid’s view: Giving up my iPod for a Walkman & Twitter is for old people, work experience whiz-kid tells bankers - & a cautionary note that the “The plural of anecdote is not data” [...]

  34. KC Says:

    I agree with everything in this article, except the title. Anecdotes are a valuable form of data. They may not be definitive or statistical, but they are valid data. This is supported by the dictionary definition of “data” as well as best practice in quality management, marketing and customer experience management. Like all data, issues of sample size, accuracy, etc come into play, as well as how you use the data (as others have pointed out).

  35. Suw Charman-Anderson Says:

    KC, the title, “The plural of anecdote is not data” is a quote - one that’s been attributed to a number of people but probably coined by economist George Stigler. (See http://bearcastle.com/blog/?p=408)

    The point is that a well-researched anecdote is a case study, but even a well researched case study does not give us information about an entire population.

    The dictionary definition of “anecdote” is:

    a short and amusing or interesting story about a real incident or person
    • an account regarded as unreliable or hearsay
    • the depiction of a minor narrative incident in a painting.

    Only the first part of the definition could possibly be considered to be “data”, and even then I think one could fairly say that the anecdote might, but does not necessarily, contain data.

    Data, on the other hand is:

    facts and statistics collected together for reference or analysis.
    • Computing: the quantities, characters, or symbols on which operations are performed by a computer, being stored and transmitted in the form of electrical signals and recorded on magnetic, optical, or mechanical recording media.
    • Philosophy: things known or assumed as facts, making the basis of reasoning or calculation.

    “Anecdote” is not, therefore, a synonym for “data”, although one *may* include data. Collecting anecdote does not guarantee one is collecting data.

    In this particular instance, people were taking anecdote as representative data regarding an entire generation. I’m sure you can see the fallacy there.

  36. Mneiae Says:

    I laughed quite a bit when I read Robson’s report. I’m a Gen Yer myself and one thing that nobody seemed to comment on was that he’s speaking for Gen Y in the UK. Costs for gaming systems, cell phone usage, TV habits, etc. are different over there. One concrete example: when I got to the part about not using Twitter, I immediately posted the article via Twitter. Unlimited text messaging is more prevalent over here, which is why services like ChaCha and Twitter are more widely used than somewhere like the UK where you pay per message. I’ve never used some of the services he calls popular and I think that taking this report seriously in the US, beyond recognizing that Gen Y is more dependent on technology, is ludicrous.

  37. Anecdote Says:

    Just came across this little ditty. I write anecdotal articles for my weekly blog and I wouldn’t dream of calling my observations “data” or “research”. I’m not even certain how helpful this report will be. It seems to be a decent indicator of how teens view media based on research already done - not much else.