Monday, July 13th, 2009
The plural of anecdote is not data
The City, and sections of the media, are getting a touch over-excited by a “research note” written for Morgan Stanley by Matthew Robson, a 15 year old on work experience. The Guardian said:
The US investment bank’s European media analysts asked Matthew Robson, an intern from a London school, to write a report on teenagers’ likes and dislikes, which made the Financial Times’ front page today.
His report, that dismissed Twitter and described online advertising as pointless, proved to be “one of the clearest and most thought-provoking insights we have seen – so we published it”, said Edward Hill-Wood, executive director of Morgan Stanley’s European media team.
“We’ve had dozens and dozens of fund managers, and several CEOs, e-mailing and calling all day.” He said the note had generated five or six times more responses than the team’s usual research.
The research note itself can be read on The Guardian’s site.
I’m going to start by giving Robson the kudos that he deserves. He has written a very well thought out piece which describes the media habits of him and his friends. In no way do I want to criticise a teenager for being thoughtful, engaged and articulate.
But one has to put this research note into context: This is one teen describing his experience. It is not a reliable description of all teens’ attitudes and behaviours, yet both Morgan Stanley and the media seem to be treating it as if Robson has Spoken The One Great Truth. “Twitter is not for teens, Morgan Stanley told by 15-year-old expert” coos The Guardian. “Note by ‘teenage scribbler’ causes sensation” says the FT in astonishment.
Neither Morgan Stanley nor the media seem to be able to tell the difference between anecdote and data. This “research note” is more note than research, and it should not be taken to be representative of all teens. A teenager in a rural setting, or in an inner city estate, or one who feels socially excluded from web culture will have a very different experience than a teen who’s well-connected enough to get himself an internship at Morgan Stanley.
What is worrying about this is not Robson’s note: He’s simply doing what most teens (and most adults) do, which is to extrapolate from his own and his friends’ experience to form generalisations about the world around him. It’s a very human thing to do, but the important thing about businesses like Morgan Stanley, and the journalists who write about them, is that they are supposed to be able to tell the difference between data and generalisations. Yet they don’t seem able to sort the wheat from the chaff. It seems yet another symptom of the group-think in the media and financial sector that led to the Great Recession, rather than an indication that we have learned anything from it.
Sarah Perez on ReadWriteWeb says:
Matthew Robson, a 15-year-old intern at analyst firm Morgan Stanley recently helped compile a report about teenage media habits. Overnight, his findings have become a sensation…which goes to show that people are either obsessed with what “the kids” are into or there’s a distinctive lack of research being done on this demographics’ media use. Robson’s report isn’t even based on any sort of statistical analysis, just good ol’ fashioned teenage honesty. And what was it that he said to cause all this attention? Only that teens aren’t into traditional media (think TV, radio, newspapers) and yet they’re eschewing some new media, too, including sites like Twitter.
Well, research has been done. danah boyd has done some excellent research into the use of the web by teens - it’s her speciality and she’s one of the foremost experts in this area. Her research would help Morgan Stanley understand the teen demographic much more clearly than any single anecdote, however well written, ever can. The fact that they haven’t ever had a clear insight into the teen demographic would seem to imply that their existing researchers and analysts aren’t doing their jobs properly. The information is out there, a lot of it is freely available, and all that remains is for someone to read it and write the report.
This story also feeds into the concept of the ‘digital native’ which, as I’ve blogged before, is a very poor way to talk about a very diverse section of the population. But because this report fits in with widely-held assumptions about teens and technology - not only does it describe ‘digital natives’, it’s written by one too - it’s immediately accepted without query or question. Morgan Stanley and the media both seem to be more interested in having their biases validated than they are in exploring the evidence to see where it leads them. Sadly, it seems that neither have been spending enough time watching CSI and drawing from it key lessons about assumptions, evidence and how to draw conclusions.
If I had been Matthew Robson’s boss at Morgan Stanley, on receiving his report I would have praised him on his good work and then asked him to look for evidence to either support or refute his points. That would have been an interesting exercise for Robson, and would have led to a research note that actually had some research in it. Instead, Morgan Stanley seem to have taken his work as gospel. I wonder why. Perhaps it was because they thought that, as a 15 year old, he’s privy to the inner workings of mysterious teen minds, a High Priest in the Digital Native Mythology?
If I relied on Morgan Stanley for anything, I’d be rather concerned right now regarding their lack of critical thinking.
Loading ...
July 13th, 2009 at 2:33 pm
Thank you very much for saying what many of us thought of the MS “report.”
July 13th, 2009 at 3:12 pm
Actually, what I was thinking was, what a brilliant bit of PR. Get an intern to write what everyone knows, write a press release and you’re all over the press for free.
July 13th, 2009 at 3:18 pm
Absolutely.
Matthew Robson’s thoughts are actually not that far away from current research, but the idea of basing corporate decisions based on a sample-size of one is scary.
July 13th, 2009 at 3:26 pm
I read it very much as a story about the reaction of financiers to a report written by a 15 year-old - that they’d clutch at anything to find out what the kids think. That’s certainly the sense of the FT piece. A story about a teen writing ‘with authority’ on media habits isn’t much of a story.
As far as I can tell, this is an agency piece, probably from PA, given that the same quote about “dozens and dozens of fund managers” turns up in all the daily stories I’ve looked at on this, and they all, apart from the FT, lead on Twitter.
July 13th, 2009 at 3:37 pm
Suw, as far as I can see (based on actually owning teenage kids and observing their friends too) he is largely correct for the UK and as Mark above notes he is not that far off current research. A lot of the US stuff you allude to is US only, and well, a bit old. The US digital past is definitley another country.
Also, he has written clearly and concisely, and has had the courage to identify a few naked emperors that everyone suspected were there anyway (Teen Twitter usage for example) which is why I think it has resonated so far.
Being put out by Morgan Stanley doesn’t hurt either.
As far as not trusting Morgan Stanley’s analysis, bearing in mind their inability to predict the cataclysms in their own industry I think this piece is a darn site more accurate
July 13th, 2009 at 3:45 pm
[...] as Suw Charman-Anderson has written, the picture is more complicated than that. I think it is even more complicated than Suw [...]
July 13th, 2009 at 3:49 pm
I think I may have to start marketing a seminar to investment banks on the subject “How to Generalise”.
You’ve reminded me that my favourite explanation of the sub-prime mortgage crisis is that it was as though the banks had said “The most weight I’ve ever put on is half a stone in a year, so I’m just going to eat cheesecake for breakfast every day”.
July 13th, 2009 at 4:25 pm
I read this blog and thought it made a lot of sense. Then I read the original memo and thought this blog was too kind. Robson’s note is rubbish, it reads like it was written by a 15 year old, showing no particular insight and all the world assumptions one expects of a teenager (I especially like the assumption that most teenagers are unaffected by most advertising - yes, go on believing that, but I bet you are wearing shoes that had a good ad campaign, buying gaming consoles with good ad campaigns, etc. I thought my generation was immune to advertising too, and was clearly wrong.)
July 13th, 2009 at 4:29 pm
I’m not sure what’s more worrying - Morgan Stanley making a big deal out of a teen’s anecdotal observations of his peers’ media usage; or that any of it is a surprise to anyone in the media business.
July 13th, 2009 at 4:30 pm
To be fair to Morgan Stanley they say in the preamble to the note “Without claiming representation or statistical accuracy” at the start of the sentence containing the lovely “so we published it” line that everyone is quoting.
The Guardian conveniently omit this preamble from their “full copy” of the note, but it can be found from the Evening Standard’s site: http://i.thisislondon.co.uk/i/pix/2009/07/TEENAGERS1.pdf
July 13th, 2009 at 4:34 pm
I’d treat it like a teen barometer for the purposes of driving research on the subject. Anecdotal, yes, but worthy of further actual investigation. In addition, there has been no predictions made for future trends (whether they would adopt certain social media or not). I’d be interested to hear about that.
July 13th, 2009 at 4:36 pm
@alan p - just devil’s advocating for a moment, surely all that proves is that your kids (and thus, their friends) are in the same social niche as the “report author”, and that two anecodes still doesn’t equal data?
July 13th, 2009 at 4:37 pm
Wall Streets response to your charge that “The plural of anecdote is not data” would be - “See the huge pile of money? - yah I made that taking anecdotal reports just like this one and trading on it - its what I do. Oh yah. Where is your pile of money?”
July 13th, 2009 at 5:09 pm
My 16 year old daughter had just told me over the weekend that she doesn’t Tweet and has no interest in Twitter. Furthermore, she said, the majority of her friends (with the exception of perhaps one or two) don’t use Twitter either.
So I found Robson’s essay, which is a window into his peer group, quite interesting.
It is what it is. I don’t think anyone is going to base any big decisions on it. But I’m sure it will spur additional research in the area.
July 13th, 2009 at 5:11 pm
@adam - oh, sure - but if Robson and my kids are not near the norm on this one I’d be dumbfounded to know where the “norm” is.
My reaction was more “yep, most of this is about right - why didn’t these banker types know this?”, and that seems to be a strong theme on the comment boards of the various articles, which actually is corroborative data rather than pure anecdote.
If I was being provocative (as if…) I’d say this gefuffle is more about a 15 yr old kid nailing this better than all the old media analysts and new media experts put together
July 13th, 2009 at 6:08 pm
I think it’s really hard to generalize “what teenagers do”. It’s like asking the question “what music is popular with teenagers”… you’d get a very varied answer from those into mainstream pop through to the extremes where they’ve discovered rave culture or taken up rock guitar.
Knowing danah and her research and pairing it with my observations of general tech use in US and my native UK, I would say that it’s entirely likely that danah paints a v North American perspective compared to a British one.
Sure, the Morgan Stanley report is poor because it’s doesn’t stand up to research quality but I think it’s indicative that some greater British-orientated research would be useful, perhaps by the BBC or IPPR.
July 13th, 2009 at 6:39 pm
Is there a UK or European version of the Pew Internet & American Life Project? http://www.pewinternet.org/
July 13th, 2009 at 7:21 pm
[...] ReadWrite Web for a break down of the findings. Also IT Pro considers teens’ Twitter defection. And Strange Attractor questions the methodology [literally just a teen and his friends] Plus Gawker snarkily says what we [...]
July 13th, 2009 at 8:55 pm
@PRoales Wall Steet’s big pile of money is mine. I’m an American taxpayer. And as for Morgan Stanley, had the US government not ‘facilitated’ their purchase by Bank of America, they would be on the deadpile with Lehman.
The fact that this is news to the media explains why one of the few sectors which is underperforming investmnent banks is the media.
July 13th, 2009 at 9:23 pm
Just got back from dinner (In Prague! Lovely evening!) to find all these comments. Some further thoughts:
@benedict If one’s best attempt at PR is to make one look like an idiot for acting surprised when a 15 year old states the obvious, well, I wouldn’t want that PR department.
@mark My post wasn’t so much about the validity or not of Matthew’s comments. He’s described - accurately I’m sure - his and his friends’ experience. But that still doesn’t mean that his experience is descriptive of all teens’. And current research, particularly that done by danah boyd and colleagues, paints a much more nuanced picture than we see in this report. As with many things, the devil is in the detail, and taking sweeping generalisations as gospel is something we should all be wary of.
@chris If a teen has an authority on being a teen, doesn’t that mean that all teens are authorities? The problem is not that MS want to know what teens think, it’s that they assume that one teen is representative.
@alan Again, this wasn’t a post about Matthew’s observations. Just as a stopped clock is right twice a day, so a teen’s observations will ring true to anyone inhabiting the same demographic segment. Many of the conclusions of US research are as valid here as they are in the US with regard to the fact that teens are not a homogeneous group and that some of them will not have the kind of access to technology that Matthew has. And again, I give him kudos for good writing, but *his* competency is not the issue and never has been. It’s the reaction to his report that’s the problem.
@gordon ha ha ha ha! I love that!
@gus I thought the note was well written given that it was written by a 15 year old. I used to have to read essays written by 18 year olds, and the vast majority of them were not as lucid, clear or well structured as Matthew’s piece. (Hell, half of them couldn’t even spell ‘technology’ — ‘teknowledgey’ anyone?) So I’m not going to criticise him on that. But the point is that he is 15, and his outlook comes with all those limitations that all 15 year olds (and clearly a lot of people older than 15) face. You’re dead right on the advertising thing though. There’s a reason that advertisers bank on ‘pester power’.
@hugh it’s frankly terrifying.
@nigel Don’t you think that preamble is full of weasel words? It’s like saying “I don’t want to be mean but…” and then promptly saying something unkind. Declaiming what you’re publishing and then publishing it anyway is equivalent to not declaiming it in the first place. It’s not like they don’t know that everyone’s going to ignore their preamble.
@andy more research is always good! As Charlie says in his post which links to this, anyone who can predict teen behaviour accurately could become very wealthy, very quickly!
@adam very true. if one anecdote doesn’t make data, neither does two.
@proales Wall Street lost their big pile of money, or did the US government’s bailouts pass you by? I suspect Wall Street made such a mess of the economy precisely because of the sort of shoddy thinking that Morgan Stanley are exhibiting now.
@Robert Undoubtedly some teens don’t use Twitter, but some do (see the comments on the Guardian’s piece). Teens, like adults, tend to move in packs and they go where their friends go. That’s why I’m on Twitter rather than Facebook - it’s where my friends are and where they went first. I wouldn’t extrapolate from even three anecdotes as to the state of play of teen adoption of Twitter or Facebook, particularly not over the long term. Indeed, ReadWriteWeb was writing not very long ago that teens and older students were fleeing Facebook… Which I think goes to prove that it’s all just a bit more complicated.
@alan What is the norm? Is it what middle class kids do? Where do rural kids sit? Or inner city kids? The thing is, you can use all the common sense in the world to express your idea of ‘the norm’ but until you have hard data to back it up, it doesn’t actually tell us anything. This kerfuffle, in my opinion, is not about a 15 year old’s opinion. His description of his experience is as valid as anyone else’s. It’s about the reaction to his report. And no, he hasn’t nailed it better than “all the … experts put together”, because he hasn’t nailed it better than danah boyd, who’s done actual real statistically valid research into the area.
@ben British oriented research would be wonderful, yes. I’d very much like to see that happen, not just to examine the use of the internet and technology by teens, but by everyone else too. Ofcom does some, and the Office for National Statistics does some, but no one really seems to address the questions that really need answering.
@Eli No, there’s no version of Pew here, although I do suggest to the government that we should have one in my report (out soon but you can read most of it here already) on the use of social media by civil society. (See the Carnegie category on the right.)
July 13th, 2009 at 9:41 pm
What I don’t understand is this:
Robson writes
“Outdoor advertising usually does not trigger a reaction in teenagers, but sometimes they will oppose it (the Benetton baby adverts)”
The Benetton baby advert appeared in 1991. That would mean that when the ad came out he would have to wait three years until he was born.
July 13th, 2009 at 10:08 pm
[...] Shared The plural of anecdote is not data [...]
July 13th, 2009 at 11:31 pm
[...] The plural of anecdote is not dataIn response to the mediastorm surrounding a 15-year old Morgan Stanley intern’s analysis of how teens use media. [...]
July 14th, 2009 at 7:49 am
[...] Strange Attractor » Blog Archive » The plural of anecdote is not data [...]
July 14th, 2009 at 9:29 am
[...] Danah Boyd wrote “Taken Out of Context: American Teen Sociality in Networked Publics” This dissertation documents my 2.5-year ethnographic study of American teens’ engagement with social network sites and the ways in which their participation supported and complicated three practices—self-presentation, peer sociality, and negotiating adult society. via Corante [...]
July 14th, 2009 at 7:54 pm
[...] Suw Charman-Anderson points out, the most interesting thing is how little seems to be understood about this age group. Every [...]
July 15th, 2009 at 2:15 am
[...] by two other teenagers. In the middle of all the media storm, my favourite opinion has been Suw Charman’s reply, who rightly points out that one anecdotal report does not constitute hard data. As a commenter [...]
July 15th, 2009 at 8:59 am
[...] I think it’s time to get some perspective about a piece of writing that is purely anecdotal. Suw is particularly eloquent on this: “The City, and sections of the media, are getting a touch over-excited by a “research [...]
July 15th, 2009 at 1:52 pm
[...] sensible reaction here Share and [...]
July 15th, 2009 at 2:47 pm
[...] as Suw Charman-Anderson notes, the main problem with Robson’s report is that he thinks his experiences are emblematic of [...]
July 17th, 2009 at 8:33 am
The only data I see here is suggestive that Morgan Stanley execs are a bunch of absent parents.
You’re right to quote danah boyd, as an example of good science on youth and social media.
There’s nothing wrong about the Robson report, except that for anyone who has bothered to speak to a teenager recently for more than 5 minutes, its plain common sense.
Like I said, imagine Morgan Stanley’s shock when they hear that teens have sex.
July 20th, 2009 at 12:32 pm
[...] Charman-Anderson wrote this in reference to a story published last week about a 15-year-old intern at Morgan Stanley who wrote a report on [...]
July 29th, 2009 at 6:51 am
[...] The kid’s view: Giving up my iPod for a Walkman & Twitter is for old people, work experience whiz-kid tells bankers - & a cautionary note that the “The plural of anecdote is not data” [...]
October 2nd, 2009 at 6:13 am
I agree with everything in this article, except the title. Anecdotes are a valuable form of data. They may not be definitive or statistical, but they are valid data. This is supported by the dictionary definition of “data” as well as best practice in quality management, marketing and customer experience management. Like all data, issues of sample size, accuracy, etc come into play, as well as how you use the data (as others have pointed out).
October 5th, 2009 at 11:38 am
KC, the title, “The plural of anecdote is not data” is a quote - one that’s been attributed to a number of people but probably coined by economist George Stigler. (See http://bearcastle.com/blog/?p=408)
The point is that a well-researched anecdote is a case study, but even a well researched case study does not give us information about an entire population.
The dictionary definition of “anecdote” is:
a short and amusing or interesting story about a real incident or person
• an account regarded as unreliable or hearsay
• the depiction of a minor narrative incident in a painting.
Only the first part of the definition could possibly be considered to be “data”, and even then I think one could fairly say that the anecdote might, but does not necessarily, contain data.
Data, on the other hand is:
facts and statistics collected together for reference or analysis.
• Computing: the quantities, characters, or symbols on which operations are performed by a computer, being stored and transmitted in the form of electrical signals and recorded on magnetic, optical, or mechanical recording media.
• Philosophy: things known or assumed as facts, making the basis of reasoning or calculation.
“Anecdote” is not, therefore, a synonym for “data”, although one *may* include data. Collecting anecdote does not guarantee one is collecting data.
In this particular instance, people were taking anecdote as representative data regarding an entire generation. I’m sure you can see the fallacy there.
October 7th, 2009 at 3:00 am
I laughed quite a bit when I read Robson’s report. I’m a Gen Yer myself and one thing that nobody seemed to comment on was that he’s speaking for Gen Y in the UK. Costs for gaming systems, cell phone usage, TV habits, etc. are different over there. One concrete example: when I got to the part about not using Twitter, I immediately posted the article via Twitter. Unlimited text messaging is more prevalent over here, which is why services like ChaCha and Twitter are more widely used than somewhere like the UK where you pay per message. I’ve never used some of the services he calls popular and I think that taking this report seriously in the US, beyond recognizing that Gen Y is more dependent on technology, is ludicrous.
October 13th, 2009 at 11:19 pm
Just came across this little ditty. I write anecdotal articles for my weekly blog and I wouldn’t dream of calling my observations “data” or “research”. I’m not even certain how helpful this report will be. It seems to be a decent indicator of how teens view media based on research already done - not much else.