Surveys: do they tell you anything?

Recently on a newsgroup a magazine editor asked about some suggested questions for a survey on responses to the publication to be put to those attending a forthcoming meeting.

It was an opportunity for me to explain why I hate surveys, especially those presented at meetings: you either fill them out at random or go for the middle answer after you realize that you've been talking to colleagues and there isn't time to read the questions let alone answer.

It's a question which comes up in the book which we've just sent to the printers, a new edition of Type & Layout, which has been out of print for several years. The author, Colin Wheildon comments "over the years of my research, David Ogilvy more than once raised the question whether I was measuring reading comprehension or merely readability. I'm grateful for his persistence in moving me to confront this important and difficult question."

In this research he was measuring the extent to which typography and design affected understanding. His answer was that to overcome statements which might just reflect that a person had perused the articles and thought they understood them, was that there had to be actual questions on content to see whether the subjects had not only read the content, or thought that they had done so, but that they could answer questions which showed they had done so.

I was, in a way, pleased that it presented such major problems for the academics and practical researchers who advised Colin on this research, because it supported something I wrote in my own book on magazine publishing:

"'Let's put a reader survey in the next issue, to find out what the reader wants!' To my mind this is like governments forming a committee: it gives the appearance of doing something without actually having to take any possibly unpopular decisions. Surveys have to be exceptionally well designed to get useful answers and can be designed to produce mostly the answers those setting the questions want to get."

I also quoted Dick Smith, the millionaire businessman and adventurer who started Australian Geographic magazine as suggesting the best way to find out what needs doing is to talk to staff and customers. He was surprised how many companies, especially large ones, which forgot to do this.

So, I suggested better results would be gained by walking among the attendees and raising the subject of the publication, and have a few other people do the same.

Did I say that I dislike surveys?