Bad Survey Design. Please Stop! -- All This ChittahChattah

At work, I’m developing a new survey to ask our new hires how the on-boarding process went. Because of all the terribly constructed surveys I’ve taken in the past, I decided to take the process of building it seriously. I searched on the web for any guides to building a good survey. One that I found concise and useful was Steve Portigal’s “Bad Survey Design. Please Stop!” on his blog “All This ChittahChattah”. I’m not sure about the blog’s title (but then again, I’m not sure about my blog’s title, either), but the article was very useful. I found many of his points informative and helpful.

He clearly states his case up front:

Survey design is a craft. If you haven’t studied it, you don’t know how to write a survey well, and the data you get is garbage. Surveys are quantitative tools. They require math to plan what does your sample size need to be to ensure that your results are valid? and to analyze regression analysis or any other buzzword anyone?. They are very tough to write. Questions have to be worded correctly and sequenced correctly.

I appreciate his enthusiasm, though I’m not sure I agree with the full argument. Along with Ben Druce (who commented some agreements and some disagreements on the blog), I don’t think that someone has to have studied design in order to benefit from others, like Steve, who have and who can make clear and informed recommendations to those of us who haven’t. Do I really need to take a long course to make a simple, 20-question survey sent to 15 people a year? No. Maybe I’m getting caught up in his use of the word “studied” – maybe he only means something less rigorous than a full academic semester/year/courseload in survey design. Maybe, but I doubt it.

His most useful arguments use a sample survey as a starting point – a survey that is poorly designed and worded. Going through the issues with the survey helps me understand his points more clearly. Bias is a huge problem and one that I’m striving to avoid. While that may not be possible in a pure sense, I’m confident that, with some serious effort, a fairly non-biased survey is possible.

In the end, Steve’s advice is sound and reasoned. It will help me build a better survey, whether or not I’ve formally studied the craft (I do agree that it is a craft – while I may never be the survey-world’s equivalent of a Thomas Moser, I might be able to whittle out some small objects of value.)