In-House Surveys: Perils & Pitfalls


I often get questions about survey instruments from prior students of professionals I’m affiliated with in one way or another.  I’m generally correct in assuming they intend to implement their own survey instrument in-house, and often without very much support from others in the organization.  Often, but not always, I think managers develop surveys to validate our own way of thinking.  The danger in this is that it can result in a type of self-fulfilling prophecy as our choice of questions and the wording of those questions can very easily “lead”.  I’ve had two of these notes in the past two weeks, so I thought I’d share my general advice. Here is one such note:
I have a question…

I am working on a project for work where we are doing a questionnaire for performance feedback. I know from doing some reasearch projects years ago that there are some fundamentals to questionnaire design to ensure that you have efficiently worded questions and structure. I also know that for most doctoral programs this is a big deal in the primary research area. Can you recommend a good book/guide on questionnaire design that could help guide us in this effort?


Hey Kevin,

Well, the issue of “scale development” is a complicated one. If you wanted to measure job satisfaction, there are many highly validated measures of that which are preferable over making your own. Reliability and validity are the issues we face when developing questionnaires. However, when you want to measure perceptions that are not captured by existing scales that have been validated in research, you end up making your own using as much common sense as you can. In this case, your best choice is to find someone who had done this in the more academic fashion (for a research paper that is scrutinized by reviewers who understand psychometrics). If that is not an expense your firm is willing to incur, you can (sometimes) “entice” an academic to work for free in exchange for giving them access to interesting data that they can use for a publication. Performance feedback is a common survey type, but the hardest question is “what do you really want to know”.  This is where an outside facilitator can really be of objective help to you. If you’re stuck with doing this yourself, common sense prevails. Simply draft up a set of questions that you think you want to ask then:

1) Have others review your questions and discuss with them “what they think the question is really asking”. This is where you find that your perfect wording is sometimes misleading.
2) Alter your wording so that you are asking your questions more plainly. Keeping is simple is the key.
3) Discuss how generalizable each question is to each respondent. Someone from one area of the firm may interpret the question quite differently from someone in another functional area.
4) Do a small pilot study launching your survey to a select random group of employees and review the results.
5) Debrief these individuals and have an open discussion about how they interpreted the questions.
6) Modify your questions as needed.
7) Add LOTS of qualifying questions to include year of tenure with the organization, year of tenure in this functional area, age, gender, ethnicity, managerial experience etc etc..
8) Launch your final survey (with fingers crossed)
9) Analyze your data using more than mean scores on items. You should ask important questions in several ways and average responses across similar questions. For instance, if it’s very important that you have a good measure of employee satisfaction with supervisors, you’ll want to know which supervisor they are referring to (not necessarily the name of that person, but the level of manager they are to that person ; direct supervisor, senior coworker etc.. Don’t assume anything. Satisfaction to one employee means something different to another employee. Hence, you need to identify the elements of management behavior/attitude that you want to ask about.
10) Finally, look through your data using statistics (t-test or ANOVA will work fine) and look for “more of the story”. You may not be looking for it, but may find that female subordinates reported statistically significantly less satisfaction with their female superiors than with their mail superiors. This type of findings raises more questions, and is what often leads to a follow up survey in order to understand it more fully.

Well, that is a mouth full. Surveys are easy to write, but very challenging to create, deploy and analyze in a valid and reliable way. If you company can afford some help, it is well worth the minor expense.

There are books out there that cover these issues, but they will essentially elaborate on what I have shared above. The statistics used in the analysis are not difficult to calculate, but interpretation is another matter. A statistically significant difference between two groups does not mean that it is necessarily material in your workplace. Conversely, a non-statistically significant result can be practically very important for your organization.

In the end, a survey of this sort will require the participation of a group of key people who share an interest in finding out the TRUTH. This is another reason that ousiders are used, they lack the biases that organization members tend to have based on their past experience.

Call to chat if I can be of more help.



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s