Designing a research questionnaire

Where should questionnaire design start?

Many scholars are sitting down and typing questions on the first day itself. Better to begin from your objectives and your conceptual model. Each item in the questionnaire needs a reason. Either the item measures a construct you need, or the item is a background variable you will use in the analysis.

1. 1

   Map constructs to objectives

   Write down each construct in your conceptual model, and next to it the objective or hypothesis it is for.

2. 2

   Find validated scales

   Search for a published scale for every construct, with reliability reported. See what licence it comes under.

3. 3

   Adapt and translate

   Change the wording to suit your setting. Translate the questionnaire with forward and back-translation, and keep a record.

4. 4

   Expert review

   Ask a few experts to judge if each item is relevant and clear. Do the corrections in the questionnaire and note down what you changed.

5. 5

   Pilot

   Try the questionnaire on respondents similar to the main sample. See the time taken, their understanding and the reliability.

6. 6

   Finalise

   Freeze the final version of the questionnaire, take ethics approval for that version, and keep it unchanged till data collection is over.

Before writing even one item, make a small table with five columns, namely construct, definition, source scale, number of items, and objective or hypothesis. Later this table becomes the instrument section of your methodology chapter, and it stops the questionnaire from growing. Many times scholars put in some “interesting” questions which nobody analyses afterwards. Every such question eats the respondent’s attention, and answers on the important items get worse.

Watch the length also. When the questionnaire goes beyond about 15–20 minutes, more people drop out and more people answer carelessly. It is worse online, and worse with busy people like nurses, bank staff, or farmers in harvest season. Note the time in the pilot, and if people are taking longer than that, some items have to go.

How do you adapt a validated scale?

A validated scale is a scale developed and tested in published research, with the reliability and validity reported. When you use a validated scale, your constructs are measured the way the literature measures them, and your results can be compared with other studies.

Finding scales

See the method sections of the most-cited papers for each construct, and recent papers in good journals of your field. Usually the appendix or the measures section give the full list of items. Choose a validated scale used in many studies, ideally one study in India or a similar setting.

Check the licence

Many scales are free for research use, but some are commercial. For example, the Multifactor Leadership Questionnaire and the Maslach Burnout Inventory are sold through Mind Garden. Using a licensed scale without permission is a copyright problem, and it can come out when the thesis goes on Shodhganga. Check the licence of the scale first, then decide.

What you can change

• Context words can normally be changed, for example “my company” to “my school”.
• Dropping items is allowed, but give the justification, and check the reliability of the shorter scale again.
• Changing the response scale is quite common, say from seven points to five. Then comparison with other studies becomes harder, so mention the change and the reason.
• Rewriting items heavily means you are practically using a new scale, and you need stronger validity evidence for it.

Keep a table in the appendix for every change, with the original item, the adapted item and the reason. Examiners like this table, and it clears their doubt before they raise it.

How do you write good items and choose a Likert scale?

Sometimes no suitable scale exists for a construct, and you have to write the items yourself. Each item you write should be short and ask one thing only, in the words your respondents speak daily.

Five points or seven?

Both are used widely, and you can defend either one in viva-voce. Five points are easier when respondents have less formal education, or when they answer on a phone. Seven points give a little more spread. The strongest reason for either is to match the original scale. Whatever you choose, label every point (strongly disagree, disagree, neither agree nor disagree, agree, strongly agree), not only the two ends. In a translated questionnaire the labels matter the most.

Should there be a neutral midpoint?

Mostly yes. Remove it and a person with no real opinion is pushed to choose a side, which adds noise. Where some respondents may not know the topic at all, a separate “don’t know” or “not applicable” option stops them hiding in the midpoint.

Reverse-worded items

Many older scales mix positive and negative worded items, so that careless answering gets caught. These reverse-worded items are giving trouble many times, because respondents misread them, especially after translation, and in the analysis they can come out as a separate factor. If the original scale has them, keep them, but check them in the pilot and recode them before the analysis. The statistical test guide explains how Likert data is analysed after that.

Demographic questions

Ask only what you will actually use in the analysis. Put these questions at the end so the main scales are answered first without disturbance. Many people feel shy to write their exact income or age, and for that ranges are fine. Use categories that suit India, like the languages people actually speak, and rural, semi-urban and urban for location. Social category should be asked only when the study needs it.

How do you translate a questionnaire into an Indian language?

A lot of PhD surveys in India are done in Tamil, Hindi, Telugu, Kannada, Malayalam, Marathi, Bengali or some other regional language. The scales, though, were written in English. A careless translation is one of the main reasons a well-known scale gives a low alpha.

The usual process comes from Brislin’s back-translation method and later guidelines, like Beaton and colleagues (2000). There are five steps in it.

1. Forward translation needs two persons who know both languages well, translating separately. At least one should know the subject and one should not, so everyday language stays in the translation.
2. Reconciliation is making one agreed version out of both translations.
3. Back-translation into English is done by a person who has not seen the original.
4. Expert comparison of the original and the back-translated version follows. The experts look for a change in meaning, not a change in words.
5. Pre-testing happens with a few people. You ask each one what they understood from every item.

Some ideas from English scales don’t have any direct translation at all. “Work engagement” or “organisational citizenship behaviour” may have no everyday word in the language. For such words explain the idea simply, because a literal term nobody knows confuses the respondent. The methodology chapter has to mention the translators, their qualifications in general terms, and what got changed also.

How do you pilot a questionnaire?

In a pilot you test the questionnaire on people like the main sample, but not from the main sample. The pilot answers practical questions about the questionnaire. How many minutes does it take to fill, and which items are confusing people? Does the online form work on a phone? Does each scale have acceptable reliability?

• Size. No single rule is there. In theses, around 30 people is common for the pilot when the aim is a first reliability check. Hertzog (2008) suggested 10 to 40 per group for pilot studies, based on what the pilot is for. Your committee may have its own preference.
• Talk to some respondents. Ask five or six people to think aloud while answering the questionnaire, or ask them afterwards what some items meant to them. This catches problems that numbers will not show.
• Check reliability and item statistics. From the pilot data, see Cronbach’s alpha for every scale and the item-total correlations. Also look for items where almost everybody gave the same answer.
• Report what changed. A pilot that led to zero changes is either rare, or the pilot is not reported fully.

Keep the pilot respondents out of the main sample, especially if the questionnaire changed after the pilot.

How do you reduce common method bias?

Suppose one person answers all the questions, in one sitting, on the same type of scale. Then part of the correlation between your constructs can come from the method only. Some people simply agree with everything. Some want to look consistent, and some answer as per that day’s mood. This is called common method bias. Reviewers in management and social science journals now ask about it routinely. The standard reference is Podsakoff, MacKenzie, Lee and Podsakoff (2003).

What you decide while designing the questionnaire helps more than any statistics done afterwards.

• Separate the measures. Keep the predictor scales and the outcome scales in different sections. Where the scales permit, give them different instructions or a different response format.
• Protect anonymity. Tell respondents plainly that no answer is right or wrong and the questionnaire is anonymous. Then they answer more honestly, and less as per what they think you want.
• Avoid identical anchors everywhere. Many times when thirty items come one after another with the same five-point agree scale, many people just tick the same column all the way down.
• Use a second source where possible. For the outcome, take the supervisor’s rating of performance, or records like sales or attendance, in place of the respondent’s own report.
• Separate in time if you can. Predictors in one round and outcomes some weeks later brings the shared-method effect down. But more people drop out between the two rounds.
• Add a marker variable. A marker variable is a short scale which, in theory, has no relation to your model. Later you can use it to test for method variance.

Many theses report only Harman’s single-factor test (no single factor explaining more than 50% of variance). Podsakoff and colleagues have pointed out that this test is weak and it rarely catches a problem. For PLS-SEM, scholars mostly report the full collinearity test proposed by Kock (2015) also, with all VIFs at or below 3.3. In the thesis, first write what you did at design stage, and after that give the tests.

Ethics and consent

Take approval from your institutional ethics committee before collecting any data. Get it for the final version of the questionnaire. First page itself should carry a short information sheet and consent statement. Tell who you are, what is the purpose of the study, that taking part is voluntary, how data is stored and how to contact you. In online forms the usual way is to keep a required consent question before the first item. Names or phone numbers, collect only if really needed and keep that file away from the answers.