Reliability and validity in research

What is the difference between reliability and validity?

Take a bathroom scale which always shows 2 kg extra. Stand on it ten times, you get the same wrong weight, so it is reliable. But it is not valid. Now take a scale whose reading changes every time you step on it. That one cannot be valid at all, because it is not even consistent. Reliability is needed before you can have validity. Only reliability is not sufficient, though.

In a questionnaire study, reliability is about whether the items of each scale measure in the same way. Validity asks if your “job satisfaction” scale is catching job satisfaction only. Suppose the scores are following the respondent’s mood that day, then validity is weak.

Mostly a survey thesis needs four of these, internal consistency, content validity, convergent validity and discriminant validity. Test-retest and criterion validity come in mainly when you are building a new instrument, or taking an existing one into a very different setting.

Cronbach’s alpha or composite reliability: which should you report?

Cronbach’s alpha (Cronbach, 1951) is known to every examiner. You get it easily in SPSS (Analyze, Scale, Reliability Analysis). The problem with alpha is its assumption that all items are tied to the construct with equal strength. Usually the items are not equal, and then alpha tends to understate reliability. Alpha also goes up just because the scale has more items.

Composite reliability (CR) does not need that assumption, since it uses the actual loading of every item. When you are doing SEM, report CR along with alpha. SmartPLS gives one more value, rho_A, and it usually comes somewhere between the two.

McDonald’s omega plays the role of CR in studies which don’t use SEM. You can calculate it in JASP, jamovi or the R package psych. Many methodologists now like omega better than alpha. Still, most Indian examiners will want to see alpha also. So give both numbers in your table.

What if alpha is below 0.70?

• First see if the reverse-coded items were recoded. Mostly a low alpha comes from one negatively worded item which nobody recoded. In SPSS you will see it as negative corrected item-total correlation.
• Look at “alpha if item deleted”. Suppose alpha jumps up clearly when one particular item is removed. Then read that item again. Did respondents understand it? Was the translation done properly?
• Consider the scale length. A scale with two or three items many times shows an alpha near 0.60, even when the scale is working fine. With two items, the Spearman-Brown coefficient is the better measure to report.
• Do not cut items one by one only to reach 0.70. Every item you delete makes the scale measure a narrower thing. If a four-item scale loses two items, defending it in the viva-voce becomes difficult.

What thresholds do examiners expect?

Most reviewers and examiners in management and social science expect the numbers in this table. They come from the methods literature as conventions. Nobody passed them as laws. Write the citation next to each number, else examiner may feel the number is your own.

Keep two cautions in mind. One, different textbooks give slightly different thresholds. That is okay. Choose any one source and cite that. After that the same cut-offs should stay in all your chapters. Two, a scale which clears every threshold is still not proven good. If the items were chosen badly from the start, the scale can pass all the tests and measure the wrong thing. For this reason content validity has to be done first.

How do you establish convergent and discriminant validity?

Convergent validity

Convergent validity checks whether the items of one construct share a big part of their variance. Loadings and AVE are what you look at. When AVE is 0.50, the construct explains half of the variance in its items, on average.

When AVE is short, start from the items having the lowest loadings. Usually an item that loads below 0.40 is taken out. An item between 0.40 and 0.70 goes only if removing it brings AVE or CR above the threshold. And the construct should keep its meaning after the removal. Write down every deletion along with its reason.

Discriminant validity

Discriminant validity checks one simple point. Constructs that are supposed to be different, are they different in your data? Many theses report the Fornell–Larcker criterion alone. Under that rule, the square root of each construct’s AVE has to be bigger than the construct’s correlation with every other construct. Henseler, Ringle and Sarstedt (2015) ran simulations and found this test many times misses real problems. In its place they proposed the heterotrait-monotrait ratio (HTMT). SmartPLS gives HTMT by default. In AMOS you can calculate it from the item correlations, or take help of a published plug-in or spreadsheet.

When HTMT crosses 0.90 for a pair of constructs, it may be that your respondents cannot separate the two. Two causes are common. Either the items overlap, or the constructs really are very close to each other (in some service studies “satisfaction” and “loyalty” are like this). You have some options here. You can drop the overlapping items. Where theory allows, both can become one construct. Else report it honestly as a limitation.

Where CFA fits

With covariance-based SEM, you run a confirmatory factor analysis of the measurement model first, and the structural model comes after it. Report the fit indices. Any error covariance you add for better fit needs a theoretical reason behind it. Examiners are questioning more and more the habit of adding covariances one by one till the fit looks good. AMOS and SmartPLS judge the measurement model differently, and our AMOS vs SmartPLS guide covers that.

How do you show content validity?

Content validity is done before you collect the data. You have to show that the items cover the construct properly. When your scale is adapted from the literature, the steps usually go like this.

1. Write down what exactly you mean by each construct, with the citation for that definition.
2. Send the draft questionnaire to an expert panel, commonly five to ten people. Take subject faculty, plus one or two practitioners from the population you are studying.
3. Ask every expert to rate how relevant each item is, often on a four-point scale. Also ask for comments on clarity.
4. Some fields want to see a number here. Then calculate the item-level content validity index. This is the share of experts who rated the item 3 or 4. One common guide, from the work of Polit and Beck, says I-CVI should be 0.78 or more when you have six or more experts. Some scholars use Lawshe’s content validity ratio instead, and there the critical value depends on panel size.
5. Change the items as per the experts’ comments, and note down every change for the methodology chapter.

After this, the pilot study checks face validity and gives you the first reliability number. How to word the items and how to run the pilot, both are there in our questionnaire design guide.

Qualitative and experimental studies

Qualitative research is judged on other criteria. Most examiners are familiar with Lincoln and Guba’s trustworthiness framework, which has credibility, transferability, dependability and confirmability. Write what you actually did, for example member checking, keeping an audit trail, peer debriefing or reflexive notes. In experimental and engineering work, reliability is about repeatability and calibration. So there you report replicates, standard deviations and how the instrument was calibrated, not alpha.

What should you report in the thesis?

Give the reliability and validity figures from your main sample, in the chapter with the measurement results. Writing only “the scale has been validated by Smith (2010), α = 0.89” will not do. Smith had his own sample, in some other country and maybe in some other language.

• One table giving, for every construct, the number of items kept, alpha, CR (plus rho_A if you used SmartPLS) and AVE.
• A loadings table with every item. Deleted items also go in it, with why each one was dropped.
• A discriminant validity table with HTMT values. Add Fornell–Larcker if your examiners are expecting it.
• For CB-SEM, the fit indices of the measurement model before and after any change, and a justification for each change.
• Where each scale came from, and the reliability its original authors reported, so the reader can compare.
• How you established content validity: how many experts, what kind of experts, and what they changed.
• Details of the pilot study: the sample size, the reliability you got in the pilot and the changes you made after it.

Place these results before any hypothesis testing, because that is the order examiners read in. When the measurement is weak, whatever tests come after it carry very little meaning. If the measurement is okay, open the statistical test guide to pick a test for your hypotheses.