Understanding reliability and validity

Tricky topics - validity trickier than reliability. At it’s core I believe reliability is about getting the same results given the same conditions and validity is about measuring what you intend to measure. Harder than it sounds.

Published 5 Feb 2014

A weekly sketch by email

Learn something new in a sketch each Sunday.

Sketchplanations in a book! I think you'll love Big Ideas Little Pictures

Prefer to listen?
Try the podcast

Like Sketchplanations?
Support me on Patreon

Keep exploring

The Barnum effect

The fun scale

A fisherman illustrates the parable of the fishing net by concluding a minimum size of fish because they never see any smaller in their nets — you get what you measure

You get what you measure

Chihuahua syndrome

Someone asks to cut in line at the photocopier with a lame reason - but the favour and reason combo mean the person mindlessly complies

Because

Accuracy and precision

The Barnum effect is our tendency to apply personal meaning to statements that could apply to many people. For statements such as the results of a personality test, the effect is stronger if: we believe they are personalised to us they are sufficiently vague — handy phrases include "at times," "a tendency to," "some of," etc. they are mostly positive — we're more inclined to accept positive evaluations of ourselves than critical ones they come from a source of authority You may recognise these tricks from a horoscope or your star sign, magicians, fortune-tellers, fortune cookies, con artists and, perhaps, marketing. There's a fascinating magician's technique you may have seen called cold reading that uses the Barnum effect with other tricks ("I see someone with heart pain in your family.") to make it seem like they know a lot of personal details about someone at a first meeting. Originally from Bertram Forer, who called it "the fallacy of personal validation," it was later known as the Forer effect until Paul Meehl suggested it should be named the Barnum effect after P. T. Barnum (now of The Greatest Showman fame) for the way he could easily fool an audience with these techniques and to emphasize that we should all know and be on the lookout for being fooled. For that reason, I called it the same here. Bertram Forer ran an experiment with his students (pdf), giving everyone the same results for their individual personality assessment. Generally, the students all thought the same description applied very well to themselves. He used these statements taken "largely from a newsstand astrology book": You have a great need for other people to like and admire you. You have a tendency to be critical of yourself. You have a great deal of unused capacity which you have not turned to your advantage. While you have some personality weaknesses, you are generally able to compensate for them. Your sexual adjustment has presented problems for you. Disciplined and self-controlled outside, you tend to be worrisome and insecure inside. At times you have serious doubts as to whether you have made the right decision or done the right thing. You prefer a certain amount of change and variety and become dissatisfied when hemmed in by restrictions and limitations. You pride yourself as an independent thinker and do not accept others' statements without satisfactory proof. You have found it unwise to be too frank in revealing yourself to others. At times you are extroverted, affable, sociable, while at other times you are introverted, wary, reserved. Some of your aspirations tend to be pretty unrealistic. Security is one of your major goals in life. Not a bad description of me... I learned about it from a discussion of the Meyers-Briggs test (I'm an ENTP btw). Here's Paul Meehl advocating for the Barnum effect: "Many psychometric reports bear a disconcerting resemblance to what my colleague Donald G. Paterson calls “personality description after the manner of P. T. Barnum” ... I suggest—and I am quite serious—that we adopt the phrase Barnum effect to stigmatize those pseudo-successful clinical procedures in which personality descriptions from tests are made to fit the patient largely or wholly by virtue of their triviality; and in which any nontrivial, but perhaps erroneous, inferences are hidden in a context of assertions or denials which carry high confidence simply because of the population base rates, regardless of the test’s validity. I think this fallacy is at least as important and frequent as others for which we have familiar labels (halo effect, leniency error, contamination, etc.). One of the best ways to increase the general sensitivity to such fallacies is to give them a name. We ought to make our clinical students as acutely aware of the Barnum effect as they are of the dangers of countertransference or the standard error of r." Meehl, P. E. (1956). Wanted—a good cook-book. American Psychologist, 11(6), 263–272. https://doi.org/10.1037/h0044164…The Barnum effect is our tendency to apply personal meaning to statements that could apply to many people. For statements such as the results of a personality test, the effect is stronger if: we believe they are personalised to us they are sufficiently vague — handy phrases include "at times," "a tendency to," "some of," etc. they are mostly positive — we're more inclined to accept positive evaluations of ourselves than critical ones they come from a source of authority You may recognise these tricks from a horoscope or your star sign, magicians, fortune-tellers, fortune cookies, con artists and, perhaps, marketing. There's a fascinating magician's technique you may have seen called cold reading that uses the Barnum effect with other tricks ("I see someone with heart pain in your family.") to make it seem like they know a lot of personal details about someone at a first meeting. Originally from Bertram Forer, who called it "the fallacy of personal validation," it was later known as the Forer effect until Paul Meehl suggested it should be named the Barnum effect after P. T. Barnum (now of The Greatest Showman fame) for the way he could easily fool an audience with these techniques and to emphasize that we should all know and be on the lookout for being fooled. For that reason, I called it the same here. Bertram Forer ran an experiment with his students (pdf), giving everyone the same results for their individual personality assessment. Generally, the students all thought the same description applied very well to themselves. He used these statements taken "largely from a newsstand astrology book": You have a great need for other people to like and admire you. You have a tendency to be critical of yourself. You have a great deal of unused capacity which you have not turned to your advantage. While you have some personality weaknesses, you are generally able to compensate for them. Your sexual adjustment has presented problems for you. Disciplined and self-controlled outside, you tend to be worrisome and insecure inside. At times you have serious doubts as to whether you have made the right decision or done the right thing. You prefer a certain amount of change and variety and become dissatisfied when hemmed in by restrictions and limitations. You pride yourself as an independent thinker and do not accept others' statements without satisfactory proof. You have found it unwise to be too frank in revealing yourself to others. At times you are extroverted, affable, sociable, while at other times you are introverted, wary, reserved. Some of your aspirations tend to be pretty unrealistic. Security is one of your major goals in life. Not a bad description of me... I learned about it from a discussion of the Meyers-Briggs test (I'm an ENTP btw). Here's Paul Meehl advocating for the Barnum effect: "Many psychometric reports bear a disconcerting resemblance to what my colleague Donald G. Paterson calls “personality description after the manner of P. T. Barnum” ... I suggest—and I am quite serious—that we adopt the phrase Barnum effect to stigmatize those pseudo-successful clinical procedures in which personality descriptions from tests are made to fit the patient largely or wholly by virtue of their triviality; and in which any nontrivial, but perhaps erroneous, inferences are hidden in a context of assertions or denials which carry high confidence simply because of the population base rates, regardless of the test’s validity. I think this fallacy is at least as important and frequent as others for which we have familiar labels (halo effect, leniency error, contamination, etc.). One of the best ways to increase the general sensitivity to such fallacies is to give them a name. We ought to make our clinical students as acutely aware of the Barnum effect as they are of the dangers of countertransference or the standard error of r." Meehl, P. E. (1956). Wanted—a good cook-book. American Psychologist, 11(6), 263–272. https://doi.org/10.1037/h0044164

Newer

RRandom

Older