Critical Thinking, Research Biases/Errors

2.1 Critical Thinking

• Clarity:
  • Is it clear what you are trying to say?
  • Have you been concise and to the point, or are you ‘waffling’?
  • Do you have support (references) for what you are asserting is a fact?
  • Have you used industry and/or academic language correctly?
    
  • Is your punctuation correct?
  • Is your sentence structure correct?
• Precision:
  • You are writing a questionnaire, for example, are your questions precise, are they asking what you need to know, and would that provide precise answers?
  • Ask yourself, what do I really want to know?
  • Are my questions precise, or are they leading questions?
  • More on writing questionnaires in chapter 8.
• Accuracy:
  • How accurate is the information you have obtained from various sources?
  • Have you ensured that the sources are reliable?
  • Have you looked at a variety of sources for information that is both positive and negative, and compared their validity?
  • While accurate information from valid, reliable sources can assist with good decision making, information from unreliable sources can lead to catastrophic errors.
• Relevance:
  • How relevant is what you have written?
  • Does what you have written relate to your topic/point, or have you gone off on a tangent where you might end up down Alice’s rabbit hole?
    
  • Is what you have written or what you are saying relevant to your audience?
• Consistency:
  • Are you consistent with what you are saying about your topic/point?
  • Is what you are writing about (saying) consistent with what you found in your literature searches?
  • Are you applying information and referencing sources correctly and consistently?
• Logical Correctness:
  • Is what you have written logical?
  • Does your argument come to a logical conclusion?
  • Read what you have written carefully. Does each sentence in each paragraph build your reasoning to a logical conclusion?
• Fairness:
  • When you have been researching your topic, have you fairly considered all points of view from reliable sources?
  • Are you satisfied you have not let personal feelings or preconceptions cloud your judgement?

Critical thinking may appear to be a field of obstacles, but taking a logical approach, and thinking carefully about what you have written, and keeping an open mind is going to deliver great results.

2.2 Barriers to Critical Thinking

While critical thinking can be logical and successful if you can answer the questions in the previous section, there are barriers that need to be addressed. These barriers are generally unconscious occurrences, but if you are aware of them, you can prevent them (or at least try to).

Anchoring bias

This occurs when a person places too much importance on the first piece of information they hear or see. For example, a person goes to a car dealer as they want to buy a new electric vehicle (EV). The first EV they see has a price tag of $73,000. The next EV has a price tag with $80,000 crossed out, and replaced with $73,000, the person sees this as an excellent deal. They have decided to place too much importance on the first price they saw.

Availability heuristic

This is when an individual overestimates the likelihood of something happening based on instances readily available in their memory. For example, an individual from Melbourne is on holiday in Alice Springs, they recall reports that crocodiles are found in waterways in the Northern Territory and kill people. The individual sees a waterway and immediately believes it is full of crocodiles (for those unaware, crocodiles do not live in Alice Springs waterways, it is too far south).

Bandwagon effect

This is a psychological phenomenon. You may find someone is more likely to adopt a belief or follow a trend based on the number of other people with that belief or following that trend. For example, State of Origin is on in Brisbane. Sam is from Darwin and is not interested in Rugby League and intends to binge-watch Stranger Things on Netflix. Sam is staying with Jo, who says, “Come and watch State of Origin tonight. Everyone is going to be there”. Sam goes to watch the State of Origin.

Blind-spot bias

Did you know that people are really quite good at spotting biases in other people? However, these same people often fail to recognise their own cognitive biases and consequently the resulting impacts on their decision-making. For example, someone may think they are being ethical by purchasing ‘fair-trade’ coffee. But they have not considered the carbon emissions created in transportation or ‘food-miles’ of the coffee.

Choice-supportive bias

Generally, this occurs sometime after a choice was made. The recollection of the choice is usually thought to be better than it might have been. That is, the focus is on the positives and the negatives are just ignored. For example, Ash wants a new phone and looks at an iPhone and an Android phone. Ash purchases the iPhone. When Ash recalls the purchase of the iPhone, the recollections are all positive – for example, all the great apps. Ash ignores the negatives, e.g. the iPhone won’t link to the Dell laptop.

Cognitive bias

The term “cognitive bias” originated in the 1970s to explain humans’ propensity to seek and find patterns in everything, even when they do not really exist. Although someone may be aware of cognitive bias they may still come up with an error, due to the error being compelling (Behimehr & Jamali, 2020). You can think of cognitive bias as our brains taking a shortcut to make information processing simpler. This can be caused by emotions, social influences, cultural conditioning, or availability of information (University of Texas, n.d.). There are many different types of cognitive biases. One of these is ‘Clustering Illusion’. This occurs when people see patterns or clusters in random happenings (data), when there is no pattern or cluster. For example, your favourite basketball player is on court and scoring baskets. They shoot 3, 2, 3, 2, so you believe the next score is going to be 3. There is no pattern here, the scores come from the player’s position on the court in relation to the 3-point line. If the player is outside the 3-point line they score 3 points, if they are inside the 3-point line they score 2 points.

Confirmation bias

This is when a person looks for and covets information that supports their strongly held beliefs. Any information that contradicts those beliefs is ignored (Nickerson, 1998). For example, in politics, many voters only seek information that supports their beliefs about their chosen candidate. Any negative information is ignored.

Group conformity

Group conformity is also referred to as social conformity, this is when a person follows a group’s (or other people’s) decisions even when it goes against their personal experiences, or better judgement (Li et al., 2019).

Conservatism bias

Here, people ignore new evidence or information that has emerged, and instead cling to prior evidence. For example, people were slow to accept the sun was the centre of the galaxy and all the planets, including the earth, orbited the sun. They maintained an earlier belief that everything in the galaxy orbited the earth.

Information bias

This happens when a person believes that the more information they have, the better the decision they can make, even when the information is irrelevant to the issue. For example, this can be applied effectively in marketing. A company provides a great deal of information on a smart TV, even if some is irrelevant to the consumer. The company hopes all the information is going to reassure consumers, so they are going to make a purchase.

Ostrich effect

This occurs when a person metaphorically puts their ‘head in the sand’. They do not want to acknowledge any possible negative information. For example, you know you are due for your annual dental appointment, but you keep finding reasons not to go for fear of what the dentist may tell you. If you do not go you are not going to hear the negative information, e.g., you need a filling, therefore you don’t need a filling.

Outcome bias

This is when a person evaluates the quality of their decision based on the outcome of the decision. For example, the score was one-one, the coach took the rookie 16-year-old off the bench and sent him on the field with 3-minutes left in the game. The rookie scored the winning goal with 5-seconds left on the clock. The coach said after the match that it was a stroke of genius on his behalf, he knew the rookie would deliver. However, if the rookie had failed the coach would have looked a fool.

Overconfidence

There are numerous people who are overly confident in their abilities. This may cause them to be overconfident and take greater risks. For example, A.J. goes for a job interview. After the interview, he heads for a coffee. A.J.’s phone is put on the table. A.J. is overly confident about how well the job interview went. The phone is on the table in anticipation of a phone call for a job offer. A.J. has failed to consider the abilities of the other job applicants. This lack of consideration that other applicants may be just as capable or even better is overconfidence bias.

Placebo effect

The placebo effect is often seen in medical treatments, most often relating to mild ailments. For example, Sam cannot sleep and has been lying on the bed for about an hour. Sam decides to get up and take a sleeping tablet. Sam does not take a sleeping tablet, but instead takes a pain reliever by accident. Because Sam believed the tablet was a sleeping tablet, they went to sleep and woke in the morning and said they had the best sleep ever. As seen in this example, if a person is convinced that a treatment is going to provide the benefit needed that it does. Even though the treatment was ‘fake’, it just has to appear ‘real’ for it to take effect on some people.

Pro-innovation bias

This occurs when someone believes an innovation should be ‘pushed out’ straight away to everyone, regardless of any limitations or weaknesses. For example, A.I.s were ‘pushed out’ to the world without actually being ready. There are privacy and ethical issues that should have been resolved in the development stage. A.I.s have demonstrated bias, they lie, they make up information and/or references. A.I. may not meet expectations and so disappoint.

Recency

This is when someone overemphasises the importance of items, ideas, information, and events that have recently occurred and therefore remembered more clearly than older ones. This relates to short-term memory. For example, you write a shopping list, you go shopping and leave the list at home. You are more likely to remember the last two or three items on the list, but you struggle to recall any at the top or middle of the list.

Salience

When the human mind focuses on something that stands out (is easy to recognise) from everything else. For example, you are reading instructions for an assessment, your lecturer bolds some of the words to get your attention. You immediately focus on these words.

Selective perception

This is when a person tends not to notice, ignore, or quickly forget any stimuli that makes them feel uncomfortable or contradicts their prior beliefs. For example, a person must select a mixed (boys and girls) netball team for the final which everyone wants to win. The person selects more girls than boys in the team because the person believes girls are better at netball than boys.

Stereotyping

A predisposition to believing that a group or individual has certain qualities because of who they are, this could be race, gender, age, etc. For example, someone sees a person who they perceive is elderly and assume the most they would do is potter away in the garden, when in fact the person is an avid scuba diver.

Survivorship bias

This occurs when you focus on those who are success stories (survivors) and ignore those who have failed. For example, you may think making money on the stock market is easy as you have heard a lot of success stories, but what about all of those who have failed?

Zero-risk bias

When a person must make a choice, they prefer the option that eliminates small risks over the alternative that would decrease large risks and provide improved results, because this still has risk. For example, you are buying a new toaster, one is good value for money but has a 15-day money-back guarantee, the other is more expensive, but has a 45-day money-back guarantee. Most people select the more expensive toaster as they feel there is no purchase risk attached.

2.3 Research Errors

Coverage error

Coverage error is when your sample does not accurately reflect the population. This most commonly occurs when your sample is biased or is not large enough (Alvarez & Van Beselaere, 2005). Note: See Chapter 10 for more on sample size.

Measurement error

Measurement error occurs when the survey is designed and implemented, and can also occur when the results are interpreted. There is a lack of completeness and the ability to interpret results, meaning they are not generalisable (Davies, 2020).

Participant or response bias or error

Participant or response bias or participant error is all about what has affected the individual participant’s responses. These biases or errors may be subconscious or deliberate and are most common in surveys where the focus is on individual opinions or behaviours. For example:

• The participant subconsciously feels the need to please the researcher and so answers the questions accordingly.
• The participant may deliberately randomly answer questions, not caring what they select as they are time poor, or not really interested in the survey, and even find it an annoyance. But they complete the survey anyway as they perceive this as helping the researcher(s), even though they are in reality skewing the data.
• The name of the survey, instructions provided to the respondent, or even question order can cause an issue.

Researcher error

Researcher error is also known as scope error. This occurs when there is an omission of required questions. That is, important questions needed to answer the research question have been left off or omitted from the survey.

Sampling error

A sampling error occurs when a specific characteristic of your sample differs from that of the whole population (McNabb, 2021). Your sub-group of interest is then over- or underrepresented therefore your study’s results are not generalisable.

Selection or sampling bias

Selection bias is also known as sampling bias. This occurs when some members of a population have a greater chance of being included in a study than others or when potential participants are unintentionally excluded from the sample. Often this is due to how the sampling process has been conducted. When selection/sampling bias occurs results from the study may not be generalisable (Rose et al., 2024).

There are three main sources of sampling/selection bias:

1. Only a section of the target population has the opportunity to participate in the study, the rest have been excluded. This may be due to a sampling frame error, or the access to participants method. For example, you are using the internet to access a group of public bus users. However, not all public bus users are also internet users. This biases your sample to public bus users who also use the internet, instead of all public bus users, therefore your results are not generalisable.
2. People who do not meet the required criteria are accidentally included in your sample. For example, you have a filter question in your questionnaire, but it is not specific enough, therefore you may have participants included who do not meet your sample criteria.
3. People may decline or are unable to participate; this is non-response bias. This bias is created when those who decline or are unable to take part in the study are summarily different from those who do participate (Rose et al., 2024).

Prevention

A census is the most accurate way to collect data and avoid errors, here everyone in the population is included, but this is only practical for a government to conduct (Australian Bureau of Statistics [ABS], n.d.). Instead, you need to conduct a survey that has a proportionate representation of the population. Further discussions on sampling are in Chapter 10 of this text.

References

Alvarez, R. M., & Van Beselaere, C. (2005). Web-based survey. In K. Kempf-Leonard (Ed.), Encyclopedia of Social Measurement, https://doi.org/10.1016/B0-12-369398-5/00390-X

Australian Bureau of Statistics. (n.d.). Errors in statistical data. https://www.abs.gov.au/websitedbs/d3310114.nsf/home/Basic+Survey+Design+-+Errors+in+Statistical+Data  

Behimehr, S., & Jamali, H. R. (2020). Cognitive biases and their effects on information behaviour of graduate students in their research projects. Journal of Information Science Theory and Practice, 8(2), 18-31. https://doi.org/10.1633/JISTaP.2020.8.2.2 

Davies, R. S. (2020). Designing surveys for evaluations and research. EdTech Books. https://edtechbooks.org/designing_surveys

Li, L., Li, K. K., & Li, J. (2019). Private but not social information validity modulates social conformity bias. Human Brain Mapping, 40(8). https://doi.org/10.1002/hbm.24536

McNabb, D. E. (2021). Research methods for political science: quantitative, qualitative, and mixed methods approaches (3rd ed.). Routledge.

Nickerson, R. S. (1998). Confirmation bias: a ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 2464-2474. https://doi.org/10.1037/1089-2680.2.2.175

Rose, S., Spinks, N., & Canhoto, A. I. (2024). Management research: applying the principles of business research methods. Routledge.