Module 32 of 47Lesson 1 of 9 · HL · Further statistics 1 · Designing a study: reliability & validity
1
Further statistics: design, regression, confidence intervals & tests · Essential understanding 1

Designing a study: reliability & validity

IB AI AHL 4.12with Prof. Kitso
💻 Using the TI-Nspire CX Student Software on a laptop
  • The screens in these notes are drawn exactly as the Student Software shows them. The keypad on the left of the software works like the handheld — click its keys with the mouse.
  • menu: click menu on the on-screen keypad (or Documents Toolbox ▸ 🔧). The menu numbers and letters are the same as in these notes.
  • enter = Enter  ·  tab = Tab (next box)  ·  esc = Esc  ·  del = Backspace.
  • Powers: type ^ (Shift+6), the power, then press → to come back down before typing more. x² = ^2 then →.
  • Fraction: type /  ·  √: type sqrt(  ·  π: type pi  ·  eˣ: type exp(  ·  store: type :=. For EE (×10ⁿ) and i, click them on the keypad.
  • New page: Ctrl+I (or the Insert menu)  ·  change page: Ctrl+← / Ctrl+→  ·  graph entry line: Ctrl+G  ·  var: Ctrl+L.
  • Radian / Degree: click Settings in the status bar at the bottom of the window. The mode (RAD or DEG) shows in the top bar.
  • IB exam mode (Press-to-Test): everything in these notes works on the non-CAS TI-Nspire with the IB exam settings. In exam mode there are no dot-product, cross-product, unit-vector or eigenvalue keys and no Diff Eq graphs — the notes show the allowed way (type the components, use polyRoots on the characteristic equation, do Euler steps on a Calculator page).
  • If you type nDeriv(, the software shows it as centralDiff( — it is the same command.
The big idea: A clever calculation on bad data gives a bad answer. Before you run a test, you must design the investigation well: ask a clear question, collect fair and relevant data, and then check that your results are reliable (repeatable) and valid (really measuring what you want).
By the end of this lesson you can:
  • follow the investigation cycle from research question to conclusion
  • choose proxy variables and a relevant, appropriate sample
  • spot what is wrong with a questionnaire question and improve it
  • test reliability with test–retest and parallel forms using a correlation coefficient
  • explain and check content validity and criterion validity
  • choose and justify an appropriate statistical test and state its assumptions
🧑🏾 Imagine this…
Kagiso wants to know: “Do students who sleep more do better at school?” Hmm. Where do you start?
  • “Do better at school” is vague. He picks a number he can measure: the end-of-term maths mark. That is a proxy.
  • He asks classmates how long they sleep. Some guess. Some lie. Would the answer be the same next week? That is a question about reliability.
  • Does the maths mark really show “doing well at school”? That is a question about validity.
Good statistics begins long before the calculator comes out.
Population and sample
the population is everyone you want to know about; the sample is the part of it you actually measure. The sample must be relevant to the research question.
Proxy variable
a variable you can measure that stands in for an idea you cannot measure directly (for example, a maths mark as a proxy for “academic ability”).
Structured / unstructured question
structured: the answer is chosen from a fixed list or scale (easy to analyse). Unstructured: the person writes what they like (rich ideas, hard to analyse).
Reliability
would you get similar results if the investigation were repeated in similar conditions? Reliable = consistent.
Test–retest
give the same test to the same group twice, some time apart. A strong correlation between the two sets of scores means the test is reliable.
Parallel forms
measure the same idea in two slightly different ways (for example, two different but similar tests) on the same people. A strong correlation means reliable.
Validity
is the investigation measuring what it claims to measure?
Content validity
experts check that the test covers the right content and nothing else (it is about trigonometry, not about reading skill). For tests in statistics it also means checking that the assumptions of the method are met.
Criterion validity
compare the test with an external standard that is accepted as a true measure. If an entrance test is valid, students who score high should also do well later.
★ The investigation cycle
  • 1. Research question → 2. Define the population → 3. Design (variables, collection method, sample, test) → 4. Collect data → 5. Explore with descriptive statistics → 6. Answer with inferential statistics → 7. Are the results reliable and valid?
  • If the answer to step 7 is “no”, go back: change the question, collect better data (for example a larger sample) or use a different test.
★ A good questionnaire question is …
  • Unbiased: it does not push you towards an answer. (“Do you agree that burgers are unhealthy?” pushes. “How often do you eat burgers?” does not.)
  • Precise: everybody reads it the same way. (“How many people live in your family?” – does that include grandparents? Say “How many people sleep in your home most nights?”)
  • Single-barrelled: it asks only one thing. (“Is the canteen cheap and tasty?” is two questions.)
  • Answerable honestly: sensitive questions can be asked anonymously, and the scale should be clear (give categories or ranges, not just “a lot”).
★ Reliability and validity at a glance
  • Reliability checks (correlation, r close to 1): test–retest (same test, twice) and parallel forms (two comparable measures). Natural variation means r need not be exactly 1.
  • Validity checks: content validity (experts, and checking the test’s assumptions) and criterion validity (agreement with an external standard).
  • A reliable test can still be invalid. A bathroom scale that always reads 3 kg too heavy is perfectly consistent, but wrong.
Question you are askingAppropriate testMain assumptions to check
Do observed counts fit a model (fair die, Poisson, normal)?χ² goodness of fitrandom sample; every expected count ≥ 5
Are two categorical variables linked?χ² test for independencerandom sample; every expected count ≥ 5
Is the population mean different from a claimed value?one-sample t test (z test if σ is known)population (roughly) normal, or large sample
Did the same people change between two occasions?paired t test on the differencesdifferences (roughly) normal
Do two numerical variables move together?correlation / regressionrelationship roughly linear (or choose a suitable curve)
Is a proportion or rate different from a claim?binomial test (or Poisson test for events in time or space)fixed n, independent trials, constant p (or events random and independent)
⚠️ Watch out
  • Reliable ≠ valid. Examiners love this. Say which one you mean, and why.
  • A correlation shows reliability only if it is strong. A weak r from a test–retest means the test is not reliable.
  • Never choose a test because it is the one you know. Choose it because it answers the question and its assumptions hold.
  • Statistics can show a link, but a test of association cannot prove cause.
💡 Command terms
Suggest – give a sensible idea with a reason. Comment on – say something useful about quality, not just describe. Justify – give reasons from the situation (“the question is subjective because …”). Always tie your answer to the context given.
🧪 Try it yourselfWhich check is it?
Decide which reliability or validity check each description is.

Worked examples

Example 1.1Improving a questionnaire question
A student council wants to know about lunch. One question on its draft questionnaire is:
“Don’t you agree that the awful, overpriced canteen food should be replaced with something decent?”
  1. (a) Give two things that are wrong with this question.
  2. (b) Write an improved version that is structured and easy to analyse.
🧠 Think it through
  1. Read the wording. Are there emotional words? Is it pushing an answer?
  2. Count how many ideas the question asks about (price? quality? replacing?).
  3. A structured question gives a fixed list or scale to choose from.
✅ Answer (a) It is biased (“awful”, “overpriced”, “Don’t you agree…” push the person to say yes). It also asks several things at once (quality, price and replacement). (b) For example: “How would you rate the quality of canteen food? 1 = very poor, 2 = poor, 3 = average, 4 = good, 5 = very good.” A separate question can ask about price.
One question, one idea. Split a double question into two single ones.
Example 1.2Test–retest reliability
Eight students sit a spelling test and, two weeks later, the same test again. Their marks (out of 100) are:
Test 16248755581396858
Test 26545726078447053
  1. (a) Find the correlation coefficient r.
  2. (b) Comment on the reliability of the test.
🧠 Think it through
  1. Same test, same students, two occasions: this is the test–retest method.
  2. Put Test 1 in one list and Test 2 in another, then find r with Linear Regression.
  3. Compare r with 1. A value close to 1 means that students who did well the first time also did well the second time.
🔢 On your TI-Nspire CX II
  1. On 1.1 Lists & Spreadsheet name column A t1 and column B t2. Type the pairs in the same order, so each row is one student.
  2. Check that both columns have the same number of rows.
    1.1*DocRAD◆At16248755581396858Bt26545726078447053CD12345678A162
  3. Add a Calculator page (ctrl💻 Ctrl doc ▾💻 File menu then 1) and open menu💻 click menu on keypad▸6: Statistics▸1: Stat Calculations▸3: Linear Regression (mx+b). Press enter ↵💻 Enter.
  4. Set X List = t1 and Y List = t2. Press enter ↵💻 Enter.
    1.1*DocRADLinear Regression (mx+b)X List:t1Y List:t2Save RegEqn to:f1Frequency List:1Category List:Include Categories:OKCancel
  5. Read r, the correlation coefficient. (The sign tells the direction; the size tells the strength.)
    1.1*DocRADLinRegMxt1,t2,1:CopyVarstat.RegEqn,f1:stat.results"Title""Linear Regression (mx+b)""RegEqn""m·x+b""m"0.872907"b"7.84592"r²"0.916576"r"0.95738"Resid""{…}"
✅ Answer (a) r = 0.957. (b) r is close to 1 (strong positive correlation), so students who scored high the first time also scored high the second time. The test is reliable.
Small differences between the two sets of marks are natural. We do not need r = 1.
Example 1.3Parallel forms
A coach wants to know whether the 100 m time and the 200 m time (both in seconds) are two reliable measures of sprint ability. Seven athletes run both:
100 m11.812.412.913.113.614.214.9
200 m24.225.126.826.327.929.030.1

A second coach compares two different quizzes (each out of 20) for eight students and finds r = 0.32.
  1. (a) Find r for the 100 m and 200 m times and comment.
  2. (b) Comment on the reliability of the second coach’s two quizzes.
🧠 Think it through
  1. Two different but comparable measures of one idea: parallel forms.
  2. Use Linear Regression on the two lists and read r.
  3. Decide: strong correlation → reliable. Weak correlation → not reliable (or the two measures are not really measuring the same thing).
🔢 On your TI-Nspire CX II
  1. On 1.1 Lists & Spreadsheet name column A t100 and column B t200. Type the pairs in the same order, so each row is one athlete.
  2. Check that both columns have the same number of rows.
    1.1*DocRAD◆At10011.812.412.913.113.614.214.9Bt20024.225.126.826.327.929.030.1CD1234567A111.8
  3. Add a Calculator page (ctrl💻 Ctrl doc ▾💻 File menu then 1) and open menu💻 click menu on keypad▸6: Statistics▸1: Stat Calculations▸3: Linear Regression (mx+b). Press enter ↵💻 Enter.
  4. Set X List = t100 and Y List = t200. Press enter ↵💻 Enter.
    1.1*DocRADLinear Regression (mx+b)X List:t100Y List:t200Save RegEqn to:f1Frequency List:1Category List:Include Categories:OKCancel
  5. Read r, the correlation coefficient. (The sign tells the direction; the size tells the strength.)
    1.1*DocRADLinRegMxt100,t200,1:CopyVarstat.RegEqn,f1:stat.results"Title""Linear Regression (mx+b)""RegEqn""m·x+b""m"1.9617"b"1.02255"r²"0.979576"r"0.989735"Resid""{…}"
✅ Answer (a) r = 0.990, a very strong positive correlation. The two measures agree, which supports the claim that they reliably measure sprint ability. (b) r = 0.32 is weak. The two quizzes do not agree, so they are not reliable parallel forms; students ranked differently on each.
Strong correlation means reliable. It does not by itself show that the measure is valid.
Example 1.4Criterion validity and content validity
A private school uses an entrance test (out of 80). Ten students’ entrance scores and their end-of-year marks (out of 100) are:
Entrance34524160472955386445
End of year48665072614063527854
  1. (a) Find r and explain which type of validity it tests.
  2. (b) The entrance test is a maths test, but many questions are long wordy paragraphs. Which type of validity is threatened, and why?
🧠 Think it through
  1. End-of-year marks are an external standard for how well the student does in school.
  2. A strong positive r suggests the entrance test predicts later success.
  3. A maths test full of reading tests two things at once: maths and reading.
🔢 On your TI-Nspire CX II
  1. On 1.1 Lists & Spreadsheet name column A entr and column B endm. Type the pairs in the same order, so each row is one student.
  2. Check that both columns have the same number of rows.
    1.1*DocRAD◆Aentr3452416047295538Bendm4866507261406352CD12345678A134
  3. Add a Calculator page (ctrl💻 Ctrl doc ▾💻 File menu then 1) and open menu💻 click menu on keypad▸6: Statistics▸1: Stat Calculations▸3: Linear Regression (mx+b). Press enter ↵💻 Enter.
  4. Set X List = entr and Y List = endm. Press enter ↵💻 Enter.
    1.1*DocRADLinear Regression (mx+b)X List:entrY List:endmSave RegEqn to:f1Frequency List:1Category List:Include Categories:OKCancel
  5. Read r, the correlation coefficient. (The sign tells the direction; the size tells the strength.)
    1.1*DocRADLinRegMxentr,endm,1:CopyVarstat.RegEqn,f1:stat.results"Title""Linear Regression (mx+b)""RegEqn""m·x+b""m"1.0082"b"11.5187"r²"0.955516"r"0.977505"Resid""{…}"
✅ Answer (a) r = 0.978, a strong positive correlation. This tests criterion validity: the test agrees with an external standard (end-of-year performance). (b) Content validity. Students with weak reading may do badly because of the reading, not the maths, so the test is not measuring only maths ability.
Example 1.5Choose an appropriate test
Choose the most appropriate test for each situation and state one assumption to check.
(a) A clinic in Francistown records whether 120 patients are “satisfied” or “not satisfied”, and whether they were seen by a doctor or a nurse. Is satisfaction linked to who they saw?
(b) A bakery claims its loaves have mean mass 700 g. A sample of 15 loaves is weighed.
(c) The number of customers arriving per minute at a till is recorded for 80 minutes. Does it follow a Poisson distribution?
(d) 12 learners take a quiz before and after a revision workshop. Did the workshop change their marks?
🧠 Think it through
  1. Name the kind of data: counts in categories, a numerical mean, or pairs of measurements.
  2. Match the question to the test (see the table in this lesson).
  3. Write one assumption that must hold for the test to be trustworthy.
✅ Answer (a) χ² test for independence; every expected count at least 5 (combine categories if not). (b) One-sample t test (σ is unknown); the masses are roughly normally distributed. (c) χ² goodness of fit to a Poisson model, with λ estimated from the data and df reduced by one; expected counts at least 5. (d) Paired t test on the differences (after − before); the differences are roughly normal.
Whenever a parameter such as λ is estimated from the data in a goodness-of-fit test, the degrees of freedom go down. You will see this in Lesson 3.

Challenge questions

⭐ Challenge 1.6A survey on phone usePaper 1 style
Onalenna wants to know whether using a phone late at night harms school performance. She asks students at her own school the question: “How much time do you waste on your phone at night?”, and uses their last maths test mark as the measure of school performance.
  1. (a) Write down one proxy variable she uses.
  2. (b) Give one reason the question is poorly worded.
  3. (c) Give one reason her sample may not be appropriate.
  4. (d) Name a method she could use to check the reliability of the question.
✏️ Try it on paper and on your calculator first. Then check.
Show the full solution
🧠 Think it through
  1. The maths mark stands in for “school performance”.
  2. Look for loaded words and vague words in the question.
  3. Who is in her sample, compared with the population she cares about?
✅ Answer (a) The last maths test mark (as a proxy for school performance). (b) “waste” is a biased, loaded word; “at night” and “how much time” are also imprecise. (c) She only asked students at her own school, so the sample may not represent all students. (d) Test–retest: ask the same students the same question again later and find the correlation between the two sets of answers.
⭐ Challenge 1.7A new reading testPaper 2 style
A literacy project designs a new reading test (out of 40). Ten learners sit it twice, three weeks apart. Their marks are:
Occasion 122311835271429332025
Occasion 224292136251730311928
Later, the same learners’ marks are compared with their teachers’ independent ratings of their reading (out of 10): r = 0.91.
  1. (a) Name the method used to compare the two occasions.
  2. (b) Find r for the two occasions and comment on the reliability.
  3. (c) Name the type of validity tested by r = 0.91, and comment.
  4. (d) An expert says the test’s passages assume knowledge of rugby, which many learners do not have. Which kind of validity is threatened? Suggest a change.
✏️ Try it on paper and on your calculator first. Then check.
Show the full solution
🧠 Think it through
  1. Same test, same group, two times: test–retest.
  2. Lists → Linear Regression → read r.
  3. Teachers’ ratings are the external standard.
  4. Does the test measure reading, or something else as well?
🔢 On your TI-Nspire CX II
  1. On 1.1 Lists & Spreadsheet name column A o1 and column B o2. Type the pairs in the same order, so each row is one learner.
  2. Check that both columns have the same number of rows.
    1.1*DocRAD◆Ao12231183527142933Bo22429213625173031CD12345678A122
  3. Add a Calculator page (ctrl💻 Ctrl doc ▾💻 File menu then 1) and open menu💻 click menu on keypad▸6: Statistics▸1: Stat Calculations▸3: Linear Regression (mx+b). Press enter ↵💻 Enter.
  4. Set X List = o1 and Y List = o2. Press enter ↵💻 Enter.
    1.1*DocRADLinear Regression (mx+b)X List:o1Y List:o2Save RegEqn to:f1Frequency List:1Category List:Include Categories:OKCancel
  5. Read r, the correlation coefficient. (The sign tells the direction; the size tells the strength.)
    1.1*DocRADLinRegMxo1,o2,1:CopyVarstat.RegEqn,f1:stat.results"Title""Linear Regression (mx+b)""RegEqn""m·x+b""m"0.821496"b"5.134"r²"0.907832"r"0.952802"Resid""{…}"
✅ Answer (a) Test–retest. (b) r = 0.953, very close to 1, so the test is reliable. (c) Criterion validity: the strong correlation with the teachers’ ratings (an external standard) suggests the test is valid. (d) Content validity. Learners without rugby knowledge could score low because of missing background, not weak reading. Use passages about everyday topics that all learners know.
Next: Lesson 2 · χ² tests with combined categories
PreviousNext