Designing and measuring for a small subpopulation

What this vignette is for

Research on a subpopulation that is a small share of the general frame runs into three problems at once, and they compound. The group is small, so a general sample yields few of its members. It is often reached through clustered or otherwise non-random designs, so the effective sample is smaller than the nominal one. And the frame may identify the group imperfectly, so even the members who are sampled may not be counted as such.

The functions here address all three. They are written as general survey methodology; the running application is estimation for Indigenous populations in Canada, where all three conditions hold together.

1. How large must the sample be?

The naive calculation is the textbook one for a proportion:

library(rmorie)
morie_sample_size_proportion(p = 0.5, moe = 0.05)
#> $n_srs
#> [1] 384.1459
#> 
#> $n_design
#> [1] 384.1459
#> 
#> $n_invite
#> [1] 384.1459
#> 
#> $p
#> [1] 0.5
#> 
#> $moe
#> [1] 0.05
#> 
#> $conf
#> [1] 0.95
#> 
#> $N
#> [1] Inf
#> 
#> $deff
#> [1] 1
#> 
#> $response_rate
#> [1] 1

which returns about 385. That number is almost never the answer for subpopulation work, because it is the number of subgroup members needed, not the number of interviews. If the group is 5% of the frame and members are selected at the same rate as everyone else, the general sample has to be roughly twenty times larger:

morie_sample_size_domain(p = 0.5, moe = 0.05, domain_prevalence = 0.05)
#> $n_domain
#> [1] 384.1459
#> 
#> $n_domain_invite
#> [1] 384.1459
#> 
#> $n_overall
#> [1] 7682.918
#> 
#> $domain_prevalence
#> [1] 0.05
#> 
#> $coverage
#> [1] 1
#> 
#> $deff
#> [1] 1
#> 
#> $response_rate
#> [1] 1
#> 
#> $moe
#> [1] 0.05
#> 
#> $conf
#> [1] 0.95

The n_overall element is near 7,700. This is the single most common reason that general-population surveys cannot report on small subpopulations: they were never sized to.

Three further factors move the requirement, and all of them move it upward.

morie_sample_size_domain(
  p = 0.5, moe = 0.05,
  domain_prevalence = 0.05,
  deff = 1.8,            # clustering by community and unequal weights
  response_rate = 0.55,  # realistic fieldwork
  coverage = 0.82        # the frame misses some members
)
#> $n_domain
#> [1] 691.4626
#> 
#> $n_domain_invite
#> [1] 1257.205
#> 
#> $n_overall
#> [1] 30663.53
#> 
#> $domain_prevalence
#> [1] 0.05
#> 
#> $coverage
#> [1] 0.82
#> 
#> $deff
#> [1] 1.8
#> 
#> $response_rate
#> [1] 0.55
#> 
#> $moe
#> [1] 0.05
#> 
#> $conf
#> [1] 0.95

The design effect multiplies the requirement, the response rate divides it, and the coverage rate divides it again. deff can be built from its parts: morie_deff_cluster() for the clustering component and morie_design_effect() for the weight component. They multiply.

morie_deff_cluster(m = 25, icc = 0.02)   # 25 interviews per community
#> [1] 1.48
morie_neff_cluster(n = 500, m = 25, icc = 0.02)
#> [1] 337.8378

2. Reaching the group: oversampling and screening

Two designs raise subgroup yield. The first is disproportionate selection: sample the group at a higher rate than the rest of the frame.

morie_oversample_factor(domain_prevalence = 0.05, target_share = 0.20)
#> $factor
#> [1] 4.75
#> 
#> $weight_ratio
#> [1] 0.2105263
#> 
#> $deff_weights
#> [1] 1.140625

factor is how much higher the selection probability must be. Note deff_weights: oversampling is not free. It creates unequal weights, which inflate the design effect, which raises the sample size requirement. Plan the two together rather than in sequence.

The second design is screening, used when the frame does not identify the group at all.

morie_screen_design(
  prevalence = 0.05, target_n = 400,
  cost_screen = 5, cost_interview = 120,
  screen_response = 0.7, interview_response = 0.85
)
#> $n_screen
#> [1] 13445.38
#> 
#> $n_eligible
#> [1] 470.5882
#> 
#> $cost_total
#> [1] 115226.9
#> 
#> $cost_per_completed_interview
#> [1] 288.0672
#> 
#> $screening_share_of_cost
#> [1] 0.5834306

screening_share_of_cost is the quantity to look at when deciding whether screening is viable: for a rare group most of the budget is spent finding people rather than interviewing them.

When strata differ in cost, as remote communities usually do, allocation should account for it:

morie_alloc_optimal(
  N_h = c(8000, 1500, 500), S_h = c(1, 1.4, 2),
  n = 900, cost_h = c(1, 3, 9)
)
#> $n_h
#> [1] 754.26087 114.31159  31.42754
#> 
#> $n_h_int
#> [1] 754 114  32
#> 
#> $share
#> [1] 0.83806764 0.12701288 0.03491948

With equal costs this reduces to Neyman allocation, also available directly as Neyman().

3. Making the sample agree with known totals

Where population totals are published for the group but their joint distribution is not, raking adjusts the weights to match each margin in turn:

df <- data.frame(
  region = c("north", "north", "south", "south"),
  group  = c("a", "b", "a", "b")
)
morie_rake(df, list(region = c(north = 60, south = 40),
                    group  = c(a = 70, b = 30)))
#> $weights
#> [1] 42 18 28 12
#> 
#> $converged
#> [1] TRUE
#> 
#> $iterations
#> [1] 1
#> 
#> $max_discrepancy
#> [1] 0

Always check converged. Raking can fail to converge when margins are inconsistent or a cell is empty, and a non-converged weight vector should not be used.

4. When the identifier is wrong

If group membership is recorded by an administrative identifier rather than by self-identification, the identifier has a sensitivity and a specificity, and neither is 1. Sensitivity below 1 means true members are recorded as non-members. Every rate computed on the recorded counts is then biased, and usually biased downward, because the group’s cases are undercounted while the denominator is not.

morie_misclass_correct(p_obs = 0.04, sensitivity = 0.75,
                       specificity = 0.999, n = 10000)
#> $p_corrected
#> [1] 0.05206943
#> 
#> $p_obs
#> [1] 0.04
#> 
#> $youden
#> [1] 0.749
#> 
#> $ratio
#> [1] 1.301736
#> 
#> $sensitivity
#> [1] 0.75
#> 
#> $specificity
#> [1] 0.999
#> 
#> $se
#> [1] 0.002616277
#> 
#> $ci
#> [1] 0.04694162 0.05719724
#> 
#> $conf
#> [1] 0.95

The correction is the Rogan-Gladen estimator. Applied to counts, it reports how many members the records miss:

morie_misclass_count(observed_count = 400, n = 10000,
                     sensitivity = 0.75, specificity = 0.999)
#> $count_corrected
#> [1] 520.6943
#> 
#> $count_observed
#> [1] 400
#> 
#> $undercount
#> [1] 120.6943
#> 
#> $undercount_pct
#> [1] 23.17949

Two cautions. The correction needs external estimates of sensitivity and specificity, normally from a linkage or validation study; it cannot be obtained from the misclassified data alone. And the interval widens sharply as the classifier degrades, which is itself the finding worth reporting when no validation study exists.

5. Measurement: does the instrument mean the same thing in both groups?

Comparing scale scores across groups presupposes that the scale measures the same construct in each. That is an empirical claim, and it is testable.

Plan the test first. Power for differential item functioning is governed by the smaller group, so a small focal group caps detection regardless of the total sample:

morie_dif_sample_size(p_reference = 0.6, odds_ratio = 1.5)              # equal groups
#> $n_focal
#> [1] 420.0423
#> 
#> $n_reference
#> [1] 420.0423
#> 
#> $n_total
#> [1] 840.0847
#> 
#> $p_focal
#> [1] 0.6923077
#> 
#> $p_reference
#> [1] 0.6
#> 
#> $odds_ratio
#> [1] 1.5
#> 
#> $ratio
#> [1] 1
#> 
#> $power
#> [1] 0.8
#> 
#> $alpha
#> [1] 0.05
morie_dif_sample_size(p_reference = 0.6, odds_ratio = 1.5, ratio = 4)   # focal is a fifth
#> $n_focal
#> [1] 265.5485
#> 
#> $n_reference
#> [1] 1062.194
#> 
#> $n_total
#> [1] 1327.742
#> 
#> $p_focal
#> [1] 0.6923077
#> 
#> $p_reference
#> [1] 0.6
#> 
#> $odds_ratio
#> [1] 1.5
#> 
#> $ratio
#> [1] 4
#> 
#> $power
#> [1] 0.8
#> 
#> $alpha
#> [1] 0.05

An instrument reported as invariant on an underpowered comparison has not been shown to be invariant; it has been shown to be untested. Report the detectable effect size alongside the DIF result.

Fit the DIF models with Difmh() (Mantel-Haenszel) or Difsib() (SIBTEST), then convert the odds ratios to the scale used for operational review:

morie_dif_delta_mh(c(1.0, 1.35, 1.9, 0.5))
#>   or_mh   delta_mh magnitude   favours
#> 1  1.00  0.0000000         A   neither
#> 2  1.35 -0.7052458         A reference
#> 3  1.90 -1.5083566         C reference
#> 4  0.50  1.6288959         C     focal

Magnitude A is negligible, B moderate, C large; favours names the group the item is easier for. For scale-level invariance, compare the nested sequence:

fits <- data.frame(
  model = c("configural", "metric", "scalar"),
  chisq = c(120.3, 128.9, 162.4),
  df    = c(48, 54, 60),
  cfi   = c(0.981, 0.979, 0.964),
  rmsea = c(0.041, 0.040, 0.052)
)
morie_invariance_compare(fits)
#>                   step delta_chisq delta_df      p_value delta_cfi delta_rmsea
#> 1 configural -> metric         8.6        6 1.973547e-01    -0.002      -0.001
#> 2     metric -> scalar        33.5        6 8.400602e-06    -0.015       0.012
#>   supported
#> 1      TRUE
#> 2     FALSE

Metric invariance holds here and scalar invariance does not, which means factor loadings are comparable but intercepts are not: group means on this scale are not comparable, though associations may be. Reporting a mean difference in that situation describes the instrument, not the groups.

Finally, report precision by group as well. A scale can be reliable in one group and not in another, and pooling conceals it:

morie_irt_theta_se(information = c(4, 9, 16))
#> [1] 0.5000000 0.3333333 0.2500000
morie_irt_marginal_reliability(se = c(0.3, 0.35, 0.4, 0.5))
#> [1] 0.844375

6. Governance is part of the method

For research with Indigenous communities in Canada, the design questions above sit inside a governance framework that is not optional and is not supplied by any software package. First Nations data governance is articulated through the OCAP principles – ownership, control, access and possession – which are a registered trademark of the First Nations Information Governance Centre and are set out by that body; Inuit and Metis governance frameworks are distinct and are articulated by their own representative organisations. A distinctions-based approach means First Nations, Inuit and Metis data are not aggregated into a single category without a reason that the relevant nations accept.

Practical consequences for the calculations above: who holds the data and who decides what is published are determined before the sample is drawn, not after the analysis; the community, not only the researcher, has a stake in whether a subgroup estimate is precise enough to publish; and a non-converged raking or an undercounted identifier is a reporting obligation, not a technical footnote.

None of this is enforced by the functions in this package. It is stated here because a design that is statistically sound and governance-blind is not usable for this work.

Summary