The project aims to address the need for design-comparable effect size measures in single-case experimental designs (SCEDs). It recognizes the limitations of existing measures like between-case standardized mean difference (BC-SMD) and the absence of suitable design-comparable effect sizes for count outcomes in SCEDs. The primary objective is to develop and evaluate methods for estimating between-case incidence rate ratio (BC-IRR) effect sizes for count data in SCEDs. Additionally, the project seeks to establish domain-specific IRR benchmarks through meta-analyses of SCED studies. It also aims to create user-friendly tools, including an R package and a web-based Shiny application. These tools will improve accessibility and simplify effect size estimation for SCED researchers.

We will investigate the performance of the proposed BC-IRR method using a comprehensive Monte Carlo simulation study. We will generate count data based on a multiple-baseline design. During data generation, we will consider several factors, including series length, number of cases, baseline trend, treatment effect, between-case variance, and overdispersion. Next, we will estimate a BC-IRR for each dataset using both the generalized linear mixed model (GLMM) and the generalized estimating equation (GEE) approach. We will then conduct statistical inferences using a Wald test. Finally, we will evaluate estimator performance using bias, mean squared error (MSE), coverage rates of the 95% confidence interval, and empirical Type I error rates.

To develop the IRR benchmarks, we will adopt an “across-studies comparison” approach. This method establishes a distribution of empirical effect sizes through meta-analysis. We will focus on two common outcome domains in SCEDs: reading and behavior. For the behavioral outcome domain, we will further distinguish between behaviors with positive and negative valence. In total, three empirical distributions of IRR effect sizes will be established, and each will consist of 200 effect size estimates.

We will search electronic databases and journals for relevant articles. We will screen studies using predefined inclusion and exclusion criteria. Included articles will be coded in terms of design type, WWC standards assessment, valence, and outcome type. The Web Plot Digitizer will be used for data extraction.

For each study, we will estimate a within-case IRR and a between-case IRR effect size for the immediate treatment effect adjusting for baseline trend. After obtaining the effect size distributions, we will calculate benchmark ranges. These ranges will be based on the quantiles of the empirical distributions. (e.g., bottom 25%, 25-50%, 50-75%, and top 25%) to serve as the empirical benchmarks.

The project is expected to yield three types of products. First, research articles will be authored to introduce the BC-IRR effect size, evaluate its performance, and establish benchmarks. These articles will demonstrate its performances in terms of the accuracy of parameter estimates and statistical inferences, and provide empirical effect size benchmarks that can aid in the interpretation of results in two specific outcome domains (i.e., reading and behavior). Second, as a key deliverable, both an R package and a user-friendly web-based calculator will be developed to ensure accessibility to these methodologies among SCED researchers. Third, the project will produce comprehensive step-by-step tutorials and virtual workshops. These tutorials and workshops will offer clear and detailed instructions on how to effectively utilize the BC-IRR method and the accompanying software tools.

Luo, W., Li, H., Baek, E., & Li, C. (2025, October 24). Between-case incidence rate ratio: A design-comparable effect size for count outcomes in single case experimental designs [Preprint]. OSF Preprints. https://doi.org/10.31219/osf.io/9gdkn_v2

Li, H., Li, C., Luo, W., & Baek, E. (2025, October 17). Between-case incidence rate ratio for single case experimental designs with count outcomes: A Monte Carlo simulation with conditional and marginal models [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/rx96g_v1