library(psych)
data(bfi, package = "psych")
X <- na.omit(bfi[, c("A1", "A2", "A3", "A4", "A5")]) # A1 is reverse-keyed, 1-6 scale
X$A1 <- (1 + 6) - X$A1 # reflect about (min + max)
a <- psych::alpha(X, check.keys = FALSE)
a$total$raw_alpha; a$total$std.alpha # Cronbach's alpha
a$feldt # 95% Feldt confidence interval
a$item.stats$r.drop # corrected item-total correlations
a$alpha.drop$raw_alpha # alpha if item dropped
sd(rowSums(X)) * sqrt(1 - a$total$raw_alpha) # standard error of measurement
Computational Provenance & Reproducibility Record Reliability and Validity · 2.0.0 · DOI 10.5281/zenodo.22943081
This Computational Provenance Record documents the statistical computing environment, software dependencies, computational provenance, and bibliographic references associated with the RAISINS Reliability and Validity module. It is intended to support computational reproducibility and software transparency. Detailed statistical methodology, mathematical derivations, and user guidance are provided separately in the official module documentation.
Any issues or updates required please comment here
Go to the App from here
1 Module Metadata
| Parameter | Specification |
|---|---|
| Module | Reliability and Validity |
| Module Version | 2.0.0 |
| DOI | 10.5281/zenodo.22943081 |
| Document Type | Computational workflow |
| Statistical Engine | R |
| R Version | 4.5.2 |
| Reproducibility | Execution Environment: Posit Connect · GCR · renv-locked |
2 Statistical Dependency Manifest
| Package | Version | Repository | Core Statistical Functions |
|---|---|---|---|
| psych | 2.6.5 |
CRAN | alpha(), omega(), splitHalf(), tenberge(), ICC(), polychoric(), KMO(), cortest.bartlett(), fa.parallel(), fa() |
| irr | 0.84.1 |
CRAN | kappam.fleiss(), kripp.alpha(), kendall(), agree() |
| lavaan | 0.6-21 |
CRAN | cfa(), fitMeasures(), standardizedSolution() |
| GPArotation | 2026.4-1 |
CRAN | Oblique and orthogonal rotation criteria used by psych::fa() |
| Hmisc | 5.2-5 |
CRAN | rcorr() (correlation matrix with p-values) |
| stats | 4.5.2 |
Base R | cor(), qbinom(), pbinom(), qnorm() |
3 Statistical Function Registry
| Analytical Role | Primary Function(s) |
|---|---|
| Reverse scoring | (scalemin + scalemax) − x for user-selected items |
| Cronbach’s alpha, item statistics | psych::alpha(check.keys = FALSE) |
| Confidence interval for alpha | Feldt interval from psych::alpha(); optional bootstrap via psych::alpha(n.iter = 500) |
| Split-half and lower bounds | psych::splitHalf(), psych::tenberge() |
| McDonald’s omega (unidimensional) | Loadings from psych::fa(nfactors = 1) |
| Omega hierarchical, bifactor omega, ECV | psych::omega() |
| Standard error of measurement | $SD\sqrt{1-\alpha}$ |
| Inter-rater reliability | irr::kappam.fleiss(), psych::ICC(), irr::kripp.alpha(), irr::kendall(), irr::agree() |
| Content validity (CVR, I-CVI, S-CVI, modified kappa) | Closed-form formulae with stats::qbinom() / stats::pbinom() |
| Correlation matrix | stats::cor(), Hmisc::rcorr(), psych::polychoric() |
| Factorability | psych::KMO(), psych::cortest.bartlett() |
| Number of factors | psych::fa.parallel() |
| Exploratory factor analysis | psych::fa() |
| Confirmatory factor analysis | lavaan::cfa(), lavaan::fitMeasures(), lavaan::standardizedSolution() |
| Convergent validity (AVE, CR) | Computed from the standardised CFA loadings |
4 Default Methods & Parameters
| Analysis | Step / Parameter | Default Method / Value |
|---|---|---|
| Data | Layout | Numeric item columns. Rows are respondents for Internal Consistency, Explore and Construct Validity; rows are raters or experts for Inter-rater and Content Validity |
| Reverse scoring | Applied only to items selected by the user, reflected about (scale minimum + scale maximum). Items are never reversed automatically; items still correlating negatively are flagged | |
| Internal Consistency |
Alpha | psych::alpha(check.keys = FALSE), so the scale is analysed exactly as reverse-scored by the user |
| Confidence interval | 95% Feldt interval (Feldt, Woodruff & Salih, 1987). Optional percentile bootstrap interval, 500 resamples | |
| Omega | Unidimensional omega from a one-factor minres solution; omega hierarchical and bifactor omega from psych::omega() when there are 6 or more items |
|
| Inter-rater Reliability |
Coefficients | Fleiss' kappa (with exact kappa), all six Shrout & Fleiss ICC forms, Kendall's W, exact and ±1-category percentage agreement |
| Krippendorff's alpha | method = "ordinal"; nominal, interval and ratio available |
|
| Content Validity |
Essential rating(s) | User-selected level(s) of the response scale |
| Critical CVR | Exact one-sided binomial critical value at α = 0.05 (0.01 and 0.10 available). May be replaced by a user-set threshold within [−1, +1] | |
| Explore | Correlation | Pearson; Spearman and polychoric available |
| Number of factors | Parallel analysis (psych::fa.parallel()); user-adjustable |
|
| Extraction | fm = "minres"; principal axis, maximum likelihood and weighted least squares available |
|
| Rotation | oblimin for 2 or more factors, none for a single factor; promax, varimax and quartimax available |
|
| Construct Validity |
Model | One-factor CFA, lavaan::cfa(estimator = "MLR"); robust (scaled) fit indices are reported. ML and WLSMV (items treated as ordered) available |
| Convergent validity | AVE = mean(λ²); CR = (Σλ)² / [(Σλ)² + Σ(1 − λ²)]. Improper solutions (Heywood cases) are detected and flagged | |
| Reproducibility | Random seed | A fixed seed is used for parallel analysis, split-half sampling and the bootstrap, so repeated runs on the same data give identical results |
5 R Code for Key Analytical Steps
The code blocks below demonstrate the computation behind each reported result using datasets supplied with the psych and irr packages.
5.1 Reverse Scoring and Internal Consistency
5.2 McDonald’s Omega and Lower Bounds
f1 <- psych::fa(X, nfactors = 1, fm = "minres", rotate = "none")
lam <- as.numeric(f1$loadings[, 1])
sum(lam)^2 / (sum(lam)^2 + sum(1 - lam^2)) # unidimensional omega
psych::splitHalf(X, check.keys = FALSE) # split-half reliabilities, lambda 2 and 6
psych::tenberge(X) # Ten Berge & Zegers lower bounds5.3 Inter-Rater Reliability
library(irr)
data(anxiety, package = "irr") # 20 subjects (rows) x 3 raters (columns)
irr::kappam.fleiss(anxiety) # Fleiss' kappa
psych::ICC(anxiety, lmer = FALSE)$results # six Shrout & Fleiss ICC forms
irr::kripp.alpha(t(as.matrix(anxiety)), method = "ordinal")
irr::kendall(anxiety, correct = TRUE) # Kendall's W
irr::agree(anxiety) # percentage agreement5.4 Content Validity (Lawshe’s CVR)
N <- 10; n_essential <- 8 # 10 experts, 8 rate the item essential
cvr <- (n_essential - N / 2) / (N / 2) # content validity ratio
icvi <- n_essential / N # item-level CVI
pval <- pbinom(n_essential - 1, N, 0.5, lower.tail = FALSE) # exact one-sided p-value
# critical CVR: smallest essential count significant at alpha = 0.05
n_crit <- min(qbinom(0.05, N, 0.5, lower.tail = FALSE) + 1, N)
2 * n_crit / N - 15.5 Factorability and Exploratory Factor Analysis
Xall <- na.omit(bfi[, 1:25])
R <- cor(Xall)
psych::KMO(R)$MSA # Kaiser-Meyer-Olkin
psych::cortest.bartlett(R, n = nrow(Xall)) # Bartlett's test of sphericity
psych::fa.parallel(Xall, fa = "both", plot = FALSE)$nfact
psych::fa(Xall, nfactors = 5, fm = "minres", rotate = "oblimin")5.6 Confirmatory Factor Analysis and Convergent Validity
library(lavaan)
fit <- lavaan::cfa("Agree =~ A1 + A2 + A3 + A4 + A5", data = X, estimator = "MLR")
lavaan::fitMeasures(fit, c("chisq.scaled", "df.scaled", "pvalue.scaled",
"cfi.robust", "tli.robust", "rmsea.robust", "srmr"))
std <- lavaan::standardizedSolution(fit)
lam <- std$est.std[std$op == "=~"]
mean(lam^2) # AVE
sum(lam)^2 / (sum(lam)^2 + sum(1 - lam^2)) # composite reliabilityExplore the entire Reliability and Validity module in preview mode using our demo datasets. To submit suggestions or report a workflow issue, please use the discussion section below, or visit the official RAISINS website.
6 RAISINS Native Statistical Framework
RAISINS uses R for all its statistical computations. Every package used to generate major results is listed and demonstrated with examples, so results can be reproduced independently. These results are then organized and formatted on the RAISINS website along with visualisation to make them easier to use and interpret. RAISINS also has its own custom-built statistical tools for managing workflows, validating results, and generating reports. Details of these are not fully covered here, they’re shared with outside researchers only on request, and are subject to licensing terms.
7 Package References
R Core Team. (2025). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/
Revelle, W. (2025). psych: Procedures for Psychological, Psychometric, and Personality Research (R package version 2.6.5). Northwestern University, Evanston, Illinois. https://doi.org/10.32614/CRAN.package.psych
Gamer, M., Lemon, J., Fellows, I., & Singh, P. (2019). irr: Various Coefficients of Interrater Reliability and Agreement (R package version 0.84.1). https://doi.org/10.32614/CRAN.package.irr
Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02
Bernaards, C. A., & Jennrich, R. I. (2005). Gradient Projection Algorithms and Software for Arbitrary Rotation Criteria in Factor Analysis. Educational and Psychological Measurement, 65(5), 676-696. https://doi.org/10.1177/0013164404272507
Harrell, F. E. (2025). Hmisc: Harrell Miscellaneous (R package version 5.2-5). https://doi.org/10.32614/CRAN.package.Hmisc
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555
McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum Associates.
Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11(1), 93-103. https://doi.org/10.1177/014662168701100107
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. https://doi.org/10.1037/0033-2909.86.2.420
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. https://doi.org/10.1037/h0031619
Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Sage Publications.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563-575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459-467. https://doi.org/10.1002/nur.20199
Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31-36. https://doi.org/10.1007/BF02291575
Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179-185. https://doi.org/10.1007/BF02289447
Allaire, J. J., Xie, Y., Dervieux, C., McPherson, J., Luraschi, J., Ushey, K., Atkins, A., Wickham, H., Cheng, J., Chang, W., & Iannone, R. (2026). rmarkdown: Dynamic Documents for R (R package version 2.31). https://github.com/rstudio/rmarkdown