Fall Study Reliability Tool
Automated inter-rater reliability assessment for clinical fall-risk research. Flags unreliable assessment items before study finalization so researchers can fix them rather than discover them in publication.
The hidden cost of reliability discovery
In fall-risk research, multiple clinicians watch the same video recordings and independently rate patient movement patterns. Agreement between raters is critical for study validity. It determines whether the assessment tool is trustworthy and publishable.
The problem: inter-rater reliability calculations happen after data collection and analysis are complete. If Krippendorff's Alpha drops below 0.667 (minimum acceptable), or you can't reach 0.80 (target), it's too late. The items are already baked into your study design. Revising the instrument mid-way through means recoding hundreds of responses or scrapping months of work.
Manual assessment
Manual calculation or basic Excel formulas. Reliability computed after all data collected. Low-confidence items discovered too late to fix. No real-time visibility into which assessment items are problematic.
Automated pipeline
Qualtrics exports flow into an automated Python pipeline. Krippendorff's Alpha computed per item, per group. Unreliable items flagged immediately. Researchers can refine the instrument, recalibrate raters, or remove problem items before finalizing the study.
How it works
The tool ingests survey responses from Qualtrics, transforms them into a format suitable for reliability analysis, computes inter-rater agreement metrics, and produces a dashboard report highlighting problem areas.
Key design decisions
Qualtrics as the single source of truth. Rather than build a custom form system, we export directly from the research team's existing survey platform. No double-entry, no manual transcription. The pipeline pulls new responses and recomputes automatically.
Krippendorff's Alpha for ordinal scale agreement. Fall-risk assessments use Likert scales, 1 through 5 ordinal ratings. Krippendorff's Alpha handles ordinal data correctly, accounts for missing values, and works with any number of raters. It's the gold standard for inter-rater reliability in clinical research.
Item-level flagging, not global thresholds. A single global reliability score hides the real problem. Some items agree well, others don't. The tool flags each item independently so researchers know exactly which assessment question is unreliable and needs refinement.
Failure recovery and data integrity
If the analysis crashes mid-computation, tested by force-killing the process at random intervals, the SQLite database preserves all prior state. Re-running the analysis picks up exactly where it left off with no data loss. For large datasets (tested up to 2,000 responses across 10 raters), the tool displays a progress indicator with estimated time remaining and supports cancellation without corrupting the database. All computations are atomic: either the full reliability matrix completes, or nothing changes.
Research foundation
Inter-rater reliability is non-negotiable in clinical research. These metrics and standards ensure that assessment tools are trustworthy before they're used to evaluate patient outcomes.
Krippendorff's Alpha
Measures agreement for ordinal and nominal data. Minimum 0.667 for exploratory research, 0.80+ for regulatory and clinical use.
Intraclass Correlation (ICC)
Assesses consistency of continuous ratings. Complements Krippendorff's for interval-scale assessments.
Rater Calibration
Multiple clinicians must be trained to apply the same standards. Low reliability often signals training gaps, not flawed items.
COSMIN Guidelines
Standards for evaluating the quality of outcome measurement instruments in medical research and practice.
Reliability benchmarks
These thresholds guide the analysis and flag items that fall short of clinical research standards.
Built with
Why these choices?
Python + Pandas. Fast data manipulation and transformation. The Qualtrics CSV export can be messy with nested columns and encoded values. Pandas cleans it up reliably.
SQLite for persistence. The database outlives any single run. Researchers can query historical reliability scores, track how items improve after recalibration, and audit the analysis pipeline.
Krippendorff's Alpha computation. Implemented from first principles so every step is auditable and transparent. No black-box calculation. Researchers can verify the math and cite the methodology.
Field deployment readiness
Zero-friction deployment
The tool runs entirely within a researcher's existing infrastructure. No IT involvement required. Researchers export their Qualtrics data as CSV, drag it into the tool, and receive reliability metrics in under 2 seconds. During pilot use across 3 Stevens research projects, zero participants needed live support after completing the 10-minute onboarding video. The most common question was "How do I interpret low alpha for a single item?" This prompted adding an inline tooltip that shows interpretation guidelines directly in the dashboard output.
What I learned
Data quality is upstream of analysis. The first version failed silently on malformed ratings, out-of-range values, missing data, and misaligned columns. Now every input gets validated hard. Garbage in equals garbage out, so I catch it before the Krippendorff's computation.
Reliability is a conversation, not a report. Researchers don't just need numbers. They need to understand which items failed and why. Low alpha might mean the item is poorly worded, the raters need more training, or the construct itself is ambiguous. A good dashboard flags the item and prompts the right questions.
Researchers don't have 3 weeks to wait for reliability feedback. They have IRB deadlines, grant reporting cycles, and data collection windows that close whether the instrument is ready or not. One Stevens research team discovered their fall risk assessment had problematic inter-rater agreement (alpha = 0.58) only after collecting 200 responses. Without the tool, they would have spent 3 weeks manually recoding and retraining raters. With it, they identified the 3 problematic items in under a minute, revised the wording in one afternoon, and were back on schedule the next day.
Post-Mortem
What I got wrong.
This tool worked for the research team that used it. But I made decisions early on that limited how far it could go.
When Krippendorff's Alpha came back low for an item, I flagged it as "unreliable" and recommended revision. That's the obvious interpretation. But after talking with the research team, I learned that some items have legitimately ambiguous cases where reasonable clinicians disagree. A patient who scores 2 or 3 on a 5-point fall risk scale might genuinely be on the boundary. Low alpha for those items doesn't mean the item is broken. It means the construct is fuzzy at that range. The tool should have distinguished between "raters disagree because the item is poorly worded" and "raters disagree because the patient is genuinely borderline." I treated all disagreement the same, and that oversimplification led to recommendations that would have removed clinically useful items.
The ETL layer expects Qualtrics CSV exports with specific column naming conventions. When a second Stevens research group wanted to use the tool with REDCap data, I couldn't accommodate them without rewriting the data ingestion. I'd hardcoded the Qualtrics format assumptions throughout the pipeline instead of building an abstraction layer that could accept different input schemas. The fix would have been straightforward: define a canonical internal format and write adapters per source system. I didn't do that because I was building for one team and one deadline. That shortcut means every new data source requires code changes instead of configuration.
The tool computes reliability on a snapshot of data. Run it once, get a report. But the research team's actual workflow is iterative: they collect responses, check reliability, revise items, retrain raters, and collect more responses. They wanted to see how reliability changed over time for each item. The SQLite database stores everything, so the data is there, but I never built the comparison view. Showing "alpha was 0.58 last month and is 0.74 now after rater retraining" would have been the most valuable feature for the team's workflow, and I didn't build it because I was thinking about the tool as a one-shot analysis, not a longitudinal instrument.
Questions You Might Ask
Answers before the interview.
If I were screening this portfolio, these are the three questions I'd ask. So here they are, answered.
Cohen's Kappa only works for two raters and nominal data. This study had variable numbers of raters (2 to 4 per item) and ordinal scales (1 through 5 Likert). ICC handles continuous data well but doesn't account for ordinal structure correctly. Krippendorff's Alpha handles any number of raters, works with ordinal data, and accounts for missing values gracefully. It's also the metric recommended by COSMIN guidelines for evaluating outcome measurement instruments, which is exactly what this research was doing. The choice wasn't arbitrary. It was the statistically appropriate metric for the data structure.
For a clinical research tool used internally by a research team, regulatory submission isn't the relevant bar. But the methodology would hold up to scrutiny. Krippendorff's Alpha is widely cited in clinical research literature, the computation is implemented from first principles with full auditability, and the SQLite database provides a complete audit trail of every input and output. If this were part of a clinical investigation used to support a device submission, the tool's output would need to be validated per 21 CFR Part 11 (electronic records) and the computation would need formal software verification. The current implementation doesn't meet those standards, but the architecture was designed with that upgrade path in mind.
The longitudinal comparison view is the clear first priority. Researchers need to see how reliability changes over time as they revise items and retrain raters. The data already exists in the database. It just needs a front-end. Second, I'd build adapters for REDCap and Google Forms, since those are the other two platforms Stevens research teams commonly use. Third, I'd add a rater performance view that shows which specific rater is the outlier on low-agreement items. Right now the tool says "this item has low reliability" but not "Rater 3 consistently disagrees with Raters 1, 2, and 4 on this item." That distinction matters for targeted rater retraining instead of retraining everyone.
Interested in this work?
I'm looking for roles where I can bring this kind of thinking. Clinical rigor, automated data pipelines, and research collaboration matter in validation and quality teams in medtech.