AGENT SKILL

experimental-design-psych

Designing psychology experiments — variables, controls, randomization, counterbalancing, and preregistration.

License 许可
MIT
所属仓库
TanNhatCMS/jarvis
最近更新
2026-11-07
Stars
★ 0
Forks
1

01 前言 Front Matter

name
experimental-design-psych
category
scientific
description
Designing psychology experiments — variables, controls, randomization, counterbalancing, and preregistration.

03 SKILL.md 正文

Overview

experimental-design-psych covers how to design behavioral experiments that can actually answer the question asked: defining constructs operationally, choosing between- vs within-subjects designs, controlling confounds, counterbalancing order effects, and preregistering the plan before data collection. Good design is cheaper than any statistical rescue — no analysis fixes a confounded experiment.

When to use

  • Turning a research question into a testable design: IVs, DVs, operational definitions.
  • Choosing between-subjects, within-subjects, or mixed designs.
  • Counterbalancing: Latin squares, ABBA, randomization for order/fatigue/practice effects.
  • Controls: placebo, active control, yoked control, sham conditions.
  • Power analysis and sample-size planning for behavioral studies.
  • Preregistration: hypotheses, exclusion criteria, analysis plan.
  • Piloting: what to check before running the full sample.

Core concepts

  • Operational definitions. A construct ("attention," "anxiety") must become a measurable procedure (reaction time on incongruent Stroop trials; STAI score). Vague constructs produce uninterpretable results — define the measure before defending the claim.
  • Between vs within. Between-subjects avoids carryover and demand effects but needs larger samples and risks group nonequivalence; within-subjects is powerful (each participant is their own control) but vulnerable to order, practice, fatigue, and asymmetric carryover (a treatment that permanently changes participants cannot be within-subjects).
  • Counterbalancing. Full counterbalancing (all orders) for small designs; Latin squares when orders multiply; ABBA for simple two-condition orders. Random order per participant is often sufficient with enough trials — but verify balance afterward.
  • Controls that actually control. A proper control differs from the experimental condition in exactly one thing. Active controls (alternative intervention) beat passive/waitlist controls, which confound treatment with attention, expectancy, and contact. Demand characteristics — participants guessing the hypothesis — are controlled by cover stories and manipulation checks, not by hoping.
  • Randomization. Random assignment (not haphazard, not alternation) is what justifies causal claims in experiments. Randomize with a documented procedure (computer-generated sequence), and conceal allocation where feasible.
  • Blinding. Participants blind to condition where possible; experimenters blind to hypotheses during data collection and to condition during coding/scoring. Unblinded experimenters leak expectations into behavior — the experimenter-expectancy effect is real and large.
  • Power. Base sample size on the smallest effect size of interest, not on optimistic pilot estimates (pilots overestimate). Within-subjects designs need fewer participants; account for planned exclusions by oversampling 10-20%.
  • Preregistration. Hypotheses, primary DV, exclusion rules, sample size, and analysis plan registered before data collection (OSF, AsPredicted). This separates confirmatory from exploratory work — the single highest-ROI practice in the field.

Practical workflow

  1. Question → hypothesis → operational definitions. Write the exact statistical test that would support each hypothesis. If you can't, the design isn't ready.
  2. Choose the design. Within-subjects unless carryover forbids it; mixed when both between- and within-factors matter.
  3. Build controls. List every alternative explanation; design each condition to rule out one. Include manipulation checks.
  4. Power analysis. Smallest effect of interest, alpha 0.05, power 0.80; add exclusion buffer.
  5. Preregister. Hypotheses, DVs, exclusions, stopping rule, full analysis plan.
  6. Pilot. n=5-10: check timing, comprehension, ceiling/floor effects, equipment, and that the manipulation actually manipulates (manipulation check data).
  7. Run blind. Experimenter scripts, automated stimulus delivery, blinded scoring. Log protocol deviations as they happen.
  8. Analyze as preregistered. Report deviations transparently; label unplanned analyses exploratory.

Common pitfalls

  • Confounded conditions differing in more than the intended manipulation.
  • Within-subjects designs with asymmetric carryover (e.g. learning that can't be unlearned).
  • Underpowered studies built on inflated pilot effect sizes.
  • No manipulation check — not knowing whether the IV actually varied.
  • Experimenter unblinded, leaking expectations (especially in clinical/developmental work).
  • Exclusion criteria invented after seeing the data.
  • HARKing: presenting exploratory findings as a priori hypotheses.

04 同仓库更多 Skills

TanNhatCMS
TanNhatCMS/jarvis

web-scrape

Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

design-taste-frontend

tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

test-driven-development

Use when implementing any feature or bugfix, before writing implementation code

★ 0
TanNhatCMS
TanNhatCMS/jarvis

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

★ 0
TanNhatCMS
TanNhatCMS/jarvis

requesting-code-review

Use when completing tasks, implementing major features, or before merging to verify work meets requirements

★ 0
TanNhatCMS
TanNhatCMS/jarvis

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).

★ 0