AGENT SKILL

heart-sense

rPPG heart rate estimation for the Rabbit R1. The user places a fingertip over the camera lens for 30 seconds; the skill extracts a blood-volume pulse signal from the red channel, applies peak detection or FFT, and returns estimated BPM with a confidence score.

License 许可
MIT
最近更新
2026-09-04
Stars
★ 0
Forks
0

01 前言 Front Matter

name
heart-sense
author
rabbit-r1-skillz
version
1.0.0
metadata
{ "openclaw": { "requires": { "npm": [ "sharp", "@opencv-tsjs/opencv4nodejs" ], "bins": [ "node" ] }, "primaryEnv": "HEART_SENSE_ENABLED" } }
description
rPPG heart rate estimation for the Rabbit R1. The user places a fingertip over the camera lens for 30 seconds; the skill extracts a blood-volume pulse signal from the red channel, applies peak detection or FFT, and returns estimated BPM with a confidence score.

02 资源文件 3

03 SKILL.md 正文

HeartSense — rPPG Heart Rate Skill

HeartSense estimates heart rate without any wearable. The user covers the R1's camera lens with a fingertip. The skill captures ~30 seconds of video frames, extracts the subtle red-channel brightness changes caused by blood pulsing through capillaries (rPPG — remote photoplethysmography), and converts that signal to beats per minute.

The R1 Creation (heart-sense/index.html) handles the UI — countdown ring, live pulse-wave visualization, BPM display, morning-baseline comparison, and a guided HRV breathing-sync session. This OpenClaw skill provides the compute: camera capture and signal processing.


Commands

Command Params Returns Description
capture { durationS: number, fps: number, ts: number } { bpm, confidence, signal_quality, raw } Capture and analyze a rPPG session

Prerequisites

  • Node.js 18+ — the skill runs as a Node.js process under the OpenClaw daemon.
  • OpenClaw daemon — must be running on the host machine paired with your R1. The Creation communicates with this skill via the shared-storage pub/sub bridge (openclaw:cmd / openclaw:resp).
  • CPU-only signal processing — no GPU required. rPPG peak detection and FFT run on CPU. The sharp and @opencv-tsjs/opencv4nodejs packages handle frame decoding and channel extraction; no CUDA or GPU driver is needed.
  • Camera access — OpenClaw must have permission to access the R1's rear camera stream. On RabbitOS 2 this is granted when the user opens the Creation and the daemon receives the capture command.

Installation

1. Place the skill

cp -r heart-sense/ ~/.openclaw/skills/heart-sense/

Or install via ClawHub when available:

clawhub install heart-sense

2. Install Node.js dependencies

cd ~/.openclaw/skills/heart-sense
npm install sharp @opencv-tsjs/opencv4nodejs

sharp is used for fast frame-buffer decoding (JPEG/raw → pixel array). @opencv-tsjs/opencv4nodejs provides the OpenCV bindings used for red-channel extraction, Gaussian smoothing, and optional cascade-based motion rejection.

3. Set the enable flag

Add to your OpenClaw environment (~/.openclaw/.env or system env):

HEART_SENSE_ENABLED=true

Optional tuning variables:

# Minimum frames required before returning a result (default: 600 = 30s × 30fps)
HEART_SENSE_MIN_FRAMES=600

# FFT window size — must be power of 2 (default: 256)
HEART_SENSE_FFT_WINDOW=256

# BPM search range (default: 40–180)
HEART_SENSE_BPM_MIN=40
HEART_SENSE_BPM_MAX=180

# Signal quality threshold below which the skill falls back to peak detection (0–1, default: 0.4)
HEART_SENSE_QUALITY_THRESHOLD=0.4

4. Verify the skill

node ~/.openclaw/skills/heart-sense/index.js --test

Expected output is a JSON object with keys bpm, confidence, signal_quality, and raw.


How rPPG Works

Remote photoplethysmography (rPPG) measures pulse rate from a camera without contact sensors. Here is what happens step by step:

  1. Illumination. When light (ambient or camera flash) hits skin, the red channel of the camera image is most sensitive to blood-volume changes. Oxygenated blood absorbs green and blue light more than red, so the red channel fluctuates as each heartbeat pushes a bolus of blood through capillaries.

  2. Frame capture. The skill requests 30 seconds of camera frames at 30 fps (900 frames). For a fingertip measurement, the R1 rear camera is ideal: the lens is small, the flashlight is close, and the fingertip blocks ambient-light noise entirely.

  3. Red-channel extraction. Each frame is decoded to an RGB array. The mean red-channel value across all pixels in the frame is computed. This produces a 1-D time series of 900 samples — the blood-volume pulse (BVP) waveform.

  4. Detrending and smoothing. A linear trend is subtracted (removes slow brightness drift from movement or lighting change). A Gaussian smoothing kernel removes high-frequency noise above ~4 Hz (240 BPM, the physiological ceiling).

  5. BPM estimation — two methods.

    • FFT (primary). A Fast Fourier Transform converts the 900-sample signal to the frequency domain. The dominant frequency peak in the 0.67–3 Hz band (40–180 BPM) is the estimated heart rate.
    • Peak detection (fallback). If signal quality is below threshold (motion, poor contact, bad lighting), the skill falls back to counting the R-peaks (systolic peaks) in the smoothed waveform and computing inter-peak intervals. This is more robust to short-duration or noisy signals.
  6. Confidence scoring. The ratio of the dominant-frequency peak power to total band power is the confidence score (0–1). High confidence (>0.7) means a clean, single-dominant frequency. Low confidence means motion, inconsistent contact, or extreme lighting.

  7. Response. The skill writes the result to openclaw:resp. The R1 Creation reads it via R1.ocReceive() and displays the BPM.


Command: capture

What the Creation sends

R1.ocSend('heart-sense', 'capture', {
  durationS: 30,   // MEASURE_DURATION_S — seconds of video to capture
  fps:       30,   // target frame rate
  ts:        Date.now(),  // request timestamp (epoch ms)
});

What the skill does, step by step

  1. Receives the command from openclaw:cmd.
  2. Opens the R1 camera stream and begins capturing frames at fps for durationS seconds.
  3. For each frame: decodes to RGB, extracts the mean red-channel value, appends to the BVP buffer.
  4. (Optional) Streams intermediate rppgFrame plugin messages to the Creation for live display: { type: "rppgFrame", bpm: number, quality: number } — the Creation renders these on the pulse-wave visualization while the user waits.
  5. After durationS seconds (or when the buffer has enough frames):
    • Detrends and smooths the BVP buffer.
    • Runs FFT; if confidence < HEART_SENSE_QUALITY_THRESHOLD, runs peak detection instead.
    • Computes bpm, confidence, and signal_quality.
  6. Writes the response to openclaw:resp.

ocReceive timeout

The Creation waits up to 45,000 ms (45 seconds) for the response. The extra 15 seconds beyond the 30-second capture window allows for signal processing time, especially on older host hardware.


Response Shape

OpenClaw writes to openclaw:resp with this structure:

{
  "status": "done",
  "ts": 1718000000000,
  "data": {
    "bpm": 72,
    "confidence": 0.84,
    "signal_quality": 0.76,
    "raw": {
      "method": "fft",
      "dominant_hz": 1.2,
      "peak_power_ratio": 0.84,
      "frame_count": 897,
      "duration_ms": 29900,
      "bvp_mean": 187.4,
      "bvp_std": 3.2
    }
  }
}

Field definitions

Field Type Range Description
bpm number 40–180 Estimated heart rate in beats per minute (integer)
confidence number 0–1 Ratio of dominant-frequency peak power to total BVP band power
signal_quality number 0–1 Overall quality score combining confidence + frame drop rate + motion rejection
raw.method string — "fft" or "peak_detection" — which algorithm produced the result
raw.dominant_hz number — The dominant frequency found (Hz); multiply by 60 to get BPM
raw.peak_power_ratio number 0–1 FFT peak power / total power in band (same as confidence when FFT is used)
raw.frame_count number — Frames actually captured (may be less than durationS × fps due to drops)
raw.duration_ms number — Actual capture duration in ms
raw.bvp_mean number — Mean red-channel value across the session (useful for lighting diagnostics)
raw.bvp_std number — Standard deviation of the BVP signal

Error response

{
  "status": "error",
  "ts": 1718000000000,
  "error": "camera_unavailable",
  "message": "Could not open camera stream. Check OpenClaw camera permissions."
}

Possible error values: camera_unavailable, too_few_frames, signal_flat, timeout.

How the Creation uses the response

const resp  = await R1.ocReceive({ timeoutMs: 45000 });
const d     = resp.data || {};
const bpm     = typeof d.bpm     === 'number' ? Math.round(d.bpm)                          : null;
const quality = typeof d.quality === 'number' ? Math.round(R1.clamp(d.quality * 5, 0, 5)) : null;

If the skill returns an error or the receive times out, the Creation falls back to a physiologically plausible estimate anchored to the morning baseline (or 72 BPM if none exists), with a small random variance. This fallback is always labeled as an estimate and is never stored as a high-confidence reading.


Accuracy Disclaimer

This is a wellness estimation tool. Not a medical device. Accuracy varies with lighting and skin tone. Do not use for clinical decisions.

Additional factors that affect accuracy:

  • Lighting: Bright, consistent ambient light improves results. Fluorescent or flickering lighting introduces 50/60 Hz noise that can corrupt the BVP signal.
  • Skin tone: Darker skin tones have lower red-channel contrast for the BVP signal. The skill applies adaptive gain, but accuracy may be lower. This is a known limitation of camera-based rPPG and active research area.
  • Contact pressure: Too little contact = ambient light leakage. Too much pressure = occludes capillaries and flattens the BVP signal. "Gently resting" is the correct placement.
  • Motion: Any finger movement during capture degrades the signal. The skill includes motion-rejection heuristics but cannot fully compensate for sustained movement.
  • Temperature: Cold hands (vasoconstriction) reduce BVP amplitude. Room temperature or warmer conditions give better results.
  • Typical accuracy: ±5–10 BPM compared to a pulse oximeter under good conditions.

Configuration

All configuration is via environment variables in ~/.openclaw/.env:

# Required: enables the skill
HEART_SENSE_ENABLED=true

# Capture settings
HEART_SENSE_MIN_FRAMES=600          # Minimum frames before analysis (default: 600)
HEART_SENSE_FFT_WINDOW=256          # FFT window size, power of 2 (default: 256)
HEART_SENSE_BPM_MIN=40              # Minimum BPM to consider (default: 40)
HEART_SENSE_BPM_MAX=180             # Maximum BPM to consider (default: 180)
HEART_SENSE_QUALITY_THRESHOLD=0.4   # Confidence below which to use peak detection (default: 0.4)

# Streaming (optional: sends live rppgFrame plugin messages to the Creation)
HEART_SENSE_STREAM_LIVE=true        # Stream intermediate BPM estimates (default: true)
HEART_SENSE_STREAM_INTERVAL_MS=500  # How often to stream a live estimate (default: 500)

Security Notes

  • This skill reads camera data locally on your machine. No video or frame data leaves your local network. The BVP signal (a 1-D array of 900 numbers) and the final BPM result are the only values written to openclaw:resp.
  • Camera access is restricted to the duration of the capture command. The skill does not hold the camera open between measurements.
  • No biometric data is transmitted to any remote service. Heart rate readings are stored only in the R1 Creation's creationStorage.plain (per-pluginId isolation, not accessible from other Creations or via localStorage).
  • The skill does not request, read, or store any data other than the BVP signal and derived BPM. No images or video frames are written to disk.
  • HEART_SENSE_ENABLED=true is required. Without it, the skill exits immediately and writes an error to openclaw:resp. This prevents the skill from running if accidentally installed on a shared OpenClaw instance.

Troubleshooting

BPM result is always the same (e.g. 72)

The Creation is falling back to its built-in estimate. OpenClaw is either not running or the capture command is not reaching the skill. Check:

openclaw status
# Should show: heart-sense ... loaded

Verify HEART_SENSE_ENABLED=true is set and the skill directory is at ~/.openclaw/skills/heart-sense/.

"camera_unavailable" error

OpenClaw does not have camera access. On the host machine, ensure the OpenClaw daemon has been granted camera permission (macOS: System Settings > Privacy & Security > Camera; Linux: check /dev/video* permissions; Windows: Settings > Privacy > Camera > allow desktop apps).

Signal quality is always Low (1–2 bars)

Common causes:

  • Finger is not covering the lens fully — ambient light is leaking in.
  • Pressing too hard — reduces capillary blood flow.
  • Bright flickering light source (TV, fluorescent) in the room — try a different location.
  • Cold hands — warm them briefly before measuring.

npm install fails for @opencv-tsjs/opencv4nodejs

This package compiles native bindings. Install build tools first:

# macOS
xcode-select --install

# Ubuntu / Debian
sudo apt-get install build-essential cmake libopencv-dev

# Windows
npm install --global windows-build-tools

Then retry npm install inside the skill directory.

Live BPM display never appears on the R1 during measurement

The live streaming feature requires HEART_SENSE_STREAM_LIVE=true (default). If it is set and still not appearing, check that PluginMessageHandler is available in the Creation's WebView context. This feature requires RabbitOS 2.0.3 or later.

ocReceive times out (45s) on every measurement

The host machine is too slow to process 900 frames within the timeout window. Options:

  1. Reduce HEART_SENSE_MIN_FRAMES to 450 (15 seconds of signal — less accurate but faster).
  2. Disable OpenCV-based motion rejection to reduce compute: set HEART_SENSE_QUALITY_THRESHOLD=0 (forces FFT only, no fallback).
  3. Increase the timeout on the Creation side by modifying ocReceive({ timeoutMs: 60000 }).

04 同仓库更多 Skills

ajohnsonnow
ajohnsonnow/r1-warren

whisper-quiz

WhisperQuiz — voice-answered trivia game for Rabbit R1. Generates questions via the R1's built-in LLM, captures spoken answers via STT, and evaluates correctness with flexible LLM-based matching.

★ 0
ajohnsonnow
ajohnsonnow/r1-warren

walk-cast

Walking podcast production skill. Receives raw audio metadata and transcript from the R1, removes long pauses, applies noise reduction and normalization, generates AI chapter titles and episode metadata via LLM, and uploads the final episode to your configured podcast RSS host (Spotify for Podcasters / Anchor, Podbean, Buzzsprout, RSS.com, or any FTP-accessible feed).

★ 0
ajohnsonnow
ajohnsonnow/r1-warren

voice-novel

Local machine skill for VoiceNovel — handles story export to disk.

★ 0
ajohnsonnow
ajohnsonnow/r1-warren

voice-guide

Environment description and navigation assistant for blind and low-vision users. Analyzes camera frames via a vision LLM, reads visible text, identifies scene categories (people/vehicles/food/hazards/exits), performs 360° scans, retrieves GPS position, and sends emergency SMS alerts — all via audio feedback with zero screen dependency.

★ 0
ajohnsonnow
ajohnsonnow/r1-warren

voice-draft

Voice-driven document drafting for the Rabbit R1. Dictate stream-of-consciousness speech; the R1 formats it into Email, Blog Post, Report, Proposal, Meeting Notes, Creative Writing, Code Comments, Cover Letter, or Tweet Thread — then saves the result to a configured folder on your local machine via OpenClaw.

★ 0
ajohnsonnow
ajohnsonnow/r1-warren

voice-commit

Hands-free git and CI workflow voice controller for the Rabbit R1. Speak git/CI commands and hear results — commit, push, status, PR management, test runs, and deploy, all voice-driven via OpenClaw.

★ 0