x-bookmark-archiver

X Bookmark Archiver

Archive your X (Twitter) bookmarks into categorized markdown files with AI-generated summaries.

Overview

This skill fetches your X bookmarks using the bird CLI, categorizes them by URL patterns, generates AI summaries using OpenAI, and saves them as organized markdown files.

Prerequisites

1. bird CLI - Install from steipete/bird
2. OpenAI API Key (optional) - Set OPENAI_API_KEY for AI-generated summaries
3. Node.js 18+

Installation

# Ensure bird CLI is installed and authenticated
bird --version

# Set OpenAI API key (optional, for AI summaries)
export OPENAI_API_KEY="sk-..."

Commands

run - Full Pipeline

Fetches new bookmarks and processes them:

node skills/x-bookmark-archiver/scripts/run.cjs

run --force - Process Existing

Skip fetch, process only pending bookmarks:

node skills/x-bookmark-archiver/scripts/run.cjs --force

fetch - Download Bookmarks Only

node skills/x-bookmark-archiver/scripts/fetch.cjs

process - Archive Pending Only

node skills/x-bookmark-archiver/scripts/process.cjs

Category Mapping

Bookmarks are automatically categorized based on URL patterns:

Category	Domains
tools	github.com, gitlab.com, github.io, huggingface.co, replicate.com, vercel.com, npmjs.com, pypi.org
articles	medium.com, substack.com, dev.to, hashnode.dev, x.com/i/article, blog.*, towardsdatascience.com
videos	youtube.com, youtu.be, vimeo.com, twitch.tv
research	arxiv.org, paperswithcode.com, semanticscholar.org, researchgate.net, dl.acm.org, ieee.org
news	techcrunch.com, theverge.com, hn.algolia.com, news.ycombinator.com, wired.com, arstechnica.com
bookmarks	fallback for unmatched URLs

Output Location

Markdown files are created in the OpenClaw workspace at:

Legacy installs (old name):

~/clawd/X-knowledge/

New installs:

~/.openclaw/workspace/X-knowledge/

With profile (OPENCLAW_PROFILE=prod):

~/.openclaw/workspace-prod/X-knowledge/

Override with environment variable:

export OPENCLAW_WORKSPACE=/custom/path
node skills/x-bookmark-archiver/scripts/run.cjs
# Creates: /custom/path/X-knowledge/

Output Structure

~/.openclaw/workspace/X-knowledge/
├── tools/
│   ├── awesome-ai-project.md
│   └── useful-cli-tool.md
├── articles/
│   ├── how-to-build-x.md
│   └── ml-best-practices.md
├── videos/
│   └── conference-talk.md
├── research/
│   └── attention-is-all-you-need.md
├── news/
│   └── latest-tech-announcement.md
└── bookmarks/
    └── misc-link.md

Markdown Template

Each archived bookmark gets a markdown file with frontmatter:

---
title: "Awesome AI Project"
type: tool
date_archived: 2026-01-31
source_tweet: https://x.com/i/web/status/1234567890
link: https://github.com/user/repo
tags: ["ai", "machine-learning", "github"]
---

This project implements a novel approach to... (AI-generated summary)

State Management

State files track processing progress:

/root/clawd/.state/
├── x-bookmark-pending.json     # Bookmarks waiting to be processed
└── x-bookmark-processed.json   # IDs of already-archived bookmarks

Environment Variables

Variable	Required	Description
OPENAI_API_KEY	No	API key for AI-generated summaries

Workflow

1. Fetch: Downloads latest 50 bookmarks from X
2. Filter: Removes already-processed bookmarks
3. Expand: Resolves t.co shortened URLs
4. Categorize: Assigns category based on URL domain
5. Enrich: Generates title, summary, tags (AI or fallback)
6. Write: Saves as markdown in X-knowledge/{category}/
7. Track: Updates processed IDs, clears pending

Customization

Adding Categories

Edit scripts/lib/categorize.cjs:

const CATEGORIES = {
  tools: ['github.com', '...'],
  your_category: ['example.com', '...'],
  // ...
};

Changing Output Directory

Edit scripts/process.cjs:

const KNOWLEDGE_DIR = 'your-directory-name';

Using Different AI Provider

Modify the generateMetadata() function in scripts/process.cjs to use your preferred API.

Testing

Run the test suite:

# Run all tests
cd skills/x-bookmark-archiver/tests
node test-all.cjs

# Run individual test suites
node lib/categorize.test.cjs
node lib/state.test.cjs
node integration.test.cjs

Test Coverage

• Unit tests: categorize.js (21 tests) - URL pattern matching
• Unit tests: state.js (9 tests) - JSON read/write operations
• Integration tests (12 tests) - Full pipeline with mock data

Manual Testing

Without bird CLI, you can test with mock data:

# Create mock pending data
cat > /tmp/test-pending.json << 'EOF'
[
  {
    "id": "test123",
    "url": "https://github.com/test/repo",
    "text": "Test bookmark"
  }
]
EOF

# Copy to state directory and process
mkdir -p /root/clawd/.state
cp /tmp/test-pending.json /root/clawd/.state/x-bookmark-pending.json
node skills/x-bookmark-archiver/scripts/process.cjs

Troubleshooting

Issue	Solution
bird: command not found	Install bird CLI from GitHub releases
No bookmarks fetched	Ensure you're logged into X in bird
AI summaries not generating	Check OPENAI_API_KEY is set
t.co links not expanding	May be network/timeout issues; will use original URL

File Structure

skills/x-bookmark-archiver/
├── SKILL.md
├── scripts/
│   ├── fetch.cjs          # Download bookmarks from X (CommonJS)
│   ├── process.cjs        # Generate markdown files (CommonJS)
│   ├── run.cjs            # Orchestrate fetch → process (CommonJS)
│   └── lib/
│       ├── categorize.cjs # URL → category mapping (CommonJS)
│       └── state.cjs      # JSON state management (CommonJS)
└── tests/
    ├── test-all.cjs
    ├── lib/
    │   ├── categorize.test.cjs
│   │   └── state.test.cjs
    ├── integration.test.cjs
    └── fixtures/
        └── sample-bookmarks.json

License

MIT