cometchat-moderation

Ground truth: capabilities and rule names are FETCHED from the live docs — {DOCS_BASE}/moderation/overview, /moderation/{rules-management,lists-management,flagged-messages,blocked-messages,reviewed-messages}, {DOCS_BASE}/rest-api/moderation-apis/overview, and the client hooks {DOCS_BASE}/ui-kit/<family>/core-features#moderation (built-in report) + {DOCS_BASE}/sdk/<platform>/ai-moderation (SDK flagging). DOCS_BASE = https://www.cometchat.com/docs; append .md. Most of this is dashboard configuration; verify the exact rule/list options and any REST shapes against the docs before asserting them.

Use this skill when

Making a chat safe/compliant: profanity/toxicity filtering, image/video safety, spam/scam detection, malware scanning of attachments, letting users report content, a moderator review queue, or an audit trail for T&S. Trust & Safety is a procurement gate for social, gaming, marketplace, dating, education and healthcare apps.

How it works

Every message runs through the rule engine as it's sent; a rule either lets it through, blocks it (hidden immediately), or flags it for a moderator. Users can also report a message manually, which flags it. Blocked/flagged items land in dashboard queues; a moderator's decision (approve/block/mark reviewed) moves it to the audit trail.

send → rules → pass → delivered
             → block → Blocked Messages
             → flag  → Flagged Messages → moderator → Reviewed Messages (audit)

1. Configure rules + lists (dashboard / REST)

• Rules Management (/moderation/rules-management): turn on the built-in rules you need and set each to block or flag. Available categories (fetch the current list — it grows):
  • Text: word-pattern match (profanity), contact/email removal, spam & scam, platform circumvention, toxicity, hate & harassment, self-harm, explicit content, privacy/sensitive-info, impersonation, violent/terroristic, non-consensual sexual content.
  • Image: unsafe/prohibited, minor-safety/CSAM, graphic violence, explicit/sexual, hate symbols, fraud (fake IDs), malware scan.
  • Video / Audio / File: malware & virus scan of attachments.
• Lists Management (/moderation/lists-management): your own keyword / regex / sentence lists, linked to rules — this is the "banned words" surface. Lists are reusable across rules.
• Programmatic setup: the Moderation REST API (/rest-api/moderation-apis/overview) manages rules + keyword lists from your server (e.g. per-tenant config, CI-seeded rules). Fetch its shapes before writing calls.

2. Wire the client (per family — built into the UI Kit)

• Report a message: the UI Kits ship a built-in Report Message action on the message — enable it via that family's moderation feature ({DOCS_BASE}/ui-kit/<family>/core-features#moderation; wire through cometchat-<family>-features). No hand-rolled report UI.
• Blocked content: rule-blocked messages simply don't render for recipients; nothing extra to build.
• SDK path (custom UI / headless): flag messages directly with the Chat SDK ({DOCS_BASE}/sdk/<platform>/ai-moderation) — the per-feature fallback when a family has no drop-in.

3. Moderator review + audit

• Flagged Messages / Blocked Messages dashboards: moderators review, approve false positives, or confirm blocks — and use them to refine rules.
• Reviewed Messages is the audit trail of every moderation decision (who did what, when) for compliance records — pair with cometchat-compliance for retention/eDiscovery of these records.

Legacy extensions — migrate, don't double-run

If the app still uses the legacy extensions (Profanity Filter, Data Masking, Image Moderation), disable them before enabling the new moderation rules. Running both processes every message twice → performance issues and message delays. Treat this as a required migration step, not optional.

Enablement note

Moderation is a dashboard/plan capability — enable it on the app (and on the production app, not just dev). npx @cometchat/skills-cli@3 features can toggle app extensions/AI after auth login; the rule/list configuration itself is done in the dashboard or via the Moderation REST API.

Common pitfalls

1. Legacy extensions left on alongside new rules — double-processing, delays. Disable the old ones first.
2. Rules configured on dev only — content ships unmoderated in production; enable on the production app.
3. Hand-rolling a report button — the UI Kit already ships Report Message; wire the built-in.
4. Block vs flag confused — block hides immediately (good for clear violations); flag routes to a human (good for borderline). Set each rule deliberately.
5. No one watching the queues — flagged content needs a moderator workflow; wire notifications/ownership.
6. Assuming a rule exists — fetch the current rule catalog; don't promise a category the docs don't list.

Verify it works

A message with a banned word/keyword is blocked or flagged per the rule · an unsafe image is caught · an attachment is malware-scanned · a user's Report action flags a message into the queue · a moderator decision appears in Reviewed Messages · legacy extensions are off · rules are enabled on the production app.