[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7EDNEA883SKEFSCYRJMCV8B":3,"skill-same:01M7EDNEA883SKEFSCYRJMCV8B":49},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":20,"frontMatter":21,"refName":22,"commitSha":23,"skillDirectory":24,"forksCount":13,"repositoryHtmlUrl":25,"repositoryHomepageUrl":6,"ownerHtmlUrl":26,"ownerBlogUrl":6,"ownerDescription":27,"updatedAt":28,"install":29,"resources":42},"01M7EDNEA883SKEFSCYRJMCV8B","telegram-channel-ingestion",null,"jonahthan433\u002FCortexOS","jonahthan433","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F211465495?v=4","01M7ECH9HPHGA89DCXHHJBTS3C","jonah.ai","01M7ECH9JCEM45DZMWPT7148S1",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjonahthan433\u002FCortexOS\u002Fblob\u002Fmain\u002Fskills\u002Fimported\u002Fsoftware-development\u002Ftelegram-channel-ingestion\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"# telegram-channel-ingestion\n\n# Telegram Channel Ingestion via Telethon MTProto\n\nClass of work: read an external Telegram channel (where your account is a **member**, not an admin) — backfill history + listen live — and persist raw messages plus parsed signals.\n\n## Golden rule: MTProto user-client, NOT Bot API\n\nA bot can read channel history only if made an admin. If the account is just a member (common for signal\u002Fscanner channels), use **Telethon MTProto user-client**.\n\n> **Session file = credential.** Store OUTSIDE the repo (e.g. `\u003Cproject>\u002F.tgsessions\u002F\u003Cname>.session`), chmod 700, add to `.gitignore`. Never log or commit.\n\n## Procedure\n\n### 1. Auth (interactive, one-time)\n\n```python\nimport os, asyncio\nfrom telethon import TelegramClient\n\nwith open(\"\u002Fopt\u002F\u003Cproject>\u002F.env\") as f:\n    for line in f:\n        line = line.strip()\n        if line and not line.startswith(\"#\") and \"=\" in line:\n            k, v = line.split(\"=\", 1)\n            os.environ.setdefault(k.strip(), v.strip())\n\nasync def main():\n    c = TelegramClient(\"\u002Fopt\u002F\u003Cproject>\u002F.tgsessions\u002F\u003Cname>\",\n                      int(os.environ[\"TELEGRAM_API_ID\"]),\n                      os.environ[\"TELEGRAM_API_HASH\"])\n    await c.connect()\n    if await c.is_user_authorized():\n        print(\"ALREADY_AUTHORIZED\")\n        return\n    await c.send_code_request(os.environ[\"TELETHON_PHONE\"])\n    code = input(\"Code: \").strip()\n    pwd = input(\"2FA password: \").strip()\n    try:\n        await c.sign_in(os.environ[\"TELETHON_PHONE\"], code)\n    except Exception as e:\n        if \"password\" in str(e).lower():\n            await c.sign_in(password=pwd)\n    await c.disconnect()\n\nasyncio.run(main())\n```\n\n**Pitfall — auth MUST be interactive.** Codes expire in ~30s. Background polls lose the race — run the script directly on the server and type code+password when prompted.\n\n**Pitfall — `os.environ` vs `pydantic-settings`.** `settings.TELEGRAM_API_ID` works but `os.environ.get(\"TELEGRAM_API_ID\")` returns None. Load `.env` manually (shown above).\n\n**Pitfall — session lock.** Kill the running server BEFORE running auth, or get `sqlite3.OperationalError: database is locked`.\n\n### 2. Parser — inspect real messages FIRST\n\nDon't guess the format. In this session the channel published \"Scout Opportunity\" lists (TOP 30 LONG\u002FSHORT with RSI15\u002FTSI15\u002FRSI4H\u002FTSI4H\u002Fdivergence\u002Ffunding), NOT trade signals. The first parser assumed entry\u002FSL\u002Ftarget and matched nothing.\n\n**Pitfall — single-pattern side detection fails on real channel variety.** Section headers arrive with emoji prefixes (🔴 LONG OPPORTUNITIES, 🟢 SHORT OPPORTUNITIES), bare keywords, inline cues (LONG:, → SHORT), and ranked list items. Use a prioritized cascade instead of one regex:\n\n1. Explicit section headers with context words (OPPORTUNITIES\u002FTOP\u002FSCOUT\u002FSETUP\u002FCONFIRM) + LONG\u002FSHORT\n2. Emoji\u002Fprefix markers (🔴🟢🟣▶▷⬆⬇) followed by LONG\u002FSHORT\n3. Inline direction cues (LONG:, SHORT:, → LONG, LONG →)\n4. Carry forward the last-detected side for lines that don't re-declare it\n\n**Pitfall — ambiguous bare headers.** A line like `LONG:` with no context word and no list items following may be data, not a header. Require a context word or emoji\u002Fprefix marker before switching side. If the next non-empty line contains a recognizable ticker pattern, treat the header as valid.\n\n**Pitfall — first-pass parser is always wrong for a new channel.** Parse a batch of real messages, print candidates, eyeball misses, iterate the regex BEFORE wiring to execution. Don't assume correctness.\n\n**Confidence scoring**: assign by rank (top-5 → high), section type (SCOUT\u002FTOP → +0.1), cap at 1.0. Low-confidence signals can route to a review queue.\n\n**Non-coin word filter**: maintain a deny-list of English words that collide with ticker patterns (TOP, LONG, SHORT, RSI, FUNDING, VOLUME, PRICE, ADMIN, UPDATE, SCAN, SCOUT, etc.). Extend when new false positives appear.\n\n**Deduplicate by (symbol, side)** keeping highest confidence per pair.\n\n**Extra model fields go to `unknown_fields` JSON.** If parser extracts fields not on the model (e.g. `rsi15`, `tsi15` when model has `rsi`), don't add DB columns. Pass known fields to constructor, stash extras in `unknown_fields = Column(JSON)`. Read back as `signal.unknown_fields.get(\"rsi15\")`.\n\n**Pitfall — stale `__pycache__` after patching parser.** Always clear `__pycache__` dirs after updating `.py` files on the target:\n```bash\nfind \u002Fopt\u002F\u003Cproject> -name __pycache__ -exec rm -rf {} + 2>\u002Fdev\u002Fnull\n```\nOtherwise the old compiled parser keeps running.\n\n```python\nfrom telethon import TelegramClient, events\nfrom telethon.errors import FloodWaitError\nfrom sqlalchemy import select  # required in this module for runtime\n\nclass TgChannelIngester:\n    def __init__(self, channel_username, backfill_limit=10000):\n        self.channel_username = channel_username\n        self.backfill_limit = backfill_limit\n        self._api_id = int(os.environ.get(\"TELEGRAM_API_ID\", \"0\"))\n        self._api_hash = os.environ.get(\"TELEGRAM_API_HASH\", \"\")\n        self._session_path = \"\u002Fopt\u002F\u003Cproject>\u002F.tgsessions\u002F\u003Cname>.session\"\n        self._client = None\n        self._running = False\n\n    async def _connect(self):\n        self._client = TelegramClient(self._session_path, self._api_id, self._api_hash)\n        await self._client.connect()\n        if not await self._client.is_user_authorized():\n            raise RuntimeError(\"Telegram not authorized\")\n\n    async def backfill(self, entity):\n        offset_id = 0\n        total = 0\n        while total \u003C self.backfill_limit:\n            batch = await self._client.get_messages(entity, limit=100,\n                                                    offset_id=offset_id, reverse=False)\n            if not batch:\n                break\n            for msg in batch:\n                if msg.message:\n                    await self._process_message(msg, entity)\n                offset_id = msg.id\n                total += 1\n            await asyncio.sleep(0.5)\n        return total\n\n    async def _on_new_message(self, event):\n        await self._process_message(event.message, event.chat)\n\n    async def start(self):\n        await self._connect()\n        entity = await self._client.get_entity(self.channel_username)\n        await self.backfill(entity)\n        self._running = True\n        self._client.add_event_handler(\n            self._on_new_message,\n            events.NewMessage(chats=[entity]),\n        )\n        while self._running:\n            await asyncio.sleep(30)\n```\n\n### 4. Restart (Proxmox CT)\n\n```bash\npct exec 106 -- bash -c \"fuser -k 8080\u002Ftcp 2>\u002Fdev\u002Fnull; sleep 2\"\npct exec 106 -- bash -c \"find \u002Fopt\u002F\u003Cproject> -name __pycache__ -exec rm -rf {} + 2>\u002Fdev\u002Fnull\"\npct exec 106 -- bash -c \"cd \u002Fopt\u002F\u003Cproject> && source venv\u002Fbin\u002Factivate && PYTHONPATH=\u002Fopt\u002F\u003Cproject> nohup python main.py > \u002Ftmp\u002Fcq.log 2>&1 &\"\n```\n\n**Pitfall — stale `__pycache__`.** Always clear after patching `.py` files.\n\n**Pitfall — port already in use.** `fuser -k \u003Cport>\u002Ftcp` before restart.\n",{},"main","38b32c99e7ed0b8e215d638e92cfa452c1ec3079","skills\u002Fimported\u002Fsoftware-development\u002Ftelegram-channel-ingestion","https:\u002F\u002Fgithub.com\u002Fjonahthan433\u002FCortexOS","https:\u002F\u002Fgithub.com\u002Fjonahthan433","AI Consultant","2026-10-24 05:13:39",{"source":7,"skillName":5,"commands":30,"prompt":40,"zipAvailable":41},[31,34,37],{"tool":32,"command":33},"npx","npx skills add jonahthan433\u002FCortexOS --skill telegram-channel-ingestion",{"tool":35,"command":36},"bunx","bunx skills add jonahthan433\u002FCortexOS --skill telegram-channel-ingestion",{"tool":38,"command":39},"pnpm","pnpm dlx skills add jonahthan433\u002FCortexOS --skill telegram-channel-ingestion","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjonahthan433\u002FCortexOS\u002Fblob\u002Fmain\u002Fskills\u002Fimported\u002Fsoftware-development\u002Ftelegram-channel-ingestion\u002FSKILL.md",true,[43],{"path":44,"resourceType":45,"resourceTypeDesc":46,"sizeBytes":47,"sourceHtmlUrl":48},"skill.json","REFERENCE","参考文档",368,"https:\u002F\u002Fgithub.com\u002Fjonahthan433\u002FCortexOS\u002Fblob\u002Fmain\u002Fskills\u002Fimported\u002Fsoftware-development\u002Ftelegram-channel-ingestion\u002Fskill.json",[50,56,62,68,74,80],{"skillNo":51,"name":52,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":53,"categories":54,"keywords":55,"publishedAt":18},"01M7EDQZDQX3Q8Y7GCQX6KN9V7","opportunity-brief","https:\u002F\u002Fgithub.com\u002Fjonahthan433\u002FCortexOS\u002Fblob\u002Fmain\u002Fskills\u002Fsales\u002Fopportunity-brief\u002FSKILL.md",[],[],{"skillNo":57,"name":58,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":59,"categories":60,"keywords":61,"publishedAt":18},"01M7EDQYA0V9Y99VKBREJJ3MRT","morning-intel","https:\u002F\u002Fgithub.com\u002Fjonahthan433\u002FCortexOS\u002Fblob\u002Fmain\u002Fskills\u002Fresearch\u002Fmorning-intel\u002FSKILL.md",[],[],{"skillNo":63,"name":64,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":65,"categories":66,"keywords":67,"publishedAt":18},"01M7EDQVX1JVY0B6PDHKKGM19G","decision-brief","https:\u002F\u002Fgithub.com\u002Fjonahthan433\u002FCortexOS\u002Fblob\u002Fmain\u002Fskills\u002Fresearch\u002Fdecision-brief\u002FSKILL.md",[],[],{"skillNo":69,"name":70,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":18},"01M7EDQV1GENZAC2SZSD8KZ3GM","weekly-review","https:\u002F\u002Fgithub.com\u002Fjonahthan433\u002FCortexOS\u002Fblob\u002Fmain\u002Fskills\u002Fproductivity\u002Fweekly-review\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":18},"01M7EDQT2KETJ1Y3CGVN9EKW2J","system-health","https:\u002F\u002Fgithub.com\u002Fjonahthan433\u002FCortexOS\u002Fblob\u002Fmain\u002Fskills\u002Fops\u002Fsystem-health\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":18},"01M7EDQQN2JK3XPWMS1M4T0F3K","backup-restore","https:\u002F\u002Fgithub.com\u002Fjonahthan433\u002FCortexOS\u002Fblob\u002Fmain\u002Fskills\u002Fops\u002Fbackup-restore\u002FSKILL.md",[],[]]