[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71CC366BAZ1WCZCDKBCFDNE":3,"skill-same:01M71CC366BAZ1WCZCDKBCFDNE":56},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":29,"linkOnly":30,"markdownContent":31,"markdownBody":32,"frontMatter":33,"refName":34,"commitSha":35,"skillDirectory":36,"forksCount":37,"repositoryHtmlUrl":38,"repositoryHomepageUrl":6,"ownerHtmlUrl":39,"ownerBlogUrl":40,"ownerDescription":6,"updatedAt":41,"install":42,"resources":55},"01M71CC366BAZ1WCZCDKBCFDNE","openai-vision",null,"Analyze images and multi-frame sequences using OpenAI GPT vision models","harbor-framework\u002Fharbor-adapters-experiments","harbor-framework","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F256467836?v=4","01M6DSWQEZXHGHAN95GH7D6PJ0","Harbor","01M71CBSVTNJ0WDQEJ7Q4TV93M",9,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fharbor-framework\u002Fharbor-adapters-experiments\u002Fblob\u002Fmain\u002Fharbor-index\u002Fdatasets\u002Fskillsbench-ocr-receipts-to-excel\u002Fenvironment\u002Fskills\u002Fopenai-vision\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28],"openai","openai vision","vision","images","analyze","sequences","frame","multi","models","gpt","2026-11-10 10:12:30",false,"---\nname: openai-vision\ndescription: Analyze images and multi-frame sequences using OpenAI GPT vision models\n---\n\n# OpenAI Vision Analysis Skill\n\n## Purpose\nThis skill enables image analysis, scene understanding, text extraction, and multi-frame comparison using OpenAI's vision-capable GPT models (e.g., `gpt-4o`, `gpt-4o-mini`). It supports single images, multiple images for comparison, and sequential frames for temporal analysis.\n\n## When to Use\n- Analyzing image content (objects, scenes, colors, spatial relationships)\n- Extracting and reading text from images (OCR via vision models)\n- Comparing multiple images to detect differences or changes\n- Processing video frames to understand temporal progression\n- Generating detailed image descriptions or captions\n- Answering questions about visual content\n\n## Required Libraries\n\nThe following Python libraries are required:\n\n```python\nfrom openai import OpenAI\nimport base64\nimport json\nimport os\nfrom pathlib import Path\n```\n\n## Input Requirements\n- **File formats**: JPG, JPEG, PNG, WEBP, non-animated GIF\n- **Image sources**: URL, Base64-encoded data, or local file paths\n- **Size limits**: Up to 20MB per image; total request payload under 50MB\n- **Maximum images**: Up to 500 images per request\n- **Image quality**: Clear, legible content; avoid watermarks or heavy distortions\n\n## Output Schema\nAnalysis results should be returned as valid JSON conforming to this schema:\n\n```json\n{\n  \"success\": true,\n  \"images_analyzed\": 1,\n  \"analysis\": {\n    \"description\": \"A detailed scene description...\",\n    \"objects\": [\n      {\"name\": \"car\", \"color\": \"red\", \"position\": \"foreground center\"},\n      {\"name\": \"tree\", \"count\": 3, \"position\": \"background\"}\n    ],\n    \"text_content\": \"Any text visible in the image...\",\n    \"colors\": [\"blue\", \"green\", \"white\"],\n    \"scene_type\": \"outdoor\u002Furban\"\n  },\n  \"comparison\": {\n    \"differences\": [\"Object X appeared\", \"Color changed from A to B\"],\n    \"similarities\": [\"Background unchanged\", \"Layout consistent\"]\n  },\n  \"metadata\": {\n    \"model_used\": \"gpt-4o\",\n    \"detail_level\": \"high\",\n    \"token_usage\": {\"prompt\": 1500, \"completion\": 200}\n  },\n  \"warnings\": []\n}\n```\n\n### Field Descriptions\n\n- `success`: Boolean indicating whether analysis completed\n- `images_analyzed`: Number of images processed in the request\n- `analysis.description`: Natural language description of the image content\n- `analysis.objects`: Array of detected objects with attributes\n- `analysis.text_content`: Any text extracted from the image\n- `analysis.colors`: Dominant colors identified\n- `analysis.scene_type`: Classification of the scene\n- `comparison`: Present when multiple images are analyzed; describes differences and similarities\n- `metadata.model_used`: The GPT model used for analysis\n- `metadata.detail_level`: Resolution level used (`low`, `high`, or `auto`)\n- `metadata.token_usage`: Token consumption for cost tracking\n- `warnings`: Array of any issues or limitations encountered\n\n\n## Code Examples\n\n### Basic Image Analysis from URL\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI()\n\ndef analyze_image_url(image_url, prompt=\"Describe this image in detail.\"):\n    \"\"\"Analyze an image from a URL using GPT-4o vision.\"\"\"\n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": [\n                    {\"type\": \"text\", \"text\": prompt},\n                    {\n                        \"type\": \"image_url\",\n                        \"image_url\": {\n                            \"url\": image_url,\n                            \"detail\": \"high\"\n                        }\n                    }\n                ]\n            }\n        ],\n        max_tokens=1000\n    )\n    return response.choices[0].message.content\n```\n\n### Image Analysis from Local File (Base64)\n\n```python\nfrom openai import OpenAI\nimport base64\n\nclient = OpenAI()\n\ndef encode_image_to_base64(image_path):\n    \"\"\"Encode a local image file to base64.\"\"\"\n    with open(image_path, \"rb\") as image_file:\n        return base64.standard_b64encode(image_file.read()).decode(\"utf-8\")\n\ndef get_image_media_type(image_path):\n    \"\"\"Determine the media type based on file extension.\"\"\"\n    ext = image_path.lower().split('.')[-1]\n    media_types = {\n        'jpg': 'image\u002Fjpeg',\n        'jpeg': 'image\u002Fjpeg',\n        'png': 'image\u002Fpng',\n        'gif': 'image\u002Fgif',\n        'webp': 'image\u002Fwebp'\n    }\n    return media_types.get(ext, 'image\u002Fjpeg')\n\ndef analyze_local_image(image_path, prompt=\"Describe this image in detail.\"):\n    \"\"\"Analyze a local image file using GPT-4o vision.\"\"\"\n    base64_image = encode_image_to_base64(image_path)\n    media_type = get_image_media_type(image_path)\n    \n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": [\n                    {\"type\": \"text\", \"text\": prompt},\n                    {\n                        \"type\": \"image_url\",\n                        \"image_url\": {\n                            \"url\": f\"data:{media_type};base64,{base64_image}\",\n                            \"detail\": \"high\"\n                        }\n                    }\n                ]\n            }\n        ],\n        max_tokens=1000\n    )\n    return response.choices[0].message.content\n```\n\n### Multi-Image Comparison\n\n```python\nfrom openai import OpenAI\nimport base64\n\nclient = OpenAI()\n\ndef compare_images(image_paths, comparison_prompt=None):\n    \"\"\"Compare multiple images and identify differences.\"\"\"\n    if comparison_prompt is None:\n        comparison_prompt = (\n            \"Compare these images carefully. \"\n            \"List all differences and similarities you observe. \"\n            \"Describe any changes in objects, colors, positions, or text.\"\n        )\n    \n    content = [{\"type\": \"text\", \"text\": comparison_prompt}]\n    \n    for i, image_path in enumerate(image_paths):\n        base64_image = encode_image_to_base64(image_path)\n        media_type = get_image_media_type(image_path)\n        \n        # Add label for each image\n        content.append({\n            \"type\": \"text\", \n            \"text\": f\"Image {i + 1}:\"\n        })\n        content.append({\n            \"type\": \"image_url\",\n            \"image_url\": {\n                \"url\": f\"data:{media_type};base64,{base64_image}\",\n                \"detail\": \"high\"\n            }\n        })\n    \n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[{\"role\": \"user\", \"content\": content}],\n        max_tokens=2000\n    )\n    return response.choices[0].message.content\n```\n\n### Multi-Frame Video Analysis\n\n```python\nfrom openai import OpenAI\nimport base64\nfrom pathlib import Path\n\nclient = OpenAI()\n\ndef analyze_video_frames(frame_paths, analysis_prompt=None):\n    \"\"\"Analyze a sequence of video frames for temporal understanding.\"\"\"\n    if analysis_prompt is None:\n        analysis_prompt = (\n            \"These are sequential frames from a video. \"\n            \"Describe what is happening over time. \"\n            \"Identify any motion, changes, or events that occur across the frames.\"\n        )\n    \n    content = [{\"type\": \"text\", \"text\": analysis_prompt}]\n    \n    for i, frame_path in enumerate(frame_paths):\n        base64_image = encode_image_to_base64(frame_path)\n        media_type = get_image_media_type(frame_path)\n        \n        content.append({\n            \"type\": \"text\",\n            \"text\": f\"Frame {i + 1}:\"\n        })\n        content.append({\n            \"type\": \"image_url\",\n            \"image_url\": {\n                \"url\": f\"data:{media_type};base64,{base64_image}\",\n                \"detail\": \"auto\"  # Use auto for frames to balance cost\n            }\n        })\n    \n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[{\"role\": \"user\", \"content\": content}],\n        max_tokens=2000\n    )\n    return response.choices[0].message.content\n```\n\n### Full Analysis with JSON Output\n\n```python\nfrom openai import OpenAI\nimport base64\nimport json\nimport os\n\nclient = OpenAI()\n\ndef analyze_image_to_json(image_path, extract_text=True):\n    \"\"\"Perform comprehensive image analysis and return structured JSON.\"\"\"\n    filename = os.path.basename(image_path)\n    \n    prompt = \"\"\"Analyze this image and return a JSON object with the following structure:\n{\n    \"description\": \"detailed scene description\",\n    \"objects\": [{\"name\": \"object name\", \"attributes\": \"color, size, position\"}],\n    \"text_content\": \"any visible text or null if none\",\n    \"colors\": [\"dominant\", \"colors\"],\n    \"scene_type\": \"indoor\u002Foutdoor\u002Fabstract\u002Fetc\",\n    \"people_count\": 0,\n    \"notable_features\": [\"list of notable visual elements\"]\n}\n\nReturn ONLY valid JSON, no other text.\"\"\"\n\n    try:\n        base64_image = encode_image_to_base64(image_path)\n        media_type = get_image_media_type(image_path)\n        \n        response = client.chat.completions.create(\n            model=\"gpt-4o\",\n            messages=[\n                {\n                    \"role\": \"user\",\n                    \"content\": [\n                        {\"type\": \"text\", \"text\": prompt},\n                        {\n                            \"type\": \"image_url\",\n                            \"image_url\": {\n                                \"url\": f\"data:{media_type};base64,{base64_image}\",\n                                \"detail\": \"high\"\n                            }\n                        }\n                    ]\n                }\n            ],\n            max_tokens=1500\n        )\n        \n        # Parse the response as JSON\n        analysis_text = response.choices[0].message.content\n        # Remove markdown code blocks if present\n        if analysis_text.startswith(\"```\"):\n            analysis_text = analysis_text.split(\"```\")[1]\n            if analysis_text.startswith(\"json\"):\n                analysis_text = analysis_text[4:]\n        analysis = json.loads(analysis_text.strip())\n        \n        result = {\n            \"success\": True,\n            \"filename\": filename,\n            \"analysis\": analysis,\n            \"metadata\": {\n                \"model_used\": \"gpt-4o\",\n                \"detail_level\": \"high\",\n                \"token_usage\": {\n                    \"prompt\": response.usage.prompt_tokens,\n                    \"completion\": response.usage.completion_tokens\n                }\n            },\n            \"warnings\": []\n        }\n        \n    except json.JSONDecodeError as e:\n        result = {\n            \"success\": False,\n            \"filename\": filename,\n            \"analysis\": {\"raw_response\": response.choices[0].message.content},\n            \"metadata\": {\"model_used\": \"gpt-4o\"},\n            \"warnings\": [f\"Failed to parse JSON: {str(e)}\"]\n        }\n    except Exception as e:\n        result = {\n            \"success\": False,\n            \"filename\": filename,\n            \"analysis\": {},\n            \"metadata\": {},\n            \"warnings\": [f\"Analysis failed: {str(e)}\"]\n        }\n    \n    return result\n\n# Usage\nresult = analyze_image_to_json(\"photo.jpg\")\nprint(json.dumps(result, indent=2))\n```\n\n### Batch Processing Directory\n\n```python\nfrom openai import OpenAI\nimport base64\nimport json\nfrom pathlib import Path\n\nclient = OpenAI()\n\ndef process_image_directory(directory_path, output_file, prompt=None):\n    \"\"\"Process all images in a directory and save results.\"\"\"\n    if prompt is None:\n        prompt = \"Describe this image briefly, including any visible text.\"\n    \n    image_extensions = {'.jpg', '.jpeg', '.png', '.webp', '.gif'}\n    results = []\n    \n    for file_path in sorted(Path(directory_path).iterdir()):\n        if file_path.suffix.lower() in image_extensions:\n            print(f\"Processing: {file_path.name}\")\n            \n            try:\n                analysis = analyze_local_image(str(file_path), prompt)\n                results.append({\n                    \"filename\": file_path.name,\n                    \"success\": True,\n                    \"analysis\": analysis\n                })\n            except Exception as e:\n                results.append({\n                    \"filename\": file_path.name,\n                    \"success\": False,\n                    \"error\": str(e)\n                })\n    \n    # Save results\n    with open(output_file, 'w') as f:\n        json.dump(results, f, indent=2)\n    \n    return results\n```\n\n\n## Detail Level Configuration\n\nThe `detail` parameter controls image resolution and token usage:\n\n```python\n# Low detail: 512x512 fixed, ~85 tokens per image\n# Best for: Quick summaries, dominant colors, general scene type\n{\"detail\": \"low\"}\n\n# High detail: Full resolution processing\n# Best for: Reading text, detecting small objects, detailed analysis\n{\"detail\": \"high\"}\n\n# Auto: Model decides based on image size\n# Best for: General use when cost vs quality tradeoff is acceptable\n{\"detail\": \"auto\"}\n```\n\n### Choosing Detail Level\n\n```python\ndef get_recommended_detail(task_type):\n    \"\"\"Recommend detail level based on task type.\"\"\"\n    high_detail_tasks = {\n        'ocr', 'text_extraction', 'document_analysis',\n        'small_object_detection', 'detailed_comparison',\n        'fine_grained_analysis'\n    }\n    \n    low_detail_tasks = {\n        'scene_classification', 'dominant_colors',\n        'general_description', 'thumbnail_preview'\n    }\n    \n    if task_type.lower() in high_detail_tasks:\n        return \"high\"\n    elif task_type.lower() in low_detail_tasks:\n        return \"low\"\n    else:\n        return \"auto\"\n```\n\n\n## Text Extraction (Vision-based OCR)\n\nFor extracting text from images using vision models:\n\n```python\ndef extract_text_from_image(image_path, preserve_layout=False):\n    \"\"\"Extract text from an image using GPT-4o vision.\"\"\"\n    if preserve_layout:\n        prompt = (\n            \"Extract ALL text visible in this image. \"\n            \"Preserve the original layout and formatting as much as possible. \"\n            \"Include headers, paragraphs, captions, and any other text. \"\n            \"Return only the extracted text, nothing else.\"\n        )\n    else:\n        prompt = (\n            \"Extract all text visible in this image. \"\n            \"Return the text in reading order (top to bottom, left to right). \"\n            \"Return only the extracted text, nothing else.\"\n        )\n    \n    return analyze_local_image(image_path, prompt)\n\n\ndef extract_structured_text(image_path):\n    \"\"\"Extract text with structure information as JSON.\"\"\"\n    prompt = \"\"\"Extract all text from this image and return as JSON:\n{\n    \"headers\": [\"list of headers\u002Ftitles\"],\n    \"paragraphs\": [\"list of paragraph texts\"],\n    \"labels\": [\"list of labels or captions\"],\n    \"other_text\": [\"any other text elements\"],\n    \"reading_order\": [\"all text in reading order\"]\n}\nReturn ONLY valid JSON.\"\"\"\n    \n    response = analyze_local_image(image_path, prompt)\n    \n    try:\n        # Clean and parse JSON\n        if response.startswith(\"```\"):\n            response = response.split(\"```\")[1]\n            if response.startswith(\"json\"):\n                response = response[4:]\n        return json.loads(response.strip())\n    except json.JSONDecodeError:\n        return {\"raw_text\": response, \"parse_error\": True}\n```\n\n\n## Error Handling\n\n### Common Issues and Solutions\n\n**Issue**: API rate limits exceeded\n\n```python\nimport time\nfrom openai import RateLimitError\n\ndef analyze_with_retry(image_path, prompt, max_retries=3):\n    \"\"\"Analyze image with exponential backoff retry.\"\"\"\n    for attempt in range(max_retries):\n        try:\n            return analyze_local_image(image_path, prompt)\n        except RateLimitError:\n            if attempt \u003C max_retries - 1:\n                wait_time = 2 ** attempt  # Exponential backoff\n                print(f\"Rate limited, waiting {wait_time}s...\")\n                time.sleep(wait_time)\n            else:\n                raise\n```\n\n**Issue**: Image too large\n\n```python\nfrom PIL import Image\nimport io\n\ndef resize_image_if_needed(image_path, max_size_mb=15):\n    \"\"\"Resize image if it exceeds size limit.\"\"\"\n    file_size_mb = os.path.getsize(image_path) \u002F (1024 * 1024)\n    \n    if file_size_mb \u003C= max_size_mb:\n        return encode_image_to_base64(image_path)\n    \n    # Resize the image\n    img = Image.open(image_path)\n    \n    # Calculate new dimensions (reduce by 50% iteratively)\n    while file_size_mb > max_size_mb:\n        new_width = img.width \u002F\u002F 2\n        new_height = img.height \u002F\u002F 2\n        img = img.resize((new_width, new_height), Image.Resampling.LANCZOS)\n        \n        # Check new size\n        buffer = io.BytesIO()\n        img.save(buffer, format='JPEG', quality=85)\n        file_size_mb = len(buffer.getvalue()) \u002F (1024 * 1024)\n    \n    # Encode resized image\n    buffer = io.BytesIO()\n    img.save(buffer, format='JPEG', quality=85)\n    return base64.standard_b64encode(buffer.getvalue()).decode(\"utf-8\")\n```\n\n**Issue**: Invalid image format\n\n```python\ndef validate_image(image_path):\n    \"\"\"Validate image before processing.\"\"\"\n    valid_extensions = {'.jpg', '.jpeg', '.png', '.gif', '.webp'}\n    \n    path = Path(image_path)\n    \n    if not path.exists():\n        return False, \"File does not exist\"\n    \n    if path.suffix.lower() not in valid_extensions:\n        return False, f\"Unsupported format: {path.suffix}\"\n    \n    try:\n        with Image.open(image_path) as img:\n            img.verify()\n        return True, \"Valid image\"\n    except Exception as e:\n        return False, f\"Invalid image: {str(e)}\"\n```\n\n\n## Quality Self-Check\n\nBefore returning results, verify:\n\n- [ ] API response was received successfully\n- [ ] Output is valid JSON (if structured output requested)\n- [ ] All requested analysis fields are present\n- [ ] Token usage is within expected bounds\n- [ ] No error messages in the response\n- [ ] For multi-image: all images were processed\n- [ ] Confidence\u002Fwarnings are included when analysis is uncertain\n\n\n## Limitations\n\n- **Medical imagery**: Not suitable for diagnostic analysis of CT scans, X-rays, or MRIs\n- **Small text**: Text smaller than ~12pt may be misread; use `detail: high`\n- **Rotated\u002Fskewed text**: Accuracy decreases with text rotation; pre-process if needed\n- **Non-Latin scripts**: Lower accuracy for complex scripts (CJK, Arabic, etc.)\n- **Object counting**: Approximate counts only; may miss or double-count similar objects\n- **Spatial precision**: Cannot provide pixel-accurate bounding boxes or measurements\n- **Panoramic\u002Ffisheye**: Distorted images reduce analysis accuracy\n- **Graphs and charts**: May misinterpret line styles, legends, or data points\n- **Metadata**: Cannot access EXIF data, camera info, or GPS coordinates from images\n- **Cost**: High-detail analysis of many images can be expensive; monitor token usage\n\n\n## Token Cost Estimation\n\nApproximate token costs for image inputs:\n\n| Detail Level | Tokens per Image | Best For |\n|-------------|------------------|----------|\n| low | ~85 tokens (fixed) | Quick classification, color detection |\n| high | 85 + 170 per 512x512 tile | OCR, detailed analysis, small objects |\n| auto | Variable | General use |\n\n```python\ndef estimate_image_tokens(image_path, detail=\"high\"):\n    \"\"\"Estimate token usage for an image.\"\"\"\n    if detail == \"low\":\n        return 85\n    \n    with Image.open(image_path) as img:\n        width, height = img.size\n    \n    # High detail: image is scaled to fit in 2048x2048, then tiled at 512x512\n    scale = min(2048 \u002F max(width, height), 1.0)\n    scaled_width = int(width * scale)\n    scaled_height = int(height * scale)\n    \n    # Ensure minimum 768 on shortest side\n    if min(scaled_width, scaled_height) \u003C 768:\n        scale = 768 \u002F min(scaled_width, scaled_height)\n        scaled_width = int(scaled_width * scale)\n        scaled_height = int(scaled_height * scale)\n    \n    # Calculate tiles\n    tiles_x = (scaled_width + 511) \u002F\u002F 512\n    tiles_y = (scaled_height + 511) \u002F\u002F 512\n    total_tiles = tiles_x * tiles_y\n    \n    return 85 + (170 * total_tiles)\n```\n\n\n## Version History\n\n- **1.0.0** (2026-01-21): Initial release with GPT-4o vision support\n","# OpenAI Vision Analysis Skill\n\n## Purpose\nThis skill enables image analysis, scene understanding, text extraction, and multi-frame comparison using OpenAI's vision-capable GPT models (e.g., `gpt-4o`, `gpt-4o-mini`). It supports single images, multiple images for comparison, and sequential frames for temporal analysis.\n\n## When to Use\n- Analyzing image content (objects, scenes, colors, spatial relationships)\n- Extracting and reading text from images (OCR via vision models)\n- Comparing multiple images to detect differences or changes\n- Processing video frames to understand temporal progression\n- Generating detailed image descriptions or captions\n- Answering questions about visual content\n\n## Required Libraries\n\nThe following Python libraries are required:\n\n```python\nfrom openai import OpenAI\nimport base64\nimport json\nimport os\nfrom pathlib import Path\n```\n\n## Input Requirements\n- **File formats**: JPG, JPEG, PNG, WEBP, non-animated GIF\n- **Image sources**: URL, Base64-encoded data, or local file paths\n- **Size limits**: Up to 20MB per image; total request payload under 50MB\n- **Maximum images**: Up to 500 images per request\n- **Image quality**: Clear, legible content; avoid watermarks or heavy distortions\n\n## Output Schema\nAnalysis results should be returned as valid JSON conforming to this schema:\n\n```json\n{\n  \"success\": true,\n  \"images_analyzed\": 1,\n  \"analysis\": {\n    \"description\": \"A detailed scene description...\",\n    \"objects\": [\n      {\"name\": \"car\", \"color\": \"red\", \"position\": \"foreground center\"},\n      {\"name\": \"tree\", \"count\": 3, \"position\": \"background\"}\n    ],\n    \"text_content\": \"Any text visible in the image...\",\n    \"colors\": [\"blue\", \"green\", \"white\"],\n    \"scene_type\": \"outdoor\u002Furban\"\n  },\n  \"comparison\": {\n    \"differences\": [\"Object X appeared\", \"Color changed from A to B\"],\n    \"similarities\": [\"Background unchanged\", \"Layout consistent\"]\n  },\n  \"metadata\": {\n    \"model_used\": \"gpt-4o\",\n    \"detail_level\": \"high\",\n    \"token_usage\": {\"prompt\": 1500, \"completion\": 200}\n  },\n  \"warnings\": []\n}\n```\n\n### Field Descriptions\n\n- `success`: Boolean indicating whether analysis completed\n- `images_analyzed`: Number of images processed in the request\n- `analysis.description`: Natural language description of the image content\n- `analysis.objects`: Array of detected objects with attributes\n- `analysis.text_content`: Any text extracted from the image\n- `analysis.colors`: Dominant colors identified\n- `analysis.scene_type`: Classification of the scene\n- `comparison`: Present when multiple images are analyzed; describes differences and similarities\n- `metadata.model_used`: The GPT model used for analysis\n- `metadata.detail_level`: Resolution level used (`low`, `high`, or `auto`)\n- `metadata.token_usage`: Token consumption for cost tracking\n- `warnings`: Array of any issues or limitations encountered\n\n\n## Code Examples\n\n### Basic Image Analysis from URL\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI()\n\ndef analyze_image_url(image_url, prompt=\"Describe this image in detail.\"):\n    \"\"\"Analyze an image from a URL using GPT-4o vision.\"\"\"\n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": [\n                    {\"type\": \"text\", \"text\": prompt},\n                    {\n                        \"type\": \"image_url\",\n                        \"image_url\": {\n                            \"url\": image_url,\n                            \"detail\": \"high\"\n                        }\n                    }\n                ]\n            }\n        ],\n        max_tokens=1000\n    )\n    return response.choices[0].message.content\n```\n\n### Image Analysis from Local File (Base64)\n\n```python\nfrom openai import OpenAI\nimport base64\n\nclient = OpenAI()\n\ndef encode_image_to_base64(image_path):\n    \"\"\"Encode a local image file to base64.\"\"\"\n    with open(image_path, \"rb\") as image_file:\n        return base64.standard_b64encode(image_file.read()).decode(\"utf-8\")\n\ndef get_image_media_type(image_path):\n    \"\"\"Determine the media type based on file extension.\"\"\"\n    ext = image_path.lower().split('.')[-1]\n    media_types = {\n        'jpg': 'image\u002Fjpeg',\n        'jpeg': 'image\u002Fjpeg',\n        'png': 'image\u002Fpng',\n        'gif': 'image\u002Fgif',\n        'webp': 'image\u002Fwebp'\n    }\n    return media_types.get(ext, 'image\u002Fjpeg')\n\ndef analyze_local_image(image_path, prompt=\"Describe this image in detail.\"):\n    \"\"\"Analyze a local image file using GPT-4o vision.\"\"\"\n    base64_image = encode_image_to_base64(image_path)\n    media_type = get_image_media_type(image_path)\n    \n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": [\n                    {\"type\": \"text\", \"text\": prompt},\n                    {\n                        \"type\": \"image_url\",\n                        \"image_url\": {\n                            \"url\": f\"data:{media_type};base64,{base64_image}\",\n                            \"detail\": \"high\"\n                        }\n                    }\n                ]\n            }\n        ],\n        max_tokens=1000\n    )\n    return response.choices[0].message.content\n```\n\n### Multi-Image Comparison\n\n```python\nfrom openai import OpenAI\nimport base64\n\nclient = OpenAI()\n\ndef compare_images(image_paths, comparison_prompt=None):\n    \"\"\"Compare multiple images and identify differences.\"\"\"\n    if comparison_prompt is None:\n        comparison_prompt = (\n            \"Compare these images carefully. \"\n            \"List all differences and similarities you observe. \"\n            \"Describe any changes in objects, colors, positions, or text.\"\n        )\n    \n    content = [{\"type\": \"text\", \"text\": comparison_prompt}]\n    \n    for i, image_path in enumerate(image_paths):\n        base64_image = encode_image_to_base64(image_path)\n        media_type = get_image_media_type(image_path)\n        \n        # Add label for each image\n        content.append({\n            \"type\": \"text\", \n            \"text\": f\"Image {i + 1}:\"\n        })\n        content.append({\n            \"type\": \"image_url\",\n            \"image_url\": {\n                \"url\": f\"data:{media_type};base64,{base64_image}\",\n                \"detail\": \"high\"\n            }\n        })\n    \n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[{\"role\": \"user\", \"content\": content}],\n        max_tokens=2000\n    )\n    return response.choices[0].message.content\n```\n\n### Multi-Frame Video Analysis\n\n```python\nfrom openai import OpenAI\nimport base64\nfrom pathlib import Path\n\nclient = OpenAI()\n\ndef analyze_video_frames(frame_paths, analysis_prompt=None):\n    \"\"\"Analyze a sequence of video frames for temporal understanding.\"\"\"\n    if analysis_prompt is None:\n        analysis_prompt = (\n            \"These are sequential frames from a video. \"\n            \"Describe what is happening over time. \"\n            \"Identify any motion, changes, or events that occur across the frames.\"\n        )\n    \n    content = [{\"type\": \"text\", \"text\": analysis_prompt}]\n    \n    for i, frame_path in enumerate(frame_paths):\n        base64_image = encode_image_to_base64(frame_path)\n        media_type = get_image_media_type(frame_path)\n        \n        content.append({\n            \"type\": \"text\",\n            \"text\": f\"Frame {i + 1}:\"\n        })\n        content.append({\n            \"type\": \"image_url\",\n            \"image_url\": {\n                \"url\": f\"data:{media_type};base64,{base64_image}\",\n                \"detail\": \"auto\"  # Use auto for frames to balance cost\n            }\n        })\n    \n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[{\"role\": \"user\", \"content\": content}],\n        max_tokens=2000\n    )\n    return response.choices[0].message.content\n```\n\n### Full Analysis with JSON Output\n\n```python\nfrom openai import OpenAI\nimport base64\nimport json\nimport os\n\nclient = OpenAI()\n\ndef analyze_image_to_json(image_path, extract_text=True):\n    \"\"\"Perform comprehensive image analysis and return structured JSON.\"\"\"\n    filename = os.path.basename(image_path)\n    \n    prompt = \"\"\"Analyze this image and return a JSON object with the following structure:\n{\n    \"description\": \"detailed scene description\",\n    \"objects\": [{\"name\": \"object name\", \"attributes\": \"color, size, position\"}],\n    \"text_content\": \"any visible text or null if none\",\n    \"colors\": [\"dominant\", \"colors\"],\n    \"scene_type\": \"indoor\u002Foutdoor\u002Fabstract\u002Fetc\",\n    \"people_count\": 0,\n    \"notable_features\": [\"list of notable visual elements\"]\n}\n\nReturn ONLY valid JSON, no other text.\"\"\"\n\n    try:\n        base64_image = encode_image_to_base64(image_path)\n        media_type = get_image_media_type(image_path)\n        \n        response = client.chat.completions.create(\n            model=\"gpt-4o\",\n            messages=[\n                {\n                    \"role\": \"user\",\n                    \"content\": [\n                        {\"type\": \"text\", \"text\": prompt},\n                        {\n                            \"type\": \"image_url\",\n                            \"image_url\": {\n                                \"url\": f\"data:{media_type};base64,{base64_image}\",\n                                \"detail\": \"high\"\n                            }\n                        }\n                    ]\n                }\n            ],\n            max_tokens=1500\n        )\n        \n        # Parse the response as JSON\n        analysis_text = response.choices[0].message.content\n        # Remove markdown code blocks if present\n        if analysis_text.startswith(\"```\"):\n            analysis_text = analysis_text.split(\"```\")[1]\n            if analysis_text.startswith(\"json\"):\n                analysis_text = analysis_text[4:]\n        analysis = json.loads(analysis_text.strip())\n        \n        result = {\n            \"success\": True,\n            \"filename\": filename,\n            \"analysis\": analysis,\n            \"metadata\": {\n                \"model_used\": \"gpt-4o\",\n                \"detail_level\": \"high\",\n                \"token_usage\": {\n                    \"prompt\": response.usage.prompt_tokens,\n                    \"completion\": response.usage.completion_tokens\n                }\n            },\n            \"warnings\": []\n        }\n        \n    except json.JSONDecodeError as e:\n        result = {\n            \"success\": False,\n            \"filename\": filename,\n            \"analysis\": {\"raw_response\": response.choices[0].message.content},\n            \"metadata\": {\"model_used\": \"gpt-4o\"},\n            \"warnings\": [f\"Failed to parse JSON: {str(e)}\"]\n        }\n    except Exception as e:\n        result = {\n            \"success\": False,\n            \"filename\": filename,\n            \"analysis\": {},\n            \"metadata\": {},\n            \"warnings\": [f\"Analysis failed: {str(e)}\"]\n        }\n    \n    return result\n\n# Usage\nresult = analyze_image_to_json(\"photo.jpg\")\nprint(json.dumps(result, indent=2))\n```\n\n### Batch Processing Directory\n\n```python\nfrom openai import OpenAI\nimport base64\nimport json\nfrom pathlib import Path\n\nclient = OpenAI()\n\ndef process_image_directory(directory_path, output_file, prompt=None):\n    \"\"\"Process all images in a directory and save results.\"\"\"\n    if prompt is None:\n        prompt = \"Describe this image briefly, including any visible text.\"\n    \n    image_extensions = {'.jpg', '.jpeg', '.png', '.webp', '.gif'}\n    results = []\n    \n    for file_path in sorted(Path(directory_path).iterdir()):\n        if file_path.suffix.lower() in image_extensions:\n            print(f\"Processing: {file_path.name}\")\n            \n            try:\n                analysis = analyze_local_image(str(file_path), prompt)\n                results.append({\n                    \"filename\": file_path.name,\n                    \"success\": True,\n                    \"analysis\": analysis\n                })\n            except Exception as e:\n                results.append({\n                    \"filename\": file_path.name,\n                    \"success\": False,\n                    \"error\": str(e)\n                })\n    \n    # Save results\n    with open(output_file, 'w') as f:\n        json.dump(results, f, indent=2)\n    \n    return results\n```\n\n\n## Detail Level Configuration\n\nThe `detail` parameter controls image resolution and token usage:\n\n```python\n# Low detail: 512x512 fixed, ~85 tokens per image\n# Best for: Quick summaries, dominant colors, general scene type\n{\"detail\": \"low\"}\n\n# High detail: Full resolution processing\n# Best for: Reading text, detecting small objects, detailed analysis\n{\"detail\": \"high\"}\n\n# Auto: Model decides based on image size\n# Best for: General use when cost vs quality tradeoff is acceptable\n{\"detail\": \"auto\"}\n```\n\n### Choosing Detail Level\n\n```python\ndef get_recommended_detail(task_type):\n    \"\"\"Recommend detail level based on task type.\"\"\"\n    high_detail_tasks = {\n        'ocr', 'text_extraction', 'document_analysis',\n        'small_object_detection', 'detailed_comparison',\n        'fine_grained_analysis'\n    }\n    \n    low_detail_tasks = {\n        'scene_classification', 'dominant_colors',\n        'general_description', 'thumbnail_preview'\n    }\n    \n    if task_type.lower() in high_detail_tasks:\n        return \"high\"\n    elif task_type.lower() in low_detail_tasks:\n        return \"low\"\n    else:\n        return \"auto\"\n```\n\n\n## Text Extraction (Vision-based OCR)\n\nFor extracting text from images using vision models:\n\n```python\ndef extract_text_from_image(image_path, preserve_layout=False):\n    \"\"\"Extract text from an image using GPT-4o vision.\"\"\"\n    if preserve_layout:\n        prompt = (\n            \"Extract ALL text visible in this image. \"\n            \"Preserve the original layout and formatting as much as possible. \"\n            \"Include headers, paragraphs, captions, and any other text. \"\n            \"Return only the extracted text, nothing else.\"\n        )\n    else:\n        prompt = (\n            \"Extract all text visible in this image. \"\n            \"Return the text in reading order (top to bottom, left to right). \"\n            \"Return only the extracted text, nothing else.\"\n        )\n    \n    return analyze_local_image(image_path, prompt)\n\n\ndef extract_structured_text(image_path):\n    \"\"\"Extract text with structure information as JSON.\"\"\"\n    prompt = \"\"\"Extract all text from this image and return as JSON:\n{\n    \"headers\": [\"list of headers\u002Ftitles\"],\n    \"paragraphs\": [\"list of paragraph texts\"],\n    \"labels\": [\"list of labels or captions\"],\n    \"other_text\": [\"any other text elements\"],\n    \"reading_order\": [\"all text in reading order\"]\n}\nReturn ONLY valid JSON.\"\"\"\n    \n    response = analyze_local_image(image_path, prompt)\n    \n    try:\n        # Clean and parse JSON\n        if response.startswith(\"```\"):\n            response = response.split(\"```\")[1]\n            if response.startswith(\"json\"):\n                response = response[4:]\n        return json.loads(response.strip())\n    except json.JSONDecodeError:\n        return {\"raw_text\": response, \"parse_error\": True}\n```\n\n\n## Error Handling\n\n### Common Issues and Solutions\n\n**Issue**: API rate limits exceeded\n\n```python\nimport time\nfrom openai import RateLimitError\n\ndef analyze_with_retry(image_path, prompt, max_retries=3):\n    \"\"\"Analyze image with exponential backoff retry.\"\"\"\n    for attempt in range(max_retries):\n        try:\n            return analyze_local_image(image_path, prompt)\n        except RateLimitError:\n            if attempt \u003C max_retries - 1:\n                wait_time = 2 ** attempt  # Exponential backoff\n                print(f\"Rate limited, waiting {wait_time}s...\")\n                time.sleep(wait_time)\n            else:\n                raise\n```\n\n**Issue**: Image too large\n\n```python\nfrom PIL import Image\nimport io\n\ndef resize_image_if_needed(image_path, max_size_mb=15):\n    \"\"\"Resize image if it exceeds size limit.\"\"\"\n    file_size_mb = os.path.getsize(image_path) \u002F (1024 * 1024)\n    \n    if file_size_mb \u003C= max_size_mb:\n        return encode_image_to_base64(image_path)\n    \n    # Resize the image\n    img = Image.open(image_path)\n    \n    # Calculate new dimensions (reduce by 50% iteratively)\n    while file_size_mb > max_size_mb:\n        new_width = img.width \u002F\u002F 2\n        new_height = img.height \u002F\u002F 2\n        img = img.resize((new_width, new_height), Image.Resampling.LANCZOS)\n        \n        # Check new size\n        buffer = io.BytesIO()\n        img.save(buffer, format='JPEG', quality=85)\n        file_size_mb = len(buffer.getvalue()) \u002F (1024 * 1024)\n    \n    # Encode resized image\n    buffer = io.BytesIO()\n    img.save(buffer, format='JPEG', quality=85)\n    return base64.standard_b64encode(buffer.getvalue()).decode(\"utf-8\")\n```\n\n**Issue**: Invalid image format\n\n```python\ndef validate_image(image_path):\n    \"\"\"Validate image before processing.\"\"\"\n    valid_extensions = {'.jpg', '.jpeg', '.png', '.gif', '.webp'}\n    \n    path = Path(image_path)\n    \n    if not path.exists():\n        return False, \"File does not exist\"\n    \n    if path.suffix.lower() not in valid_extensions:\n        return False, f\"Unsupported format: {path.suffix}\"\n    \n    try:\n        with Image.open(image_path) as img:\n            img.verify()\n        return True, \"Valid image\"\n    except Exception as e:\n        return False, f\"Invalid image: {str(e)}\"\n```\n\n\n## Quality Self-Check\n\nBefore returning results, verify:\n\n- [ ] API response was received successfully\n- [ ] Output is valid JSON (if structured output requested)\n- [ ] All requested analysis fields are present\n- [ ] Token usage is within expected bounds\n- [ ] No error messages in the response\n- [ ] For multi-image: all images were processed\n- [ ] Confidence\u002Fwarnings are included when analysis is uncertain\n\n\n## Limitations\n\n- **Medical imagery**: Not suitable for diagnostic analysis of CT scans, X-rays, or MRIs\n- **Small text**: Text smaller than ~12pt may be misread; use `detail: high`\n- **Rotated\u002Fskewed text**: Accuracy decreases with text rotation; pre-process if needed\n- **Non-Latin scripts**: Lower accuracy for complex scripts (CJK, Arabic, etc.)\n- **Object counting**: Approximate counts only; may miss or double-count similar objects\n- **Spatial precision**: Cannot provide pixel-accurate bounding boxes or measurements\n- **Panoramic\u002Ffisheye**: Distorted images reduce analysis accuracy\n- **Graphs and charts**: May misinterpret line styles, legends, or data points\n- **Metadata**: Cannot access EXIF data, camera info, or GPS coordinates from images\n- **Cost**: High-detail analysis of many images can be expensive; monitor token usage\n\n\n## Token Cost Estimation\n\nApproximate token costs for image inputs:\n\n| Detail Level | Tokens per Image | Best For |\n|-------------|------------------|----------|\n| low | ~85 tokens (fixed) | Quick classification, color detection |\n| high | 85 + 170 per 512x512 tile | OCR, detailed analysis, small objects |\n| auto | Variable | General use |\n\n```python\ndef estimate_image_tokens(image_path, detail=\"high\"):\n    \"\"\"Estimate token usage for an image.\"\"\"\n    if detail == \"low\":\n        return 85\n    \n    with Image.open(image_path) as img:\n        width, height = img.size\n    \n    # High detail: image is scaled to fit in 2048x2048, then tiled at 512x512\n    scale = min(2048 \u002F max(width, height), 1.0)\n    scaled_width = int(width * scale)\n    scaled_height = int(height * scale)\n    \n    # Ensure minimum 768 on shortest side\n    if min(scaled_width, scaled_height) \u003C 768:\n        scale = 768 \u002F min(scaled_width, scaled_height)\n        scaled_width = int(scaled_width * scale)\n        scaled_height = int(scaled_height * scale)\n    \n    # Calculate tiles\n    tiles_x = (scaled_width + 511) \u002F\u002F 512\n    tiles_y = (scaled_height + 511) \u002F\u002F 512\n    total_tiles = tiles_x * tiles_y\n    \n    return 85 + (170 * total_tiles)\n```\n\n\n## Version History\n\n- **1.0.0** (2026-01-21): Initial release with GPT-4o vision support\n",{"name":5,"description":7},"main","92a1ffe8fca31d0c743232b87e3b61d08d55fad7","harbor-index\u002Fdatasets\u002Fskillsbench-ocr-receipts-to-excel\u002Fenvironment\u002Fskills\u002Fopenai-vision",13,"https:\u002F\u002Fgithub.com\u002Fharbor-framework\u002Fharbor-adapters-experiments","https:\u002F\u002Fgithub.com\u002Fharbor-framework","https:\u002F\u002Fharborframework.com\u002F","2026-06-29 03:15:37",{"source":8,"skillName":5,"commands":43,"prompt":53,"zipAvailable":54},[44,47,50],{"tool":45,"command":46},"npx","npx skills add harbor-framework\u002Fharbor-adapters-experiments --skill openai-vision",{"tool":48,"command":49},"bunx","bunx skills add harbor-framework\u002Fharbor-adapters-experiments --skill openai-vision",{"tool":51,"command":52},"pnpm","pnpm dlx skills add harbor-framework\u002Fharbor-adapters-experiments --skill openai-vision","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fharbor-framework\u002Fharbor-adapters-experiments\u002Fblob\u002Fmain\u002Fharbor-index\u002Fdatasets\u002Fskillsbench-ocr-receipts-to-excel\u002Fenvironment\u002Fskills\u002Fopenai-vision\u002FSKILL.md",true,[],[57,78,94],{"skillNo":58,"name":59,"nameZh":6,"description":60,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":61,"categories":62,"keywords":68,"publishedAt":29},"01M71CC7RBYF4HZ1148W6KN8ZS","video-frame-extraction","Extract frames from video files and save them as images using OpenCV","https:\u002F\u002Fgithub.com\u002Fharbor-framework\u002Fharbor-adapters-experiments\u002Fblob\u002Fmain\u002Fharbor-index\u002Fdatasets\u002Fskillsbench-ocr-receipts-to-excel\u002Fenvironment\u002Fskills\u002Fvideo-frame-extraction\u002FSKILL.md",[63],{"categoryNo":64,"categoryName":65,"taxonomyType":66,"taxonomyTypeDesc":67},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[69,25,70,71,72,73,74,75,76,22,77],"Video","video frame extraction","extraction","files","save","opencv","extract frames","as","extract",{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":85,"publishedAt":29},"01M71CC1HXZW936NPEA2127FXR","image-ocr","Extract text content from images using Tesseract OCR via Python","https:\u002F\u002Fgithub.com\u002Fharbor-framework\u002Fharbor-adapters-experiments\u002Fblob\u002Fmain\u002Fharbor-index\u002Fdatasets\u002Fskillsbench-ocr-receipts-to-excel\u002Fenvironment\u002Fskills\u002Fimage-ocr\u002FSKILL.md",[84],{"categoryNo":64,"categoryName":65,"taxonomyType":66,"taxonomyTypeDesc":67},[86,87,88,77,89,90,91,92,93,22],"image","image ocr","ocr","content","tesseract","via","python","text",{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":29},"01M71CBW35P5SBZ4XAB9Z2B7H7","generate-greeting","Generate a greeting message and write it to a file.","https:\u002F\u002Fgithub.com\u002Fharbor-framework\u002Fharbor-adapters-experiments\u002Fblob\u002Fmain\u002Fexamples\u002Fdatasets\u002Fhello\u002Fhello-skills\u002Fenvironment\u002Fskills\u002Fgenerate-greeting\u002FSKILL.md",[],[101,102,103,104,105,106,107],"generate greeting","generate","greeting","write","it","file","message"]