openevidence-prod-checklist

OpenEvidence Production Checklist

Overview

Comprehensive checklist for deploying OpenEvidence clinical AI integrations to production healthcare environments.

Prerequisites

• Completed development and testing phases
• Signed BAA with OpenEvidence
• Security review completed
• Compliance team sign-off

Production Readiness Checklist

1. Legal & Compliance

• [ ] Business Associate Agreement (BAA) signed

  • Contact: compliance@openevidence.com
  • Required for any PHI handling

• [ ] API Terms of Service accepted

  • Review: https://www.openevidence.com/policies/api

• [ ] Clinical use disclaimers in place

  • All clinical outputs must include appropriate disclaimers
  • "For informational purposes only. Verify with current guidelines."

• [ ] Liability and malpractice considerations reviewed

  • Legal review of clinical decision support usage

• [ ] Data processing agreement documented

  • What data is sent, stored, processed

2. Security

• [ ] Production API keys generated

  • Separate keys from development/staging
  • Stored in secret manager (not environment variables)

• [ ] Key rotation schedule established

  • Recommend: 90-day rotation cycle

• [ ] IP allowlist configured (if available)

  • Restrict API access to known IP ranges

• [ ] TLS 1.2+ enforced

  • No fallback to older protocols

• [ ] PHI sanitization implemented

  • Review openevidence-security-basics

• [ ] Webhook signature verification active

  • All webhook endpoints validate signatures

• [ ] Audit logging enabled

  • HIPAA-compliant access logging
  • Minimum 6-year retention

3. Infrastructure

• [ ] High availability configuration

  // Multiple API endpoints for failover
  const config = {
    primaryEndpoint: 'https://api.openevidence.com',
    fallbackEndpoint: 'https://api-fallback.openevidence.com',
    timeout: 30000,
    retries: 3,
  };
  

• [ ] Circuit breaker implemented

  import CircuitBreaker from 'opossum';
  
  const breaker = new CircuitBreaker(clinicalQuery, {
    timeout: 30000,
    errorThresholdPercentage: 50,
    resetTimeout: 30000,
  });
  

• [ ] Rate limit handling configured

  • Review openevidence-rate-limits
  • Request queuing for traffic spikes

• [ ] Timeout configuration appropriate

  • Clinical queries: 15-30s
  • DeepConsult: 5-10 minutes

• [ ] Health check endpoint implemented

  app.get('/health/openevidence', async (req, res) => {
    try {
      await client.health.check();
      res.json({ status: 'healthy' });
    } catch (error) {
      res.status(503).json({ status: 'unhealthy', error: error.message });
    }
  });
  

4. Monitoring & Alerting

• [ ] Metrics collection active

  • Request count, latency, error rate
  • Review openevidence-observability

• [ ] Alert rules configured

  Alert	Threshold	Severity
  Error rate > 5%	5 min window	Warning
  Error rate > 20%	5 min window	Critical
  P95 latency > 15s	5 min window	Warning
  Health check failed	2 consecutive	Critical

• [ ] Dashboard created

  • Real-time visibility into OpenEvidence integration health

• [ ] Log aggregation configured

  • Centralized logging for debugging
  • PHI excluded from logs

5. Error Handling & Resilience

• [ ] Graceful degradation implemented

  async function clinicalQueryWithFallback(question: string) {
    try {
      return await openEvidenceQuery(question);
    } catch (error) {
      // Log error, alert, return cached or default response
      await alertService.send('OpenEvidence unavailable');
      return {
        answer: 'Clinical evidence service temporarily unavailable. Please consult current guidelines directly.',
        fallback: true,
      };
    }
  }
  

• [ ] Retry logic tested

  • Exponential backoff with jitter
  • Respects Retry-After headers

• [ ] Error classification implemented

  • Review openevidence-common-errors

6. Performance

• [ ] Response caching strategy

  • Cache frequently queried clinical information
  • Cache TTL appropriate for clinical data freshness

• [ ] Connection pooling configured

  • Reuse HTTP connections

• [ ] Query optimization reviewed

  • Clear, specific clinical questions
  • Appropriate context provided

• [ ] Load testing completed

  • Test at 2x expected peak traffic
  • Verify rate limits not exceeded

7. Documentation & Training

• [ ] Runbook created

  • Incident response procedures
  • Review openevidence-incident-runbook

• [ ] On-call training completed

  • Team knows how to triage OpenEvidence issues

• [ ] User documentation updated

  • How to use clinical decision support
  • Limitations and appropriate use

• [ ] Clinical staff training

  • Understanding AI-assisted decision support
  • When to rely on vs. question AI recommendations

8. Testing & Validation

• [ ] Integration tests passing

  • Against sandbox environment

• [ ] End-to-end tests passing

  • Full workflow validation

• [ ] Clinical validation completed

  • Sample queries reviewed by clinical team
  • Responses appropriate and accurate

• [ ] Performance benchmarks met

  • P50 < 5s, P95 < 15s for clinical queries
  • P50 < 3min, P95 < 5min for DeepConsult

9. Deployment

• [ ] Blue-green or canary deployment planned

  • Gradual rollout with monitoring

• [ ] Rollback procedure documented

  • Quick revert if issues detected

• [ ] Feature flags configured

  • Ability to disable OpenEvidence quickly

• [ ] Database migrations completed

  • Audit log tables, cache tables ready

10. Post-Launch

• [ ] Monitoring dashboard reviewed daily (first week)

• [ ] Error reports triaged immediately

• [ ] User feedback collection active

• [ ] Performance baseline established

Production Configuration Example

// config/openevidence.production.ts
export const productionConfig = {
  // Credentials from secret manager
  credentials: {
    source: 'secret-manager',
    apiKeyPath: 'projects/prod/secrets/openevidence-api-key',
    orgIdPath: 'projects/prod/secrets/openevidence-org-id',
  },

  // Endpoints
  api: {
    baseUrl: 'https://api.openevidence.com',
    timeout: 30000,
    retries: 3,
  },

  // Rate limiting
  rateLimits: {
    enabled: true,
    maxConcurrent: 10,
    requestsPerMinute: 100,
  },

  // Circuit breaker
  circuitBreaker: {
    enabled: true,
    errorThreshold: 50,
    resetTimeout: 30000,
  },

  // Caching
  cache: {
    enabled: true,
    ttlSeconds: 3600,
    maxEntries: 10000,
  },

  // Monitoring
  monitoring: {
    metricsEnabled: true,
    tracingEnabled: true,
    logLevel: 'info',
  },
};

Output

• Completed production readiness checklist
• Configuration validated
• All stakeholders signed off
• Go-live approved

Resources

• OpenEvidence
• OpenEvidence Security

Next Steps

For version upgrades, see openevidence-upgrade-migration.