
Implementation Guide: Manage between-session check-in messages and triage responses before next appointment
Step-by-step implementation guide for deploying AI to manage between-session check-in messages and triage responses before next appointment for Allied & Mental Health clients.
Hardware Procurement
No Specialized Hardware Required
No Specialized Hardware Required
$0 — This is a cloud-native, software-defined solution
This solution runs entirely on cloud infrastructure managed by the MSP. Clinicians access the dashboard via existing workstations, laptops, or mobile devices. No edge hardware, specialized servers, cameras, or microphones are needed at the practice. Confirm that clinicians have a modern web browser (Chrome, Edge, Safari) and a smartphone capable of receiving push notifications or SMS alerts.
Software Procurement
OpenAI API (GPT-4.1-mini) — Primary LLM
$0.40/1M input tokens, $1.60/1M output tokens. Estimated $50–$150/month for 200 patients. MSP resale: $100–$250/month
Primary LLM for processing patient check-in responses, sentiment analysis, triage classification, and generating structured pre-appointment summaries. Handles 90%+ of routine message processing at low cost.
OpenAI API (GPT-5.4) — Escalation LLM
$2.50/1M input tokens, $10.00/1M output tokens. Estimated $20–$80/month for escalated messages only. MSP resale: $40–$140/month
Higher-capability LLM used only for messages flagged as clinically significant by the primary model or crisis detection layer. Provides deeper reasoning and more nuanced triage for edge cases.
Twilio Programmable Messaging (SMS)
$0.0079/SMS sent or received + $2/month per phone number. Estimated $30–$100/month for 200 patients. MSP resale: $75–$200/month
HIPAA-eligible SMS delivery platform for sending check-in prompts to patients and receiving their responses. Twilio signs BAAs for eligible products. Provides webhook-based inbound message routing to the agent.
$149.99/month (Group Plan, 1 provider + 1 support) + $50/month per additional provider. MSP resale: $250–$400/month
Recommended EHR with full GraphQL API and webhook support. Provides the appointment schedule data (to trigger check-ins), patient roster, and the ability to push pre-appointment summaries back into the patient record. HIPAA, SOC-2, PIPEDA, GDPR, and PCI compliant.
AWS ECS Fargate (Container Compute)
$50–$150/month for agent containers (2 vCPU, 4GB RAM typical). MSP resale: $100–$300/month
Serverless container hosting for the AI agent core service, API gateway, and clinician dashboard backend. No server management required. Runs within HIPAA-eligible AWS infrastructure with BAA.
AWS RDS for PostgreSQL (Database)
$30–$80/month (direct); MSP resale: $60–$160/month
Managed PostgreSQL database for storing patient check-in conversations, triage results, clinician configurations, audit logs, and session summaries. Encrypted at rest (AES-256) and in transit (TLS 1.2+).
AWS SQS (Message Queue)
$1–$5/month at typical volume. Included in cloud hosting markup.
Asynchronous message queue for decoupling inbound patient SMS reception from AI processing. Ensures no messages are lost during processing spikes and provides retry logic for failed LLM calls.
AWS S3 (Encrypted Object Storage)
$5–$15/month. Included in cloud hosting markup.
Encrypted storage for conversation archives, audit logs, consent records, and system backups. Server-side encryption with AWS KMS. Required for HIPAA audit trail retention (minimum 6 years).
AWS KMS (Key Management Service)
$1–$3/month per key + $0.03/10K requests. Included in cloud hosting markup.
Centralized encryption key management for all data at rest (RDS, S3) and application-level encryption of PHI fields. Customer-managed keys provide full control over data encryption lifecycle.
$0 license cost; runs on existing AWS infrastructure. MSP labor for setup included in implementation.
Workflow orchestration engine that connects the appointment schedule polling, check-in message dispatch, inbound message routing, LLM processing pipeline, triage logic, and clinician notification delivery. Self-hosted on AWS for full HIPAA data control.
Auth0 (Identity & Access Management)
Free tier for up to 7,500 MAU; $23/month for Professional. MSP resale: $50–$100/month
Provides SSO, MFA, and role-based access control for the clinician dashboard. Supports HIPAA-eligible configurations with audit logging of all authentication events. License type: SaaS per-user.
Prerequisites
- Practice must have a HIPAA-compliant EHR system — Healthie is strongly recommended for full API integration; if using SimplePractice or TherapyNotes (no public API), the system will operate as a standalone platform with manual data handoff
- Practice must have a documented patient consent workflow for AI-assisted between-session communication, reviewed by a healthcare attorney
- Practice must execute Business Associate Agreements with the MSP and all subprocessors (OpenAI, AWS, Twilio)
- Practice must have at least one designated Clinical Safety Officer (licensed clinician) who reviews escalated messages and participates in system design
- If the practice treats substance use disorders, a separate 42 CFR Part 2-compliant consent form must be developed (compliance deadline: February 16, 2026) — consult healthcare attorney
- Practice must have business internet connectivity (25+ Mbps minimum) and clinicians must have smartphones capable of receiving push notifications or SMS alerts
- Practice must provide a dedicated phone number or approve provisioning of a new Twilio number for patient check-in SMS
- Practice must complete or update their HIPAA Security Risk Assessment — HHS provides a free SRA Tool (v3.6) or the MSP can perform a third-party assessment ($2,000–$10,000)
- Practice staff must agree to complete HIPAA compliance training covering AI-assisted communication workflows
- DNS domain must be available or existing domain accessible for the clinician dashboard (e.g., checkin.practicename.com)
- An AWS account must be created or an existing account designated, with an Organization-level BAA signed via AWS Artifact
- A valid SSL/TLS certificate must be provisioned for the dashboard domain (AWS Certificate Manager recommended)
Installation Steps
...
Step 1: Execute All Business Associate Agreements
Before any technical work begins, execute BAAs with every vendor that will handle PHI. This is a legal prerequisite for HIPAA compliance. The MSP must sign a BAA with the practice (MSP as Business Associate), then execute sub-BA agreements or confirm BAA coverage with AWS, OpenAI, Twilio, Healthie, and Auth0.
- AWS BAA: Sign via AWS Artifact in the AWS Management Console — Navigate to: AWS Console → AWS Artifact → Agreements → AWS Business Associate Addendum → Accept
- OpenAI BAA: Apply via https://privacy.openai.com/policies — available for API customers without enterprise contract
- Twilio BAA: Request via https://www.twilio.com/en-us/hipaa — execute before enabling messaging
- Healthie BAA: Included in subscription — confirm execution during onboarding
- Auth0 BAA: Available on paid plans — contact sales or enable via dashboard
Do NOT proceed to any step involving PHI until ALL BAAs are fully executed and filed. Maintain a BAA registry spreadsheet with vendor name, date signed, renewal date, and responsible contact. Store copies in a secure, access-controlled location.
Step 2: Provision HIPAA-Eligible AWS Infrastructure
Set up the foundational AWS infrastructure within a HIPAA-eligible configuration. This includes creating a dedicated VPC with private subnets, enabling CloudTrail for audit logging, configuring KMS encryption keys, and setting up the core services (ECS, RDS, SQS, S3).
# Install and configure AWS CLI
pip install awscli
aws configure --profile checkin-agent
# Create dedicated VPC with private subnets
aws ec2 create-vpc --cidr-block 10.0.0.0/16 --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=checkin-agent-vpc},{Key=Environment,Value=production},{Key=HIPAA,Value=true}]'
# Create private subnets in two AZs for high availability
aws ec2 create-subnet --vpc-id <VPC_ID> --cidr-block 10.0.1.0/24 --availability-zone us-east-1a --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=private-1a}]'
aws ec2 create-subnet --vpc-id <VPC_ID> --cidr-block 10.0.2.0/24 --availability-zone us-east-1b --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=private-1b}]'
# Create public subnets for ALB
aws ec2 create-subnet --vpc-id <VPC_ID> --cidr-block 10.0.10.0/24 --availability-zone us-east-1a --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=public-1a}]'
aws ec2 create-subnet --vpc-id <VPC_ID> --cidr-block 10.0.11.0/24 --availability-zone us-east-1b --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=public-1b}]'
# Enable CloudTrail for audit logging
aws cloudtrail create-trail --name checkin-agent-audit --s3-bucket-name checkin-agent-audit-logs --is-multi-region-trail --enable-log-file-validation
aws cloudtrail start-logging --name checkin-agent-audit
# Create KMS key for data encryption
aws kms create-key --description 'CheckIn Agent PHI encryption key' --tags TagKey=HIPAA,TagValue=true
# Create encrypted S3 bucket for audit logs and archives
aws s3api create-bucket --bucket checkin-agent-data-<ACCOUNT_ID> --region us-east-1
aws s3api put-bucket-encryption --bucket checkin-agent-data-<ACCOUNT_ID> --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"<KMS_KEY_ID>"},"BucketKeyEnabled":true}]}'
aws s3api put-public-access-block --bucket checkin-agent-data-<ACCOUNT_ID> --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
# Create SQS queue for message processing
aws sqs create-queue --queue-name checkin-inbound-messages --attributes '{"KmsMasterKeyId":"<KMS_KEY_ID>","MessageRetentionPeriod":"1209600","VisibilityTimeout":"120"}'
# Create SQS dead-letter queue for failed messages
aws sqs create-queue --queue-name checkin-inbound-messages-dlq --attributes '{"KmsMasterKeyId":"<KMS_KEY_ID>","MessageRetentionPeriod":"1209600"}'Use us-east-1 or us-west-2 as the primary region — both are HIPAA eligible. Enable VPC Flow Logs for network audit trail. All S3 buckets must have public access blocked. Use Terraform or CloudFormation for reproducible infrastructure-as-code deployments in production — the manual CLI commands above are for initial setup understanding.
Step 3: Deploy PostgreSQL Database with Encryption
Provision an AWS RDS PostgreSQL instance with encryption at rest (KMS), encryption in transit (TLS), automated backups, and multi-AZ deployment for reliability. This database stores patient check-in conversations, triage results, consent records, and audit logs.
# Create DB subnet group using private subnets
aws rds create-db-subnet-group --db-subnet-group-name checkin-agent-db-subnets --db-subnet-group-description 'Private subnets for RDS' --subnet-ids <PRIVATE_SUBNET_1A_ID> <PRIVATE_SUBNET_1B_ID>
# Create security group for RDS (allow only from agent containers)
aws ec2 create-security-group --group-name checkin-rds-sg --description 'RDS access from agent only' --vpc-id <VPC_ID>
aws ec2 authorize-security-group-ingress --group-id <RDS_SG_ID> --protocol tcp --port 5432 --source-group <ECS_SG_ID>
# Create encrypted RDS instance
aws rds create-db-instance \
--db-instance-identifier checkin-agent-db \
--db-instance-class db.t3.medium \
--engine postgres \
--engine-version 16.4 \
--master-username checkin_admin \
--master-user-password '<GENERATE_STRONG_PASSWORD>' \
--allocated-storage 20 \
--max-allocated-storage 100 \
--storage-encrypted \
--kms-key-id <KMS_KEY_ID> \
--vpc-security-group-ids <RDS_SG_ID> \
--db-subnet-group-name checkin-agent-db-subnets \
--multi-az \
--backup-retention-period 35 \
--deletion-protection \
--no-publicly-accessible
# After instance is available, connect and create the schema
psql -h <RDS_ENDPOINT> -U checkin_admin -d postgres
# Run schema creation (see custom_ai_components for full schema)Store the master password in AWS Secrets Manager, NOT in code or config files. The backup retention period of 35 days exceeds the HIPAA minimum. Enable Performance Insights for monitoring. Multi-AZ deployment ensures automatic failover. The --no-publicly-accessible flag is critical — the database must only be reachable from within the VPC.
Step 4: Configure Twilio for HIPAA-Compliant Messaging
Set up a Twilio account with HIPAA-eligible configuration, provision a phone number for patient check-in SMS, configure inbound webhook routing, and enable message retention controls.
# 5. Install Twilio SDK in the agent project
pip install twilio
# 6. Store credentials in AWS Secrets Manager
aws secretsmanager create-secret --name checkin-agent/twilio --secret-string '{"account_sid":"<TWILIO_SID>","auth_token":"<TWILIO_TOKEN>","phone_number":"+1XXXXXXXXXX"}'Setting Twilio message retention to 0 days minimizes PHI stored at Twilio. All message content is persisted in the MSP-managed encrypted database instead. Validate the Twilio webhook signature on every inbound request to prevent spoofing. Consider using Twilio Messaging Services for better deliverability and compliance features including opt-out handling (STOP keyword).
Step 5: Set Up Healthie EHR API Integration
Configure API access to Healthie for pulling appointment schedules, patient contact information, and pushing pre-appointment summaries. Healthie uses a GraphQL API with webhook support for real-time event notifications.
# 2. Store API key in AWS Secrets Manager
aws secretsmanager create-secret --name checkin-agent/healthie --secret-string '{"api_key":"<HEALTHIE_API_KEY>","api_url":"https://api.gethealthie.com/graphql"}'
# 3. Test API connectivity
curl -X POST https://api.gethealthie.com/graphql \
-H 'Content-Type: application/json' \
-H 'Authorization: Basic <HEALTHIE_API_KEY>' \
-d '{"query": "{ appointments(filter: \"upcoming\") { id date patient { id first_name last_name phone_number } } }"}'If the practice uses SimplePractice or TherapyNotes (no public API), skip this step. Instead, implement a manual CSV upload workflow or schedule-based polling through a shared Google Calendar. This significantly reduces automation but is the only option for closed-API EHRs. Document this limitation for the client and recommend Healthie migration for full functionality.
Step 6: Deploy n8n Workflow Automation Engine
Deploy a self-hosted n8n instance on AWS ECS Fargate to orchestrate the check-in workflow. n8n connects appointment schedule polling, message dispatch timing, inbound message routing, and clinician notification logic without custom code for the orchestration layer.
# Create ECR repository for n8n
aws ecr create-repository --repository-name checkin-agent/n8n
# Pull and push n8n Docker image
docker pull n8nio/n8n:latest
docker tag n8nio/n8n:latest <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/checkin-agent/n8n:latest
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com
docker push <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/checkin-agent/n8n:latest
# Create ECS task definition for n8n (save as n8n-task-def.json)
cat > n8n-task-def.json << 'EOF'
{
"family": "checkin-n8n",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"cpu": "1024",
"memory": "2048",
"executionRoleArn": "<ECS_EXECUTION_ROLE_ARN>",
"taskRoleArn": "<ECS_TASK_ROLE_ARN>",
"containerDefinitions": [{
"name": "n8n",
"image": "<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/checkin-agent/n8n:latest",
"portMappings": [{"containerPort": 5678, "protocol": "tcp"}],
"environment": [
{"name": "N8N_ENCRYPTION_KEY", "value": "<GENERATE_RANDOM_KEY>"},
{"name": "DB_TYPE", "value": "postgresdb"},
{"name": "DB_POSTGRESDB_HOST", "value": "<RDS_ENDPOINT>"},
{"name": "DB_POSTGRESDB_DATABASE", "value": "n8n"},
{"name": "DB_POSTGRESDB_USER", "value": "n8n_user"},
{"name": "DB_POSTGRESDB_PASSWORD", "value": "<N8N_DB_PASSWORD>"},
{"name": "N8N_BASIC_AUTH_ACTIVE", "value": "true"},
{"name": "N8N_BASIC_AUTH_USER", "value": "admin"},
{"name": "N8N_BASIC_AUTH_PASSWORD", "value": "<ADMIN_PASSWORD>"}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/checkin-n8n",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "n8n"
}
}
}]
}
EOF
aws ecs register-task-definition --cli-input-json file://n8n-task-def.json
# Create ECS service
aws ecs create-service --cluster checkin-agent --service-name n8n --task-definition checkin-n8n --desired-count 1 --launch-type FARGATE --network-configuration 'awsvpcConfiguration={subnets=[<PRIVATE_SUBNET_1A>,<PRIVATE_SUBNET_1B>],securityGroups=[<N8N_SG_ID>],assignPublicIp=DISABLED}'n8n stores workflow definitions in its own database — create a separate 'n8n' database in the same RDS instance. The n8n encryption key encrypts stored credentials; back it up securely in Secrets Manager. Access n8n via the ALB on a restricted path (e.g., /n8n/) with IP allowlisting for MSP admin IPs only. Never expose n8n directly to the internet.
Step 7: Build and Deploy the AI Agent Core Service
Build the core Python application that contains the LLM integration, crisis detection engine, triage logic, conversation management, and API endpoints for Twilio webhooks, Healthie webhooks, and the clinician dashboard. Deploy as a Docker container on ECS Fargate.
# Create project structure
mkdir -p checkin-agent/{api,services,models,prompts,tests}
cd checkin-agent
# Create requirements.txt
cat > requirements.txt << 'EOF'
fastapi==0.115.0
uvicorn==0.30.0
openai==1.52.0
twilio==9.3.0
httpx==0.27.0
psycopg2-binary==2.9.9
sqlalchemy==2.0.35
alembic==1.13.0
pydantic==2.9.0
python-jose[cryptography]==3.3.0
boto3==1.35.0
python-dotenv==1.0.1
structlog==24.4.0
sentry-sdk[fastapi]==2.14.0
EOF
# Build Docker image
cat > Dockerfile << 'EOF'
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8000"]
EOF
# Build and push to ECR
aws ecr create-repository --repository-name checkin-agent/core
docker build -t checkin-agent-core .
docker tag checkin-agent-core:latest <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/checkin-agent/core:latest
docker push <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/checkin-agent/core:latest
# Deploy ECS task and service (similar pattern to n8n)
# Use 2 vCPU, 4GB RAM task definition
# Set desired count to 2 for high availability
# Configure ALB health check on /health endpointSee the custom_ai_components section for complete implementation code for all service modules including the FastAPI application, crisis detection engine, triage classifier, prompt templates, and database models. The container must have IAM role access to Secrets Manager, SQS, S3, and KMS. Use AWS Application Load Balancer with ACM certificate for HTTPS termination.
Step 8: Deploy the Clinician Dashboard (Frontend)
Build and deploy a React-based clinician dashboard that displays triaged patient check-in summaries, allows clinicians to review conversations, acknowledge alerts, and configure per-patient check-in preferences. Deploy as a static site on S3 + CloudFront with the API backend on ECS.
# Create React app
npx create-react-app clinician-dashboard --template typescript
cd clinician-dashboard
# Install dependencies
npm install @auth0/auth0-react axios react-router-dom @mui/material @emotion/react @emotion/styled date-fns recharts
# Build for production
npm run build
# Create S3 bucket for frontend hosting
aws s3 mb s3://checkin-dashboard-<PRACTICE_ID>
aws s3 website s3://checkin-dashboard-<PRACTICE_ID> --index-document index.html --error-document index.html
# Upload build files
aws s3 sync build/ s3://checkin-dashboard-<PRACTICE_ID>/ --delete
# Create CloudFront distribution with custom domain
aws cloudfront create-distribution --distribution-config file://cloudfront-config.json
# Configure Route53 DNS for checkin.practicename.com → CloudFrontThe dashboard is intentionally simple — it is a read-mostly interface. Key views: (1) Today's alerts sorted by urgency, (2) Pre-appointment summaries for upcoming sessions, (3) Full conversation history per patient, (4) System settings. Auth0 provides MFA-enforced login. All API calls go through the ALB to the ECS backend. See custom_ai_components for dashboard specification.
Step 9: Configure Auth0 for Clinician Authentication
Set up Auth0 as the identity provider for the clinician dashboard with mandatory MFA, role-based access control (Clinician, Admin, ReadOnly), and audit logging of all authentication events.
aws secretsmanager create-secret --name checkin-agent/auth0 --secret-string '{"domain":"<TENANT>.auth0.com","client_id":"<CLIENT_ID>","client_secret":"<CLIENT_SECRET>","audience":"https://api.checkin.practicename.com"}'MFA is mandatory for HIPAA compliance — do not allow users to skip MFA enrollment. Configure Auth0 Log Streams to forward authentication events to CloudWatch for centralized audit logging. Set session timeout to 15 minutes of inactivity per HIPAA best practices.
Step 10: Configure n8n Workflows for Check-In Orchestration
Create the four core n8n workflows that orchestrate the agent: (1) Appointment Completion → Schedule Check-In, (2) Timed Check-In Dispatch, (3) Inbound Response Processing, (4) Pre-Appointment Summary Generation.
- Access n8n at https://n8n.internal.checkin.practicename.com (MSP admin only)
- Workflow 1: Appointment Completion Trigger — Trigger: Webhook node receiving Healthie appointment.completed webhook. Steps: (1) Webhook (POST /healthie/webhook/appointment-completed), (2) Function node: Extract patient_id, appointment_date, clinician_id, (3) HTTP Request: Query Healthie API for patient phone number, (4) Function node: Calculate check-in time (appointment_date + 24 hours), (5) HTTP Request: POST to agent API /api/schedule-checkin with body: {patient_id, phone, clinician_id, send_at, appointment_id}
- Workflow 2: Timed Check-In Dispatch (runs every 5 minutes) — Trigger: Cron node (*/5 * * * *). Steps: (1) HTTP Request: GET /api/pending-checkins?due_before=now, (2) SplitInBatches: Process each pending check-in, (3) HTTP Request: POST /api/send-checkin/{checkin_id}, (4) This triggers the Twilio SMS send from the agent core
- Workflow 3: Inbound Response Processing — Trigger: Webhook (POST /twilio/inbound) forwarded from agent API. Steps: (1) Agent core receives Twilio webhook, validates signature, (2) Agent core enqueues message to SQS, (3) This n8n workflow polls SQS (or receives webhook from agent), (4) HTTP Request: POST /api/process-response/{message_id}, (5) Agent core runs crisis detection + LLM triage, (6) If crisis: HTTP Request POST /api/alert/crisis/{clinician_id}, (7) If urgent: HTTP Request POST /api/alert/urgent/{clinician_id}
- Workflow 4: Pre-Appointment Summary (runs every 30 minutes) — Trigger: Cron node (*/30 * * * *). Steps: (1) HTTP Request: GET /api/upcoming-appointments?within_hours=24, (2) For each appointment with check-in data: HTTP Request POST /api/generate-summary/{patient_id}/{appointment_id}, (3) Agent core generates LLM summary of all check-in exchanges, (4) HTTP Request: POST Healthie API to create chart note with summary
- Import workflows via n8n CLI or manually create in n8n UI
Each workflow should have error handling nodes that send failure alerts to the MSP monitoring channel (Slack/PagerDuty). The 5-minute polling for check-in dispatch provides near-real-time behavior without overloading the system. For practices not on Healthie, replace Workflow 1 with a manual trigger or shared calendar polling approach.
Step 11: Run Database Migrations and Seed Initial Data
Apply the database schema to the RDS PostgreSQL instance, create initial configuration records, and seed the crisis keyword dictionary.
# Connect to RDS via bastion or VPN
psql -h <RDS_ENDPOINT> -U checkin_admin -d checkin_agent
# Create the application database and user
CREATE DATABASE checkin_agent;
CREATE USER checkin_app WITH ENCRYPTED PASSWORD '<APP_DB_PASSWORD>';
GRANT ALL PRIVILEGES ON DATABASE checkin_agent TO checkin_app;
# Run Alembic migrations from the agent container
docker exec -it <AGENT_CONTAINER_ID> alembic upgrade head
# Seed crisis keywords (run via agent API or direct SQL)
INSERT INTO crisis_keywords (keyword, severity, category) VALUES
('kill myself', 'critical', 'suicidal_ideation'),
('want to die', 'critical', 'suicidal_ideation'),
('end my life', 'critical', 'suicidal_ideation'),
('suicide', 'critical', 'suicidal_ideation'),
('suicidal', 'critical', 'suicidal_ideation'),
('self-harm', 'high', 'self_harm'),
('cutting myself', 'high', 'self_harm'),
('hurt myself', 'high', 'self_harm'),
('overdose', 'critical', 'self_harm'),
('kill someone', 'critical', 'homicidal_ideation'),
('hurt someone', 'high', 'homicidal_ideation'),
('not safe', 'high', 'safety_concern'),
('being abused', 'high', 'safety_concern'),
('no reason to live', 'critical', 'suicidal_ideation'),
('better off dead', 'critical', 'suicidal_ideation'),
('plan to end', 'critical', 'suicidal_ideation'),
('goodbye letter', 'critical', 'suicidal_ideation'),
('giving away things', 'high', 'suicidal_ideation'),
('stockpiling pills', 'critical', 'suicidal_ideation'),
('gun', 'high', 'means_access'),
('weapon', 'high', 'means_access');
# Seed initial clinician accounts
# (These are synced from Healthie or manually entered)
INSERT INTO clinicians (external_id, name, email, phone, role) VALUES
('<HEALTHIE_PROVIDER_ID>', 'Dr. Jane Smith', 'jane@practice.com', '+1XXXXXXXXXX', 'clinician');The crisis keyword list should be reviewed and expanded by the Clinical Safety Officer before go-live. Use Alembic for all schema changes going forward to maintain migration history. The application database user should have limited permissions (no DROP, no SUPERUSER). Store the app database password in Secrets Manager and inject via ECS environment.
Step 12: Configure Monitoring, Alerting, and Audit Logging
Set up comprehensive monitoring for system health, message delivery, crisis detection events, and HIPAA-required audit logging. Configure alerts for critical failures that could result in missed patient messages or crisis responses.
# Create CloudWatch Log Groups
aws logs create-log-group --log-group-name /checkin-agent/core --retention-in-days 2557
aws logs create-log-group --log-group-name /checkin-agent/n8n --retention-in-days 2557
aws logs create-log-group --log-group-name /checkin-agent/audit --retention-in-days 2557
# Create SNS topic for critical alerts
aws sns create-topic --name checkin-agent-critical-alerts
aws sns subscribe --topic-arn <TOPIC_ARN> --protocol email --notification-endpoint msp-oncall@mspcompany.com
aws sns subscribe --topic-arn <TOPIC_ARN> --protocol sms --notification-endpoint +1XXXXXXXXXX
# Create CloudWatch Alarms
# Alarm: Agent service unhealthy
aws cloudwatch put-metric-alarm --alarm-name checkin-agent-unhealthy --metric-name HealthyHostCount --namespace AWS/ApplicationELB --statistic Minimum --period 60 --evaluation-periods 3 --threshold 1 --comparison-operator LessThanThreshold --alarm-actions <TOPIC_ARN> --dimensions Name=TargetGroup,Value=<TARGET_GROUP_ARN>
# Alarm: SQS dead-letter queue has messages (failed processing)
aws cloudwatch put-metric-alarm --alarm-name checkin-dlq-messages --metric-name ApproximateNumberOfMessagesVisible --namespace AWS/SQS --statistic Sum --period 300 --evaluation-periods 1 --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold --alarm-actions <TOPIC_ARN> --dimensions Name=QueueName,Value=checkin-inbound-messages-dlq
# Alarm: No check-ins sent in 24 hours (during business days)
# Custom metric published by agent core
aws cloudwatch put-metric-alarm --alarm-name checkin-no-sends-24h --metric-name CheckInsSent --namespace CheckinAgent --statistic Sum --period 86400 --evaluation-periods 1 --threshold 1 --comparison-operator LessThanThreshold --alarm-actions <TOPIC_ARN>
# Alarm: Crisis detection triggered (informational to MSP)
aws cloudwatch put-metric-alarm --alarm-name checkin-crisis-detected --metric-name CrisisDetected --namespace CheckinAgent --statistic Sum --period 300 --evaluation-periods 1 --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold --alarm-actions <TOPIC_ARN>
# Set up CloudWatch dashboard
aws cloudwatch put-dashboard --dashboard-name CheckinAgent --dashboard-body file://dashboard.jsonHIPAA requires audit logs to be retained for a minimum of 6 years (2,557 days). The 'no check-ins sent' alarm catches silent failures where the system appears healthy but is not actually processing. The crisis detection alarm notifies the MSP so they can verify the clinician was properly alerted. Consider integrating with PagerDuty or OpsGenie for on-call rotation.
Step 13: Clinical Safety Testing and Crisis Detection Validation
Before any patient contact, perform exhaustive testing of the crisis detection system, triage accuracy, and escalation pathways. This step requires the Clinical Safety Officer (licensed clinician) to participate directly.
# Run the automated crisis detection test suite
cd checkin-agent
python -m pytest tests/test_crisis_detection.py -v --tb=long
# Run the triage accuracy test suite
python -m pytest tests/test_triage_classifier.py -v --tb=long
# Run end-to-end test scenarios (uses test phone numbers)
python -m pytest tests/test_e2e_scenarios.py -v --tb=long
# Manual test: Send crisis message via Twilio test number
curl -X POST https://api.checkin.practicename.com/test/simulate-inbound \
-H 'Authorization: Bearer <TEST_TOKEN>' \
-H 'Content-Type: application/json' \
-d '{"from": "+15551234567", "body": "I have been thinking about ending my life. I have pills saved up.", "test_mode": true}'
# Verify: Crisis alert received by test clinician within 60 seconds
# Verify: Patient receives 988 Lifeline information
# Verify: Conversation flagged as CRISIS in dashboard
# Verify: Audit log entry created
# Manual test: Send message with ambiguous language
curl -X POST https://api.checkin.practicename.com/test/simulate-inbound \
-H 'Authorization: Bearer <TEST_TOKEN>' \
-H 'Content-Type: application/json' \
-d '{"from": "+15551234567", "body": "I just feel like giving up on everything. Nothing matters anymore.", "test_mode": true}'
# Verify: Escalated to GPT-5.4 for deeper analysis
# Verify: Classified as at minimum URGENT
# Verify: Clinician alert triggeredDO NOT go live until the Clinical Safety Officer has signed off on crisis detection accuracy. Test at least 50 crisis scenarios, 50 ambiguous scenarios, and 100 routine scenarios. Document all test results and clinician sign-off in the compliance file. The crisis detection system should err heavily toward false positives — it is far better to over-alert clinicians than to miss a genuine crisis. Target: 100% recall on explicit crisis language, >90% recall on ambiguous crisis language.
Step 14: Patient Consent and Onboarding Configuration
Configure the patient consent workflow, opt-in process, and initial onboarding messages. Patients must explicitly consent to AI-assisted check-in communications before receiving any messages.
# Create consent template in the system
curl -X POST https://api.checkin.practicename.com/api/admin/consent-template \
-H 'Authorization: Bearer <ADMIN_TOKEN>' \
-H 'Content-Type: application/json' \
-d '{
"template_name": "standard_checkin_consent",
"version": "1.0",
"text": "I consent to receive automated between-session check-in messages from [Practice Name] via SMS. I understand that: (1) These messages are generated with AI assistance and are NOT a substitute for therapy or emergency services. (2) My responses will be reviewed by my clinician. (3) If I am in crisis, I should call 988 (Suicide & Crisis Lifeline), text HOME to 741741, or call 911. (4) I can opt out at any time by replying STOP. (5) Standard messaging rates may apply.",
"requires_signature": true,
"sud_variant": false
}'
# Create SUD-specific consent template (42 CFR Part 2)
curl -X POST https://api.checkin.practicename.com/api/admin/consent-template \
-H 'Authorization: Bearer <ADMIN_TOKEN>' \
-H 'Content-Type: application/json' \
-d '{
"template_name": "sud_checkin_consent",
"version": "1.0",
"text": "[ADDITIONAL CONSENT FOR SUBSTANCE USE DISORDER TREATMENT] In addition to the standard check-in consent, I specifically consent to the use of my substance use disorder treatment information in automated check-in communications. I understand this information receives additional federal protections under 42 CFR Part 2 and this consent may be revoked at any time.",
"requires_signature": true,
"sud_variant": true
}'
# Configure opt-in onboarding message
curl -X POST https://api.checkin.practicename.com/api/admin/message-template \
-H 'Authorization: Bearer <ADMIN_TOKEN>' \
-H 'Content-Type: application/json' \
-d '{
"template_name": "opt_in_confirmation",
"text": "Hi {first_name}, this is {practice_name}. Your therapist has invited you to receive between-session check-in messages. Reply YES to opt in or STOP at any time to opt out. Note: This is not an emergency service. If you are in crisis, call 988."
}'Consent must be collected BEFORE the first check-in message is sent. The practice should collect consent during the therapy session (in-person or telehealth) and record it in the EHR. The system should verify consent status before dispatching any check-in. The SUD consent template addresses 42 CFR Part 2 requirements — this is legally necessary for practices treating substance use disorders. Have the healthcare attorney review both consent templates before deployment.
Step 15: Pilot Deployment with Limited Patient Cohort
Launch the system with a small cohort of 10–20 patients selected by the Clinical Safety Officer. Monitor all interactions closely for 2 weeks before expanding.
# Enable pilot mode in system configuration
curl -X PUT https://api.checkin.practicename.com/api/admin/config \
-H 'Authorization: Bearer <ADMIN_TOKEN>' \
-H 'Content-Type: application/json' \
-d '{
"pilot_mode": true,
"max_active_patients": 20,
"require_clinician_review_before_send": true,
"enhanced_logging": true
}'
# Enroll pilot patients (after consent collected)
curl -X POST https://api.checkin.practicename.com/api/patients/enroll \
-H 'Authorization: Bearer <CLINICIAN_TOKEN>' \
-H 'Content-Type: application/json' \
-d '{
"patient_id": "<HEALTHIE_PATIENT_ID>",
"phone": "+1XXXXXXXXXX",
"clinician_id": "<CLINICIAN_ID>",
"consent_date": "2025-XX-XX",
"consent_type": "standard_checkin_consent",
"checkin_preferences": {
"frequency": "post_session",
"delay_hours": 24,
"tone": "warm_professional"
}
}'
# Monitor pilot via CloudWatch dashboard and daily review meetings
# Schedule daily 15-minute check-in with Clinical Safety Officer for first week
# Schedule every-other-day check-in for second weekDuring pilot, enable 'require_clinician_review_before_send' to have the clinician approve each outbound check-in before it is sent. This is training-wheel mode — disable after pilot when clinician is comfortable with message quality. Keep enhanced logging enabled throughout pilot for debugging. The pilot criteria for proceeding to full rollout: (1) Zero missed crisis detections, (2) Clinician satisfaction with summary quality, (3) Patient satisfaction/engagement rate > 60%, (4) No HIPAA incidents.
Step 16: Full Rollout and Production Hardening
After successful pilot completion and Clinical Safety Officer sign-off, transition to full production mode. Disable training wheels, expand to all consented patients, and apply production hardening.
# Disable pilot mode
curl -X PUT https://api.checkin.practicename.com/api/admin/config \
-H 'Authorization: Bearer <ADMIN_TOKEN>' \
-H 'Content-Type: application/json' \
-d '{
"pilot_mode": false,
"max_active_patients": 500,
"require_clinician_review_before_send": false,
"enhanced_logging": false
}'
# Enable WAF on ALB for production protection
aws wafv2 create-web-acl --name checkin-agent-waf --scope REGIONAL --default-action '{"Allow":{}}' --rules file://waf-rules.json --visibility-config '{"SampledRequestsEnabled":true,"CloudWatchMetricsEnabled":true,"MetricName":"checkin-waf"}'
# Enable AWS Backup for automated snapshots
aws backup create-backup-plan --backup-plan file://backup-plan.json
# Configure auto-scaling for ECS
aws application-autoscaling register-scalable-target --service-namespace ecs --resource-id service/checkin-agent/core --scalable-dimension ecs:service:DesiredCount --min-capacity 2 --max-capacity 6
aws application-autoscaling put-scaling-policy --service-namespace ecs --resource-id service/checkin-agent/core --scalable-dimension ecs:service:DesiredCount --policy-name cpu-scaling --policy-type TargetTrackingScaling --target-tracking-scaling-policy-configuration '{"TargetValue":70.0,"PredefinedMetricSpecification":{"PredefinedMetricType":"ECSServiceAverageCPUUtilization"},"ScaleOutCooldown":60,"ScaleInCooldown":300}'Production requires minimum 2 agent instances for high availability. Enable WAF with rate limiting (100 requests/minute per IP) and AWS managed rule groups for common attack protection. Configure daily RDS snapshots retained for 35 days and weekly S3 cross-region replication for disaster recovery. Document the full production configuration in the client's operational runbook.
Custom AI Components
Patient Check-In Prompt Generator
Type: prompt Generates personalized, clinically appropriate between-session check-in messages tailored to the patient's treatment context and clinician preferences. Supports multiple tone options (warm_professional, casual_supportive, structured_clinical) and customizable check-in domains (mood, sleep, medication adherence, homework completion, interpersonal, substance use).
Implementation
System Prompt (stored in database, editable by admin)
User Prompt Template
# services/checkin_generator.py
import openai
import random
from typing import Optional
from models.database import CheckIn, Patient, ClinicianPreferences
from services.secrets import get_secret
DOMAINS = [
'mood',
'sleep',
'coping_skills',
'homework_completion',
'interpersonal',
'general_wellbeing',
'substance_use', # Only if patient consented under SUD consent
'medication_adherence',
'anxiety_level',
'daily_functioning'
]
FALLBACK_MESSAGES = {
'warm_professional': {
'mood': 'Hi {first_name}, this is {practice_name}. How have you been feeling since our last session? Reply anytime. Not for emergencies — call 988 if in crisis.',
'sleep': 'Hi {first_name}, {practice_name} here. How has your sleep been this week? Not for emergencies — call 988 if in crisis.',
'general_wellbeing': 'Hi {first_name}, this is {practice_name} checking in. How are things going? Not for emergencies — call 988 if in crisis.',
},
'casual_supportive': {
'mood': 'Hey {first_name}! Just checking in from {practice_name}. How are you feeling today? Not for emergencies — call 988 if in crisis.',
'general_wellbeing': 'Hey {first_name}! {practice_name} here. How\'s your week going? Not for emergencies — call 988 if in crisis.',
},
'structured_clinical': {
'mood': '{first_name}, {practice_name} check-in: Rate your mood today 1-10 (1=lowest, 10=best). Not for emergencies — call 988.',
'anxiety_level': '{first_name}, {practice_name} check-in: Rate your anxiety today 1-10 (1=none, 10=severe). Not for emergencies — call 988.',
}
}
class CheckInGenerator:
def __init__(self):
secrets = get_secret('checkin-agent/openai')
self.client = openai.OpenAI(api_key=secrets['api_key'])
def select_domain(self, patient: Patient, preferences: ClinicianPreferences, checkin_number: int) -> str:
"""Select which domain to ask about based on clinician preferences and rotation."""
allowed_domains = preferences.checkin_domains
# Remove substance_use if patient doesn't have SUD consent
if not patient.has_sud_consent:
allowed_domains = [d for d in allowed_domains if d != 'substance_use']
# Rotate through domains
return allowed_domains[checkin_number % len(allowed_domains)]
async def generate_checkin_message(
self,
patient: Patient,
preferences: ClinicianPreferences,
checkin_number: int,
total_checkins: int,
days_since_session: int
) -> str:
"""Generate a personalized check-in message using LLM with fallback."""
domain = self.select_domain(patient, preferences, checkin_number)
tone = preferences.tone or 'warm_professional'
try:
response = self.client.chat.completions.create(
model='gpt-4.1-mini',
messages=[
{'role': 'system', 'content': SYSTEM_PROMPT_CHECKIN_GENERATOR},
{'role': 'user', 'content': USER_PROMPT_CHECKIN.format(
first_name=patient.first_name,
tone=tone,
domain=domain,
days_since_session=days_since_session,
checkin_number=checkin_number,
total_checkins=total_checkins,
practice_name=preferences.practice_name
)}
],
max_tokens=200,
temperature=0.7
)
message = response.choices[0].message.content.strip()
# Safety check: ensure crisis disclaimer is present
if '988' not in message and 'emergency' not in message.lower():
message += ' Not for emergencies — call 988 if in crisis.'
return message
except Exception as e:
# Fallback to template if LLM fails
fallback = FALLBACK_MESSAGES.get(tone, FALLBACK_MESSAGES['warm_professional'])
template = fallback.get(domain, fallback.get('general_wellbeing', fallback[list(fallback.keys())[0]]))
return template.format(
first_name=patient.first_name,
practice_name=preferences.practice_name
)Crisis Detection Engine
Type: skill Dual-layer crisis detection system that combines fast keyword/regex pattern matching (Layer 1) with LLM-based contextual analysis (Layer 2). Layer 1 runs on every inbound message in milliseconds. Layer 2 runs when Layer 1 flags potential concern or when Layer 1 is clean but the message contains emotional language that warrants deeper analysis. Designed to maximize recall (never miss a crisis) at the cost of some false positives.
Crisis LLM System Prompt
# services/crisis_detection.py
import re
from enum import Enum
from typing import Tuple, List, Optional
from dataclasses import dataclass
import openai
from models.database import CrisisKeyword, get_session
from services.secrets import get_secret
class CrisisLevel(str, Enum):
NONE = 'none'
ELEVATED = 'elevated' # Warrants LLM review
HIGH = 'high' # Alert clinician within 1 hour
CRITICAL = 'critical' # Alert clinician immediately + send crisis resources
@dataclass
class CrisisDetectionResult:
level: CrisisLevel
layer1_matched_keywords: List[str]
layer1_level: CrisisLevel
layer2_level: Optional[CrisisLevel]
layer2_reasoning: Optional[str]
confidence: float
recommended_action: str
crisis_resources_sent: bool
EMOTIONAL_INTENSITY_PATTERNS = [
r'\bcan\'?t\s+(go\s+on|take\s+(it|this|anymore)|do\s+this)\b',
r'\b(hopeless|worthless|pointless|empty|numb|trapped)\b',
r'\b(nobody\s+cares|no\s+one\s+cares|alone|burden)\b',
r'\b(give\s+up|giving\s+up|done\s+with)\b',
r'\b(hate\s+my(self)?|disgusted\s+with\s+my(self)?)\b',
r'\b(dark\s+place|dark\s+thoughts|intrusive\s+thoughts)\b',
r'\b(relapse[d]?|using\s+again|started\s+drinking)\b',
r'\b(panic|panicking|can\'?t\s+breathe|anxiety\s+attack)\b'
]
class CrisisDetectionEngine:
def __init__(self):
secrets = get_secret('checkin-agent/openai')
self.client = openai.OpenAI(api_key=secrets['api_key'])
self._load_keywords()
self._compile_patterns()
def _load_keywords(self):
"""Load crisis keywords from database."""
session = get_session()
keywords = session.query(CrisisKeyword).all()
self.keyword_map = {}
for kw in keywords:
self.keyword_map[kw.keyword.lower()] = {
'severity': kw.severity,
'category': kw.category
}
session.close()
def _compile_patterns(self):
"""Compile regex patterns for emotional intensity detection."""
self.emotional_patterns = [re.compile(p, re.IGNORECASE) for p in EMOTIONAL_INTENSITY_PATTERNS]
def _layer1_keyword_scan(self, message: str) -> Tuple[CrisisLevel, List[str]]:
"""Fast keyword/regex scan. Returns highest severity level and matched keywords."""
message_lower = message.lower()
matched = []
max_level = CrisisLevel.NONE
# Check exact keyword matches
for keyword, info in self.keyword_map.items():
if keyword in message_lower:
matched.append(keyword)
if info['severity'] == 'critical':
max_level = CrisisLevel.CRITICAL
elif info['severity'] == 'high' and max_level != CrisisLevel.CRITICAL:
max_level = CrisisLevel.HIGH
# Check emotional intensity patterns (elevate to ELEVATED if not already higher)
if max_level == CrisisLevel.NONE:
for pattern in self.emotional_patterns:
match = pattern.search(message)
if match:
matched.append(f'pattern:{match.group()}')
max_level = CrisisLevel.ELEVATED
return max_level, matched
async def _layer2_llm_assessment(self, message: str, layer1_context: str) -> Tuple[CrisisLevel, str, float]:
"""Deep LLM-based contextual crisis assessment. Uses GPT-5.4 for maximum accuracy."""
try:
response = self.client.chat.completions.create(
model='gpt-5.4',
messages=[
{'role': 'system', 'content': CRISIS_LLM_SYSTEM_PROMPT},
{'role': 'user', 'content': f'Patient message: "{message}"\n\nKeyword scan context: {layer1_context}'}
],
max_tokens=200,
temperature=0.0, # Deterministic for safety-critical
response_format={'type': 'json_object'}
)
import json
result = json.loads(response.choices[0].message.content)
level = CrisisLevel(result['level'].lower())
reasoning = result.get('reasoning', 'No reasoning provided')
confidence = float(result.get('confidence', 0.8))
return level, reasoning, confidence
except Exception as e:
# If LLM fails, escalate to HIGH as safety default
return CrisisLevel.HIGH, f'LLM assessment failed ({str(e)}), defaulting to HIGH', 0.5
async def assess(self, message: str) -> CrisisDetectionResult:
"""Full dual-layer crisis assessment."""
# Layer 1: Fast keyword scan
layer1_level, matched_keywords = self._layer1_keyword_scan(message)
# Determine if Layer 2 is needed
run_layer2 = (
layer1_level != CrisisLevel.NONE or # Any keyword match
len(message) > 50 # Longer messages get LLM review regardless
)
layer2_level = None
layer2_reasoning = None
final_confidence = 1.0 if layer1_level == CrisisLevel.CRITICAL else 0.7
if run_layer2:
layer1_context = f'Keywords matched: {matched_keywords}' if matched_keywords else 'No keyword matches'
layer2_level, layer2_reasoning, final_confidence = await self._layer2_llm_assessment(message, layer1_context)
# Final level = highest of both layers
final_level = layer1_level
if layer2_level and self._level_to_int(layer2_level) > self._level_to_int(layer1_level):
final_level = layer2_level
# Determine recommended action
action_map = {
CrisisLevel.CRITICAL: 'IMMEDIATE_CLINICIAN_ALERT_AND_CRISIS_RESOURCES',
CrisisLevel.HIGH: 'CLINICIAN_ALERT_WITHIN_1_HOUR',
CrisisLevel.ELEVATED: 'FLAG_FOR_CLINICIAN_REVIEW',
CrisisLevel.NONE: 'ROUTINE_PROCESSING'
}
return CrisisDetectionResult(
level=final_level,
layer1_matched_keywords=matched_keywords,
layer1_level=layer1_level,
layer2_level=layer2_level,
layer2_reasoning=layer2_reasoning,
confidence=final_confidence,
recommended_action=action_map[final_level],
crisis_resources_sent=(final_level == CrisisLevel.CRITICAL)
)
@staticmethod
def _level_to_int(level: CrisisLevel) -> int:
return {CrisisLevel.NONE: 0, CrisisLevel.ELEVATED: 1, CrisisLevel.HIGH: 2, CrisisLevel.CRITICAL: 3}[level]
CRISIS_RESPONSE_MESSAGE = """If you are in immediate danger, please call 911.
National Suicide Prevention Lifeline: Call or text 988
Crisis Text Line: Text HOME to 741741
Your therapist has been notified and will reach out to you as soon as possible. You are not alone."""Triage Classifier and Response Router
Type: agent Classifies each patient response into four triage levels (Routine, Needs Attention, Urgent, Crisis) based on the crisis detection result and additional LLM-based clinical content analysis. Routes the classified response to the appropriate workflow: routine responses are logged and included in the next pre-appointment summary; higher levels trigger escalation workflows with appropriate timing.
Implementation:
Triage System Prompt
# services/triage_classifier.py
# services/triage_classifier.py
from enum import Enum
from typing import Optional
from dataclasses import dataclass
import openai
import json
from services.crisis_detection import CrisisDetectionEngine, CrisisLevel, CrisisDetectionResult
from services.secrets import get_secret
from services.notifications import NotificationService
from models.database import (
CheckInResponse, TriageResult, PatientConversation,
get_session
)
import structlog
logger = structlog.get_logger()
class TriageLevel(str, Enum):
ROUTINE = 'routine'
NEEDS_ATTENTION = 'needs_attention'
URGENT = 'urgent'
CRISIS = 'crisis'
@dataclass
class TriageOutput:
level: TriageLevel
clinical_themes: list[str]
mood_indicator: Optional[str] # positive, neutral, negative, distressed
summary_for_clinician: str
follow_up_recommended: bool
crisis_result: Optional[CrisisDetectionResult]
TRIAGE_SYSTEM_PROMPT = """
You are a clinical triage assistant for a mental health practice. You analyze patient check-in responses to categorize their clinical significance and extract key themes for the clinician.
You are NOT a therapist. You do NOT respond to the patient. You only analyze their message for the clinician's review.
For each message, provide:
1. clinical_themes: List of 1-5 clinical themes present (e.g., ["sleep_disruption", "medication_concerns", "positive_coping"])
2. mood_indicator: One of: positive, neutral, negative, distressed
3. summary: A 1-2 sentence clinical summary for the clinician written in professional clinical language
4. follow_up_recommended: Whether the clinician should follow up before next session (true/false)
5. triage_level: One of: routine, needs_attention, urgent
- routine: Patient is doing well or experiencing normal range of emotions
- needs_attention: Clinically noteworthy content that the clinician should review before next session
- urgent: Significant clinical concern requiring prompt clinician response (within hours, not immediate)
Note: Do NOT classify as 'crisis' — crisis detection is handled by a separate system. Focus on clinical content analysis.
Respond in this exact JSON format:
{"clinical_themes": [...], "mood_indicator": "...", "summary": "...", "follow_up_recommended": true/false, "triage_level": "..."}
"""
class TriageClassifier:
def __init__(self):
secrets = get_secret('checkin-agent/openai')
self.client = openai.OpenAI(api_key=secrets['api_key'])
self.crisis_engine = CrisisDetectionEngine()
self.notifications = NotificationService()
async def classify_and_route(self, patient_id: str, message: str, checkin_id: str) -> TriageOutput:
"""Full triage pipeline: crisis detection → clinical analysis → routing."""
# Step 1: Crisis detection (always runs first)
crisis_result = await self.crisis_engine.assess(message)
# Step 2: If CRITICAL crisis, fast-path to crisis routing
if crisis_result.level == CrisisLevel.CRITICAL:
output = TriageOutput(
level=TriageLevel.CRISIS,
clinical_themes=['crisis', crisis_result.layer2_reasoning or 'crisis_keywords_detected'],
mood_indicator='distressed',
summary_for_clinician=f'CRISIS DETECTED: {crisis_result.layer2_reasoning or "Crisis keywords matched: " + ", ".join(crisis_result.layer1_matched_keywords)}',
follow_up_recommended=True,
crisis_result=crisis_result
)
await self._route_crisis(patient_id, message, output, checkin_id)
return output
# Step 3: LLM clinical analysis for non-crisis messages
try:
response = self.client.chat.completions.create(
model='gpt-4.1-mini', # Cost-efficient for routine triage
messages=[
{'role': 'system', 'content': TRIAGE_SYSTEM_PROMPT},
{'role': 'user', 'content': f'Patient check-in response: "{message}"'}
],
max_tokens=300,
temperature=0.0,
response_format={'type': 'json_object'}
)
analysis = json.loads(response.choices[0].message.content)
# Combine crisis detection level with clinical triage
triage_level = TriageLevel(analysis['triage_level'])
# Elevate triage if crisis detection found HIGH or ELEVATED
if crisis_result.level == CrisisLevel.HIGH:
triage_level = TriageLevel.URGENT
elif crisis_result.level == CrisisLevel.ELEVATED and triage_level == TriageLevel.ROUTINE:
triage_level = TriageLevel.NEEDS_ATTENTION
output = TriageOutput(
level=triage_level,
clinical_themes=analysis.get('clinical_themes', []),
mood_indicator=analysis.get('mood_indicator', 'neutral'),
summary_for_clinician=analysis.get('summary', 'Unable to generate summary'),
follow_up_recommended=analysis.get('follow_up_recommended', False),
crisis_result=crisis_result
)
except Exception as e:
logger.error('triage_llm_failed', error=str(e), patient_id=patient_id)
# Fallback: if crisis engine found something, use that; otherwise needs_attention as safety default
fallback_level = TriageLevel.NEEDS_ATTENTION if crisis_result.level != CrisisLevel.NONE else TriageLevel.NEEDS_ATTENTION
output = TriageOutput(
level=fallback_level,
clinical_themes=['analysis_failed'],
mood_indicator='unknown',
summary_for_clinician=f'Automated analysis failed. Manual review required. Raw message available in dashboard.',
follow_up_recommended=True,
crisis_result=crisis_result
)
# Step 4: Route based on triage level
await self._route(patient_id, message, output, checkin_id)
return output
async def _route_crisis(self, patient_id: str, message: str, output: TriageOutput, checkin_id: str):
"""Crisis routing: immediate clinician alert + crisis resources to patient."""
logger.critical('crisis_detected', patient_id=patient_id, checkin_id=checkin_id)
# Send crisis resources to patient IMMEDIATELY
from services.messaging import MessagingService
messaging = MessagingService()
await messaging.send_crisis_resources(patient_id)
# Alert clinician IMMEDIATELY (SMS + push + dashboard)
await self.notifications.send_crisis_alert(
patient_id=patient_id,
summary=output.summary_for_clinician,
urgency='immediate'
)
# Store with highest priority
await self._store_triage_result(patient_id, message, output, checkin_id)
async def _route(self, patient_id: str, message: str, output: TriageOutput, checkin_id: str):
"""Standard routing based on triage level."""
if output.level == TriageLevel.URGENT:
await self.notifications.send_urgent_alert(
patient_id=patient_id,
summary=output.summary_for_clinician
)
elif output.level == TriageLevel.NEEDS_ATTENTION:
await self.notifications.flag_for_review(
patient_id=patient_id,
summary=output.summary_for_clinician
)
# ROUTINE messages are just stored — included in next pre-appointment summary
await self._store_triage_result(patient_id, message, output, checkin_id)
async def _store_triage_result(self, patient_id: str, message: str, output: TriageOutput, checkin_id: str):
"""Persist triage result to database."""
session = get_session()
result = TriageResult(
checkin_id=checkin_id,
patient_id=patient_id,
raw_message=message, # Encrypted at database level
triage_level=output.level.value,
clinical_themes=output.clinical_themes,
mood_indicator=output.mood_indicator,
summary=output.summary_for_clinician,
follow_up_recommended=output.follow_up_recommended,
crisis_detected=(output.level == TriageLevel.CRISIS),
crisis_keywords=output.crisis_result.layer1_matched_keywords if output.crisis_result else [],
crisis_llm_reasoning=output.crisis_result.layer2_reasoning if output.crisis_result else None
)
session.add(result)
session.commit()
session.close()Pre-Appointment Summary Generator
Type: skill Generates a concise clinical summary of all between-session check-in interactions for a given patient before their next appointment. The summary is formatted for clinical utility — organized by theme, with mood trajectory, key concerns flagged, and recommended discussion points. Can be pushed to Healthie as a chart note or displayed in the clinician dashboard.
Implementation:
Pre-Appointment Summary System Prompt
# services/summary_generator.py
import openai
import json
from datetime import datetime, timedelta
from typing import List, Optional
from models.database import TriageResult, CheckIn, Patient, get_session
from services.secrets import get_secret
import structlog
logger = structlog.get_logger()
SUMMARY_SYSTEM_PROMPT = """
You are a clinical summarization assistant for a mental health practice. You create concise pre-appointment summaries from between-session check-in data.
Your audience is a licensed mental health clinician (therapist, psychologist, counselor, social worker). Write in professional clinical language.
For each summary, provide:
1. overview: 1-2 sentence overall assessment of the patient's between-session functioning
2. mood_trajectory: Brief description of mood pattern across check-ins (improving, stable, declining, variable)
3. key_themes: List of clinical themes that emerged across check-ins
4. concerns: Any items flagged as needing attention or urgent during the between-session period
5. positive_indicators: Any positive developments, coping successes, or progress indicators
6. suggested_discussion_points: 2-4 suggested items for the upcoming session based on check-in data
7. raw_data_note: Note that the full conversation transcripts are available in the dashboard
RULES:
1. Be concise — clinicians are time-pressed. Total summary should be under 300 words.
2. Use clinical language but avoid jargon that might not transfer across disciplines.
3. Clearly distinguish between patient-reported data and your analytical observations.
4. If any check-in was flagged as crisis or urgent, prominently note this at the top.
5. Do not make diagnostic impressions or treatment recommendations — that is the clinician's role.
Respond in JSON format:
{"overview": "...", "mood_trajectory": "...", "key_themes": [...], "concerns": [...], "positive_indicators": [...], "suggested_discussion_points": [...], "has_crisis_flag": true/false, "crisis_summary": "...or null"}
"""
class SummaryGenerator:
def __init__(self):
secrets = get_secret('checkin-agent/openai')
self.client = openai.OpenAI(api_key=secrets['api_key'])
async def generate_summary(
self,
patient_id: str,
appointment_id: str,
since_date: Optional[datetime] = None
) -> dict:
"""Generate pre-appointment summary from all check-ins since last appointment."""
session = get_session()
# Get all triage results since last appointment
query = session.query(TriageResult).filter(
TriageResult.patient_id == patient_id
)
if since_date:
query = query.filter(TriageResult.created_at >= since_date)
results = query.order_by(TriageResult.created_at.asc()).all()
if not results:
return {
'overview': 'No between-session check-in data available for this patient.',
'mood_trajectory': 'N/A',
'key_themes': [],
'concerns': [],
'positive_indicators': [],
'suggested_discussion_points': ['Discuss engagement with between-session check-ins'],
'has_crisis_flag': False,
'crisis_summary': None,
'checkin_count': 0
}
# Format check-in data for LLM
checkin_data = []
has_crisis = False
for r in results:
entry = {
'date': r.created_at.strftime('%Y-%m-%d %H:%M'),
'triage_level': r.triage_level,
'mood': r.mood_indicator,
'themes': r.clinical_themes,
'summary': r.summary,
'patient_message_excerpt': r.raw_message[:200] # Limit for context window
}
checkin_data.append(entry)
if r.crisis_detected:
has_crisis = True
session.close()
try:
response = self.client.chat.completions.create(
model='gpt-5.4', # Use higher-quality model for clinical summaries
messages=[
{'role': 'system', 'content': SUMMARY_SYSTEM_PROMPT},
{'role': 'user', 'content': f'Patient check-in data ({len(checkin_data)} check-ins):\n\n{json.dumps(checkin_data, indent=2)}'}
],
max_tokens=800,
temperature=0.1, # Low temperature for clinical accuracy
response_format={'type': 'json_object'}
)
summary = json.loads(response.choices[0].message.content)
summary['checkin_count'] = len(checkin_data)
summary['generated_at'] = datetime.utcnow().isoformat()
summary['appointment_id'] = appointment_id
return summary
except Exception as e:
logger.error('summary_generation_failed', error=str(e), patient_id=patient_id)
# Return structured fallback
return {
'overview': f'Automated summary generation failed. {len(results)} check-in responses available for manual review in dashboard.',
'mood_trajectory': 'Manual review required',
'key_themes': list(set(theme for r in results for theme in (r.clinical_themes or []))),
'concerns': [r.summary for r in results if r.triage_level in ('urgent', 'crisis')],
'positive_indicators': [],
'suggested_discussion_points': ['Review between-session check-in responses in dashboard'],
'has_crisis_flag': has_crisis,
'crisis_summary': 'Crisis event detected during between-session period. Review immediately.' if has_crisis else None,
'checkin_count': len(results),
'generation_error': str(e)
}
def format_for_ehr(self, summary: dict) -> str:
"""Format summary as plain text suitable for EHR chart note."""
lines = []
lines.append('=== BETWEEN-SESSION CHECK-IN SUMMARY (AI-GENERATED) ===')
lines.append(f'Check-ins reviewed: {summary.get("checkin_count", "N/A")}')
lines.append(f'Generated: {summary.get("generated_at", "N/A")}')
lines.append('')
if summary.get('has_crisis_flag'):
lines.append('⚠️ CRISIS EVENT DETECTED DURING THIS PERIOD')
lines.append(f'Details: {summary.get("crisis_summary", "See dashboard for details")}')
lines.append('')
lines.append(f'OVERVIEW: {summary.get("overview", "N/A")}')
lines.append(f'MOOD TRAJECTORY: {summary.get("mood_trajectory", "N/A")}')
lines.append('')
if summary.get('concerns'):
lines.append('CONCERNS:')
for c in summary['concerns']:
lines.append(f' • {c}')
lines.append('')
if summary.get('positive_indicators'):
lines.append('POSITIVE INDICATORS:')
for p in summary['positive_indicators']:
lines.append(f' • {p}')
lines.append('')
if summary.get('suggested_discussion_points'):
lines.append('SUGGESTED DISCUSSION POINTS:')
for d in summary['suggested_discussion_points']:
lines.append(f' • {d}')
lines.append('')
lines.append('Note: This summary was generated by AI from patient check-in messages.')
lines.append('Full conversation transcripts available in the clinician dashboard.')
lines.append('=== END SUMMARY ===')
return '\n'.join(lines)Database Schema and Models
Type: integration Complete SQLAlchemy database models and schema for the check-in agent system. Includes tables for patients, clinicians, check-ins, responses, triage results, consent records, crisis events, and audit logs. All PHI columns are marked for application-level encryption in addition to database-level encryption at rest.
Implementation:
# Database Schema and Models
# models/database.py
from datetime import datetime
from typing import Optional, List
from sqlalchemy import (
create_engine, Column, String, Integer, Boolean, DateTime,
Text, Float, ForeignKey, Enum as SQLEnum, JSON, Index
)
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, relationship
from sqlalchemy.dialects.postgresql import UUID, ARRAY
import uuid
import enum
Base = declarative_base()
class TriageLevelEnum(str, enum.Enum):
ROUTINE = 'routine'
NEEDS_ATTENTION = 'needs_attention'
URGENT = 'urgent'
CRISIS = 'crisis'
class ConsentTypeEnum(str, enum.Enum):
STANDARD = 'standard_checkin_consent'
SUD = 'sud_checkin_consent'
class Patient(Base):
__tablename__ = 'patients'
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
external_id = Column(String(255), unique=True, nullable=False) # Healthie patient ID
first_name = Column(String(255), nullable=False) # PHI - encrypted at app level
last_name = Column(String(255), nullable=False) # PHI - encrypted at app level
phone = Column(String(20), nullable=False) # PHI - encrypted at app level
email = Column(String(255), nullable=True) # PHI - encrypted at app level
clinician_id = Column(UUID(as_uuid=True), ForeignKey('clinicians.id'), nullable=False)
is_active = Column(Boolean, default=True)
has_sud_consent = Column(Boolean, default=False)
opted_in = Column(Boolean, default=False)
opted_in_at = Column(DateTime, nullable=True)
opted_out_at = Column(DateTime, nullable=True)
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
clinician = relationship('Clinician', back_populates='patients')
consents = relationship('ConsentRecord', back_populates='patient')
checkins = relationship('CheckIn', back_populates='patient')
__table_args__ = (
Index('idx_patients_external_id', 'external_id'),
Index('idx_patients_clinician', 'clinician_id'),
Index('idx_patients_active', 'is_active', 'opted_in'),
)
class Clinician(Base):
__tablename__ = 'clinicians'
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
external_id = Column(String(255), unique=True, nullable=False)
name = Column(String(255), nullable=False)
email = Column(String(255), nullable=False)
phone = Column(String(20), nullable=False)
role = Column(String(50), default='clinician')
is_active = Column(Boolean, default=True)
notification_preferences = Column(JSON, default=lambda: {
'crisis_sms': True, 'crisis_email': True, 'crisis_push': True,
'urgent_sms': True, 'urgent_email': True,
'daily_digest': True, 'digest_time': '08:00'
})
checkin_preferences = Column(JSON, default=lambda: {
'tone': 'warm_professional',
'checkin_domains': ['mood', 'sleep', 'coping_skills', 'general_wellbeing'],
'delay_hours': 24,
'max_checkins_between_sessions': 2,
'practice_name': 'Your Practice'
})
created_at = Column(DateTime, default=datetime.utcnow)
patients = relationship('Patient', back_populates='clinician')
class ConsentRecord(Base):
__tablename__ = 'consent_records'
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
patient_id = Column(UUID(as_uuid=True), ForeignKey('patients.id'), nullable=False)
consent_type = Column(SQLEnum(ConsentTypeEnum), nullable=False)
version = Column(String(20), nullable=False)
granted_at = Column(DateTime, nullable=False)
revoked_at = Column(DateTime, nullable=True)
collected_by = Column(String(255), nullable=False) # Clinician who collected
collection_method = Column(String(50), nullable=False) # in_person, telehealth, portal
patient = relationship('Patient', back_populates='consents')
__table_args__ = (
Index('idx_consent_patient', 'patient_id', 'consent_type'),
)
class CheckIn(Base):
__tablename__ = 'checkins'
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
patient_id = Column(UUID(as_uuid=True), ForeignKey('patients.id'), nullable=False)
clinician_id = Column(UUID(as_uuid=True), ForeignKey('clinicians.id'), nullable=False)
appointment_id = Column(String(255), nullable=True) # Source appointment in EHR
scheduled_send_at = Column(DateTime, nullable=False)
actual_sent_at = Column(DateTime, nullable=True)
message_text = Column(Text, nullable=True) # The check-in prompt sent
domain = Column(String(50), nullable=True)
status = Column(String(20), default='pending') # pending, sent, responded, expired
twilio_message_sid = Column(String(255), nullable=True)
checkin_number = Column(Integer, default=1)
created_at = Column(DateTime, default=datetime.utcnow)
patient = relationship('Patient', back_populates='checkins')
responses = relationship('CheckInResponse', back_populates='checkin')
__table_args__ = (
Index('idx_checkins_pending', 'status', 'scheduled_send_at'),
Index('idx_checkins_patient', 'patient_id', 'created_at'),
)
class CheckInResponse(Base):
__tablename__ = 'checkin_responses'
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
checkin_id = Column(UUID(as_uuid=True), ForeignKey('checkins.id'), nullable=False)
patient_id = Column(UUID(as_uuid=True), ForeignKey('patients.id'), nullable=False)
raw_message = Column(Text, nullable=False) # PHI - encrypted at app level
received_at = Column(DateTime, default=datetime.utcnow)
twilio_message_sid = Column(String(255), nullable=True)
processed = Column(Boolean, default=False)
processed_at = Column(DateTime, nullable=True)
checkin = relationship('CheckIn', back_populates='responses')
triage_result = relationship('TriageResult', back_populates='response', uselist=False)
class TriageResult(Base):
__tablename__ = 'triage_results'
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
response_id = Column(UUID(as_uuid=True), ForeignKey('checkin_responses.id'), nullable=False)
patient_id = Column(UUID(as_uuid=True), ForeignKey('patients.id'), nullable=False)
checkin_id = Column(String(255), nullable=False)
triage_level = Column(SQLEnum(TriageLevelEnum), nullable=False)
clinical_themes = Column(ARRAY(String), default=[])
mood_indicator = Column(String(50), nullable=True)
summary = Column(Text, nullable=False)
follow_up_recommended = Column(Boolean, default=False)
crisis_detected = Column(Boolean, default=False)
crisis_keywords = Column(ARRAY(String), default=[])
crisis_llm_reasoning = Column(Text, nullable=True)
clinician_acknowledged = Column(Boolean, default=False)
clinician_acknowledged_at = Column(DateTime, nullable=True)
created_at = Column(DateTime, default=datetime.utcnow)
response = relationship('CheckInResponse', back_populates='triage_result')
__table_args__ = (
Index('idx_triage_patient_date', 'patient_id', 'created_at'),
Index('idx_triage_level', 'triage_level', 'clinician_acknowledged'),
Index('idx_triage_crisis', 'crisis_detected'),
)
class CrisisKeyword(Base):
__tablename__ = 'crisis_keywords'
id = Column(Integer, primary_key=True, autoincrement=True)
keyword = Column(String(255), unique=True, nullable=False)
severity = Column(String(20), nullable=False) # critical, high
category = Column(String(50), nullable=False)
is_active = Column(Boolean, default=True)
added_by = Column(String(255), nullable=True)
created_at = Column(DateTime, default=datetime.utcnow)
class CrisisEvent(Base):
__tablename__ = 'crisis_events'
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
patient_id = Column(UUID(as_uuid=True), ForeignKey('patients.id'), nullable=False)
triage_result_id = Column(UUID(as_uuid=True), ForeignKey('triage_results.id'), nullable=False)
clinician_id = Column(UUID(as_uuid=True), ForeignKey('clinicians.id'), nullable=False)
crisis_level = Column(String(20), nullable=False)
clinician_notified_at = Column(DateTime, nullable=False)
clinician_responded_at = Column(DateTime, nullable=True)
crisis_resources_sent_at = Column(DateTime, nullable=False)
resolution_notes = Column(Text, nullable=True)
resolved_at = Column(DateTime, nullable=True)
created_at = Column(DateTime, default=datetime.utcnow)
class AuditLog(Base):
__tablename__ = 'audit_logs'
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
timestamp = Column(DateTime, default=datetime.utcnow, nullable=False)
user_id = Column(String(255), nullable=True) # Auth0 user ID or 'system'
action = Column(String(100), nullable=False) # e.g., 'view_conversation', 'acknowledge_alert'
resource_type = Column(String(50), nullable=False) # e.g., 'patient', 'triage_result'
resource_id = Column(String(255), nullable=False)
details = Column(JSON, nullable=True)
ip_address = Column(String(45), nullable=True)
user_agent = Column(String(500), nullable=True)
__table_args__ = (
Index('idx_audit_timestamp', 'timestamp'),
Index('idx_audit_user', 'user_id', 'timestamp'),
Index('idx_audit_resource', 'resource_type', 'resource_id'),
)
class AppointmentSummary(Base):
__tablename__ = 'appointment_summaries'
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
patient_id = Column(UUID(as_uuid=True), ForeignKey('patients.id'), nullable=False)
appointment_id = Column(String(255), nullable=False)
checkin_count = Column(Integer, default=0)
summary_json = Column(JSON, nullable=False)
summary_text = Column(Text, nullable=False) # Plain text for EHR
pushed_to_ehr = Column(Boolean, default=False)
pushed_to_ehr_at = Column(DateTime, nullable=True)
generated_at = Column(DateTime, default=datetime.utcnow)
__table_args__ = (
Index('idx_summary_appointment', 'appointment_id'),
Index('idx_summary_patient', 'patient_id', 'generated_at'),
)
# Database connection
def get_engine():
from services.secrets import get_secret
db_secrets = get_secret('checkin-agent/database')
url = f"postgresql://{db_secrets['username']}:{db_secrets['password']}@{db_secrets['host']}:5432/{db_secrets['database']}?sslmode=require"
return create_engine(url, pool_size=10, max_overflow=20)
def get_session():
engine = get_engine()
Session = sessionmaker(bind=engine)
return Session()FastAPI Application and API Endpoints
Type: integration The main FastAPI application that serves as the API gateway for all system interactions: Twilio webhook ingestion, Healthie webhook processing, clinician dashboard API, admin configuration, and internal agent operations. Includes request validation, authentication middleware, audit logging, and CORS configuration.
Implementation:
# FastAPI Application
# api/main.py
from fastapi import FastAPI, Request, HTTPException, Depends, BackgroundTasks
from fastapi.middleware.cors import CORSMiddleware
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from pydantic import BaseModel, Field
from typing import Optional, List
from datetime import datetime, timedelta
import hmac
import hashlib
import structlog
import boto3
import json
from services.triage_classifier import TriageClassifier
from services.checkin_generator import CheckInGenerator
from services.summary_generator import SummaryGenerator
from services.messaging import MessagingService
from services.auth import verify_token, require_role
from models.database import (
get_session, Patient, CheckIn, CheckInResponse,
TriageResult, Clinician, AuditLog, AppointmentSummary, CrisisEvent
)
logger = structlog.get_logger()
app = FastAPI(title='Between-Session Check-In Agent', version='1.0.0')
security = HTTPBearer()
app.add_middleware(
CORSMiddleware,
allow_origins=['https://checkin.practicename.com'],
allow_credentials=True,
allow_methods=['GET', 'POST', 'PUT', 'DELETE'],
allow_headers=['*'],
)
triage_classifier = TriageClassifier()
checkin_generator = CheckInGenerator()
summary_generator = SummaryGenerator()
messaging = MessagingService()
# ============ Health Check ============
@app.get('/health')
async def health_check():
return {'status': 'healthy', 'timestamp': datetime.utcnow().isoformat()}
# ============ Twilio Webhook ============
@app.post('/twilio/inbound')
async def twilio_inbound_webhook(request: Request, background_tasks: BackgroundTasks):
"""Receive inbound SMS from patients via Twilio."""
# Validate Twilio signature
form_data = await request.form()
from_number = form_data.get('From', '')
body = form_data.get('Body', '')
message_sid = form_data.get('MessageSid', '')
# Validate Twilio request signature
# (implement signature validation using twilio.request_validator)
# Check for opt-out
if body.strip().upper() in ['STOP', 'UNSUBSCRIBE', 'CANCEL', 'QUIT']:
background_tasks.add_task(handle_opt_out, from_number)
return {'status': 'opt_out_processed'}
# Check for opt-in confirmation
if body.strip().upper() == 'YES':
background_tasks.add_task(handle_opt_in, from_number)
return {'status': 'opt_in_processed'}
# Enqueue for processing
sqs = boto3.client('sqs')
sqs.send_message(
QueueUrl='https://sqs.us-east-1.amazonaws.com/<ACCOUNT_ID>/checkin-inbound-messages',
MessageBody=json.dumps({
'from': from_number,
'body': body,
'message_sid': message_sid,
'received_at': datetime.utcnow().isoformat()
})
)
# Also process immediately for crisis detection (don't wait for queue)
background_tasks.add_task(process_inbound_message, from_number, body, message_sid)
return {'status': 'received'}
async def process_inbound_message(from_number: str, body: str, message_sid: str):
"""Process an inbound patient message through the triage pipeline."""
session = get_session()
# Find patient by phone number
patient = session.query(Patient).filter(
Patient.phone == from_number,
Patient.is_active == True,
Patient.opted_in == True
).first()
if not patient:
logger.warning('message_from_unknown_number', phone=from_number[-4:]) # Log only last 4 digits
session.close()
return
# Find the most recent pending check-in for this patient
checkin = session.query(CheckIn).filter(
CheckIn.patient_id == patient.id,
CheckIn.status == 'sent'
).order_by(CheckIn.actual_sent_at.desc()).first()
if checkin:
checkin.status = 'responded'
# Store the response
response = CheckInResponse(
checkin_id=checkin.id if checkin else None,
patient_id=patient.id,
raw_message=body,
twilio_message_sid=message_sid
)
session.add(response)
session.commit()
response_id = response.id
session.close()
# Run triage classification (includes crisis detection)
await triage_classifier.classify_and_route(
patient_id=str(patient.id),
message=body,
checkin_id=str(checkin.id) if checkin else str(response_id)
)
# ============ Healthie Webhook ============
@app.post('/healthie/webhook/appointment-completed')
async def healthie_appointment_completed(request: Request, background_tasks: BackgroundTasks):
"""Handle completed appointment webhook from Healthie."""
payload = await request.json()
appointment_id = payload.get('resource_id')
patient_external_id = payload.get('patient_id')
background_tasks.add_task(schedule_checkins_for_appointment, appointment_id, patient_external_id)
return {'status': 'received'}
async def schedule_checkins_for_appointment(appointment_id: str, patient_external_id: str):
"""Schedule check-in messages after an appointment completes."""
session = get_session()
patient = session.query(Patient).filter(
Patient.external_id == patient_external_id,
Patient.opted_in == True
).first()
if not patient:
session.close()
return
clinician = session.query(Clinician).filter(Clinician.id == patient.clinician_id).first()
prefs = clinician.checkin_preferences or {}
delay_hours = prefs.get('delay_hours', 24)
max_checkins = prefs.get('max_checkins_between_sessions', 2)
for i in range(max_checkins):
send_at = datetime.utcnow() + timedelta(hours=delay_hours * (i + 1))
checkin = CheckIn(
patient_id=patient.id,
clinician_id=clinician.id,
appointment_id=appointment_id,
scheduled_send_at=send_at,
checkin_number=i + 1,
status='pending'
)
session.add(checkin)
session.commit()
session.close()
# ============ Agent Operations API ============
@app.get('/api/pending-checkins')
async def get_pending_checkins(credentials: HTTPAuthorizationCredentials = Depends(security)):
"""Get check-ins that are due to be sent."""
verify_token(credentials.credentials, required_role='system')
session = get_session()
pending = session.query(CheckIn).filter(
CheckIn.status == 'pending',
CheckIn.scheduled_send_at <= datetime.utcnow()
).all()
result = [{'id': str(c.id), 'patient_id': str(c.patient_id), 'clinician_id': str(c.clinician_id), 'checkin_number': c.checkin_number} for c in pending]
session.close()
return result
@app.post('/api/send-checkin/{checkin_id}')
async def send_checkin(checkin_id: str, credentials: HTTPAuthorizationCredentials = Depends(security)):
"""Generate and send a check-in message."""
verify_token(credentials.credentials, required_role='system')
session = get_session()
checkin = session.query(CheckIn).filter(CheckIn.id == checkin_id).first()
if not checkin:
raise HTTPException(status_code=404)
patient = session.query(Patient).filter(Patient.id == checkin.patient_id).first()
clinician = session.query(Clinician).filter(Clinician.id == checkin.clinician_id).first()
prefs_obj = type('Prefs', (), clinician.checkin_preferences)()
prefs_obj.checkin_domains = clinician.checkin_preferences.get('checkin_domains', ['mood'])
prefs_obj.tone = clinician.checkin_preferences.get('tone', 'warm_professional')
prefs_obj.practice_name = clinician.checkin_preferences.get('practice_name', 'Your Practice')
days_since = (datetime.utcnow() - checkin.created_at).days
max_checkins = clinician.checkin_preferences.get('max_checkins_between_sessions', 2)
message = await checkin_generator.generate_checkin_message(
patient=patient,
preferences=prefs_obj,
checkin_number=checkin.checkin_number,
total_checkins=max_checkins,
days_since_session=days_since
)
# Send via Twilio
twilio_sid = await messaging.send_sms(patient.phone, message)
checkin.message_text = message
checkin.actual_sent_at = datetime.utcnow()
checkin.status = 'sent'
checkin.twilio_message_sid = twilio_sid
session.commit()
session.close()
return {'status': 'sent', 'twilio_sid': twilio_sid}
@app.post('/api/generate-summary/{patient_id}/{appointment_id}')
async def generate_appointment_summary(patient_id: str, appointment_id: str, credentials: HTTPAuthorizationCredentials = Depends(security)):
"""Generate pre-appointment summary."""
verify_token(credentials.credentials, required_role='system')
summary = await summary_generator.generate_summary(patient_id, appointment_id)
summary_text = summary_generator.format_for_ehr(summary)
session = get_session()
record = AppointmentSummary(
patient_id=patient_id,
appointment_id=appointment_id,
checkin_count=summary.get('checkin_count', 0),
summary_json=summary,
summary_text=summary_text
)
session.add(record)
session.commit()
session.close()
return summary
# ============ Clinician Dashboard API ============
@app.get('/api/dashboard/alerts')
async def get_alerts(credentials: HTTPAuthorizationCredentials = Depends(security)):
"""Get pending alerts for the authenticated clinician."""
user = verify_token(credentials.credentials, required_role='clinician')
session = get_session()
clinician = session.query(Clinician).filter(Clinician.external_id == user['sub']).first()
alerts = session.query(TriageResult).join(CheckInResponse).join(CheckIn).filter(
CheckIn.clinician_id == clinician.id,
TriageResult.triage_level.in_(['crisis', 'urgent', 'needs_attention']),
TriageResult.clinician_acknowledged == False
).order_by(
TriageResult.triage_level.desc(),
TriageResult.created_at.desc()
).all()
result = [{
'id': str(a.id),
'patient_id': str(a.patient_id),
'level': a.triage_level.value,
'summary': a.summary,
'themes': a.clinical_themes,
'mood': a.mood_indicator,
'crisis': a.crisis_detected,
'timestamp': a.created_at.isoformat()
} for a in alerts]
# Audit log
log = AuditLog(user_id=user['sub'], action='view_alerts', resource_type='dashboard', resource_id='alerts', details={'count': len(result)})
session.add(log)
session.commit()
session.close()
return result
@app.post('/api/dashboard/alerts/{alert_id}/acknowledge')
async def acknowledge_alert(alert_id: str, credentials: HTTPAuthorizationCredentials = Depends(security)):
"""Clinician acknowledges an alert."""
user = verify_token(credentials.credentials, required_role='clinician')
session = get_session()
triage = session.query(TriageResult).filter(TriageResult.id == alert_id).first()
if not triage:
raise HTTPException(status_code=404)
triage.clinician_acknowledged = True
triage.clinician_acknowledged_at = datetime.utcnow()
log = AuditLog(user_id=user['sub'], action='acknowledge_alert', resource_type='triage_result', resource_id=alert_id)
session.add(log)
session.commit()
session.close()
return {'status': 'acknowledged'}
@app.get('/api/dashboard/patient/{patient_id}/conversations')
async def get_patient_conversations(patient_id: str, credentials: HTTPAuthorizationCredentials = Depends(security)):
"""Get full conversation history for a patient."""
user = verify_token(credentials.credentials, required_role='clinician')
session = get_session()
checkins = session.query(CheckIn).filter(
CheckIn.patient_id == patient_id
).order_by(CheckIn.created_at.desc()).limit(50).all()
conversations = []
for c in checkins:
responses = session.query(CheckInResponse).filter(CheckInResponse.checkin_id == c.id).all()
conv = {
'checkin_id': str(c.id),
'sent_at': c.actual_sent_at.isoformat() if c.actual_sent_at else None,
'prompt': c.message_text,
'status': c.status,
'responses': []
}
for r in responses:
triage = session.query(TriageResult).filter(TriageResult.response_id == r.id).first()
conv['responses'].append({
'message': r.raw_message,
'received_at': r.received_at.isoformat(),
'triage_level': triage.triage_level.value if triage else None,
'summary': triage.summary if triage else None,
'mood': triage.mood_indicator if triage else None
})
conversations.append(conv)
log = AuditLog(user_id=user['sub'], action='view_conversations', resource_type='patient', resource_id=patient_id)
session.add(log)
session.commit()
session.close()
return conversations
@app.get('/api/dashboard/summaries/upcoming')
async def get_upcoming_summaries(credentials: HTTPAuthorizationCredentials = Depends(security)):
"""Get pre-appointment summaries for upcoming appointments."""
user = verify_token(credentials.credentials, required_role='clinician')
session = get_session()
clinician = session.query(Clinician).filter(Clinician.external_id == user['sub']).first()
# Get summaries generated in last 48 hours
cutoff = datetime.utcnow() - timedelta(hours=48)
summaries = session.query(AppointmentSummary).join(Patient).filter(
Patient.clinician_id == clinician.id,
AppointmentSummary.generated_at >= cutoff
).order_by(AppointmentSummary.generated_at.desc()).all()
result = [{
'id': str(s.id),
'patient_id': str(s.patient_id),
'appointment_id': s.appointment_id,
'summary': s.summary_json,
'checkin_count': s.checkin_count,
'generated_at': s.generated_at.isoformat()
} for s in summaries]
session.close()
return resultNotification and Messaging Service
Type: integration Handles all outbound communications: patient SMS via Twilio, clinician alerts via SMS/email/push, and crisis resource delivery. Implements retry logic, delivery confirmation tracking, and notification throttling to prevent alert fatigue.
Implementation:
# Notification and Messaging Service
# services/messaging.py
from twilio.rest import Client as TwilioClient
import boto3
from datetime import datetime
from services.secrets import get_secret
import structlog
logger = structlog.get_logger()
CRISIS_RESPONSE_SMS = """If you are in immediate danger, please call 911.
988 Suicide & Crisis Lifeline: Call or text 988
Crisis Text Line: Text HOME to 741741
Your therapist has been notified and will reach out to you as soon as possible. You are not alone."""
class MessagingService:
def __init__(self):
twilio_secrets = get_secret('checkin-agent/twilio')
self.twilio = TwilioClient(twilio_secrets['account_sid'], twilio_secrets['auth_token'])
self.from_number = twilio_secrets['phone_number']
self.ses = boto3.client('ses', region_name='us-east-1')
self.sns = boto3.client('sns', region_name='us-east-1')
async def send_sms(self, to_number: str, body: str) -> str:
"""Send SMS via Twilio. Returns message SID."""
try:
message = self.twilio.messages.create(
body=body,
from_=self.from_number,
to=to_number
)
logger.info('sms_sent', to_last4=to_number[-4:], sid=message.sid)
return message.sid
except Exception as e:
logger.error('sms_send_failed', to_last4=to_number[-4:], error=str(e))
raise
async def send_crisis_resources(self, patient_id: str):
"""Send crisis resources to patient immediately."""
from models.database import get_session, Patient
session = get_session()
patient = session.query(Patient).filter(Patient.id == patient_id).first()
if patient:
await self.send_sms(patient.phone, CRISIS_RESPONSE_SMS)
logger.critical('crisis_resources_sent', patient_id=str(patient_id))
session.close()
class NotificationService:
def __init__(self):
self.messaging = MessagingService()
self.ses = boto3.client('ses', region_name='us-east-1')
async def send_crisis_alert(self, patient_id: str, summary: str, urgency: str = 'immediate'):
"""Send crisis alert to clinician via all channels."""
from models.database import get_session, Patient, Clinician
session = get_session()
patient = session.query(Patient).filter(Patient.id == patient_id).first()
clinician = session.query(Clinician).filter(Clinician.id == patient.clinician_id).first()
prefs = clinician.notification_preferences or {}
alert_text = f'CRISIS ALERT - {patient.first_name} {patient.last_name[0]}.: {summary[:200]}. Log in to dashboard immediately.'
if prefs.get('crisis_sms', True):
await self.messaging.send_sms(clinician.phone, alert_text)
if prefs.get('crisis_email', True):
self.ses.send_email(
Source='alerts@checkin.practicename.com',
Destination={'ToAddresses': [clinician.email]},
Message={
'Subject': {'Data': f'🚨 CRISIS ALERT - Patient Check-In System'},
'Body': {'Text': {'Data': alert_text}}
}
)
from models.database import CrisisEvent
event = CrisisEvent(
patient_id=patient_id,
clinician_id=str(clinician.id),
crisis_level='critical',
clinician_notified_at=datetime.utcnow(),
crisis_resources_sent_at=datetime.utcnow()
)
session.add(event)
session.commit()
session.close()
logger.critical('crisis_alert_sent', patient_id=str(patient_id), clinician_id=str(clinician.id))
async def send_urgent_alert(self, patient_id: str, summary: str):
"""Send urgent alert to clinician via SMS and email."""
from models.database import get_session, Patient, Clinician
session = get_session()
patient = session.query(Patient).filter(Patient.id == patient_id).first()
clinician = session.query(Clinician).filter(Clinician.id == patient.clinician_id).first()
alert_text = f'URGENT - {patient.first_name} {patient.last_name[0]}.: {summary[:200]}. Please review in dashboard within 1 hour.'
if clinician.notification_preferences.get('urgent_sms', True):
await self.messaging.send_sms(clinician.phone, alert_text)
session.close()
async def flag_for_review(self, patient_id: str, summary: str):
"""Flag for clinician review — included in daily digest, no immediate alert."""
logger.info('flagged_for_review', patient_id=str(patient_id), summary=summary[:100])
# These are surfaced in the dashboard and daily digest email
# services/secrets.py
import boto3
import json
from functools import lru_cache
@lru_cache(maxsize=10)
def get_secret(secret_name: str) -> dict:
"""Retrieve secret from AWS Secrets Manager."""
client = boto3.client('secretsmanager', region_name='us-east-1')
response = client.get_secret_value(SecretId=secret_name)
return json.loads(response['SecretString'])
# services/auth.py
import httpx
from jose import jwt, JWTError
from fastapi import HTTPException
from functools import lru_cache
from services.secrets import get_secret
@lru_cache(maxsize=1)
def get_auth0_config():
return get_secret('checkin-agent/auth0')
def verify_token(token: str, required_role: str = None) -> dict:
"""Verify Auth0 JWT token and optionally check role."""
config = get_auth0_config()
jwks_url = f"https://{config['domain']}/.well-known/jwks.json"
try:
# In production, cache JWKS keys
jwks = httpx.get(jwks_url).json()
unverified_header = jwt.get_unverified_header(token)
rsa_key = None
for key in jwks['keys']:
if key['kid'] == unverified_header['kid']:
rsa_key = key
break
if not rsa_key:
raise HTTPException(status_code=401, detail='Invalid token')
payload = jwt.decode(
token,
rsa_key,
algorithms=['RS256'],
audience=config['audience'],
issuer=f"https://{config['domain']}/"
)
if required_role and required_role != 'system':
roles = payload.get('https://checkin.practicename.com/roles', [])
if required_role not in roles:
raise HTTPException(status_code=403, detail='Insufficient permissions')
return payload
except JWTError as e:
raise HTTPException(status_code=401, detail=f'Token validation failed: {str(e)}')
def require_role(role: str):
def decorator(func):
async def wrapper(*args, credentials=None, **kwargs):
if credentials:
verify_token(credentials.credentials, required_role=role)
return await func(*args, credentials=credentials, **kwargs)
return wrapper
return decoratorTesting & Validation
- CRISIS DETECTION - Explicit keywords: Send test messages containing each crisis keyword in the database (e.g., 'I want to kill myself', 'I have been cutting myself', 'I have a plan to end my life') and verify that ALL are classified as CRITICAL within 5 seconds, clinician receives SMS/email alert within 60 seconds, and patient receives 988/crisis resources within 30 seconds
- CRISIS DETECTION - Implicit/ambiguous: Send test messages with indirect crisis language (e.g., 'I just feel like disappearing', 'Everyone would be better off without me', 'I wrote letters to my family saying goodbye') and verify that ALL are classified as at least HIGH/URGENT and trigger clinician alert
- CRISIS DETECTION - False positive rate: Send 100 routine positive messages (e.g., 'Feeling great today!', 'Had a good week', 'Sleep has been better') and verify that <5% are incorrectly escalated above ROUTINE
- TRIAGE ACCURACY - Needs Attention: Send messages indicating clinically noteworthy but non-crisis content (e.g., 'I stopped taking my medication because of side effects', 'Haven't been sleeping well at all this week', 'Got into a big fight with my partner') and verify classification as NEEDS_ATTENTION
- TRIAGE ACCURACY - Routine: Send simple positive or neutral responses (e.g., 'Doing well, thanks for checking in', 'Mood is about a 7 today', 'Practiced the breathing exercises you suggested') and verify classification as ROUTINE
- SMS DELIVERY - Outbound: Trigger a check-in for a test patient and verify the SMS is received on the test phone within 2 minutes, contains the patient's first name, matches the configured tone, and includes the crisis disclaimer
- SMS DELIVERY - Inbound: Reply to a check-in SMS from the test phone and verify the response appears in the database, is processed through triage within 30 seconds, and appears in the clinician dashboard
- SMS DELIVERY - Opt-out: Reply STOP from the test phone and verify the patient is marked as opted_out in the database, no further check-ins are scheduled, and a confirmation message is sent
- HEALTHIE INTEGRATION - Webhook: Complete a test appointment in Healthie and verify the webhook is received, a check-in is scheduled for 24 hours later (or configured delay), and the correct patient and clinician are associated
- HEALTHIE INTEGRATION - Summary push: Generate a pre-appointment summary and verify it is formatted correctly and can be pushed to Healthie as a chart note via the GraphQL API
- CLINICIAN DASHBOARD - Authentication: Attempt to access the dashboard without authentication and verify a 401 response. Log in with valid credentials and verify access. Attempt to access another clinician's patient data and verify a 403 response
- CLINICIAN DASHBOARD - Alert display: Create test triage results at each level and verify they appear in the dashboard sorted by urgency (Crisis > Urgent > Needs Attention), with correct patient information and summaries
- CLINICIAN DASHBOARD - Acknowledge: Click acknowledge on a test alert and verify the alert is marked as acknowledged in the database with timestamp and clinician ID recorded in the audit log
- PRE-APPOINTMENT SUMMARY - Quality: Generate summaries for test patients with varied check-in histories (2-5 check-ins, mix of moods) and have the Clinical Safety Officer review for clinical accuracy, appropriate language, and utility
- ENCRYPTION VERIFICATION - At rest: Verify RDS encryption is enabled by checking the instance configuration. Verify S3 bucket encryption by attempting to upload without encryption headers. Verify KMS key is being used for encryption
- ENCRYPTION VERIFICATION - In transit: Use ssllabs.com to test the dashboard domain for TLS configuration. Verify the API only accepts HTTPS connections. Verify the database connection requires SSL (sslmode=require)
- AUDIT LOGGING - PHI access: Access a patient's conversation history via the dashboard and verify an audit log entry is created with the clinician's user ID, timestamp, action type, and resource ID
- AUDIT LOGGING - Completeness: Review CloudTrail logs for all API calls to AWS services (RDS, S3, SQS, Secrets Manager) and verify they are captured. Review application audit logs for all PHI access events
- LOAD TESTING - Simulate 200 concurrent patient responses arriving within a 5-minute window and verify all are processed through triage within 10 minutes, no messages are lost (check SQS dead-letter queue is empty), and crisis detection latency remains under 10 seconds
- FAILOVER - Kill one of the two ECS agent instances and verify the ALB routes traffic to the surviving instance with no message loss. Verify the replacement task launches within 2 minutes via ECS service auto-recovery
- BACKUP/RESTORE - Trigger a manual RDS snapshot, create a test restoration instance, and verify all data is intact and accessible. Delete the test instance after verification
Client Handoff
Client Handoff Checklist
Training Sessions (Minimum 3 sessions, 60-90 minutes each)
Session 1: Clinician Training (All clinicians)
- How the check-in system works end-to-end (patient receives SMS → AI processes → dashboard shows results)
- Dashboard walkthrough: viewing alerts, acknowledging alerts, reviewing conversations, reading pre-appointment summaries
- Understanding triage levels: what Routine, Needs Attention, Urgent, and Crisis mean and what action each requires
- Setting personal notification preferences (SMS, email, timing)
- Configuring check-in preferences per patient (tone, domains, frequency)
- What the AI can and cannot do: it is NOT a therapist, NOT for crisis intervention, NOT a diagnostic tool
- How to handle crisis alerts: verify patient safety, document actions, close the crisis event in the system
- How to discuss the check-in system with patients and collect consent
Session 2: Admin Training (Practice manager/owner)
- System configuration: adding/removing clinicians, managing patient roster
- Consent management: tracking consent records, handling revocations
- Understanding the audit log and how to pull compliance reports
- Monthly usage and cost review (understanding LLM token costs, SMS costs)
- How to request changes to check-in prompts, crisis keywords, or triage behavior
- Escalation path to MSP for technical issues
Session 3: Patient Onboarding Process Training (All clinical staff)
- How to explain the check-in system to patients in session
- How to collect and document consent (standard and SUD-specific)
- How to enroll a patient in the system
- How to handle patient questions about AI, privacy, and data use
- How to handle opt-outs and re-enrollments
Documentation to Deliver
Success Criteria Review
Maintenance
Ongoing Maintenance Plan
Daily (Automated + Spot Check)
- Automated: CloudWatch monitors system health, message delivery, and crisis detection
- Automated: SQS dead-letter queue alarm triggers if any messages fail processing
- MSP spot check (5 min): Review CloudWatch dashboard for anomalies — check message volumes, error rates, latency
Weekly (30 minutes MSP time)
- Review system metrics: messages sent, responses received, triage distribution, response rates
- Check for any unacknowledged urgent/crisis alerts older than 24 hours — follow up with clinician
- Review dead-letter queue for any failed messages and investigate root cause
- Check LLM API usage and costs against budget
- Verify backup completion (RDS snapshots, S3 replication)
Monthly (2-3 hours MSP time)
- Generate and deliver monthly usage report to practice: patient engagement rates, triage breakdown, cost summary
- Review and update crisis keyword dictionary with Clinical Safety Officer input
- Test crisis detection with 5-10 new test scenarios to verify accuracy
- Apply security patches to container images (rebuild and redeploy)
- Review Auth0 login audit logs for suspicious activity
- Review and rotate any expiring API keys or certificates
- Check for LLM model updates from OpenAI — evaluate if prompt adjustments are needed
Quarterly (4-6 hours MSP time)
- Comprehensive triage accuracy audit: sample 50 random triage results and have Clinical Safety Officer review for accuracy
- Prompt tuning: adjust check-in prompts and triage prompts based on clinician feedback and accuracy audit
- HIPAA compliance review: verify all BAAs are current, review access logs, update risk assessment if needed
- Performance review: analyze response latency, cost trends, scalability needs
- Clinician satisfaction survey: collect feedback on summary quality, alert relevance, dashboard usability
- Update system documentation for any changes made during the quarter
Annually
- Full HIPAA Security Risk Assessment update
- Penetration testing of all internet-facing components
- BAA renewal verification for all vendors
- Comprehensive disaster recovery test (full restore from backup)
- AI model evaluation: assess whether newer LLM models offer better cost/quality and migrate if warranted
- Annual clinician refresher training (1 hour)
- Review 42 CFR Part 2 compliance posture (especially for first year with February 2026 compliance deadline)
SLA Considerations
- System availability: 99.5% uptime (allows ~3.65 hours downtime/month)
- Crisis alert delivery: Within 2 minutes of crisis detection — this is the highest priority SLA
- Routine message processing: Within 15 minutes of receipt
- Dashboard availability: Within 4 hours of reported outage
- MSP response time: Critical (crisis system down): 30 minutes. High (message delivery failure): 2 hours. Medium (dashboard issues): 4 hours. Low (feature requests): 5 business days.
Escalation Path
Model Retraining / Prompt Update Triggers
- Triage accuracy drops below 85% on quarterly audit
- New OpenAI or Anthropic model release with significant quality improvement
- Clinician feedback indicates consistent misclassification patterns
- New clinical modalities added to the practice (e.g., group therapy, couples therapy)
- Regulatory changes requiring updated consent language or crisis protocols
- Patient population changes (e.g., new age groups, languages)
Alternatives
...
Pre-Built Mental Health Chatbot Platform (Wysa or Woebot)
Instead of building a custom AI agent, license an existing clinical AI chatbot platform like Wysa ($74.99/year individual, custom enterprise pricing) or Woebot (custom enterprise pricing). These platforms offer pre-built, clinically validated conversational AI for mental health check-ins with existing evidence base and FDA considerations. The MSP would handle integration and IT management rather than full custom development.
Keragon + StackAI No-Code Approach
Use Keragon ($49-399/month) for healthcare-specific workflow automation combined with StackAI for no-code HIPAA-compliant chatbot building. Both platforms are HIPAA-ready with BAAs included. This approach eliminates most custom coding — the check-in flows, LLM integration, and EHR connections are configured via visual drag-and-drop interfaces. The clinician dashboard would use Keragon's built-in notification features plus a simple portal.
Spruce Health as Primary Platform
Use Spruce Health ($24/user/month) as both the secure messaging platform and the clinician communication hub, with a lightweight AI processing layer that intercepts Spruce messages via API. Spruce provides HIPAA-compliant messaging, phone, video, and team collaboration already used by many mental health practices. The AI agent would be a thin service that monitors Spruce conversations and injects triage metadata.
Azure OpenAI + Azure Stack (Single Vendor)
Deploy entirely on Microsoft Azure using Azure OpenAI Service for LLM inference, Azure App Service for the agent, Azure Database for PostgreSQL, Azure Communication Services for SMS, and Azure AD B2C for authentication. This consolidates all vendors under a single Microsoft BAA and Azure HIPAA compliance umbrella.
Manual Triage with AI-Assisted Summaries Only
Deploy a simplified system that sends check-in messages and collects responses, but skips automated triage entirely. All patient responses are displayed chronologically in the clinician dashboard for manual review. The AI only generates pre-appointment summaries from the raw conversation data. No autonomous crisis detection — clinicians review all messages themselves.
Want early access to the full toolkit?