Developer Guide

This guide is for people who change, review, operate, or release the code. User onboarding stays in User Guide; IRB/auditor evidence stays in IRB/Auditor Profile.

Start by Role

Reader	Goal	Start here
Pipeline developer	Change extraction, PHI scrub, cleanup, publish, variables, or lineage behavior.	Architecture, then Dataset Extraction
PDF pipeline developer	Change PDF extraction, redaction, merge, or snapshot fallback.	PDF Extraction, then PHI Architecture
Agent/tool developer	Add or change assistant tools without breaking file-zone and PHI gates.	Agent Instructions (for AI Coding Assistants), then API Reference
Privacy or security reviewer	Inspect load-bearing controls, invariants, and tests.	PHI Architecture, Sandbox: Subprocess-Isolated Code Execution, then Testing
Maintainer	Run verification, restore reviewed snapshots, and prepare releases.	Operations, then Production Readiness
Documentation contributor	Keep README and Sphinx organized by audience.	Documentation Style, then Contributing

Contents

Architecture & Decisions

• Architecture
  • System Overview
  • The Five-Tier Zone Model
  • Core Components
  • Pipeline Modules
  • Supporting Services
  • Data Flow
  • Security Boundaries
  • Design Principles
  • See Also
• PHI Architecture
  • The Four Tiers (plus audit and one out-of-zone tier)
  • The Eight-Action Scrub Catalog (Step 1.6)
  • The Agent-Boundary Three-Gate Stack
  • The PDF Orchestrator’s Redact-Then-Call Posture
  • The Integrity Chain
  • Log Hygiene
  • KeyStore
  • Subprocess Sandbox
  • Module Map
  • IRB Benchmark Cross-Reference
  • When You Touch This Code
  • See Also
• Architecture Decisions (ADRs)
  • ADR-001 — Single-study, local-first runtime
  • ADR-002 — HMAC-SHA256 pseudonymization with sidecar key (no vault)
  • ADR-003 — SANT per-subject date jitter over HIPAA year-only
  • ADR-004 — Rule+allowlist over Microsoft Presidio
  • ADR-005 — tmpfs staging as an operator opt-in (not default)
  • ADR-006 — External-API PDF extraction refused by default
  • ADR-007 — Four-tier architecture (RED / AMBER / GREEN / GREEN-PROTECT)
  • ADR-008 — Agent boundary PHI + k-anon gate as defence-in-depth
  • ADR-009 — Counts-only audit reports (never raw values)
  • ADR-010 — Subprocess + rlimits sandbox for run_python_analysis
  • ADR-011 — KeyStore (in-memory API-key registry)
  • ADR-012 — Two-way PDF orchestrator (pdfplumber + redacted-text LLM merge)
  • ADR-013 — Single reviewed snapshot baseline
  • ADR-014 — Parallel extraction phase (3-worker ThreadPoolExecutor)
  • ADR-015 — l-diversity (l=2) on row-returning tools
• References
  • Primary Regulations
  • Standards & Frameworks
  • Techniques
  • Benchmarks & Comparative Studies
  • Tools & Libraries Cited in Decisions
  • Reading Order for a New Contributor
• Tech Stack
  • Runtime — language and tooling
  • Runtime — pipeline
  • Runtime — agent
  • Runtime — security
  • Runtime — observability
  • Development
  • Custom type stubs
  • Pinning policy

Pipeline Components

• Dataset Extraction
  • Overview
  • Data Flow
  • Source
  • Output
  • JSONL Record Schema
  • Zone Enforcement
  • CLI Usage
  • Downstream Handoff
  • Key Files
• PDF Extraction
  • Two co-existing extraction paths
  • Dispatch
  • Orchestrator path (the default)
  • Capability gate details
  • Legacy raw-PDF API path
  • Output schema (per-form)
  • CLI usage
  • Key files
  • Testing
  • Downstream usage
  • Licensing
• Operations
  • Prerequisites
  • Pipeline Run
  • Artifact Rebuild
  • Cleanup
  • Security Verification
  • Quality Checks
  • Debug and Troubleshooting
  • Known Limitations
  • AI Assistant
  • Trio-Bundle Snapshot Maintenance
• Production Readiness
  • Scope
  • Release Gate
  • Deployment Boundary
  • Security Headers
  • Monitoring
  • Backups and Restore
  • Secret and Key Rotation
  • Incident Response
  • Operational Non-Negotiables
  • External References
• Sandbox: Subprocess-Isolated Code Execution
  • Threat Model
  • Architecture
  • The macOS Asymmetry
  • Configurable Knobs
  • Code Persistence and Replication
  • Tests
  • Future Work
  • When You Touch This Code
• Versioning
  • Version Source of Truth
  • Automatic Version Bumping
  • Manual Bumping
  • Version Validation
  • Checking the Current Version

Reference & Status

• API Reference
  • Core Modules
  • Extraction Modules
  • Security Modules
  • Utility Modules
  • Extraction Modules (continued)
  • AI Assistant Modules
  • Telemetry
  • Extraction Modules (continued)
  • Utility Modules (continued)
  • Sandbox Subprocess
  • Extraction I/O Helpers
  • Indices and Tables
• Project Status
  • Implemented
  • Verification
  • IRB Conformance
  • Known Follow-Ups
• Production Backlog
  • Supply Chain
  • Runtime Resilience
  • Security Headers
  • Observability
  • Data Retention
  • Deployment Packaging
• Documentation Style
  • Style Basis
  • Documentation Boundary
  • Profile-first Rule
  • User-facing Page Pattern
  • Developer-facing Page Pattern
  • IRB-facing Page Pattern
  • Language Rules
  • Freshness Checks
  • Change Checklist

Contributing

• Contributing
  • Branch And PR Workflow
  • Attribution
  • Local Setup
  • Before Opening A PR
  • Coding Rules
  • Documentation Rules
  • Adding A PHI Rule
  • Adding An Agent Tool
  • Adding Provider Support
  • Review Focus
• Testing
  • Active Commands
  • Current Test Layout
  • What Each Gate Proves
  • PHI-Critical Coverage
  • Writing Tests
  • CI Behavior
• Agent Instructions (for AI Coding Assistants)
  • Orientation
  • Quick reference
  • Architecture (two-world)
  • Tech stack
  • Critical conventions
  • Web UI architecture
  • Key files
  • Documentation

Working Rules

• Preserve the raw → staging → published bundle → agent-boundary PHI model described in PHI Architecture.

• Keep implementation changes, tests, and documentation in the same PR when behavior changes.

• Run the smallest focused tests first, then the repo gates required by Testing.

• Keep README brief. Put durable detail in Sphinx and link to it.