Databook
activeopen governmentAn award-winning transparency tool that combines over 50 official NYC open datasets — agencies, people, jobs, capital projects, schools, districts and procurement — into one searchable picture of how city government works.
This update resolves API connection timeout errors by routing heavy DuckDB scans to a dedicated thread executor, preventing Postgres connection pool starvation. It also configures the Postgres pool to open fully on startup and adds Sentry error alerts to prevent silent failures in weekly data refresh scripts.
Added vendor and contract profile data enrichment and implemented a dedicated DuckDB executor to prevent API event loop blocking and connection timeouts.
Added Sentry Cron monitoring for payroll and version-controlled the curated crosswalk. Updated the NYCHA curated crosswalk to support reviewed rejections, and documented crosswalk normalization and false-positive fixes. Warmed the Postgres database pool baseline to prevent API cold-start connection storms.
This pull request adds Sentry Cron monitoring to the monthly payroll refresh script to alert on failures or runs exceeding 12 hours. It also version-controls the curated NYCHA crosswalk CSV file to prevent data loss and adds one newly approved vendor mapping.
This update adds support for durable fuzzy-match rejections in NYCHA curated crosswalks by storing rejected pairs with a NULL ID so they are not re-proposed. It also sets the curated file path to a default location for automatic weekly runs and updates the logs to report confirmations and rejections separately.
This pull request fixes API timeout bursts by increasing the minimum Postgres connection pool size from 2 to 10. It also adds startup logging to track the pool's state.