Databook
activeopen governmentAn award-winning transparency tool that combines over 50 official NYC open datasets — agencies, people, jobs, capital projects, schools, districts and procurement — into one searchable picture of how city government works.
Fixed a bug where the unmapped-entity scan never executed by moving it to a new end-of-cycle scheduler sweep. Additionally, registered the "civillist" dataset as normalized to enable its entity scanning, added six new tests, and resolved a gap identifying one unmapped agency.
This update fixes a database seeding bug by making the NORMALIZED_DATASETS membership the single owner of the needs_normalization column. It removes the redundant DIRECT_SOCRATA_OVERRIDE configuration and prevents incorrect column overwrites, resolving a 27-dataset drift issue without changing production data.
This update fixes a bug where registry deactivations were bypassed in production due to a development-only path check. The deactivation logic has been moved to run unconditionally on every execution, and five new tests were added to prevent future regression without altering other production ingest behaviors.
Deactivated the duplicate cpdb_commitments registry entry to prevent redundant, failing normalizer runs.
Added a new dataset-staleness monitoring script that alerts when Socrata datasets fall behind their source or go empty. Additionally, the import process now explicitly distinguishes empty-at-source errors from headerless responses.
This update prevents pipeline failures when an NYC Open Data source publishes an empty dataset with a valid CSV header. Instead of raising an error, the system now logs a warning, caches the ETag to prevent redundant reprocessing, and continues serving the last good output.