Databook
activeopen governmentAn award-winning transparency tool that combines over 50 official NYC open datasets — agencies, people, jobs, capital projects, schools, districts and procurement — into one searchable picture of how city government works.
Improved database performance by filtering raw contract IDs to enable Parquet partition pruning. This optimization speeds up contract detail queries.
Added a New York Department of State legal-entity card to vendor profiles, displaying business age, legal form, and registration details. The integration uses an offline DuckDB-to-Postgres crosswalk script to match vendors with the NY DOS registry, achieving a 43% unambiguous match rate.
Added the Doing Business Database (LL34) to vendor profiles, linking 4,474 vendors to over 31,000 associated people and registered lobbyists. This update integrates a new daily ingestion script, implements conservative name-matching to prevent false positives, and resolves source data issues like truncated dates.
Optimized the contract detail endpoint by filtering on the raw contract_id column instead of using a regex function, enabling Parquet row-group pruning. The change also materializes filtered rows once to avoid duplicate scans, resulting in up to a 34x speedup for contract queries.
This update enables five PASSPort vendor sub-tables to load and refresh daily via a new post-ingest hook. It surfaces new vendor profile sections for ownership, leadership, and performance ratings, while adding contract-specific MOCS ratings. It also fixes a bug where post-ingest hooks were silently skipped during extractor dataset processing.
Surfaces existing database and API data on vendor and contract profiles. Key additions include SBS certified-business profiles matched by normalized name, Checkbook paid-to-date actuals, contract payment windows, and current contract value tiles. It also links transaction payees to PASSPort IDs and improves labelling for NYCHA and transaction blocks.