Databook

Updated the CheckbookNYC extractor to validate that HTTP 200 responses contain valid XML before parsing. If a non-XML error page is returned due to throttling, the extractor now retries the request with exponential backoff instead of silently breaking the pagination loop and truncating the fiscal year data.

Added NYCHA contracts, budget, and revenue domains to the data, API, and frontend layers. Implemented retry and backoff logic for CheckbookNYC throttling, and optimized the budget refresh process to update only recent fiscal years. Adjusted budget defaults to the latest fully-closed fiscal year.

Added a Capital Projects mini-dashboard and spending chart to the procurement section, ensuring field consistency with capital projects. Improved spending data ingestion speed by 7x, automated weekly lake refreshes, optimized cache pre-warming, and increased memory limits to prevent API out-of-memory errors.

Fixed API and database memory leaks by capping DuckDB memory, serializing startup cache pre-warming, and limiting subdataset rows. Resolved OOM crashes by scoping the contract spend-map and bounding the digital-reform cache.

This pull request adds a planning document outlining the scope and roadmap for expanding the Contracts domain to match CheckbookNYC capabilities. The plan details how to deliver new features like contract utilization rates, payment timelines, and sub-vendor breakdowns using existing data pipelines, alongside a phased timeline for future API and frontend updates.

Added budget, revenue, and M/WBE spending domains with corresponding API endpoints and frontend lenses. Integrated Checkbook spending data, sub-vendor dashboards, and enhanced contract profiles with CSV export. Redesigned and consolidated the agency profile pages, collapsing procurement subsections and adding a hero component.