Databook

Updated public documentation to correct outdated information regarding the Checkbook data lake refresh crons, which are now active. The updates also revise the system health check documentation to accurately reflect all eight active monitoring checks and document the retirement of the newsletter repository copy due to security improvements.

This update improves the reliability of the public API by ensuring that temporary, incomplete data during system restarts is never cached. This prevents contract pages from temporarily displaying blank spend figures, ensuring users always see accurate financial data.

Updated public developer documentation to record active Cloudflare caching rules for the /oce/ and /get/ API endpoints, which significantly improve response times. The documentation also details the behavior of the scraper challenge rules and custom firewall settings to ensure reliable public API access.

This update significantly improves the loading speed of procurement, contract, and vendor pages. By adding database indexes to the contracts, vendors, and solicitations tables, the system can now instantly retrieve specific records instead of scanning through entire datasets.

Resolved a cron collision issue during organization refreshes.

Changed the NYCHA data refresh schedule from weekly to monthly to reduce system load, and updated the staleness alert threshold to 40 days to match. Also added a documentation file outlining data publisher improvement requests for the Comptroller's CheckbookNYC and MOCS teams.