AI Contract Classifier — prompt, inputs & assumptions

For expert review. This document describes exactly how the "Digital Services Analysis" pages on databook.nyc classify NYC technology contracts. Everything below is editorial/AI-derived enrichment layered on top of official, normalized government data — it is a prompt to investigate, not a determination. We are seeking feedback on whether the prompt, the inputs it relies on, and the outputs it produces are sound enough to publish and to inform renewal decisions.

• Where it runs: an offline batch job over contracts held by vendors tagged digital_services. Results are stored and read by the public pages; a human curated override is never overwritten by re-runs.
• Model / settings: Google Gemini 3.5 Flash, temperature = 0, forced JSON output, ~20 contracts per request.
• Source code: api/classify_digital_contracts.py.
• Status: v2 — the classifier now reads richer, multi-source scope text (see §1), not just the contract title. This directly addresses the biggest v1 critique. ~639 expiring-before-2030 contracts are classified today.

---

1. What the model sees (tiered inputs)

The model no longer judges from a bare title. For each contract we assemble the best available description by joining four official, already-normalized sources, and the prompt tells the model to weigh them in priority order (richest first):

Priority	Field(s) given to model	Source (all official NYC data)	How joined	Coverage*
1	contract_purpose, expense_category	Checkbook NYC (checkbook_contract_meta)	by normalized contract ID	~88%
2	scope_description, notice_category, selection_method, special_case_reason	City Record / CROL notice for the PIN (longest description wins)	by PIN/EPIN	subset
3	commodity (Main Commodity), solicitation_name	PASSPort solicitations	EPIN-prefix crosswalk	~8%
4	contract_title, program, industry, vendor_name, agency	PASSPort / MOCS contracts	base record	100%
context	contract_type, term, award_amount vs current_amount, status, procurement_method	PASSPort	base record	100%

* Share of the classified set for which that input is present and non-empty. The Checkbook contract purpose — the most authoritative statement of what a contract actually buys — is available for ~88% of contracts, so most judgments now rest on a real scope description. Where higher tiers are missing the model falls back to the title (a known weak case — see Assumption 2).

It still does NOT see: the full statement of work / deliverables, line items, actual spend, number of integrations, data sensitivity/classification, or user counts. The richest input is typically one to a few sentences.

---

2. The exact prompt (system instruction)

You classify New York City government technology contracts to help the city decide which EXPIRING contracts it should NOT renew.

Each contract comes with several fields — weigh them in roughly this priority when they are present and non-empty: contract_purpose and expense_category (from Checkbook NYC, the most authoritative description of what the contract buys), then scope_description and notice_category (from the City Record notice), then commodity and solicitation_name (from the originating solicitation), then contract_title, program and industry. The title alone is often vague; always prefer the richer contract_purpose / scope_description / commodity when available. contract_type, term, award_amount vs current_amount, status, procurement_method, selection_method and special_case_reason are context for the renewal judgment.

For each contract, judge:

• tech_relevant: Is this genuinely a digital / IT / software / data service or product? Return false for physical-world work mis-tagged as tech: pest control, painting, ship/drydock repair, physical security guards, fuel, waste hauling, generators/power equipment, office furniture/filing, medical supplies, building/HVAC systems, courier/delivery.
• is_license: Is this primarily the purchase of a SOFTWARE LICENSE or subscription (vs. custom development, staffing, consulting, hardware)?
• license_product: If is_license, the product/platform name (e.g. "Microsoft Office 365", "ArcGIS", "Salesforce"); else "".
• license_purpose: If is_license, what the software is used for, one short phrase; else "".
• function_category: A 2-4 word capability bucket. Prefer reusing common buckets: "Website/CMS", "GIS/mapping", "ERP/financials", "Case management", "Cybersecurity", "Data/analytics", "Office productivity", "Staffing/consulting", "Hardware/infrastructure", "Telecom/network", "Identity/access", "Payments", "Document management", "Non-tech". Use the commodity field as a strong hint when present.
• build_vs_buy: How plausibly could the city replace this by building its own solution with open-source software + modern AI tooling? "high" = simple, commoditized software a small team could now stand up (basic websites, forms, simple dashboards, chatbots); "medium" = feasible but real effort/integration; "low" = specialized/regulated/hardware/deeply-integrated platforms or non-tech.
• rationale: One sentence justifying the build_vs_buy rating, citing the scope/commodity evidence you relied on.

Base your judgment on the evidence provided; do not invent capabilities not implied by the fields. Return one result object per contract, preserving the given id.

---

3. The output (per contract)

Field	Type	Meaning
tech_relevant	true/false	Is this genuinely a digital/IT service (vs. mis-tagged physical work)?
is_license	true/false	Is this primarily a software license/subscription?
license_product	text	Product name, if a license
license_purpose	text	One-phrase use of the software, if a license
function_category	text	One of ~14 capability buckets (see prompt)
build_vs_buy	high / medium / low	How replaceable with open-source + AI
rationale	text	One sentence justifying the build-vs-buy rating

How each output is used on the site: tech_relevant=false hides the contract from digital totals (cleanup of mis-tagged vendors); build_vs_buy=high raises a "Build-your-own candidate" flag; licenses get a badge + "Licenses only" filter; function_category powers a category filter.

---

4. Worked examples (live output)

These are real rows from the current run, chosen to show the tiered inputs at work. Note how several titles are just a vendor name + procurement code — the classification is only possible because of the richer Checkbook/solicitation/ notice text.

Opaque title → decoded by richer input

Contract title (input tier 4)	Richer input used (tier 1–3)	Result
K Systems Solutions LLC 25MI021301R0X00	"New and Renewal Licenses for Existing SurveyMonkey Software"	license · SurveyMonkey · Data/analytics · build=high
Amendment-SPRUCE TECHNOLOGY INC-22EI014301R0J0	"Online referral portal for Early Intervention"	Website/CMS · build=high
BIT Nintex Workflow Software 6300039X	(purpose) Nintex workflow automation	license · Nintex Workflow · Office productivity · build=high
Maintenance and support of Track IT	"Maintenance and support of Track It!"	license · Track-It! · Case management · build=high ("basic IT ticketing… can easily be built")

"Build-your-own" candidates (build=high)

Title / scope	Result + rationale (verbatim, truncated)
OrgChart Now Software Subscription	Office productivity · "Org chart visualization is a highly commoditized capability that can easily be built…"
GoDiagram 10 Team License (MTA project)	Data/analytics · "GoDiagram is a software library for creating interactive diagrams, which can easily…"
MWBE Public Building Dashboard and Data Analysis	Data/analytics · "Building a public building dashboard and performing data analysis is highly feasible…"

Non-tech catches (tech_relevant = false → removed from digital totals)

Title	Richer input	Result
CSB Multiyear High and Low Pressure Boiler…	"High and Low Pressure Boilers Maintenance Services"	Non-tech — physical boiler maintenance
Janitorial Services at DYFJ Secure Detention…	(purpose) janitorial services	Non-tech
Interagency Communications Committee Support	"…Committee Support Renewal" (meeting support)	Non-tech — professional committee support

Specialized / not easily replaceable (build=low)

Title	Result + rationale
IT Consulting Services for CurRent NYC	Staffing/consulting · "professional IT consulting and development services…"
TLC Connect Quality Assurance Consultant	Staffing/consulting · "professional services contract for a Quality Assurance consultant…"

---

5. Assumptions & judgment calls baked in (please critique)

1. The framing is renewal-oriented, not neutral. The prompt's first line states the goal is deciding which contracts NOT to renew — this may bias the model toward "replaceable." Should this instead be a neutral capability classifier whose output a human weighs separately?
2. Build-vs-buy is inferred from a short scope, not a full SOW. v2 feeds the Checkbook contract purpose / notice / commodity (mitigating the v1 "title only" problem for ~88% of contracts), but the richest input is still typically one to a few sentences. It has no view of integration footprint, data sensitivity, compliance regime, user base, or maintenance burden — the things that usually decide build-vs-buy in practice. And ~12% still fall back to the title alone.
3. The "open-source + AI can replace it" premise is itself an assumption. It encodes an optimistic prior about the city's in-house build and ongoing maintenance/security capacity, and ignores total cost of ownership. "A small team could stand it up" is doing a lot of work.
4. tech_relevant is a hard-coded exclusion list (pest control, drydock, guards, fuel, waste…), not a principled definition. Dual-use cases (e.g. "security systems" = cybersecurity vs. physical alarm) are ambiguous; novel non-tech categories can slip through.
5. is_license collapses a spectrum — pure license vs. SaaS subscription vs. license-plus-implementation-plus-support bundles are lumped together; "primarily a license" is undefined.
6. function_category is a fixed, NYC-agnostic taxonomy chosen by us, not derived from how NYC actually organizes IT. Free-text drift is still possible.
7. No confidence or abstention. The model is asked to base its judgment on the evidence and not invent capabilities, but there is still no "insufficient information" output — a contract with only a thin title still gets a confident rating, indistinguishable from one backed by a full Checkbook purpose.
8. Three coarse build-vs-buy tiers with thresholds defined only by examples ("basic websites, forms, chatbots" = high); no notion of who builds/maintains it, or over what time horizon.
9. Single model, single pass. No second-model check or ensemble; human review is optional (supported via a curated flag, but not required).
10. Input coverage is uneven and silent. Higher-tier inputs (notice text, commodity) are present for only a minority of contracts, so two contracts with the same rating may rest on very different evidence — and the output does not record which tier of input it used.

---

6. Questions we'd like expert input on

1. Neutral classifier vs. explicitly renewal-oriented framing — which is appropriate for a public, official-data product?
2. With richer scope now available for most contracts, is a build-vs-buy rating publishable (labeled heuristic), or does a credible call still require inputs we don't feed (full SOW, # of integrations, data classification, incumbency)?
3. Should build_vs_buy carry an explicit confidence and an "insufficient information" value — and should the output record which input tier it relied on?
4. Is the function_category taxonomy right for NYC, or should it map to an existing city/federal IT category scheme (e.g. the Checkbook expense categories, NIGP commodity codes)?
5. Where is human review required vs. optional before a flag is shown publicly?
6. Is the tech_relevant denylist the right mechanism, or should "digital service" be defined positively (e.g. from commodity/expense codes)?

---

Generated to support expert review of the databook.nyc Digital Services Analysis classifier (v2 — tiered inputs). Proposed changes can be wired back into api/classify_digital_contracts.py and re-run.