Documentation
Dataset index
All 18 phases
Each independently runnable, resumable, and testable offline via --selftest.
| # | Dataset | Source | Cadence | Module |
|---|---|---|---|---|
| 1 | Senate stock trades (PTRs) | efdsearch.senate.gov | Daily | scrape_senate.py |
| 2 | House stock trades (PTRs) | disclosures-clerk.house.gov | Daily | scrape_house.py |
| 3 | Ticker resolution | embedded in filing text | — | entities.py |
| 4 | Daily incremental run | — | Daily (cron) | daily.py |
| 5 | Read-only REST API | — | — | api.py |
| 6 | Corporate lobbying | LDA.gov | As needed | scrape_lobbying.py |
| 7 | Government contracts | USAspending v2 | Rolling 7-day | scrape_contracts.py |
| 8 | Entity resolution engine | SEC company_tickers.json | — | entities.py |
| 9 | Insider trades (Form 4) | SEC bulk data + EDGAR daily index | Daily gap-fill | scrape_insiders.py |
| 10 | 13F institutional holdings | SEC Form 13F data sets | Quarterly | scrape_13f.py |
| 11 | Off-exchange short volume | FINRA daily files | Daily | scrape_short_volume.py |
| 12 | Patents | USPTO Open Data Portal | As needed | scrape_patents.py |
| 13 | Corporate/PAC donations | OpenFEC | As needed | scrape_donors.py |
| 14 | Wikipedia pageviews | Wikimedia REST API | As needed | scrape_pageviews.py |
| 15 | Politician net worth (floor est.) | derived: trades × live prices | On demand | networth.py |
| 16 | Executive compensation | SEC XBRL Frames API | As needed | scrape_execcomp.py |
| 17 | Donald Trump 278-T trades | ProPublica DocumentCloud mirror | Manual | scrape_trump.py |
| 18 | Senate Annual Financial Disclosure | efdsearch.senate.gov | Annual | scrape_senate_annual.py |
Prev: Quick start
Next: API reference