u15data — Architecture Document

Version: 1.23 Status: Draft — schema and infrastructure confirmed; all 15 institutions' faculty-level data classified (193 faculties, 47 sub-units, 19 excluded, 259 rows); Phase 1 robots.txt audit resolved at institution level and re-audited at faculty-unit level; people-collection schema applied and now holding data: 972 directory-page rows (212 ai_likely, 110 ai_unsure, 22 ai_rejected, 4 manual_needed, 624 proposed) for 166 of the 193 faculties, and 158 saved robots.txt texts (Sections 3.11, 3.15, 5.14); discovery, checking and import scripts built on shared robots and scheduling modules (Sections 5.12 to 5.14); people scraper not yet written; human-review pass not yet started Last updated: 2026-11-08 (v1.23, evening)

1. Purpose

1.1. Compile faculty and staff contact records for all 15 U15 Canadian universities, for academic/research outreach purposes.

1.2. Collection is limited to information already published on official institutional pages, and only from sources/units that explicitly or implicitly permit automated collection (per robots.txt and Terms of Use, checked at the faculty/unit level, not assumed at the institution level).

1.3. Priority: faculty first, staff second. Both remain in scope (per the original decision to cover "faculty and staff, all 15 institutions"), but collection and validation work should sequence faculty ahead of staff where the two can be separated — e.g., when auditing or collecting a unit with both, faculty records take priority; staff can follow once faculty coverage for that unit is established.

2. Server placement and naming

2.1. Project folder: /opt/apps/u15data/

2.2. Follows the server's standard one-app-per-folder, dedicated-venv pattern (per /opt/apps/CONVENTIONS.md), not the shared-database exception used by zoomsort/zoom-downloader.

2.3. Database: u15data_db

2.4. Application role: u15data_app

2.5. Naming follows the xab_db / xab_app reference pair, now recorded in /opt/apps/CONVENTIONS.md as the standard pattern: <project>_db (database) owned by <project>_app (role).

2.6. Docs for this project live in /opt/apps/u15data/docs/, per server convention.

3. Database schema

Three tables, normalized so audit/compliance data (which changes rarely) is separated from contact data (which refreshes on a recurring cycle).

3.1. Table A — institutions

Reference table, one row per U15 institution.

Field Type Notes
institution_id PK
name text
province text
central_directory_url text nullable — existence and location of a central directory is unconfirmed for all 15 as of v1.7
has_central_directory enum yes / no / unknown — default unknown; tracks whether a central, institution-wide staff directory exists at all, separate from whether it can be collected
robots_txt_url text
robots_txt_status enum no_file / empty_file / has_rules_permissive / has_rules_restrictive — same definitions as Section 5.1; nullable where ambiguous pending confirmation
tos_url text nullable
tos_summary text nullable
automated_collection_allowed enum yes / no / manual_only — nullable; not set until a directory's actual location has been checked against both robots.txt and Terms of Use for that specific path, since a domain-wide robots.txt finding alone doesn't establish this
last_checked_date date
notes text nullable
sunshine_list_url text nullable — see Section 5.10
sunshine_list_mechanism enum centralized_provincial / per_institution_posting / annual_report_to_legislature / not_applicable / unknown — how (or whether) this institution's compensation disclosure is actually published; default unknown
sunshine_list_threshold numeric nullable — the compensation threshold above which disclosure applies, in CAD; varies by province and can change year to year
sunshine_list_status enum confirmed / unconfirmed — whether the specific source has actually been located and verified, not just the general provincial mechanism; default unconfirmed
directory_usage_terms text nullable — exact wording of any usage restriction found stated directly on a directory page itself (e.g., a prohibition on bulk mailing or solicitation), captured verbatim rather than paraphrased, since a paraphrase could blur exactly what's prohibited. Distinct from tos_url/tos_summary, which cover the institution's general Terms of Use — this field is specifically for restrictions found on the directory page itself, which may be stricter or more specific.
directory_terms_captured_date date nullable — when directory_usage_terms was captured
classification_notes text nullable — added 2026-09-27. A short plain-language paragraph per institution summarizing what the faculty count shown publicly excludes and why (e.g. college-level groupings, historic affiliated colleges, administrative bodies) — written for the public-facing faculty-directories page (Section 6), not internal use. See Section 3.8 for the companion institution_sources table.

3.2. Table B — faculty_units

Faculty/school-level dataset. Originally scoped as a fallback table for institutions without a usable central directory (see Section 5.2's original two-tier strategy); superseded 2026-09-26 — populated for all 15 institutions regardless of central-directory status, since faculty-level data has independent value beyond being a fallback. Superseded again 2026-09-27: the original flat structure conflated true top-level faculties with nested schools, colleges, and departments, which had both been captured as peer rows during discovery (see Section 7.2, now resolved, and the companion u15data_faculty_discovery_log v1.1 Section 4 for the full resolution of the resulting row-count discrepancy). unit_level and parent_unit_id, added below, resolve this.

Field Type Notes
unit_id PK
institution_id FK → institutions
unit_name text e.g., "Faculty of Engineering," "Registrar's Office"
home_page_url text nullable — the unit's own website. Distinct from directory_url: confirming a unit's home page is a much lower bar than confirming it has an actual staff-listing page. Added 2026-09-26 to stop conflating the two.
directory_url text nullable — only populated when a genuine staff/professor-listing page has been confirmed (not just a home page). As of 2026-09-26, confirmed for a minority of the 275 units on file; most have a confirmed home_page_url but no confirmed directory_url yet.
robots_txt_url text
robots_txt_status enum no_file / empty_file / has_rules_permissive / has_rules_restrictive — see Section 5.1 for definitions; deliberately distinguishes "no robots.txt exists" from "a robots.txt exists and is blank," since these are different facts about the site even though both currently mean no automated-access restriction is stated
tos_url text nullable
tos_summary text nullable
automated_collection_allowed enum yes / no / manual_only
last_checked_date date
notes text nullable — added 2026-09-26 alongside the discovery process; free-text provenance/context (e.g. "confirmed via automated link discovery," "under Faculty of Health Sciences")
ai_confirmed boolean default false — added 2026-09-26. True when a link was found on a real, fetched page (not guessed/pattern-inferred) and judged relevant by AI review of that page's actual links. A weaker bar than human_confirmed, not a replacement for it.
human_confirmed boolean default false — added 2026-09-26. Whether a person has actually verified the row. As of v1.21, false on all rows — the human-review pass has not started.
unit_level text nullable, CHECK constrained to faculty / sub_unit / excluded — added 2026-09-27. faculty is a true top-level unit (the project's actual unit of analysis); sub_unit is nested within a specific faculty (a school, college, or department captured during discovery but not itself a faculty), with parent_unit_id set; excluded is a real, on-file unit that is neither — administrative/coordinating bodies (e.g. most institutions' Graduate Studies), historic affiliated colleges, or college-level groupings that organize faculties without being one. As of v1.21, all 259 rows across all 15 institutions are classified. Corrected 2026-11-08 (v1.22): the live database holds 193 faculty, 47 sub_unit and 19 excluded (total 259); v1.21 and the earlier discovery log stated 191, 41 and 27. The 191 was an addition slip (the per-institution counts 17+13+6+7+17+13+17+18+14+15+10+17+17+6+12 add to 193); the 41 and 27 could not be traced to any source, and the database is authoritative. Full per-institution reasoning and every source consulted is in the companion document u15data_unit_classification_sources v1.1.
parent_unit_id integer nullable, self-referencing FK to faculty_units.unit_id, ON DELETE RESTRICT (a faculty row cannot be deleted while a sub_unit still points at it) — added 2026-09-27. NULL for faculty and excluded rows; set to the parent faculty's unit_id for every sub_unit row. In a small number of cases (e.g. UBC's School of Dental Hygiene-equivalent nesting) a sub_unit's parent is itself another sub_unit, not a top-level faculty — the schema permits this chain but the public-facing export (Section 6) shows faculty-level rows only.

3.8. Table H — institution_sources

Added 2026-09-27. One row per external reference link used to verify an institution's real faculty structure during the unit-level classification pass (Section 3.2). Distinct from discovery_seeds (Section 3.6), which holds the original "list of faculties" pages used for the initial discovery run — institution_sources holds the (often several) additional sources consulted afterward to confirm which discovered units are true faculties versus nested sub-units, and to catch gaps the original discovery missed. Displayed on the public faculty-directories page (Section 6) alongside each institution's classification_notes.

Field Type Notes
source_id PK
institution_id FK → institutions
url text
description text nullable — one line on what this source confirmed
added_date date default current date

3.3. Table C — contacts

One row per person.

Field Type Notes
contact_id PK
unit_id FK → faculty_units institution is derivable via this FK, not duplicated
full_name text
title text
department text nullable — some faculties have no departments
email text
source_url text exact page the record was collected from
collected_date date
last_verified_date date nullable
status enum active / stale / removed
opt_out_flag boolean default false
collection_method enum automated / manual — see Section 5.8
snapshot_html_path text path to archived raw HTML of the source page on the server (Section 5.9); nullable only if archiving failed
notes text nullable

3.4. Table D — project_tasks

Tracks project progress as a checklist, feeding the status export (Section 6.9) — not a single summary line, but a per-task status so the WordPress-facing page can show actual checkmarks.

Field Type Notes
task_id PK
task_name text
status enum not_started / in_progress / complete
notes text nullable
last_updated date

3.5. Table E — sunshine_list_entries

Validation/QA data, kept deliberately separate from contacts — this is government-disclosed reference data used to check coverage, not collected-for-outreach data, and mixing the two would blur what each table is for. See Section 5.10 for the workflow this supports.

Field Type Notes
entry_id PK
institution_id FK → institutions
full_name text
position text
employer text as disclosed — sometimes differs from the institution itself (e.g. a federated college)
compensation numeric nullable — CAD, as disclosed
disclosure_year integer
source_url text
matched_contact_id FK → contacts nullable — filled in during the name-matching pass (Section 5.10); null means no match found yet or ever
match_confidence enum exact / fuzzy / unmatched — default unmatched until the matching pass runs
last_checked_date date

3.6. Table F — discovery_seeds

Added 2026-09-26, supporting the automated faculty-page discovery process (Section 5.11). Each row is one "list of faculties" page used as a starting point for a discovery run. Re-running discovery on an institution later adds a new row rather than overwriting the old one, preserving history of what the seed was on a given date.

Field Type Notes
seed_id PK
institution_id FK → institutions
seed_url text the master "list of faculties" page for this institution — distinct from an institution's plain homepage, and distinct from any one faculty's own page
acquired_date date
notes text nullable — how the seed was found, and any confidence caveats (e.g. "JavaScript-rendered, acquired via manual browser paste instead of automated fetch")

3.7. Table G — discovery_html_snapshots

Added 2026-09-26. One row per successful fetch of a seed page by the discovery script (Section 5.11). Raw HTML is stored on disk, not in Postgres directly, per the same reasoning as contacts.snapshot_html_path (Section 3.3) — keeps the database itself lean as repeated runs accumulate snapshots.

Field Type Notes
snapshot_id PK
seed_id FK → discovery_seeds
file_path text path to the archived raw HTML on disk, under /opt/apps/u15data/discovery_output/html/
fetched_date date
http_status integer

3.9. People-collection tables — overview (added 2026-11-08, v1.22)

Applied to u15data_db on 2026-11-08 with u15data_schema_people_collection_v1.0.sql (SQL scripts now live in /opt/apps/u15data/sql/). Full column lists, rules and reasoning are in u15data_people_collection_design (v1.1 or later); this section only records what exists. Privileges for u15data_app are least-privilege: the history tables (runs, events, snapshots) allow insert and select only.

3.10. Table I — collection_runs

After the import of 2026-11-08: 8 rows (2 checker, 2 discovery, 3 robots_fetch, 1 import; run ids 9 to 16, ids 2 to 8 were used up by a dry run). One row per script run of any kind (run_type: discovery / checker / scraper / robots_fetch / import), with script name and version, start and finish times and the run's time-stamped folder.

3.11. Table J — faculty_directory_pages

One row per directory address for a unit (unique on unit_id plus address). Carries the checker's verdict and counts and a status: proposed, ai_likely, ai_unsure, ai_rejected, manual_needed, ai_verified, human_confirmed, human_rejected. Scripts set and clear only the ai_ and manual_ statuses; they never change a human status. faculty_units.directory_url is set only when a page becomes human_confirmed. The public status page counts any non-null directory_url as confirmed (it does not look at the ai or human flags), so 17 on-file addresses already show as confirmed on the public page although none has been verified; 6 of the 15 checked so far redirect elsewhere and are probably stale. Review needed.

Contents after the import of 2026-11-08 (v1.23): 972 rows for 166 of the 193 faculties: ai_likely 212, ai_unsure 110, ai_rejected 22, manual_needed 4, proposed 624; 972 event rows. These count addresses, not faculties: one faculty can have several. Of the proposed rows, 590 and 33 are links and path probes found by discovery that no checker run visited, and 1 is an on-file address nobody checked. The 27 faculties with no row (193 minus 166) are those with no address at all: the manual-route faculties, Montréal Médecine, UBC Medicine, and any faculty where discovery found nothing; the list is kept in u15data_PENDING_ITEMS.

3.12. Table K — faculty_directory_page_events

One row for every status change of a directory page (old status, new status, reason, run, actor), so confirming and unconfirming leave a trail.

3.13. Table L — people and Table M — person_review_candidates

people holds one row per real person, keyed by a numeric person_id that is never reused or deleted. Merging sets merged_into_person_id and nothing else, so it can be undone. A person is matched automatically only on the same email or the same profile address. A match on name alone never merges: it adds a pair to person_review_candidates (same name at the same institution, or at another institution) for a reviewer. Cross-appointed people with the same email in two units are one person automatically.

3.14. contacts changes and view contacts_current

contacts.email now allows no value. New columns: profile_url, profile_robots_status, run_id, page_id, listing_snapshot_path, person_id, person_match_basis, person_match_review, with a unique index on (run_id, unit_id, person_id). Each run adds rows (history). The view contacts_current shows the latest row per unit and person; everyday queries use the view. contacts is still empty.

3.15. Table N — robots_txt_snapshots

The actual robots.txt text for each site checked, with fetch time and HTTP status. Loaded on 2026-11-08 (v1.23): 158 snapshots from three folders (the robots audit 2026-11-08_153739: 148 sites; the discovery run 2026-11-08_155845: 6; the checker run 2026-11-08_164614: 4), each attached to a collection_runs row of type robots_fetch. Seven snapshots have no text because the site could not be read at that moment (the table has no error column; the saved files in the run folders carry the error text). The first discovery and checker runs of 2026-11-08 saved no robots text; the audit covers those sites.

4.1. Collection permission is audited at the faculty_units level, not assumed institution-wide — individual faculties/units may carry stricter (or looser) robots.txt or Terms of Use than their parent institution.

4.2. Units flagged automated_collection_allowed = no are excluded from automated collection; they may be handled manually or dropped, never overridden.

4.3. Data minimization: only fields already published on the official page are collected (name, title, department, email). No personal/non-institutional contact data.

4.4. CASL implied-consent reasoning applies to outreach messaging (Phase 6), not to the collection method itself — collection legality is governed separately by each unit's own terms, per 4.1.

4.5. opt_out_flag and a suppression mechanism are required fields from the outset, not added later.

4.6. When a directory page itself states a usage restriction (e.g., a prohibition on bulk mailing or solicitation), the exact wording is captured verbatim in institutions.directory_usage_terms (Section 3.1) — not paraphrased, and not silently noted only in passing. First instance found: Dalhousie's directory page (2026-09-22), which explicitly states directory data "may not be used to provide addresses for bulk mailings" and that "solicitation of business, information, contributions, or other response from individuals listed by mail or other means is forbidden." This kind of restriction is checked separately from automated_collection_allowed (which reflects technical/robots.txt permission) — a directory can be technically scrapable while still carrying an explicit usage prohibition that governs what the data may be used for once collected.

4.7. Added 2026-11-08 (v1.22). How robots.txt is read. All scripts must use the shared module u15data_robots_v1_1.py, which follows RFC 9309: wildcards and the end-of-address marker, longest matching rule wins, the group for our own user agent (or *), and Crawl-delay honored per site (never less than the 2-second floor). A robots.txt that cannot be read (403, 5xx, timeout) is treated as disallowed, not as permission; 404 and 410 mean no restrictions. Every redirect step is its own request, checked against the robots.txt of the site it goes to. The text of each robots.txt used is saved. Python's built-in urllib.robotparser must not be used (on the server's Python 3.12.3 it ignores wildcards, longest-match and Crawl-delay).

5. Phase plan

5.1. Phase 1 — Faculty/unit-level source mapping and permission audit, all 15 institutions. Research only (search/fetch), no code. Output: MD table matching the faculty_units schema (Section 3.2). Institution-level robots.txt findings for all 15 institutions are loaded into the live database (institutions table) as of 2026-09-22 — the markdown audit log (u15data_phase1_audit) remains the narrative record of methodology and reasoning, but the database is now the authoritative source for the data itself. Methodology: robots.txt checks are run from a local machine, not institutional infrastructure, to avoid IP-reputation risk (Section 7.4); fetched via a standard browser user-agent, since default tooling user-agents can be blocked by CDN/WAF configurations for reasons unrelated to bot policy; checked in batches to keep review manageable. robots_txt_status values are defined precisely, since "no restriction found" can mean different things worth keeping distinct for future re-checks: no_file = the URL returns HTTP 404, no robots.txt exists at all; empty_file = a robots.txt exists (HTTP 200) but specifies no rules; has_rules_permissive = rules exist but none block anything relevant to directory/contact pages; has_rules_restrictive = rules exist that do block directory/contact-relevant paths, whether sitewide or scoped to a specific unit. no_file and empty_file currently have the same practical effect (no stated restriction) but are logged separately because they are different facts about the site and a future re-check should be able to tell "nothing changed" from "a robots.txt was added since we last checked."

5.2. Phase 2 — Review of Phase 1 findings; finalize which units are in-scope for automated collection. Originally a two-tier strategy where faculty-level mapping (faculty_units) was only populated as a fallback for institutions without a usable central directory. Superseded 2026-09-26: faculty_units is now populated for all 15 institutions regardless of central-directory status — see Section 5.11.

5.3. Phase 3 — Schema finalization (this document) and infrastructure provisioning (database, role, folder) on the server. Requires explicit go-ahead before any code is written. Complete as of 2026-09-22: u15data_db database and u15data_app role created (ownership/grant pattern matches the xab_db/xab_app reference — owned by postgres, u15data_app granted access); all three tables (institutions, faculty_units, contacts) created per the Section 3 schema; /opt/apps/u15data/docs/ and /opt/apps/u15data/snapshots/ folders created, owned by apps.

5.4. Phase 4 — Build site-specific collectors for in-scope units only; respects robots.txt/rate limits; no login-walled or CAPTCHA-protected pages.

5.5. Phase 5 — Load into u15data_db; log source_url and collected_date per record.

5.6. Phase 6 — Outreach compliance: sender identification, opt-out handling, suppression list enforcement.

5.7. Phase 7 — Recurring refresh cycle; directories change over time, so this is not a one-time pull.

5.8. Manual collection and upload pathway (complement to Phase 4). For units flagged manual_only in faculty_units, or where automated collection isn't worth building for a small unit, a person manually visits the page and enters what's there through a simple internal upload interface (form entry or CSV upload) rather than a scraper. Records entered this way land in the same contacts table as automated records, distinguished only by the collection_method field (Section 3.3), so downstream use of the data doesn't need to know or care how a given record arrived. This pathway is designed to be fast and low-friction, since its purpose is to make manual entry a genuine, easy option rather than a fallback nobody uses in practice.

5.9. Source page archiving (applies to both automated and manual collection). Every time a contact record is collected, the original source page's raw HTML is saved on the server — not just referenced by URL — so it can be re-analyzed later even if the live page changes or disappears, and so accuracy of the automated collector can be manually verified against what the page actually contained at collection time. No retention limit: archives are kept indefinitely, on the assumption that 15 institutions' worth of HTML won't grow large enough to justify a deletion policy. Exact storage path convention is an open item (Section 7.6).

5.10. Phase 8 — Sunshine list validation (QA/coverage check, run after meaningful collection exists — not before, since there's nothing to validate yet). Uses provincial public-sector compensation disclosure ("sunshine list") data as an independent, government-published cross-check on collection completeness. Important limitation, stated plainly rather than glossed over: sunshine lists only disclose employees above a compensation threshold, so this validates coverage of senior/high-earning faculty and administrators specifically — it says nothing about coverage of junior faculty or general staff, which is most of contacts. It also never contains email addresses, so it can only validate, never collect.

Initial research (2026-09-22) found real unevenness across the 7 provinces our 15 institutions span, which shapes the workflow below: a. Ontario (6 institutions: Toronto, Waterloo, Ottawa, Queen's, McMaster, Western) — centralized, single provincial website, $100,000 threshold, well-established since 1996. The clean case. b. British Columbia (UBC) — centrally disclosed, but the threshold appears to have changed over time (older sources: $75,000; a more recent structured source: $125,000, "through multiple parts") — needs direct confirmation, not assumption. c. Alberta (Alberta, Calgary) — not centralized; disclosed via each university's own annual report, on that university's own site. d. Manitoba (Manitoba) — legally required above $50,000, but not required to be published online; historically library-access-only. May not be usable for this workflow at all — needs confirming rather than assuming it's accessible. e. Saskatchewan (Saskatchewan) — conflicting signals found: older reporting says disclosure law doesn't apply to universities; a 2026 University of Regina FAQ describes a $125,000 threshold now in place "in alignment with other Saskatchewan organizations." Needs a direct, current, per-institution check. f. Nova Scotia (Dalhousie) — required since 2016 at $100,000, but disclosed per-institution rather than centrally; Dalhousie's own specific page not yet located. g. Quebec (Laval, McGill, Montréal) — no dedicated sunshine-list legislation; senior administrator compensation is technically public but embedded in annual reports each university submits to the National Assembly's Culture and Education committee — unstructured, and limited to senior administrators by nature.

Workflow: a. Source confirmation — for each of the 15 institutions, confirm the actual current disclosure source (URL, format, threshold, centralized vs. per-institution), populating the new sunshine_list_url / sunshine_list_mechanism / sunshine_list_threshold / sunshine_list_status fields on institutions (Section 3.1). Comparable in effort to the robots.txt audit already done for Phase 1. b. Ingest confirmed sources into sunshine_list_entries (Section 3.5) — kept separate from contacts deliberately, since this is validation/reference data, not collected-for-outreach data. c. Name-matching pass: for each sunshine_list_entries row belonging to one of our 15 institutions, check for a matching name in contacts, populating matched_contact_id and match_confidence. Needs fuzzy matching (middle names, name variants, maiden names) — exact string matching alone will under-report real matches. d. Produce a coverage report: senior/high-earning people who appear on a sunshine list but have no match in contacts — a signal of possible gaps, explicitly scoped to high earners, not a general completeness claim.

5.11. Automated faculty-page discovery. Built 2026-09-26 to replace search-engine-based discovery (Section 7.12) for finding each institution's individual faculties/schools and their real websites. Two distinct scripts: a. discover_pages.py (v1.0) — given a discovery_seeds row (a master "list of faculties" page), fetches it directly with requests, saves the raw HTML to disk (discovery_html_snapshots), and extracts every link's text and destination to a JSON file for review. Does not write to faculty_units directly — a script finding a link on a real page is stronger evidence than a search-engine snippet, but still gets reviewed (AI pass, then human pass) before being treated as confirmed, consistent with the project's standing no-guessing rule (Section 1, standing instruction). b. Identifies itself honestly: User-Agent string names the project and links to its own public status page (u15data-research/1.0 (+https://u15data.schillo.world/; ...)), rather than posing as a browser. Checks robots.txt for the target domain before every fetch and skips if disallowed (confirmed working against Alberta, which is JavaScript-rendered and also robots.txt-blocked at the seed page). Two-second delay between requests. c. Known limitation, not yet worked around: like most simple fetchers, it does not execute JavaScript. Sites where the real faculty list only renders client-side (Alberta's ualberta.ca/faculties) return incomplete/empty content. Current workaround for these cases is manual: the person visits the page in their own browser and pastes the real content back, which is treated the same as any other confirmed source. A headless-browser fallback (e.g. Playwright) would remove this limitation but hasn't been built, since the manual workaround has been sufficient so far and adds a real dependency. d. Milestone, 2026-09-26: every one of the 15 institutions now has real faculty/school-level data — 275 units total, every one with a confirmed home_page_url, none guessed. directory_url (an actual staff-listing page, not just a home page) is confirmed for a minority of these 275 so far; extending that is separate, ongoing work. human_confirmed is false on all 275 rows — the review pass has not started (Section 7.13). e. Raw methodology detail (which seed page was used for which institution, and per-institution confidence notes) lives in the separate u15data_faculty_discovery_log document, matching how Phase 1's raw robots.txt findings live in u15data_phase1_audit rather than in this document.

5.12. Faculty-level robots audit and the 2026-11-08 reader correction. The first discovery and checker runs of 2026-11-08 used a robots reader with three flaws: it ignored wildcards, longest-match and Crawl-delay; the discovery script treated an unreadable robots.txt as permission; and the checker followed redirects before checking the final site's robots.txt. A retrospective audit (audit_robots_v1.0.py, run folder discovery_output/robots_audit/2026-11-08_153739/, one request per site, robots.txt text saved) re-tested everything those runs requested against the correct rules. Findings, stated as found: a. Of the 36 faculties the discovery run recorded as blocked by robots.txt, 14 (9 at Ottawa, 5 at Waterloo) are in fact allowed under the correct reader and are being re-run (Section 5.14). 22 remain blocked or unknowable: 19 at sites whose robots.txt answers 403 or cannot be reached (17 University of Alberta faculties, McMaster Health Sciences and Toronto Arts & Science, treated as disallowed under 4.7) and 3 with a real Disallow: / (McMaster Humanities, Laval Faculté des sciences de l'administration, Rotman). All 22 are kept in the database with their addresses and go to the manual route (Section 5.8); nothing is deleted. b. A first prediction that McGill's robots.txt had been violated was wrong: the wildcard rule that looked relevant belongs to the archive.org_bot group, not to us. McGill's group for * sets Crawl-delay: 5. The audit also records that our scripts waited only 1 to 2 seconds between requests to the same site, so Crawl-delay was not honored for McGill (5 seconds) or Queen's (10 seconds). This is a conduct gap, now fixed in the scripts (Section 5.13); McGill's automated_collection_allowed = manual_only setting from v1.19 should be revisited (open item 7.18). c. A small number of requests went to sites whose robots.txt could not be read (www.med.ubc.ca, www.landfood.ubc.ca, www.med.umontreal.ca and one Laval repertoire host), and one checker request followed a redirect to a final address disallowed by robots.txt (dalu.sharepoint.com/sites/engineering, apparently Dalhousie's). The audit summary file in the run folder lists each one. d. University of Alberta, McMaster Health Sciences and Toronto Arts & Science return 403 for robots.txt to our script. Whether this depends on our user agent or on the server's address is not known; a comparison of curl from a personal computer and from the server was proposed and results are not yet recorded (open item 7.19).

5.13. Shared modules and the run model (built 2026-11-08; committed ee511a4, cd86a8c, d5986ee). Current versions: u15data_robots_v1_2.py, u15data_scheduler_v1_1.py, discover_faculty_directories_v1.4.py, summarize_faculty_candidates_v1.1.py, check_faculty_directories_v1.5.py, import_directory_results_v1.1.py (earlier versions are kept beside them; the 2026-11-08 afternoon runs used robots v1.1, scheduler v1.0, discovery v1.3 and checker v1.4). Robots module v1.2 retries a failed robots.txt read three times (waits of 3 and 10 seconds) when the failure looks passing, and re-reads a site that stayed unreadable after 10 minutes if asked about it again; it was added after one dropped connection marked UBC Land and Food Systems as disallowed for a whole checker run. u15data_robots_v1_1.py is the robots reader of Section 4.7 (with parallel pre-fetch and saving of the robots.txt text). u15data_scheduler_v1_0.py rotates requests across websites: each site has its own queue and not-before time (at least 2 seconds, or its Crawl-delay if longer); when a site is waiting, the script works on another site and sleeps only when nothing at all is ready; failed requests go to the back of the queue and are retried no sooner than 60 seconds later, up to 3 attempts; a Retry-After from a site is honored. Scripts built on them: discover_faculty_directories_v1.3.py, summarize_faculty_candidates_v1.1.py, check_faculty_directories_v1.4.py (the checker's every redirect step is robots-checked and a disallowed step is reported as ROBOTS_DISALLOWED without being fetched). Every run writes into its own time-stamped folder (UTC), never overwriting: discovery_output/runs/<stamp>_discovery_v1.3/ (html/, links/, robots/, discovery.log, run_info.json, summaries/), discovery_output/checks/<stamp>/, discovery_output/robots_audit/<stamp>/. Raw run output is kept out of Git; scripts and documents are committed (local Git only, run as sudo -u apps git -C /opt/apps/u15data ..., commit messages end with the Claude-Session line only). Earlier script versions are kept alongside the new ones. The collector runs on this server; the earlier checks from a personal computer were only for working things out (this closes open item 7.4).

5.14. Faculty directory pages, status as of the evening of 2026-11-08. Discovery v1.2 covered the faculty-level units (home page links plus 16 guessed directory paths); checker v1.3 checked 318 addresses. The 18-faculty re-run (14 faculties that the corrected robots reader allows, 3 that had failed to load, and UBC Medicine) was done with discovery v1.3, summary v1.1 and checker v1.4: 16 faculties loaded (Ottawa 9, Waterloo 5, Laval FAAAD, UBC Land and Food Systems); Montréal Médecine and UBC Medicine were again recorded as blocked because their robots.txt could not be read. The checker visited 30 addresses; seven faculties got a LIKELY_DIRECTORY proposal needing no person (Ottawa Common Law, Ottawa Social Sciences and all five Waterloo faculties); the others need a person, and several proposals are plainly wrong or too broad (details in u15data_PENDING_ITEMS). All of it, plus every other address discovery found, is now in faculty_directory_pages (Section 3.11) through import_directory_results_v1.1.py (dry run by default, --apply to commit; report in discovery_output/imports/2026-11-08_183404/). Status rule: LIKELY_DIRECTORY becomes ai_likely (ai_unsure when flagged with anything other than paginated or redirected); PARTIAL_OR_UNSURE and the scope and JavaScript verdicts become ai_unsure; no-directory verdicts become ai_rejected; ROBOTS_DISALLOWED becomes manual_needed; addresses found by discovery but not visited, and on-file addresses nobody checked, are proposed. Pages already ai_verified, human_confirmed or human_rejected are never overwritten by an import. Retry later: since about 16:46 UTC both UBC hosts (www.med.ubc.ca, www.landfood.ubc.ca) reset every connection from this server (curl exit 56), although Land and Food Systems answered at 15:58 and 16:12; its one manual_needed row is from a dropped connection, not a restriction, and both faculties are on the retry list (discovery_output/retry_ubc_unit_ids_2026-11-08.txt, ids 181 and 179). www.med.umontreal.ca no longer resolves (curl exit 6), so Montréal Médecine needs a correct address from a person; it is not to be guessed.

5.15. People scraper (Phase 4, not yet written). Design in u15data_people_collection_design: reads only addresses with status ai_likely, ai_verified or human_confirmed; records everyone found with their position text, email and profile address (tenure-track filtering happens later, not in the scraper); records the directory address and the run for every row; checks robots.txt for every listing and profile page; confirms an address (ai_verified) when it extracts people and unconfirms it (ai_unsure or manual_needed) when it fails, writing an event each time. It will not be written until the user approves the design.

6. WordPress / i2hub.ca Integration

6.1. Final design, after working through the alternatives below: a plain static JSON file, regenerated by an offline export script (not a live database query), served directly by Nginx from u15data.schillo.world — no application, no Flask/Gunicorn, no /api/ route. This is functionally the Static Partner Import Pattern already proven elsewhere on i2hub.ca (per ERICA_Methods_Pattern_Inventory_v2, Section 2), pointed at our own server's file instead of a Google Drive file.

6.2. On the WordPress side: a PHP Code Snippet performs a server-side HTTP fetch of that JSON file, renders it as HTML, and returns it via a shortcode dropped into an Elementor Shortcode widget on a PMPro-gated page — the same shape as the site's existing Data- snippets that already fetch from Google Drive/Apps Script.

6.3. Why a file, not a direct database connection: PHP (running on WordPress's server) can only reach things exposed somewhere on the network. Exposing Postgres itself to the network would mean opening it to the whole internet, not just to WordPress specifically, which breaks the server's own firewall convention (only SSH/80/443 open) and is a materially larger attack surface than exposing one narrow, read-only file. The static file is a deliberate boundary, not an unnecessary intermediate step.

6.4. Why no iframe, no live viewer page: CORS (the reason originally given for an iframe) only applies to JavaScript running in a visitor's browser fetching cross-domain — it does not apply to PHP's server-side fetch. Since this status page only needs to show current numbers on load, not refresh itself live in the visitor's browser, there's no CORS problem to design around in the first place. An iframe-embedded viewer page would only be justified if the page itself needed to be interactive/live-updating, which it doesn't.

6.5. Rejected alternatives, in order of consideration: a. Live REST API (Flask/Gunicorn querying Postgres per request) — unnecessary complexity for data that only needs to be as current as the last manual export; also the "emerging main method" pattern site-wide, but with no working precedent anywhere on i2hub.ca yet. b. Iframe-embedded viewer page on u15data.schillo.world — more setup than needed; only justified for genuinely interactive/live content. c. Reusing the Google Drive JSON pattern instead of our own server — considered as the simplest option overall (zero new server infrastructure), but rejected in favor of keeping this on our own infrastructure once the actual complexity of the static-file approach became clear (a plain Nginx-served file, not a live app) — the earlier stated benefit (not depending on Google Drive staying configured correctly) applies without much added cost once no application layer is needed. d. Piggybacking the file onto the existing xab.schillo.world subdomain instead of provisioning a new one — would avoid new Nginx/SSL setup entirely, but breaks the "one subdomain per app" server convention and mixes two unrelated projects' content under one address. Rejected in favor of a clean, dedicated subdomain.

6.6. Subdomain: u15data.schillo.world, per the xab.conf Nginx configuration reference on this server. Live as of 2026-09-22: DNS A record added at Network Solutions (u15data → 134.87.8.156, same server as xab), Nginx server block created serving /opt/apps/u15data/static/, SSL certificate issued and deployed via Certbot (expires 2026-12-21, auto-renewal scheduled). Verified reachable over both HTTP and HTTPS, currently serving a placeholder status.json. Remaining work is content, not infrastructure: the export script (Postgres → real status.json) and the WordPress-side PHP snippet, both of which are code and await explicit go-ahead per Section 1's standing instruction.

6.7. Scope for now: status/progress dashboard only (phase tracking, audit findings/counts) — no contact-level data. Consistent with the earlier decision to keep contacts data out of any WordPress-facing surface until a separate access-control design is done for it specifically.

6.8. Known, accepted gap: access control is page-level only. The WordPress page is PMPro-gated, but the raw file URL is not independently protected — anyone with the direct link could view it without logging in. Accepted for now given the low sensitivity of status/summary data. Must be revisited before this pattern is reused for anything higher-sensitivity, such as actual contact records.

6.9. Export script: /opt/apps/u15data/export_status.py, run manually (/opt/apps/u15data/venv/bin/python3 /opt/apps/u15data/export_status.py) whenever there's new progress to publish. Reads project_tasks and institutions from u15data_db, writes /opt/apps/u15data/static/status.json, which Nginx serves directly. Live and verified as of 2026-09-22 — confirmed returning real task and institution-summary data over HTTPS. Workflow going forward: update project_tasks via SQL, then re-run the script; the script itself does not need to change for routine updates.

6.10. PHP snippet naming: no ERICA prefix — this project is not part of ERICA, so the site's ERICA Data- snippet naming convention doesn't apply here. Named simply "u15data Status" on the WordPress side.

6.11. Complete and verified live as of 2026-09-22: PHP Code Snippet added (server-side fetch of status.json, 5-minute transient cache, renders task checklist + institution summary via [u15data_status] shortcode), WordPress page created, Shortcode widget placed via Elementor, page restricted to the PMPro membership level, and access confirmed both ways — visible to a member, blocked when logged out. This closes out the WordPress integration work described in this section.

6.12. Documentation hosting pattern: full versioned history of both markdown documents (this one and the Phase 1 audit log) lives in /opt/apps/u15data/docs/, per the server's own convention that each app keeps its docs in its own docs/ subfolder — every version, kept permanently, matching the project's standing rule that every document carries a version number in both its filename and its header. Separately, one stable-named "latest" copy of each (architecture_latest.md, phase1_audit_latest.md) lives in /opt/apps/u15data/static/docs/, served directly by Nginx alongside status.json, and linked from the WordPress status page. The stable names mean the PHP snippet's links never need editing when a document's version number changes — only two file operations are needed per update (move the new versioned file into docs/, overwrite the matching _latest.md in static/docs/), not a snippet edit.

6.13. Status page display fix: the rendered lists (task checklist, robots.txt status breakdown, central directory breakdown, documentation links) previously showed default bullet points on some lists but not others, since bullet-removal had only been applied to the task list individually. Fixed by adding a single scoped <style> block covering every list inside the status box, rather than styling each list separately.

7. Open items

7.1. Exact refresh cadence for Phase 7 — not yet defined.

7.2. Resolved 2026-09-27: faculty_units gained a unit_level field (Section 3.2) distinguishing true top-level faculties from nested sub-units and excluded administrative/historic entities, rather than relying on unit_name free text alone.

7.3. Conversion of this document to .docx can be done once content is stable; not needed while still in active revision.

7.4. Resolved 2026-11-08: the collector runs on this server (Section 5.13). Original question: where the automated collector (Phase 4) physically runs from — raised during Phase 1 robots.txt testing, where running checks from uOttawa infrastructure was identified as a reputational/IP-blocking risk to institutional systems generally, not just this project. Needs a deliberate decision (local machine, separate non-institutional host, etc.) rather than defaulting to "runs on the same server as everything else."

7.5. Resolved 2026-09-22: /profiles/ confirmed, via Drupal's own source repository, as Drupal core's default "installation profiles" directory — boilerplate present in Drupal's default robots.txt template across many versions, unrelated to staff/user profile pages and not directory-relevant. Applied to all 5 affected institutions (Calgary, Laval, Manitoba, Toronto, Ottawa), each re-coded from NULL to has_rules_permissive in the database. institutions.by_robots_txt_status no longer has a pending_confirmation bucket — Phase 1's institution-level robots.txt audit is now fully resolved for all 15.

7.6. Storage path convention for the HTML archives (Section 5.9) — not yet defined. Likely lives under /opt/apps/u15data/ in its own subfolder (e.g., a snapshots/ directory organized by unit and date), but this should be confirmed rather than assumed. Retention is settled — kept indefinitely, no deletion policy — so this item is scoped to path/naming only.

7.7. Resolved — see Section 6.6.

7.8. Resolved — no wildcard exists; DNS is managed at Network Solutions with individual A records per subdomain, matching xab's existing setup. u15data record added the same way. See Section 6.6.

7.9. Resolved — see Section 6.11. PHP snippet and WordPress page are live and verified.

7.10. Sunshine list source confirmation for all 15 institutions — not yet started (Section 5.10). Real unevenness found across provinces during initial research; several sources (BC's current threshold, Manitoba's online accessibility, Saskatchewan's applicability to universities, Nova Scotia's Dalhousie-specific page) need direct confirmation rather than assumption before sunshine_list_status can be marked confirmed for any institution.

7.11. Subdomain-specific robots.txt gap, found 2026-09-22 while attempting to set automated_collection_allowed: for institutions where the actual central directory lives on a different subdomain than the one checked in Phase 1 (Alberta: apps.ualberta.ca vs. checked ualberta.ca; Calgary: contacts.ucalgary.ca; McMaster: directories.mcmaster.ca; Dalhousie: directory.dal.ca), the Phase 1 robots.txt result does not actually tell us whether the directory subdomain itself permits automated access — robots.txt is per-domain, not institution-wide. automated_collection_allowed is deliberately left NULL for these 4 institutions until their directory subdomains are checked directly, rather than inferring permission from the main domain's result.

7.12. Resolved 2026-09-26, superseded by Section 5.11: search-engine-based discovery of individual faculty pages proved unreliable and, in a small number of cases, produced guessed URLs that turned out to be wrong (an Ottawa faculty home page and a Queen's graduate-studies page were both corrected after the fact — see the faculty discovery log's changelog for specifics). Replaced with the discover_pages.py script, which fetches real pages directly rather than relying on search snippets.

7.13. Human-review pass on faculty_units — not started. All rows currently carry human_confirmed = false. ai_confirmed = true reflects that a link was found on an actually-fetched page and judged relevant, which is a real evidentiary bar, but is not a substitute for a person checking it.

7.14. directory_url (an actual confirmed staff-listing page, distinct from home_page_url) is populated for only a minority of the 193 faculty-level rows (corrected from 191 in v1.22). Extending this is separate, ongoing work — most units currently have a confirmed home page but not yet a confirmed staff directory within it.

7.15. Two institutions (McGill, University of Toronto) and several individual faculties elsewhere still lack a directory_url beyond their home page even where one likely exists (e.g. Ottawa's Faculty of Medicine has one confirmed; most other Ottawa faculties don't yet). Toronto in particular has all 19 divisions logged with a home page only, since each division runs its own separate site with no common pattern found yet.

7.16. McMaster's automated_collection_allowed — revisited 2026-09-26. McMaster's robots.txt explicitly blocks a list of named AI-company crawlers (including ClaudeBot and anthropic-ai by name), though its general wildcard rule only blocks /wp-admin/. Originally treated as equivalent to a no. Revised after discussion: the block targets AI companies' own commercial/training crawlers specifically, not a custom, honestly-identified research tool built for a bounded academic project — a meaningfully different activity even though a coding assistant helped build the tool. discover_pages.py's honest self-identification (Section 5.11b) is what makes this distinction real rather than a rationalization: the script does not pose as a browser or as one of the named blocked bots.

7.17. Resolved 2026-11-08: the 191 versus 193 faculty count. 193 is correct (Section 3.2); the unit-level split is 193 / 47 / 19. The classification sources document (u15data_unit_classification_sources v1.1) and the faculty discovery log should be read with that correction.

7.18. McGill automated_collection_allowed = manual_only (v1.19) was set when McGill's robots.txt was thought to block directory pages. The 2026-11-08 audit found the blocking rule belongs to archive.org_bot, and McGill's group for all other agents only asks for Crawl-delay: 5. The user decides whether to revise the setting.

7.19. Whether the 403 on robots.txt from University of Alberta, McMaster Health Sciences and Toronto Arts & Science depends on our user agent or on the server's address. Comparison of curl from a personal computer and from the server has been proposed; results not yet recorded. Until known, the 19 affected faculties are on the manual route.

7.20. Meaning of institutions.automated_collection_allowed at faculty level. Recommendation to be confirmed at the scraper stage: the unit-level field gates collection and the institution-level field describes only the central directory.

7.21. Sunshine list (Section 5.10, item 7.10): source confirmation started 2026-11-08 in this project; findings so far are in u15data_HANDOVER_sunshine_list v1.0 and are not yet entered here or in the database. Several notes in Section 5.10 are out of date: Manitoba publishes an online PDF; the University of Saskatchewan posts a list; Dalhousie's own disclosure documents exist; the Alberta threshold appears to be $130,000 for 2026 (the earlier $125,000 is the University of Regina's). All to be re-verified on the actual pages before being recorded.

7.22. Directory addresses needing correction: Montréal Médecine home address (dead), the Dalhousie faculty whose on-file address redirects to dalu.sharepoint.com/sites/engineering, which robots.txt disallows (the faculty to be identified from the checker output), and the 17 on-file directory_url values that the public page already counts as confirmed (item 3.11).

7.23. Stale text in institutions.classification_notes for Ottawa (says 9 faculties, there are 10) and Montréal (says 12 faculties, there are 13), noted 2026-09 and not yet refreshed. Fixing them is a change to the export, to be decided.

7.24. UBC retry: after the connection resets of 2026-11-08 (5.14), retry UBC Medicine and UBC Land and Food Systems no earlier than the next day, starting with one request for each host's robots.txt, and do not work around a block. Tracked in u15data_PENDING_ITEMS.

7.25. When the people scraper is designed it must handle several ai_likely addresses for one faculty (212 ai_likely rows exist for far fewer faculties), so that the same person is not read twice from overlapping pages.

8. Version history

Version Date Change
1.0 2026-09-22 Initial architecture document: schema (3 tables), server naming, phase plan, legal/ethical constraints
1.1 2026-09-22 Added manual collection/upload pathway (Section 5.8) and collection_method field to contacts table; added open items on collector hosting location and the /profiles/ path ambiguity found during Phase 1 testing
1.2 2026-09-22 Added source page archiving (Section 5.9) — both HTML and PDF/screenshot, applies to automated and manual collection alike; added snapshot_html_path and snapshot_pdf_path fields to contacts table; added open item on archive storage path/retention
1.3 2026-09-22 Archiving scoped to HTML only, PDF/screenshot dropped per decision; removed snapshot_pdf_path field; retention settled as indefinite (no deletion policy) — open item narrowed to storage path convention only
1.4 2026-09-22 Added Phase 1 methodology note (Section 5.1) and pointed to the separate u15data_phase1_audit log document, which now tracks actual robots.txt findings; this document stays scoped to decisions/structure rather than raw results
1.5 2026-09-22 Redefined robots_txt_status enum to distinguish no_file (404, no robots.txt exists) from empty_file (robots.txt exists but has no rules) rather than collapsing both into "allowed"; added precise definitions for all four status values to Section 5.1, prompted by the Waterloo 404 finding
1.6 2026-09-22 Marked Phase 3 complete (Section 5.3): database, role, tables, and folders provisioned and verified against \du, \l, \dt, and ls -la output
1.7 2026-09-22 Moved domain-level audit fields (robots_txt_url, robots_txt_status, tos_url, tos_summary, automated_collection_allowed, last_checked_date) onto institutions; added has_central_directory; redefined faculty_units as a targeted fallback table rather than a blanket requirement for all 15 institutions; added two-tier strategy to Section 5.2
1.8 2026-09-22 Institution-level robots.txt findings for all 15 institutions loaded into the live institutions table (after catching and cleaning up an accidental duplicate insert of 7 rows); database is now the authoritative source for this data, markdown audit log retained for methodology/reasoning
1.9 2026-09-22 Added Section 6 (WordPress / i2hub.ca Integration): adopted the Server-Hosted Static Content Pattern per ERICA_Methods_Pattern_Inventory_v2, rejected the unproven live REST API pattern for now, accepted the known page-level-only access gap for this low-sensitivity data, and flagged the still-unconfirmed subdomain as an open item; corrected section numbering (Open items moved to Section 7, Version history to Section 8)
1.10 2026-09-22 Confirmed subdomain u15data.schillo.world from xab.conf's live Nginx configuration; resolved the open item from v1.9
1.11 2026-09-22 Finalized WordPress integration design after working through simplicity trade-offs: static JSON file served by Nginx (no application layer), PHP snippet does a server-side fetch (no iframe needed — corrected earlier CORS reasoning, which only applies to browser-side fetches), no direct database exposure. Rewrote Section 7 to document the reasoning and rejected alternatives explicitly. Added open item on DNS setup method (Section 7.8).
1.12 2026-09-22 WordPress integration infrastructure completed and verified live: DNS A record (Network Solutions), Nginx server block, SSL certificate (Certbot, expires 2026-12-21) — u15data.schillo.world/status.json confirmed reachable over HTTPS serving a placeholder file. Resolved open item 7.8 (no wildcard DNS in use). Remaining work scoped to code (export script, PHP snippet), not infrastructure.
1.13 2026-09-22 Added project_tasks table (Section 3.4) as a checklist rather than a single summary string, per feedback that a task-by-task view is more useful than a one-line status; export script (Section 6.9) written, run, and verified live serving real data at u15data.schillo.world/status.json. Also fixed a section-numbering bug introduced in v1.9: the WordPress Integration section had been numbered 7 with nothing occupying 6, cascading a stale gap through v1.9–v1.12; renumbered to close the gap (WordPress Integration is now Section 6, Open items Section 7, Version history Section 8) and corrected all internal cross-references; caught and fixed one collateral corruption from the renumbering script (the IP address 134.87.8.156 was briefly mis-edited to 134.87.7.156 in Section 6.6, now restored)
1.14 2026-09-22 PHP Code Snippet and WordPress page (Elementor Shortcode widget + PMPro restriction) built, deployed, and verified live — access confirmed both as a member and logged out. Corrected naming: not ERICA-branded, named "u15data Status." Added corresponding project_tasks row. This closes out the WordPress integration work in full.
1.15 2026-09-22 Added documentation hosting pattern (Section 6.12): versioned docs live permanently in /opt/apps/u15data/docs/, stable-named "latest" copies in /opt/apps/u15data/static/docs/ are what the status page actually links to, so version bumps don't require snippet edits. Fixed inconsistent bullet-point styling across all rendered lists on the status page (Section 6.13), not just the task list.
1.16 2026-09-22 Added faculty-first priority note (Section 1.3) — both faculty and staff remain in scope, faculty sequenced first. Added Phase 8, sunshine list validation (Section 5.10), as a QA/coverage-check workflow using provincial public-sector compensation disclosure data; documented real per-province unevenness found in initial research (Ontario centralized and clean, BC/Saskatchewan/Nova Scotia needing direct confirmation, Manitoba possibly inaccessible online, Quebec unstructured/legislature-report-based). Added sunshine_list_* fields to institutions (Section 3.1) and new Table E sunshine_list_entries (Section 3.5), kept deliberately separate from contacts. Added open item 7.10.
1.17 2026-09-22 Resolved the /profiles/ ambiguity (open item 7.5) for all 5 affected institutions (Calgary, Laval, Manitoba, Toronto, Ottawa): confirmed via Drupal's own source repository as core CMS boilerplate, not staff/user profile content. Database updated (robots_txt_status re-coded from NULL to has_rules_permissive for all 5), status page re-exported. Phase 1's institution-level robots.txt audit is now fully closed for all 15 institutions.
1.18 2026-09-22 Completed central directory discovery for all 15 institutions (has_central_directory/central_directory_url populated in the database): 5 confirmed with a usable URL (Alberta, Calgary, Dalhousie, McMaster, and McGill, though McGill's is robots.txt-blocked), Toronto confirmed to exist but login-gated (inaccessible to us), UBC confirmed to exist but exact URL not yet found, Waterloo has an unconfirmed lead (WatIAM), Western has a partial lead (staff-coordinator list only), and 6 institutions (Laval, Manitoba, Montréal, Ottawa, Queen's, Saskatchewan) show no central directory at all — appear to be genuinely per-faculty/department only. Added directory_usage_terms and directory_terms_captured_date fields (Section 3.1) and Section 4.6, to capture verbatim any usage restriction stated on a directory page itself; first instance found and recorded: Dalhousie's explicit prohibition on bulk mailing/solicitation.
1.19 2026-09-22 Set automated_collection_allowed where determinable: no for the 8 institutions with no accessible central directory (Laval, Manitoba, Montréal, Ottawa, Queen's, Saskatchewan, Toronto, Western), manual_only for McGill (directory exists but is robots.txt-blocked). Left NULL for the other 6 rather than guess — discovered and documented a real methodological gap (Section 7.11): for Alberta, Calgary, McMaster, and Dalhousie, the confirmed directory lives on a different subdomain than the one checked in Phase 1's robots.txt pass, so that result doesn't actually establish permission for the directory itself; UBC and Waterloo still lack a confirmed URL at all. Also added a Permissions column with expandable details to the review tool (review.html), and fixed a badge-label bug (has rules permissive → permissive).
1.20 2026-09-26 Major scope shift: faculty-level data collection extended from fallback-only to all 15 institutions (Section 5.2 superseded). Added home_page_url, notes, ai_confirmed, and human_confirmed fields to faculty_units (Section 3.2), separating "unit's own website" from "confirmed staff-listing page" for the first time. Added two new tables, discovery_seeds and discovery_html_snapshots (Sections 3.6–3.7), and built discover_pages.py v1.0 (Section 5.11), a script that fetches real pages directly rather than relying on search-engine snippets, self-identifies honestly in its User-Agent, and checks robots.txt before every fetch. Milestone: all 15 institutions now have real faculty/school-level data — 275 units total, every one with a confirmed home_page_url, none guessed (caught and corrected two guessed URLs along the way — an Ottawa faculty page and a Queen's graduate-studies page — per open item 7.12). directory_url and human_confirmed remain open (7.13–7.15). Revisited and revised the McMaster automated_collection_allowed call (7.16): distinguished a custom, honestly-identified research tool from the AI-company crawlers McMaster's robots.txt explicitly names. Raw per-institution methodology now lives in the new companion document u15data_faculty_discovery_log.
1.21 2026-09-27 Resolved the ambiguity behind v1.20's flat 275-row count (Section 3.2, Section 7.2 now closed): added unit_level (faculty / sub_unit / excluded) and self-referencing parent_unit_id to faculty_units, and reclassified all 259 confirmed rows across all 15 institutions by direct verification against each institution's own current official sources — 191 true top-level faculties, 41 sub-units, 27 excluded. Along the way, found and corrected several genuine discovery gaps predating this version: Dalhousie was missing 3 faculties, Universite Laval 2, Universite de Montreal 2 (a third remains unresolved), and Calgary's entire faculty set had been pointing to University Calendar description pages rather than each faculty's real site (corrected, and one missing faculty added). Added classification_notes to institutions (Section 3.1) and a new table, institution_sources (Section 3.8), to hold the reference links used for this verification; both are now shown on the public faculty-directories page (Section 6), which was updated to show faculty-level rows only rather than all confirmed units. Full per-institution reasoning and sources: u15data_unit_classification_sources v1.1. Resolution of the 275-vs-251 row-count discrepancy specifically: u15data_faculty_discovery_log v1.1, Section 4.
1.22 2026-11-08 Corrected the faculty-level counts to 193 / 47 / 19 (Section 3.2, item 7.17). Added the people-collection schema (Sections 3.9 to 3.15: collection_runs, faculty_directory_pages and events, people, person_review_candidates, robots_txt_snapshots, contacts changes, contacts_current). Added the robots reading rules (4.7), the faculty-level robots audit and its findings (5.12, including the McGill correction and the Crawl-delay gap), the shared robots and scheduler modules, time-stamped run folders and local Git (5.13), directory-page status (5.14) and the people scraper outline (5.15). Collector location settled: this server (7.4). New open items 7.18 to 7.23.
1.23 2026-11-08 Directory-page data imported: 972 faculty_directory_pages rows (212 ai_likely, 110 ai_unsure, 22 ai_rejected, 4 manual_needed, 624 proposed) for 166 of the 193 faculties, 972 events, 158 robots.txt snapshots, 8 collection_runs rows (Sections 3.10, 3.11, 3.15). Robots module v1.2 (retry of failed robots.txt reads), scheduler v1.1, discovery v1.4, checker v1.5 and import script v1.1 added (5.13). Section 5.14 rewritten with the 18-faculty re-run results, the status rule and the UBC connection resets. New open items 7.24 (UBC retry) and 7.25 (several ai_likely addresses per faculty).