Version: 1.23 Status: Draft — schema and infrastructure confirmed; all 15 institutions' faculty-level data classified (193 faculties, 47 sub-units, 19 excluded, 259 rows); Phase 1 robots.txt audit resolved at institution level and re-audited at faculty-unit level; people-collection schema applied and now holding data: 972 directory-page rows (212 ai_likely, 110 ai_unsure, 22 ai_rejected, 4 manual_needed, 624 proposed) for 166 of the 193 faculties, and 158 saved robots.txt texts (Sections 3.11, 3.15, 5.14); discovery, checking and import scripts built on shared robots and scheduling modules (Sections 5.12 to 5.14); people scraper not yet written; human-review pass not yet started Last updated: 2026-11-08 (v1.23, evening)
1.1. Compile faculty and staff contact records for all 15 U15 Canadian universities, for academic/research outreach purposes.
1.2. Collection is limited to information already published on official institutional pages, and only from sources/units that explicitly or implicitly permit automated collection (per robots.txt and Terms of Use, checked at the faculty/unit level, not assumed at the institution level).
1.3. Priority: faculty first, staff second. Both remain in scope (per the original decision to cover "faculty and staff, all 15 institutions"), but collection and validation work should sequence faculty ahead of staff where the two can be separated — e.g., when auditing or collecting a unit with both, faculty records take priority; staff can follow once faculty coverage for that unit is established.
2.1. Project folder: /opt/apps/u15data/
2.2. Follows the server's standard one-app-per-folder, dedicated-venv
pattern (per /opt/apps/CONVENTIONS.md), not the
shared-database exception used by zoomsort/zoom-downloader.
2.3. Database: u15data_db
2.4. Application role: u15data_app
2.5. Naming follows the xab_db / xab_app
reference pair, now recorded in /opt/apps/CONVENTIONS.md as
the standard pattern: <project>_db (database) owned
by <project>_app (role).
2.6. Docs for this project live in
/opt/apps/u15data/docs/, per server convention.
Three tables, normalized so audit/compliance data (which changes rarely) is separated from contact data (which refreshes on a recurring cycle).
institutionsReference table, one row per U15 institution.
| Field | Type | Notes |
|---|---|---|
| institution_id | PK | |
| name | text | |
| province | text | |
| central_directory_url | text | nullable — existence and location of a central directory is unconfirmed for all 15 as of v1.7 |
| has_central_directory | enum | yes / no / unknown — default unknown; tracks whether a
central, institution-wide staff directory exists at all, separate from
whether it can be collected |
| robots_txt_url | text | |
| robots_txt_status | enum | no_file / empty_file / has_rules_permissive / has_rules_restrictive — same definitions as Section 5.1; nullable where ambiguous pending confirmation |
| tos_url | text | nullable |
| tos_summary | text | nullable |
| automated_collection_allowed | enum | yes / no / manual_only — nullable; not set until a directory's actual location has been checked against both robots.txt and Terms of Use for that specific path, since a domain-wide robots.txt finding alone doesn't establish this |
| last_checked_date | date | |
| notes | text | nullable |
| sunshine_list_url | text | nullable — see Section 5.10 |
| sunshine_list_mechanism | enum | centralized_provincial / per_institution_posting /
annual_report_to_legislature / not_applicable / unknown — how (or
whether) this institution's compensation disclosure is actually
published; default unknown |
| sunshine_list_threshold | numeric | nullable — the compensation threshold above which disclosure applies, in CAD; varies by province and can change year to year |
| sunshine_list_status | enum | confirmed / unconfirmed — whether the specific source has actually
been located and verified, not just the general provincial mechanism;
default unconfirmed |
| directory_usage_terms | text | nullable — exact wording of any usage restriction found stated
directly on a directory page itself (e.g., a prohibition on bulk mailing
or solicitation), captured verbatim rather than paraphrased, since a
paraphrase could blur exactly what's prohibited. Distinct from
tos_url/tos_summary, which cover the
institution's general Terms of Use — this field is specifically for
restrictions found on the directory page itself, which may be stricter
or more specific. |
| directory_terms_captured_date | date | nullable — when directory_usage_terms was captured |
| classification_notes | text | nullable — added 2026-09-27. A short plain-language paragraph per
institution summarizing what the faculty count shown publicly excludes
and why (e.g. college-level groupings, historic affiliated colleges,
administrative bodies) — written for the public-facing
faculty-directories page (Section 6), not internal use. See Section 3.8
for the companion institution_sources table. |
faculty_unitsFaculty/school-level dataset. Originally scoped as a fallback table
for institutions without a usable central directory (see Section 5.2's
original two-tier strategy); superseded 2026-09-26 — populated for all
15 institutions regardless of central-directory status, since
faculty-level data has independent value beyond being a fallback.
Superseded again 2026-09-27: the original flat
structure conflated true top-level faculties with nested schools,
colleges, and departments, which had both been captured as peer rows
during discovery (see Section 7.2, now resolved, and the companion
u15data_faculty_discovery_log v1.1 Section 4 for the full
resolution of the resulting row-count discrepancy).
unit_level and parent_unit_id, added below,
resolve this.
| Field | Type | Notes |
|---|---|---|
| unit_id | PK | |
| institution_id | FK → institutions | |
| unit_name | text | e.g., "Faculty of Engineering," "Registrar's Office" |
| home_page_url | text | nullable — the unit's own website. Distinct from
directory_url: confirming a unit's home page is a much
lower bar than confirming it has an actual staff-listing page. Added
2026-09-26 to stop conflating the two. |
| directory_url | text | nullable — only populated when a genuine staff/professor-listing
page has been confirmed (not just a home page). As of 2026-09-26,
confirmed for a minority of the 275 units on file; most have a confirmed
home_page_url but no confirmed directory_url
yet. |
| robots_txt_url | text | |
| robots_txt_status | enum | no_file / empty_file / has_rules_permissive / has_rules_restrictive — see Section 5.1 for definitions; deliberately distinguishes "no robots.txt exists" from "a robots.txt exists and is blank," since these are different facts about the site even though both currently mean no automated-access restriction is stated |
| tos_url | text | nullable |
| tos_summary | text | nullable |
| automated_collection_allowed | enum | yes / no / manual_only |
| last_checked_date | date | |
| notes | text | nullable — added 2026-09-26 alongside the discovery process; free-text provenance/context (e.g. "confirmed via automated link discovery," "under Faculty of Health Sciences") |
| ai_confirmed | boolean | default false — added 2026-09-26. True when a link was found on a
real, fetched page (not guessed/pattern-inferred) and judged relevant by
AI review of that page's actual links. A weaker bar than
human_confirmed, not a replacement for it. |
| human_confirmed | boolean | default false — added 2026-09-26. Whether a person has actually
verified the row. As of v1.21, false on all rows — the
human-review pass has not started. |
| unit_level | text | nullable, CHECK constrained to faculty /
sub_unit / excluded — added 2026-09-27.
faculty is a true top-level unit (the project's actual unit
of analysis); sub_unit is nested within a specific faculty
(a school, college, or department captured during discovery but not
itself a faculty), with parent_unit_id set;
excluded is a real, on-file unit that is neither —
administrative/coordinating bodies (e.g. most institutions' Graduate
Studies), historic affiliated colleges, or college-level groupings that
organize faculties without being one. As of v1.21, all 259 rows across
all 15 institutions are classified. Corrected 2026-11-08
(v1.22): the live database holds 193 faculty, 47
sub_unit and 19 excluded (total 259); v1.21
and the earlier discovery log stated 191, 41 and 27. The 191 was an
addition slip (the per-institution counts
17+13+6+7+17+13+17+18+14+15+10+17+17+6+12 add to 193); the 41 and 27
could not be traced to any source, and the database is authoritative.
Full per-institution reasoning and every source consulted is in the
companion document u15data_unit_classification_sources
v1.1. |
| parent_unit_id | integer | nullable, self-referencing FK to faculty_units.unit_id,
ON DELETE RESTRICT (a faculty row cannot be deleted while a
sub_unit still points at it) — added 2026-09-27. NULL for
faculty and excluded rows; set to the parent
faculty's unit_id for every sub_unit row. In a
small number of cases (e.g. UBC's School of Dental Hygiene-equivalent
nesting) a sub_unit's parent is itself another sub_unit, not a top-level
faculty — the schema permits this chain but the public-facing export
(Section 6) shows faculty-level rows only. |
institution_sourcesAdded 2026-09-27. One row per external reference link used to verify
an institution's real faculty structure during the unit-level
classification pass (Section 3.2). Distinct from
discovery_seeds (Section 3.6), which holds the original
"list of faculties" pages used for the initial discovery run —
institution_sources holds the (often several) additional
sources consulted afterward to confirm which discovered units are true
faculties versus nested sub-units, and to catch gaps the original
discovery missed. Displayed on the public faculty-directories page
(Section 6) alongside each institution's
classification_notes.
| Field | Type | Notes |
|---|---|---|
| source_id | PK | |
| institution_id | FK → institutions | |
| url | text | |
| description | text | nullable — one line on what this source confirmed |
| added_date | date | default current date |
contactsOne row per person.
| Field | Type | Notes |
|---|---|---|
| contact_id | PK | |
| unit_id | FK → faculty_units | institution is derivable via this FK, not duplicated |
| full_name | text | |
| title | text | |
| department | text | nullable — some faculties have no departments |
| text | ||
| source_url | text | exact page the record was collected from |
| collected_date | date | |
| last_verified_date | date | nullable |
| status | enum | active / stale / removed |
| opt_out_flag | boolean | default false |
| collection_method | enum | automated / manual — see Section 5.8 |
| snapshot_html_path | text | path to archived raw HTML of the source page on the server (Section 5.9); nullable only if archiving failed |
| notes | text | nullable |
project_tasksTracks project progress as a checklist, feeding the status export (Section 6.9) — not a single summary line, but a per-task status so the WordPress-facing page can show actual checkmarks.
| Field | Type | Notes |
|---|---|---|
| task_id | PK | |
| task_name | text | |
| status | enum | not_started / in_progress / complete |
| notes | text | nullable |
| last_updated | date |
sunshine_list_entriesValidation/QA data, kept deliberately separate from
contacts — this is government-disclosed reference data used
to check coverage, not collected-for-outreach data, and mixing the two
would blur what each table is for. See Section 5.10 for the workflow
this supports.
| Field | Type | Notes |
|---|---|---|
| entry_id | PK | |
| institution_id | FK → institutions | |
| full_name | text | |
| position | text | |
| employer | text | as disclosed — sometimes differs from the institution itself (e.g. a federated college) |
| compensation | numeric | nullable — CAD, as disclosed |
| disclosure_year | integer | |
| source_url | text | |
| matched_contact_id | FK → contacts | nullable — filled in during the name-matching pass (Section 5.10); null means no match found yet or ever |
| match_confidence | enum | exact / fuzzy / unmatched — default unmatched until the
matching pass runs |
| last_checked_date | date |
discovery_seedsAdded 2026-09-26, supporting the automated faculty-page discovery process (Section 5.11). Each row is one "list of faculties" page used as a starting point for a discovery run. Re-running discovery on an institution later adds a new row rather than overwriting the old one, preserving history of what the seed was on a given date.
| Field | Type | Notes |
|---|---|---|
| seed_id | PK | |
| institution_id | FK → institutions | |
| seed_url | text | the master "list of faculties" page for this institution — distinct from an institution's plain homepage, and distinct from any one faculty's own page |
| acquired_date | date | |
| notes | text | nullable — how the seed was found, and any confidence caveats (e.g. "JavaScript-rendered, acquired via manual browser paste instead of automated fetch") |
discovery_html_snapshotsAdded 2026-09-26. One row per successful fetch of a seed page by the
discovery script (Section 5.11). Raw HTML is stored on disk, not in
Postgres directly, per the same reasoning as
contacts.snapshot_html_path (Section 3.3) — keeps the
database itself lean as repeated runs accumulate snapshots.
| Field | Type | Notes |
|---|---|---|
| snapshot_id | PK | |
| seed_id | FK → discovery_seeds | |
| file_path | text | path to the archived raw HTML on disk, under
/opt/apps/u15data/discovery_output/html/ |
| fetched_date | date | |
| http_status | integer |
Applied to u15data_db on 2026-11-08 with
u15data_schema_people_collection_v1.0.sql (SQL scripts now
live in /opt/apps/u15data/sql/). Full column lists, rules
and reasoning are in u15data_people_collection_design (v1.1
or later); this section only records what exists. Privileges for
u15data_app are least-privilege: the history tables (runs,
events, snapshots) allow insert and select only.
collection_runsAfter the import of 2026-11-08: 8 rows (2 checker, 2 discovery, 3 robots_fetch, 1 import; run ids 9 to 16, ids 2 to 8 were used up by a dry run). One row per script run of any kind (run_type: discovery / checker / scraper / robots_fetch / import), with script name and version, start and finish times and the run's time-stamped folder.
faculty_directory_pagesOne row per directory address for a unit (unique on unit_id plus
address). Carries the checker's verdict and counts and a status:
proposed, ai_likely, ai_unsure, ai_rejected, manual_needed, ai_verified,
human_confirmed, human_rejected. Scripts set and clear only the ai_ and
manual_ statuses; they never change a human status.
faculty_units.directory_url is set only when a page becomes
human_confirmed. The public status page counts any non-null
directory_url as confirmed (it does not look at the ai or
human flags), so 17 on-file addresses already show as confirmed on the
public page although none has been verified; 6 of the 15 checked so far
redirect elsewhere and are probably stale. Review needed.
Contents after the import of 2026-11-08 (v1.23): 972 rows for 166 of
the 193 faculties: ai_likely 212, ai_unsure 110, ai_rejected 22,
manual_needed 4, proposed 624; 972 event rows. These count addresses,
not faculties: one faculty can have several. Of the
proposed rows, 590 and 33 are links and path probes found
by discovery that no checker run visited, and 1 is an on-file address
nobody checked. The 27 faculties with no row (193 minus 166) are those
with no address at all: the manual-route faculties, Montréal Médecine,
UBC Medicine, and any faculty where discovery found nothing; the list is
kept in u15data_PENDING_ITEMS.
faculty_directory_page_eventsOne row for every status change of a directory page (old status, new status, reason, run, actor), so confirming and unconfirming leave a trail.
people and Table M —
person_review_candidatespeople holds one row per real person, keyed by a numeric
person_id that is never reused or deleted. Merging sets
merged_into_person_id and nothing else, so it can be
undone. A person is matched automatically only on the same email or the
same profile address. A match on name alone never merges: it adds a pair
to person_review_candidates (same name at the same
institution, or at another institution) for a reviewer. Cross-appointed
people with the same email in two units are one person
automatically.
contacts changes and view
contacts_currentcontacts.email now allows no value. New columns:
profile_url, profile_robots_status, run_id, page_id,
listing_snapshot_path, person_id, person_match_basis,
person_match_review, with a unique index on (run_id, unit_id,
person_id). Each run adds rows (history). The view
contacts_current shows the latest row per unit and person;
everyday queries use the view. contacts is still empty.
robots_txt_snapshotsThe actual robots.txt text for each site checked, with fetch time and
HTTP status. Loaded on 2026-11-08 (v1.23): 158 snapshots from three
folders (the robots audit 2026-11-08_153739: 148 sites; the discovery
run 2026-11-08_155845: 6; the checker run 2026-11-08_164614: 4), each
attached to a collection_runs row of type robots_fetch.
Seven snapshots have no text because the site could not be read at that
moment (the table has no error column; the saved files in the run
folders carry the error text). The first discovery and checker runs of
2026-11-08 saved no robots text; the audit covers those sites.
4.1. Collection permission is audited at the
faculty_units level, not assumed institution-wide —
individual faculties/units may carry stricter (or looser) robots.txt or
Terms of Use than their parent institution.
4.2. Units flagged automated_collection_allowed = no are
excluded from automated collection; they may be handled manually or
dropped, never overridden.
4.3. Data minimization: only fields already published on the official page are collected (name, title, department, email). No personal/non-institutional contact data.
4.4. CASL implied-consent reasoning applies to outreach messaging (Phase 6), not to the collection method itself — collection legality is governed separately by each unit's own terms, per 4.1.
4.5. opt_out_flag and a suppression mechanism are
required fields from the outset, not added later.
4.6. When a directory page itself states a usage restriction (e.g., a
prohibition on bulk mailing or solicitation), the exact wording is
captured verbatim in institutions.directory_usage_terms
(Section 3.1) — not paraphrased, and not silently noted only in passing.
First instance found: Dalhousie's directory page (2026-09-22), which
explicitly states directory data "may not be used to provide addresses
for bulk mailings" and that "solicitation of business, information,
contributions, or other response from individuals listed by mail or
other means is forbidden." This kind of restriction is checked
separately from automated_collection_allowed (which
reflects technical/robots.txt permission) — a directory can be
technically scrapable while still carrying an explicit usage prohibition
that governs what the data may be used for once collected.
4.7. Added 2026-11-08 (v1.22). How robots.txt is read. All scripts
must use the shared module u15data_robots_v1_1.py, which
follows RFC 9309: wildcards and the end-of-address marker, longest
matching rule wins, the group for our own user agent (or
*), and Crawl-delay honored per site (never
less than the 2-second floor). A robots.txt that cannot be read (403,
5xx, timeout) is treated as disallowed, not as permission; 404 and 410
mean no restrictions. Every redirect step is its own request, checked
against the robots.txt of the site it goes to. The text of each
robots.txt used is saved. Python's built-in
urllib.robotparser must not be used (on the server's Python
3.12.3 it ignores wildcards, longest-match and Crawl-delay).
5.1. Phase 1 — Faculty/unit-level source mapping and permission
audit, all 15 institutions. Research only (search/fetch), no code.
Output: MD table matching the faculty_units schema (Section
3.2). Institution-level robots.txt findings for all 15
institutions are loaded into the live database
(institutions table) as of 2026-09-22 — the
markdown audit log (u15data_phase1_audit) remains the
narrative record of methodology and reasoning, but the database is now
the authoritative source for the data itself. Methodology: robots.txt
checks are run from a local machine, not institutional infrastructure,
to avoid IP-reputation risk (Section 7.4); fetched via a standard
browser user-agent, since default tooling user-agents can be blocked by
CDN/WAF configurations for reasons unrelated to bot policy; checked in
batches to keep review manageable. robots_txt_status values
are defined precisely, since "no restriction found" can mean different
things worth keeping distinct for future re-checks: no_file
= the URL returns HTTP 404, no robots.txt exists at all;
empty_file = a robots.txt exists (HTTP 200) but specifies
no rules; has_rules_permissive = rules exist but none block
anything relevant to directory/contact pages;
has_rules_restrictive = rules exist that do block
directory/contact-relevant paths, whether sitewide or scoped to a
specific unit. no_file and empty_file
currently have the same practical effect (no stated restriction) but are
logged separately because they are different facts about the site and a
future re-check should be able to tell "nothing changed" from "a
robots.txt was added since we last checked."
5.2. Phase 2 — Review of Phase 1 findings; finalize which units are
in-scope for automated collection. Originally a two-tier strategy where
faculty-level mapping (faculty_units) was only populated as
a fallback for institutions without a usable central directory.
Superseded 2026-09-26: faculty_units is
now populated for all 15 institutions regardless of central-directory
status — see Section 5.11.
5.3. Phase 3 — Schema finalization (this document) and infrastructure
provisioning (database, role, folder) on the server. Requires explicit
go-ahead before any code is written. Complete as of
2026-09-22: u15data_db database and
u15data_app role created (ownership/grant pattern matches
the xab_db/xab_app reference — owned by
postgres, u15data_app granted access); all
three tables (institutions, faculty_units,
contacts) created per the Section 3 schema;
/opt/apps/u15data/docs/ and
/opt/apps/u15data/snapshots/ folders created, owned by
apps.
5.4. Phase 4 — Build site-specific collectors for in-scope units only; respects robots.txt/rate limits; no login-walled or CAPTCHA-protected pages.
5.5. Phase 5 — Load into u15data_db; log source_url and
collected_date per record.
5.6. Phase 6 — Outreach compliance: sender identification, opt-out handling, suppression list enforcement.
5.7. Phase 7 — Recurring refresh cycle; directories change over time, so this is not a one-time pull.
5.8. Manual collection and upload pathway (complement to Phase 4).
For units flagged manual_only in
faculty_units, or where automated collection isn't worth
building for a small unit, a person manually visits the page and enters
what's there through a simple internal upload interface (form entry or
CSV upload) rather than a scraper. Records entered this way land in the
same contacts table as automated records, distinguished
only by the collection_method field (Section 3.3), so
downstream use of the data doesn't need to know or care how a given
record arrived. This pathway is designed to be fast and low-friction,
since its purpose is to make manual entry a genuine, easy option rather
than a fallback nobody uses in practice.
5.9. Source page archiving (applies to both automated and manual collection). Every time a contact record is collected, the original source page's raw HTML is saved on the server — not just referenced by URL — so it can be re-analyzed later even if the live page changes or disappears, and so accuracy of the automated collector can be manually verified against what the page actually contained at collection time. No retention limit: archives are kept indefinitely, on the assumption that 15 institutions' worth of HTML won't grow large enough to justify a deletion policy. Exact storage path convention is an open item (Section 7.6).
5.10. Phase 8 — Sunshine list validation (QA/coverage check, run
after meaningful collection exists — not before, since there's nothing
to validate yet). Uses provincial public-sector compensation disclosure
("sunshine list") data as an independent, government-published
cross-check on collection completeness. Important limitation, stated
plainly rather than glossed over: sunshine lists only disclose employees
above a compensation threshold, so this validates coverage of
senior/high-earning faculty and administrators specifically — it says
nothing about coverage of junior faculty or general staff, which is most
of contacts. It also never contains email addresses, so it
can only validate, never collect.
Initial research (2026-09-22) found real unevenness across the 7 provinces our 15 institutions span, which shapes the workflow below: a. Ontario (6 institutions: Toronto, Waterloo, Ottawa, Queen's, McMaster, Western) — centralized, single provincial website, $100,000 threshold, well-established since 1996. The clean case. b. British Columbia (UBC) — centrally disclosed, but the threshold appears to have changed over time (older sources: $75,000; a more recent structured source: $125,000, "through multiple parts") — needs direct confirmation, not assumption. c. Alberta (Alberta, Calgary) — not centralized; disclosed via each university's own annual report, on that university's own site. d. Manitoba (Manitoba) — legally required above $50,000, but not required to be published online; historically library-access-only. May not be usable for this workflow at all — needs confirming rather than assuming it's accessible. e. Saskatchewan (Saskatchewan) — conflicting signals found: older reporting says disclosure law doesn't apply to universities; a 2026 University of Regina FAQ describes a $125,000 threshold now in place "in alignment with other Saskatchewan organizations." Needs a direct, current, per-institution check. f. Nova Scotia (Dalhousie) — required since 2016 at $100,000, but disclosed per-institution rather than centrally; Dalhousie's own specific page not yet located. g. Quebec (Laval, McGill, Montréal) — no dedicated sunshine-list legislation; senior administrator compensation is technically public but embedded in annual reports each university submits to the National Assembly's Culture and Education committee — unstructured, and limited to senior administrators by nature.
Workflow: a. Source confirmation — for each of the 15 institutions,
confirm the actual current disclosure source (URL, format, threshold,
centralized vs. per-institution), populating the new
sunshine_list_url / sunshine_list_mechanism /
sunshine_list_threshold / sunshine_list_status
fields on institutions (Section 3.1). Comparable in effort
to the robots.txt audit already done for Phase 1. b. Ingest confirmed
sources into sunshine_list_entries (Section 3.5) — kept
separate from contacts deliberately, since this is
validation/reference data, not collected-for-outreach data. c.
Name-matching pass: for each sunshine_list_entries row
belonging to one of our 15 institutions, check for a matching name in
contacts, populating matched_contact_id and
match_confidence. Needs fuzzy matching (middle names, name
variants, maiden names) — exact string matching alone will under-report
real matches. d. Produce a coverage report: senior/high-earning people
who appear on a sunshine list but have no match in contacts
— a signal of possible gaps, explicitly scoped to high earners, not a
general completeness claim.
5.11. Automated faculty-page discovery. Built 2026-09-26 to replace
search-engine-based discovery (Section 7.12) for finding each
institution's individual faculties/schools and their real websites. Two
distinct scripts: a. discover_pages.py (v1.0) — given a
discovery_seeds row (a master "list of faculties" page),
fetches it directly with requests, saves the raw HTML to
disk (discovery_html_snapshots), and extracts every link's
text and destination to a JSON file for review. Does not write to
faculty_units directly — a script finding a link on a real
page is stronger evidence than a search-engine snippet, but still gets
reviewed (AI pass, then human pass) before being treated as confirmed,
consistent with the project's standing no-guessing rule (Section 1,
standing instruction). b. Identifies itself honestly: User-Agent string
names the project and links to its own public status page
(u15data-research/1.0 (+https://u15data.schillo.world/; ...)),
rather than posing as a browser. Checks robots.txt for the target domain
before every fetch and skips if disallowed (confirmed working against
Alberta, which is JavaScript-rendered and also robots.txt-blocked at the
seed page). Two-second delay between requests. c. Known limitation, not
yet worked around: like most simple fetchers, it does not execute
JavaScript. Sites where the real faculty list only renders client-side
(Alberta's ualberta.ca/faculties) return incomplete/empty
content. Current workaround for these cases is manual: the person visits
the page in their own browser and pastes the real content back, which is
treated the same as any other confirmed source. A headless-browser
fallback (e.g. Playwright) would remove this limitation but hasn't been
built, since the manual workaround has been sufficient so far and adds a
real dependency. d. Milestone, 2026-09-26: every one of the 15
institutions now has real faculty/school-level data — 275 units
total, every one with a confirmed home_page_url,
none guessed. directory_url (an actual staff-listing page,
not just a home page) is confirmed for a minority of these 275 so far;
extending that is separate, ongoing work. human_confirmed
is false on all 275 rows — the review pass has not started
(Section 7.13). e. Raw methodology detail (which seed page was used for
which institution, and per-institution confidence notes) lives in the
separate u15data_faculty_discovery_log document, matching
how Phase 1's raw robots.txt findings live in
u15data_phase1_audit rather than in this document.
5.12. Faculty-level robots audit and the 2026-11-08 reader
correction. The first discovery and checker runs of 2026-11-08 used a
robots reader with three flaws: it ignored wildcards, longest-match and
Crawl-delay; the discovery script treated an unreadable robots.txt as
permission; and the checker followed redirects before checking the final
site's robots.txt. A retrospective audit
(audit_robots_v1.0.py, run folder
discovery_output/robots_audit/2026-11-08_153739/, one
request per site, robots.txt text saved) re-tested everything those runs
requested against the correct rules. Findings, stated as found: a. Of
the 36 faculties the discovery run recorded as blocked by robots.txt, 14
(9 at Ottawa, 5 at Waterloo) are in fact allowed under the correct
reader and are being re-run (Section 5.14). 22 remain blocked or
unknowable: 19 at sites whose robots.txt answers 403 or cannot be
reached (17 University of Alberta faculties, McMaster Health Sciences
and Toronto Arts & Science, treated as disallowed under 4.7) and 3
with a real Disallow: / (McMaster Humanities, Laval Faculté
des sciences de l'administration, Rotman). All 22 are kept in the
database with their addresses and go to the manual route (Section 5.8);
nothing is deleted. b. A first prediction that McGill's robots.txt had
been violated was wrong: the wildcard rule that looked relevant belongs
to the archive.org_bot group, not to us. McGill's group for
* sets Crawl-delay: 5. The audit also records
that our scripts waited only 1 to 2 seconds between requests to the same
site, so Crawl-delay was not honored for McGill (5 seconds) or Queen's
(10 seconds). This is a conduct gap, now fixed in the scripts (Section
5.13); McGill's automated_collection_allowed = manual_only
setting from v1.19 should be revisited (open item 7.18). c. A small
number of requests went to sites whose robots.txt could not be read (www.med.ubc.ca, www.landfood.ubc.ca, www.med.umontreal.ca and one
Laval repertoire host), and one checker request followed a redirect to a
final address disallowed by robots.txt
(dalu.sharepoint.com/sites/engineering, apparently
Dalhousie's). The audit summary file in the run folder lists each one.
d. University of Alberta, McMaster Health Sciences and Toronto Arts
& Science return 403 for robots.txt to our script. Whether this
depends on our user agent or on the server's address is not known; a
comparison of curl from a personal computer and from the
server was proposed and results are not yet recorded (open item
7.19).
5.13. Shared modules and the run model (built 2026-11-08; committed
ee511a4, cd86a8c, d5986ee). Current versions:
u15data_robots_v1_2.py,
u15data_scheduler_v1_1.py,
discover_faculty_directories_v1.4.py,
summarize_faculty_candidates_v1.1.py,
check_faculty_directories_v1.5.py,
import_directory_results_v1.1.py (earlier versions are kept
beside them; the 2026-11-08 afternoon runs used robots v1.1, scheduler
v1.0, discovery v1.3 and checker v1.4). Robots module v1.2 retries a
failed robots.txt read three times (waits of 3 and 10 seconds) when the
failure looks passing, and re-reads a site that stayed unreadable after
10 minutes if asked about it again; it was added after one dropped
connection marked UBC Land and Food Systems as disallowed for a whole
checker run. u15data_robots_v1_1.py is the robots reader of
Section 4.7 (with parallel pre-fetch and saving of the robots.txt text).
u15data_scheduler_v1_0.py rotates requests across websites:
each site has its own queue and not-before time (at least 2 seconds, or
its Crawl-delay if longer); when a site is waiting, the script works on
another site and sleeps only when nothing at all is ready; failed
requests go to the back of the queue and are retried no sooner than 60
seconds later, up to 3 attempts; a Retry-After from a site is honored.
Scripts built on them:
discover_faculty_directories_v1.3.py,
summarize_faculty_candidates_v1.1.py,
check_faculty_directories_v1.4.py (the checker's every
redirect step is robots-checked and a disallowed step is reported as
ROBOTS_DISALLOWED without being fetched). Every run writes into its own
time-stamped folder (UTC), never overwriting:
discovery_output/runs/<stamp>_discovery_v1.3/ (html/,
links/, robots/, discovery.log, run_info.json, summaries/),
discovery_output/checks/<stamp>/,
discovery_output/robots_audit/<stamp>/. Raw run
output is kept out of Git; scripts and documents are committed (local
Git only, run as sudo -u apps git -C /opt/apps/u15data ...,
commit messages end with the Claude-Session line only). Earlier script
versions are kept alongside the new ones. The collector runs on this
server; the earlier checks from a personal computer were only for
working things out (this closes open item 7.4).
5.14. Faculty directory pages, status as of the evening of
2026-11-08. Discovery v1.2 covered the faculty-level units (home page
links plus 16 guessed directory paths); checker v1.3 checked 318
addresses. The 18-faculty re-run (14 faculties that the corrected robots
reader allows, 3 that had failed to load, and UBC Medicine) was done
with discovery v1.3, summary v1.1 and checker v1.4: 16 faculties loaded
(Ottawa 9, Waterloo 5, Laval FAAAD, UBC Land and Food Systems); Montréal
Médecine and UBC Medicine were again recorded as blocked because their
robots.txt could not be read. The checker visited 30 addresses; seven
faculties got a LIKELY_DIRECTORY proposal needing no person (Ottawa
Common Law, Ottawa Social Sciences and all five Waterloo faculties); the
others need a person, and several proposals are plainly wrong or too
broad (details in u15data_PENDING_ITEMS). All of it, plus
every other address discovery found, is now in
faculty_directory_pages (Section 3.11) through
import_directory_results_v1.1.py (dry run by default,
--apply to commit; report in
discovery_output/imports/2026-11-08_183404/). Status rule:
LIKELY_DIRECTORY becomes ai_likely (ai_unsure when flagged with anything
other than paginated or redirected); PARTIAL_OR_UNSURE and the scope and
JavaScript verdicts become ai_unsure; no-directory verdicts become
ai_rejected; ROBOTS_DISALLOWED becomes manual_needed; addresses found by
discovery but not visited, and on-file addresses nobody checked, are
proposed. Pages already ai_verified, human_confirmed or human_rejected
are never overwritten by an import. Retry later: since about 16:46 UTC
both UBC hosts (www.med.ubc.ca, www.landfood.ubc.ca) reset every
connection from this server (curl exit 56), although Land and Food
Systems answered at 15:58 and 16:12; its one manual_needed
row is from a dropped connection, not a restriction, and both faculties
are on the retry list
(discovery_output/retry_ubc_unit_ids_2026-11-08.txt, ids
181 and 179). www.med.umontreal.ca no longer resolves (curl
exit 6), so Montréal Médecine needs a correct address from a person; it
is not to be guessed.
5.15. People scraper (Phase 4, not yet written). Design in
u15data_people_collection_design: reads only addresses with
status ai_likely, ai_verified or human_confirmed; records everyone found
with their position text, email and profile address (tenure-track
filtering happens later, not in the scraper); records the directory
address and the run for every row; checks robots.txt for every listing
and profile page; confirms an address (ai_verified) when it extracts
people and unconfirms it (ai_unsure or manual_needed) when it fails,
writing an event each time. It will not be written until the user
approves the design.
6.1. Final design, after working through the alternatives below: a
plain static JSON file, regenerated by an offline export script (not a
live database query), served directly by Nginx from
u15data.schillo.world — no application, no Flask/Gunicorn,
no /api/ route. This is functionally the Static Partner
Import Pattern already proven elsewhere on i2hub.ca (per
ERICA_Methods_Pattern_Inventory_v2, Section 2), pointed at
our own server's file instead of a Google Drive file.
6.2. On the WordPress side: a PHP Code Snippet performs a server-side
HTTP fetch of that JSON file, renders it as HTML, and returns it via a
shortcode dropped into an Elementor Shortcode widget on a PMPro-gated
page — the same shape as the site's existing Data- snippets
that already fetch from Google Drive/Apps Script.
6.3. Why a file, not a direct database connection: PHP (running on WordPress's server) can only reach things exposed somewhere on the network. Exposing Postgres itself to the network would mean opening it to the whole internet, not just to WordPress specifically, which breaks the server's own firewall convention (only SSH/80/443 open) and is a materially larger attack surface than exposing one narrow, read-only file. The static file is a deliberate boundary, not an unnecessary intermediate step.
6.4. Why no iframe, no live viewer page: CORS (the reason originally given for an iframe) only applies to JavaScript running in a visitor's browser fetching cross-domain — it does not apply to PHP's server-side fetch. Since this status page only needs to show current numbers on load, not refresh itself live in the visitor's browser, there's no CORS problem to design around in the first place. An iframe-embedded viewer page would only be justified if the page itself needed to be interactive/live-updating, which it doesn't.
6.5. Rejected alternatives, in order of consideration: a. Live REST
API (Flask/Gunicorn querying Postgres per request) — unnecessary
complexity for data that only needs to be as current as the last manual
export; also the "emerging main method" pattern site-wide, but with no
working precedent anywhere on i2hub.ca yet. b. Iframe-embedded viewer
page on u15data.schillo.world — more setup than needed;
only justified for genuinely interactive/live content. c. Reusing the
Google Drive JSON pattern instead of our own server — considered as the
simplest option overall (zero new server infrastructure), but rejected
in favor of keeping this on our own infrastructure once the actual
complexity of the static-file approach became clear (a plain
Nginx-served file, not a live app) — the earlier stated benefit (not
depending on Google Drive staying configured correctly) applies without
much added cost once no application layer is needed. d. Piggybacking the
file onto the existing xab.schillo.world subdomain instead
of provisioning a new one — would avoid new Nginx/SSL setup entirely,
but breaks the "one subdomain per app" server convention and mixes two
unrelated projects' content under one address. Rejected in favor of a
clean, dedicated subdomain.
6.6. Subdomain: u15data.schillo.world, per the
xab.conf Nginx configuration reference on this server.
Live as of 2026-09-22: DNS A record added at Network
Solutions (u15data → 134.87.8.156, same server
as xab), Nginx server block created serving
/opt/apps/u15data/static/, SSL certificate issued and
deployed via Certbot (expires 2026-12-21, auto-renewal scheduled).
Verified reachable over both HTTP and HTTPS, currently serving a
placeholder status.json. Remaining work is content, not
infrastructure: the export script (Postgres → real
status.json) and the WordPress-side PHP snippet, both of
which are code and await explicit go-ahead per Section 1's standing
instruction.
6.7. Scope for now: status/progress dashboard only (phase tracking,
audit findings/counts) — no contact-level data. Consistent with the
earlier decision to keep contacts data out of any
WordPress-facing surface until a separate access-control design is done
for it specifically.
6.8. Known, accepted gap: access control is page-level only. The WordPress page is PMPro-gated, but the raw file URL is not independently protected — anyone with the direct link could view it without logging in. Accepted for now given the low sensitivity of status/summary data. Must be revisited before this pattern is reused for anything higher-sensitivity, such as actual contact records.
6.9. Export script: /opt/apps/u15data/export_status.py,
run manually
(/opt/apps/u15data/venv/bin/python3 /opt/apps/u15data/export_status.py)
whenever there's new progress to publish. Reads
project_tasks and institutions from
u15data_db, writes
/opt/apps/u15data/static/status.json, which Nginx serves
directly. Live and verified as of 2026-09-22 —
confirmed returning real task and institution-summary data over HTTPS.
Workflow going forward: update project_tasks via SQL, then
re-run the script; the script itself does not need to change for routine
updates.
6.10. PHP snippet naming: no ERICA prefix — this project is not part
of ERICA, so the site's ERICA Data- snippet naming
convention doesn't apply here. Named simply "u15data Status" on the
WordPress side.
6.11. Complete and verified live as of 2026-09-22:
PHP Code Snippet added (server-side fetch of status.json,
5-minute transient cache, renders task checklist + institution summary
via [u15data_status] shortcode), WordPress page created,
Shortcode widget placed via Elementor, page restricted to the PMPro
membership level, and access confirmed both ways — visible to a member,
blocked when logged out. This closes out the WordPress integration work
described in this section.
6.12. Documentation hosting pattern: full versioned history of both
markdown documents (this one and the Phase 1 audit log) lives in
/opt/apps/u15data/docs/, per the server's own convention
that each app keeps its docs in its own docs/ subfolder —
every version, kept permanently, matching the project's standing rule
that every document carries a version number in both its filename and
its header. Separately, one stable-named "latest" copy of each
(architecture_latest.md,
phase1_audit_latest.md) lives in
/opt/apps/u15data/static/docs/, served directly by Nginx
alongside status.json, and linked from the WordPress status
page. The stable names mean the PHP snippet's links never need editing
when a document's version number changes — only two file operations are
needed per update (move the new versioned file into docs/,
overwrite the matching _latest.md in
static/docs/), not a snippet edit.
6.13. Status page display fix: the rendered lists (task checklist,
robots.txt status breakdown, central directory breakdown, documentation
links) previously showed default bullet points on some lists but not
others, since bullet-removal had only been applied to the task list
individually. Fixed by adding a single scoped <style>
block covering every list inside the status box, rather than styling
each list separately.
7.1. Exact refresh cadence for Phase 7 — not yet defined.
7.2. Resolved 2026-09-27: faculty_units gained a
unit_level field (Section 3.2) distinguishing true
top-level faculties from nested sub-units and excluded
administrative/historic entities, rather than relying on
unit_name free text alone.
7.3. Conversion of this document to .docx can be done
once content is stable; not needed while still in active revision.
7.4. Resolved 2026-11-08: the collector runs on this server (Section 5.13). Original question: where the automated collector (Phase 4) physically runs from — raised during Phase 1 robots.txt testing, where running checks from uOttawa infrastructure was identified as a reputational/IP-blocking risk to institutional systems generally, not just this project. Needs a deliberate decision (local machine, separate non-institutional host, etc.) rather than defaulting to "runs on the same server as everything else."
7.5. Resolved 2026-09-22: /profiles/ confirmed, via
Drupal's own source repository, as Drupal core's default "installation
profiles" directory — boilerplate present in Drupal's default robots.txt
template across many versions, unrelated to staff/user profile pages and
not directory-relevant. Applied to all 5 affected institutions (Calgary,
Laval, Manitoba, Toronto, Ottawa), each re-coded from NULL
to has_rules_permissive in the database.
institutions.by_robots_txt_status no longer has a
pending_confirmation bucket — Phase 1's institution-level
robots.txt audit is now fully resolved for all 15.
7.6. Storage path convention for the HTML archives (Section 5.9) —
not yet defined. Likely lives under /opt/apps/u15data/ in
its own subfolder (e.g., a snapshots/ directory organized
by unit and date), but this should be confirmed rather than assumed.
Retention is settled — kept indefinitely, no deletion policy — so this
item is scoped to path/naming only.
7.7. Resolved — see Section 6.6.
7.8. Resolved — no wildcard exists; DNS is managed at Network
Solutions with individual A records per subdomain, matching
xab's existing setup. u15data record added the
same way. See Section 6.6.
7.9. Resolved — see Section 6.11. PHP snippet and WordPress page are live and verified.
7.10. Sunshine list source confirmation for all 15 institutions — not
yet started (Section 5.10). Real unevenness found across provinces
during initial research; several sources (BC's current threshold,
Manitoba's online accessibility, Saskatchewan's applicability to
universities, Nova Scotia's Dalhousie-specific page) need direct
confirmation rather than assumption before
sunshine_list_status can be marked confirmed
for any institution.
7.11. Subdomain-specific robots.txt gap, found 2026-09-22 while
attempting to set automated_collection_allowed: for
institutions where the actual central directory lives on a different
subdomain than the one checked in Phase 1 (Alberta:
apps.ualberta.ca vs. checked ualberta.ca;
Calgary: contacts.ucalgary.ca; McMaster:
directories.mcmaster.ca; Dalhousie:
directory.dal.ca), the Phase 1 robots.txt result does not
actually tell us whether the directory subdomain itself permits
automated access — robots.txt is per-domain, not institution-wide.
automated_collection_allowed is deliberately left
NULL for these 4 institutions until their directory
subdomains are checked directly, rather than inferring permission from
the main domain's result.
7.12. Resolved 2026-09-26, superseded by Section 5.11:
search-engine-based discovery of individual faculty pages proved
unreliable and, in a small number of cases, produced guessed URLs that
turned out to be wrong (an Ottawa faculty home page and a Queen's
graduate-studies page were both corrected after the fact — see the
faculty discovery log's changelog for specifics). Replaced with the
discover_pages.py script, which fetches real pages directly
rather than relying on search snippets.
7.13. Human-review pass on faculty_units — not started.
All rows currently carry human_confirmed = false.
ai_confirmed = true reflects that a link was found on an
actually-fetched page and judged relevant, which is a real evidentiary
bar, but is not a substitute for a person checking it.
7.14. directory_url (an actual confirmed staff-listing
page, distinct from home_page_url) is populated for only a
minority of the 193 faculty-level rows (corrected from 191
in v1.22). Extending this is separate, ongoing work — most units
currently have a confirmed home page but not yet a confirmed staff
directory within it.
7.15. Two institutions (McGill, University of Toronto) and several
individual faculties elsewhere still lack a directory_url
beyond their home page even where one likely exists (e.g. Ottawa's
Faculty of Medicine has one confirmed; most other Ottawa faculties don't
yet). Toronto in particular has all 19 divisions logged with a home page
only, since each division runs its own separate site with no common
pattern found yet.
7.16. McMaster's automated_collection_allowed —
revisited 2026-09-26. McMaster's robots.txt explicitly blocks a list of
named AI-company crawlers (including ClaudeBot and
anthropic-ai by name), though its general wildcard rule
only blocks /wp-admin/. Originally treated as equivalent to
a no. Revised after discussion: the block targets AI
companies' own commercial/training crawlers specifically, not a custom,
honestly-identified research tool built for a bounded academic project —
a meaningfully different activity even though a coding assistant helped
build the tool. discover_pages.py's honest
self-identification (Section 5.11b) is what makes this distinction real
rather than a rationalization: the script does not pose as a browser or
as one of the named blocked bots.
7.17. Resolved 2026-11-08: the 191 versus 193 faculty count. 193 is
correct (Section 3.2); the unit-level split is 193 / 47 / 19. The
classification sources document
(u15data_unit_classification_sources v1.1) and the faculty
discovery log should be read with that correction.
7.18. McGill automated_collection_allowed = manual_only
(v1.19) was set when McGill's robots.txt was thought to block directory
pages. The 2026-11-08 audit found the blocking rule belongs to
archive.org_bot, and McGill's group for all other agents
only asks for Crawl-delay: 5. The user decides whether to
revise the setting.
7.19. Whether the 403 on robots.txt from University of Alberta,
McMaster Health Sciences and Toronto Arts & Science depends on our
user agent or on the server's address. Comparison of curl
from a personal computer and from the server has been proposed; results
not yet recorded. Until known, the 19 affected faculties are on the
manual route.
7.20. Meaning of
institutions.automated_collection_allowed at faculty level.
Recommendation to be confirmed at the scraper stage: the unit-level
field gates collection and the institution-level field describes only
the central directory.
7.21. Sunshine list (Section 5.10, item 7.10): source confirmation
started 2026-11-08 in this project; findings so far are in
u15data_HANDOVER_sunshine_list v1.0 and are not yet entered
here or in the database. Several notes in Section 5.10 are out of date:
Manitoba publishes an online PDF; the University of Saskatchewan posts a
list; Dalhousie's own disclosure documents exist; the Alberta threshold
appears to be $130,000 for 2026 (the earlier $125,000 is the University
of Regina's). All to be re-verified on the actual pages before being
recorded.
7.22. Directory addresses needing correction: Montréal Médecine home
address (dead), the Dalhousie faculty whose on-file address redirects to
dalu.sharepoint.com/sites/engineering, which robots.txt
disallows (the faculty to be identified from the checker output), and
the 17 on-file directory_url values that the public page
already counts as confirmed (item 3.11).
7.23. Stale text in institutions.classification_notes
for Ottawa (says 9 faculties, there are 10) and Montréal (says 12
faculties, there are 13), noted 2026-09 and not yet refreshed. Fixing
them is a change to the export, to be decided.
7.24. UBC retry: after the connection resets of 2026-11-08 (5.14),
retry UBC Medicine and UBC Land and Food Systems no earlier than the
next day, starting with one request for each host's robots.txt, and do
not work around a block. Tracked in
u15data_PENDING_ITEMS.
7.25. When the people scraper is designed it must handle several ai_likely addresses for one faculty (212 ai_likely rows exist for far fewer faculties), so that the same person is not read twice from overlapping pages.
| Version | Date | Change |
|---|---|---|
| 1.0 | 2026-09-22 | Initial architecture document: schema (3 tables), server naming, phase plan, legal/ethical constraints |
| 1.1 | 2026-09-22 | Added manual collection/upload pathway (Section 5.8) and
collection_method field to contacts table;
added open items on collector hosting location and the
/profiles/ path ambiguity found during Phase 1 testing |
| 1.2 | 2026-09-22 | Added source page archiving (Section 5.9) — both HTML and
PDF/screenshot, applies to automated and manual collection alike; added
snapshot_html_path and snapshot_pdf_path
fields to contacts table; added open item on archive
storage path/retention |
| 1.3 | 2026-09-22 | Archiving scoped to HTML only, PDF/screenshot dropped per decision;
removed snapshot_pdf_path field; retention settled as
indefinite (no deletion policy) — open item narrowed to storage path
convention only |
| 1.4 | 2026-09-22 | Added Phase 1 methodology note (Section 5.1) and pointed to the
separate u15data_phase1_audit log document, which now
tracks actual robots.txt findings; this document stays scoped to
decisions/structure rather than raw results |
| 1.5 | 2026-09-22 | Redefined robots_txt_status enum to distinguish
no_file (404, no robots.txt exists) from
empty_file (robots.txt exists but has no rules) rather than
collapsing both into "allowed"; added precise definitions for all four
status values to Section 5.1, prompted by the Waterloo 404 finding |
| 1.6 | 2026-09-22 | Marked Phase 3 complete (Section 5.3): database, role, tables, and
folders provisioned and verified against \du,
\l, \dt, and ls -la output |
| 1.7 | 2026-09-22 | Moved domain-level audit fields (robots_txt_url, robots_txt_status,
tos_url, tos_summary, automated_collection_allowed, last_checked_date)
onto institutions; added
has_central_directory; redefined faculty_units
as a targeted fallback table rather than a blanket requirement for all
15 institutions; added two-tier strategy to Section 5.2 |
| 1.8 | 2026-09-22 | Institution-level robots.txt findings for all 15 institutions loaded
into the live institutions table (after catching and
cleaning up an accidental duplicate insert of 7 rows); database is now
the authoritative source for this data, markdown audit log retained for
methodology/reasoning |
| 1.9 | 2026-09-22 | Added Section 6 (WordPress / i2hub.ca Integration): adopted the
Server-Hosted Static Content Pattern per
ERICA_Methods_Pattern_Inventory_v2, rejected the unproven
live REST API pattern for now, accepted the known page-level-only access
gap for this low-sensitivity data, and flagged the still-unconfirmed
subdomain as an open item; corrected section numbering (Open items moved
to Section 7, Version history to Section 8) |
| 1.10 | 2026-09-22 | Confirmed subdomain u15data.schillo.world from
xab.conf's live Nginx configuration; resolved the open item
from v1.9 |
| 1.11 | 2026-09-22 | Finalized WordPress integration design after working through simplicity trade-offs: static JSON file served by Nginx (no application layer), PHP snippet does a server-side fetch (no iframe needed — corrected earlier CORS reasoning, which only applies to browser-side fetches), no direct database exposure. Rewrote Section 7 to document the reasoning and rejected alternatives explicitly. Added open item on DNS setup method (Section 7.8). |
| 1.12 | 2026-09-22 | WordPress integration infrastructure completed and verified live:
DNS A record (Network Solutions), Nginx server block, SSL certificate
(Certbot, expires 2026-12-21) —
u15data.schillo.world/status.json confirmed reachable over
HTTPS serving a placeholder file. Resolved open item 7.8 (no wildcard
DNS in use). Remaining work scoped to code (export script, PHP snippet),
not infrastructure. |
| 1.13 | 2026-09-22 | Added project_tasks table (Section 3.4) as a checklist
rather than a single summary string, per feedback that a task-by-task
view is more useful than a one-line status; export script (Section 6.9)
written, run, and verified live serving real data at
u15data.schillo.world/status.json. Also fixed a
section-numbering bug introduced in v1.9: the WordPress Integration
section had been numbered 7 with nothing occupying 6, cascading a stale
gap through v1.9–v1.12; renumbered to close the gap (WordPress
Integration is now Section 6, Open items Section 7, Version history
Section 8) and corrected all internal cross-references; caught and fixed
one collateral corruption from the renumbering script (the IP address
134.87.8.156 was briefly mis-edited to
134.87.7.156 in Section 6.6, now restored) |
| 1.14 | 2026-09-22 | PHP Code Snippet and WordPress page (Elementor Shortcode widget +
PMPro restriction) built, deployed, and verified live — access confirmed
both as a member and logged out. Corrected naming: not ERICA-branded,
named "u15data Status." Added corresponding project_tasks
row. This closes out the WordPress integration work in full. |
| 1.15 | 2026-09-22 | Added documentation hosting pattern (Section 6.12): versioned docs
live permanently in /opt/apps/u15data/docs/, stable-named
"latest" copies in /opt/apps/u15data/static/docs/ are what
the status page actually links to, so version bumps don't require
snippet edits. Fixed inconsistent bullet-point styling across all
rendered lists on the status page (Section 6.13), not just the task
list. |
| 1.16 | 2026-09-22 | Added faculty-first priority note (Section 1.3) — both faculty and
staff remain in scope, faculty sequenced first. Added Phase 8, sunshine
list validation (Section 5.10), as a QA/coverage-check workflow using
provincial public-sector compensation disclosure data; documented real
per-province unevenness found in initial research (Ontario centralized
and clean, BC/Saskatchewan/Nova Scotia needing direct confirmation,
Manitoba possibly inaccessible online, Quebec
unstructured/legislature-report-based). Added
sunshine_list_* fields to institutions
(Section 3.1) and new Table E sunshine_list_entries
(Section 3.5), kept deliberately separate from contacts.
Added open item 7.10. |
| 1.17 | 2026-09-22 | Resolved the /profiles/ ambiguity (open item 7.5) for
all 5 affected institutions (Calgary, Laval, Manitoba, Toronto, Ottawa):
confirmed via Drupal's own source repository as core CMS boilerplate,
not staff/user profile content. Database updated
(robots_txt_status re-coded from NULL to
has_rules_permissive for all 5), status page re-exported.
Phase 1's institution-level robots.txt audit is now fully closed for all
15 institutions. |
| 1.18 | 2026-09-22 | Completed central directory discovery for all 15 institutions
(has_central_directory/central_directory_url
populated in the database): 5 confirmed with a usable URL (Alberta,
Calgary, Dalhousie, McMaster, and McGill, though McGill's is
robots.txt-blocked), Toronto confirmed to exist but login-gated
(inaccessible to us), UBC confirmed to exist but exact URL not yet
found, Waterloo has an unconfirmed lead (WatIAM), Western has a partial
lead (staff-coordinator list only), and 6 institutions (Laval, Manitoba,
Montréal, Ottawa, Queen's, Saskatchewan) show no central directory at
all — appear to be genuinely per-faculty/department only. Added
directory_usage_terms and
directory_terms_captured_date fields (Section 3.1) and
Section 4.6, to capture verbatim any usage restriction stated on a
directory page itself; first instance found and recorded: Dalhousie's
explicit prohibition on bulk mailing/solicitation. |
| 1.19 | 2026-09-22 | Set automated_collection_allowed where determinable:
no for the 8 institutions with no accessible central
directory (Laval, Manitoba, Montréal, Ottawa, Queen's, Saskatchewan,
Toronto, Western), manual_only for McGill (directory exists
but is robots.txt-blocked). Left NULL for the other 6
rather than guess — discovered and documented a real methodological gap
(Section 7.11): for Alberta, Calgary, McMaster, and Dalhousie, the
confirmed directory lives on a different subdomain than the one checked
in Phase 1's robots.txt pass, so that result doesn't actually establish
permission for the directory itself; UBC and Waterloo still lack a
confirmed URL at all. Also added a Permissions column with expandable
details to the review tool (review.html), and fixed a
badge-label bug (has rules permissive →
permissive). |
| 1.20 | 2026-09-26 | Major scope shift: faculty-level data collection extended from
fallback-only to all 15 institutions (Section 5.2 superseded). Added
home_page_url, notes,
ai_confirmed, and human_confirmed fields to
faculty_units (Section 3.2), separating "unit's own
website" from "confirmed staff-listing page" for the first time. Added
two new tables, discovery_seeds and
discovery_html_snapshots (Sections 3.6–3.7), and built
discover_pages.py v1.0 (Section 5.11), a script that
fetches real pages directly rather than relying on search-engine
snippets, self-identifies honestly in its User-Agent, and checks
robots.txt before every fetch. Milestone: all 15 institutions now have
real faculty/school-level data — 275 units total, every one with a
confirmed home_page_url, none guessed (caught and corrected
two guessed URLs along the way — an Ottawa faculty page and a Queen's
graduate-studies page — per open item 7.12). directory_url
and human_confirmed remain open (7.13–7.15). Revisited and
revised the McMaster automated_collection_allowed call
(7.16): distinguished a custom, honestly-identified research tool from
the AI-company crawlers McMaster's robots.txt explicitly names. Raw
per-institution methodology now lives in the new companion document
u15data_faculty_discovery_log. |
| 1.21 | 2026-09-27 | Resolved the ambiguity behind v1.20's flat 275-row count (Section
3.2, Section 7.2 now closed): added unit_level
(faculty / sub_unit / excluded)
and self-referencing parent_unit_id to
faculty_units, and reclassified all 259 confirmed rows
across all 15 institutions by direct verification against each
institution's own current official sources — 191 true top-level
faculties, 41 sub-units, 27 excluded. Along the way, found and corrected
several genuine discovery gaps predating this version: Dalhousie was
missing 3 faculties, Universite Laval 2, Universite de Montreal 2 (a
third remains unresolved), and Calgary's entire faculty set had been
pointing to University Calendar description pages rather than each
faculty's real site (corrected, and one missing faculty added). Added
classification_notes to institutions (Section
3.1) and a new table, institution_sources (Section 3.8), to
hold the reference links used for this verification; both are now shown
on the public faculty-directories page (Section 6), which was updated to
show faculty-level rows only rather than all confirmed
units. Full per-institution reasoning and sources:
u15data_unit_classification_sources v1.1. Resolution of the
275-vs-251 row-count discrepancy specifically:
u15data_faculty_discovery_log v1.1, Section 4. |
| 1.22 | 2026-11-08 | Corrected the faculty-level counts to 193 / 47 / 19 (Section 3.2, item 7.17). Added the people-collection schema (Sections 3.9 to 3.15: collection_runs, faculty_directory_pages and events, people, person_review_candidates, robots_txt_snapshots, contacts changes, contacts_current). Added the robots reading rules (4.7), the faculty-level robots audit and its findings (5.12, including the McGill correction and the Crawl-delay gap), the shared robots and scheduler modules, time-stamped run folders and local Git (5.13), directory-page status (5.14) and the people scraper outline (5.15). Collector location settled: this server (7.4). New open items 7.18 to 7.23. |
| 1.23 | 2026-11-08 | Directory-page data imported: 972
faculty_directory_pages rows (212 ai_likely, 110 ai_unsure,
22 ai_rejected, 4 manual_needed, 624 proposed) for 166 of the 193
faculties, 972 events, 158 robots.txt snapshots, 8
collection_runs rows (Sections 3.10, 3.11, 3.15). Robots
module v1.2 (retry of failed robots.txt reads), scheduler v1.1,
discovery v1.4, checker v1.5 and import script v1.1 added (5.13).
Section 5.14 rewritten with the 18-faculty re-run results, the status
rule and the UBC connection resets. New open items 7.24 (UBC retry) and
7.25 (several ai_likely addresses per faculty). |