Version: 1.3 Status: All 15 institutions have real
faculty/school-level data, and (as of v1.1) every row has been
classified as faculty, sub_unit, or
excluded (schema v1.21). The true count of top-level
faculties across all 15 institutions is 193 (corrected
in v1.2; v1.1 said 191), out of 259 total confirmed rows (193 faculty,
47 sub_unit, 19 excluded; v1.1 said 191, 41 and 27). Section 6 (new in
v1.2) records the 2026-11-08 faculty-directory discovery and checking,
and the robots.txt audit. directory_url (an actual
staff-listing page) confirmed for a minority of faculty-level units so
far. human_confirmed is false on all rows --
review pass not yet started. This document is the narrative/methodology
record; the database (faculty_units,
discovery_seeds, discovery_html_snapshots) is
the authoritative source for the data itself. Last updated: 2026-11-08
(evening)
1.1. Two-step process per institution: first find the institution's
own "list of faculties" master page (a search task), then run
discover_pages.py (architecture document Section 5.11)
against that page to extract real links directly from the fetched HTML,
rather than relying on search-engine snippets.
1.2. Every link the script finds is reviewed before being treated as a genuine faculty (filtering out navigation, social media, phone numbers, and similar noise) -- see Section 3 for specific cases where this mattered.
1.3. Where a seed page turned out to be JavaScript-rendered and the script returned empty or incomplete content (Alberta, and McGill's first attempt), the person visited the page directly in their own browser and pasted the real content back. This is treated as equally valid to an automated fetch.
1.4. Consistent with the project's standing no-guessing rule: a
unit's home page and its confirmed staff-directory page are tracked as
two separate fields (home_page_url,
directory_url). A name without a confirmed URL is recorded
honestly as unconfirmed (NULL), never filled with a
plausible-looking guess.
(Unchanged from v1.0 -- see that version, or the live
discovery_seeds table, for the full per-institution seed
list and notes.)
(Unchanged from v1.0, items 3.1 through 3.7 -- see that version for the full list of pre-script guessed-URL corrections and duplicate-row cleanups.)
4.1. This v1.0 document originally stated a total of "275 units total, every one with a confirmed home_page_url." At the start of the unit-level reclassification work (v1.21 schema change), the live database was found to contain 251 rows, not 275 -- a 24-row gap that prompted a dedicated investigation (see the handover and unit-classification-sources documents for the full trail).
4.2. The investigation considered and ruled out several candidate explanations, including a suspected-but-unconfirmed silently-failed UBC insert (see item 3.7 above), before the real explanation emerged during the reclassification work itself: the original "275" figure, and the 251-row count checked against it, were both counting a mix of true top-level faculties and nested sub-units (schools, colleges, departments) as if they were peers. Some institutions' seed pages captured only top-level faculties; others (e.g. Waterloo, whose seed "mixes core Faculties and constituent Schools, both captured," per Section 1's original methodology note) captured both levels flattened into one list. Neither 275 nor 251 was ever a clean count of the same thing.
4.3. The reclassification pass (v1.21 schema:
unit_level, parent_unit_id) resolved this by
separating every row into faculty (true top-level, the
project's actual unit of analysis), sub_unit (nested under
a specific faculty, with parent_unit_id recorded), or
excluded (real but not a faculty -- administrative bodies,
historic affiliated colleges, and similar). This also surfaced and
corrected a small number of genuine gaps missed by the original
discovery passes (e.g. Dalhousie was missing Agriculture, Medicine, and
Open Learning & Career Development; Universite Laval was missing
FAAAD and FESP; Calgary's 13 rows all pointed to University Calendar
pages instead of real faculty websites, and one whole faculty -- School
of Public Policy -- had never been captured at all).
4.4. The reconciled total is 259 rows, of which 193 are true
top-level faculties (stated as 191 in v1.1; see item 4.6). Full
per-institution source citations for the reclassification work are in
u15data_unit_classification_sources_v1.1.md.
4.5. The practical lesson, worth carrying forward: a raw row count is
only meaningful once the unit-of-analysis is unambiguous. Future
discovery or audit passes on this project should classify
unit_level at the time of insertion, not after the fact, to
avoid a repeat of this exact ambiguity.
4.6. Correction (v1.2, 2026-11-08). The 191 faculties stated above was an addition slip: the per-institution faculty counts (Alberta 17, Dalhousie 13, McMaster 6, Queen's 7, McGill 11, Calgary 14, Laval 17, Montréal 13, Manitoba 15, Ottawa 10, Saskatchewan 17, Toronto 17, Waterloo 6, Western 12, UBC 18) add to 193, and the live database holds 193. The split of the remaining 66 rows is 47 sub_unit and 19 excluded in the database; the 41 and 27 in v1.1 could not be traced to any source. The database is authoritative.
6.1. Aim. For each of the 193 faculty-level units, find the page that
lists its faculty members (the directory page), so that the people
scraper (see u15data_people_collection_design) can later
read it. The unit's home page and its directory page remain two separate
fields (home_page_url, directory_url), as in
item 1.4.
6.2. Method. discover_faculty_directories (v1.2 for the
first full run) fetches each faculty's home page, saves the raw HTML,
keeps links whose text or address look like a staff directory (English
and French keywords), and probes 16 common directory paths inside the
faculty's own section of the site.
summarize_faculty_candidates (v1.0) scores and ranks the
candidates and labels each faculty EASY, CHECK or WORK.
check_faculty_directories (v1.3) then visits the best
candidates and the address already on file and records, as counts only
(no names, no emails, no page content), whether the page loads, looks
like a list of people, is paginated, or is probably built by JavaScript,
and proposes one address per faculty.
6.3. Results of the first full run (run folder
discovery_output/runs/2026-11-08_run_v1.2/): of 193
faculties, 154 produced a result file, 36 were recorded as blocked by
robots.txt and 3 failed to fetch (Laval FAAAD, Montréal Médecine, UBC
Land and Food Systems). The checker run
discovery_output/checks/2026-11-08_141658/ checked 318
addresses and wrote proposal rows for 157 faculties, 14 of them without
a proposed address. Of the 15 faculties that already had a directory
address on file and were checked, 6 redirect elsewhere and are probably
stale.
6.4. Flaws found in the first scripts, and corrections. Stated plainly so later readers know what the first run did: a. The robots.txt reader was Python's built-in one, which ignores wildcards, longest-match and Crawl-delay (confirmed on the server's Python 3.12.3). b. Discovery treated an unreadable robots.txt as permission, and the checker followed redirects before checking the final site's robots.txt (one request went to an address that robots.txt disallows). c. Requests to one site were spaced 1 to 2 seconds apart whatever the site's Crawl-delay (McGill asks 5 seconds, Queen's 10). d. A 2 minute 16 second stall in the checker was probably a robots.txt read with no time limit. e. Results: the first checker version mislabelled an address on file that redirects to the proposed address, proposed an address for a faculty with no people on it, and over-flagged needs_person; all fixed in checker v1.3. f. The first version also waited a fixed 2 seconds after every request. It was replaced by rotation across sites, so waiting for one site never stops work on another.
6.5. Robots audit of 2026-11-08. audit_robots_v1.0.py
(run folder
discovery_output/robots_audit/2026-11-08_153739/) fetched
robots.txt once per site, saved the text, and re-tested every address
the discovery and checker runs had requested against the correct rules.
Findings: of the 36 faculties recorded as blocked, 14 (Ottawa 9,
Waterloo 5) are allowed under the correct reader and are being re-run;
22 remain, 19 where robots.txt returns 403 or cannot be reached
(University of Alberta 17, McMaster Health Sciences, Toronto Arts &
Science) and 3 with a real Disallow: / (McMaster
Humanities, Laval Faculté des sciences de l'administration, Rotman).
These 22 stay in the database with their addresses and go to a manual
route; nothing is deleted. A first prediction that McGill's robots.txt
had been violated was wrong: the wildcard rule belongs to the
archive.org_bot group; what McGill asks of other agents is
Crawl-delay: 5, which the first run did not honor. A
prediction that all 36 blocks were false was also wrong (14 of 36 were).
Whether the 403s depend on our user agent or on the server's address is
not yet known.
6.6. Rebuilt scripts (committed ee511a4, 2026-11-08):
u15data_robots_v1_1.py (standards-following reader,
unreadable robots.txt counts as disallowed, text saved),
u15data_scheduler_v1_0.py (rotation across sites with
per-site not-before times, retry later, Retry-After honored),
discover_faculty_directories_v1.3.py (each redirect step
robots-checked; each faculty's result file states what happened,
including the reason for blocked or failed; everything in a time-stamped
run folder), summarize_faculty_candidates_v1.1.py (reads
that status; --run-dir, --unit-ids-file),
check_faculty_directories_v1.4.py (per-hop robots checks,
hops column, robots text saved). All were tested offline
against local test servers (normal pages, 404, unreadable robots.txt, a
disallowed page, a redirect into a disallowed area, retries) and have
been run on real sites only in the re-run described next.
6.7. Re-run of 2026-11-08 afternoon, finished. 18 faculties (Ottawa
9, Waterloo 5, Laval FAAAD, Montréal Médecine, UBC Land and Food
Systems, UBC Medicine) through discovery v1.3, run folder
discovery_output/runs/2026-11-08_155845_discovery_v1.3/,
15:58 to 16:39 UTC (Ottawa and Waterloo have no Crawl-delay and ran at
the 2-second floor; Ottawa faculties took 2 to 4 minutes each). Result:
16 faculties loaded; Montréal Médecine and UBC Medicine were recorded as
blocked because their robots.txt could not be read. Summary v1.1: EASY
11, CHECK 5, WORK 2 (Ottawa 4 EASY and 5 CHECK; Waterloo 5 EASY; Laval 1
EASY; UBC Land and Food 1 EASY; Montréal and UBC Medicine WORK).
6.8. Checker v1.4 on that summary (run folder
discovery_output/checks/2026-11-08_164614/, 16:46 to 16:53
UTC): 30 addresses, 16 faculties. Seven faculties received a
LIKELY_DIRECTORY proposal needing no person: Ottawa Common Law, Ottawa
Social Sciences and all five Waterloo faculties (Waterloo Arts to be
looked at first: its address ends in faculty-and-staff-resources). Eight
are PARTIAL_OR_UNSURE and need a person, and some proposals are clearly
wrong or too broad: Ottawa Arts proposes a news article; Ottawa
Education, Health Sciences and Science all propose the same
university-wide page about-us/faculty-staff; Laval FAAAD proposes a
dean's-office page. Montréal Médecine, UBC Medicine and UBC Land and
Food Systems have no proposal.
6.9. A weakness found and fixed. UBC Land and Food Systems' best address was marked ROBOTS_DISALLOWED in that checker run although the site's robots.txt had been read normally (HTTP 200, only /wp-admin/ and two bot rules disallowed) in the discovery run 30 minutes earlier. The saved snapshot of the checker run shows "Connection aborted: Remote end closed connection without response": one dropped connection, kept by the robots module for the whole run. Robots module v1.2 now retries a failed read (3 attempts, waits of 3 and 10 seconds) and re-reads a site that stayed unreadable after 10 minutes if asked about it again; scheduler v1.1, discovery v1.4 and checker v1.5 use it (committed cd86a8c). A later curl test (3 tries each, 4 seconds apart) found that both UBC hosts (www.med.ubc.ca and www.landfood.ubc.ca) now reset every connection (curl exit 56), and that www.med.umontreal.ca no longer resolves (curl exit 6). Land and Food Systems had answered at 15:58 and 16:12 UTC, so the change happened after our runs; the cause is not known and nothing was done to get around it. Both UBC faculties are on the retry list (ids 181 and 179, no earlier than the next day, one robots.txt request per host first); Montréal Médecine needs a correct address from a person.
6.10. Import into the database (2026-11-08 18:34 UTC,
import_directory_results_v1.1.py, dry run first, report in
discovery_output/imports/2026-11-08_183404/; committed
d5986ee). Loaded: both checker runs (318 and 30 address rows), every
other address that the two discovery runs found (590 and 33 rows, status
proposed), 1 on-file address nobody had checked (proposed), and 158
robots.txt snapshots. Result in faculty_directory_pages:
972 rows for 166 of the 193 faculties (ai_likely 212, ai_unsure 110,
ai_rejected 22, manual_needed 4, proposed 624), 972 event rows, 8
collection_runs rows. These count addresses, not faculties.
The status rule is in the architecture document, Section 5.14. The 27
faculties with no row are those with no address at all (manual route and
others); their list is in the pending-items document.
6.11. Still open: the correct address for Montréal Médecine; the UBC retry; the manual route for the 22 robots-blocked faculties and for faculties without any address; review of the proposals by people; the 17 on-file addresses that the public status page already counts as confirmed although unverified.
| Version | Date | Change |
|---|---|---|
| 1.0 | 2026-09-26 | Initial version, written once all 15 institutions reached 275 confirmed faculty/school units with a home page each. |
| 1.1 | 2026-09-27 | Added Section 4, resolving the "275 vs 251" discrepancy discovered during the v1.21 unit-level reclassification pass. Updated Status line to reflect the reconciled total (191 true faculties out of 259 rows). |
| 1.2 | 2026-11-08 | Corrected the faculty count to 193 and the split to 193 / 47 / 19 (Section 4.6). Added Section 6: the 2026-11-08 faculty-directory discovery and checking, the flaws found in the first scripts, the robots audit and its findings, the rebuilt scripts and the 18-faculty re-run. |
| 1.3 | 2026-11-08 | Section 6 completed: re-run results (6.7), checker results and proposals (6.8), the dropped-connection weakness and robots module v1.2 and the UBC connection resets (6.9), the import of 972 directory-page rows and 158 robots.txt snapshots (6.10), open items (6.11). |