u15data — Faculty Discovery Log

Version: 1.3 Status: All 15 institutions have real faculty/school-level data, and (as of v1.1) every row has been classified as faculty, sub_unit, or excluded (schema v1.21). The true count of top-level faculties across all 15 institutions is 193 (corrected in v1.2; v1.1 said 191), out of 259 total confirmed rows (193 faculty, 47 sub_unit, 19 excluded; v1.1 said 191, 41 and 27). Section 6 (new in v1.2) records the 2026-11-08 faculty-directory discovery and checking, and the robots.txt audit. directory_url (an actual staff-listing page) confirmed for a minority of faculty-level units so far. human_confirmed is false on all rows -- review pass not yet started. This document is the narrative/methodology record; the database (faculty_units, discovery_seeds, discovery_html_snapshots) is the authoritative source for the data itself. Last updated: 2026-11-08 (evening)

1. Methodology

1.1. Two-step process per institution: first find the institution's own "list of faculties" master page (a search task), then run discover_pages.py (architecture document Section 5.11) against that page to extract real links directly from the fetched HTML, rather than relying on search-engine snippets.

1.2. Every link the script finds is reviewed before being treated as a genuine faculty (filtering out navigation, social media, phone numbers, and similar noise) -- see Section 3 for specific cases where this mattered.

1.3. Where a seed page turned out to be JavaScript-rendered and the script returned empty or incomplete content (Alberta, and McGill's first attempt), the person visited the page directly in their own browser and pasted the real content back. This is treated as equally valid to an automated fetch.

1.4. Consistent with the project's standing no-guessing rule: a unit's home page and its confirmed staff-directory page are tracked as two separate fields (home_page_url, directory_url). A name without a confirmed URL is recorded honestly as unconfirmed (NULL), never filled with a plausible-looking guess.

2. Seeds used, by institution

(Unchanged from v1.0 -- see that version, or the live discovery_seeds table, for the full per-institution seed list and notes.)

3. Corrections made along the way

(Unchanged from v1.0, items 3.1 through 3.7 -- see that version for the full list of pre-script guessed-URL corrections and duplicate-row cleanups.)

4. Resolution of the v1.0 "275 vs 251" discrepancy

4.1. This v1.0 document originally stated a total of "275 units total, every one with a confirmed home_page_url." At the start of the unit-level reclassification work (v1.21 schema change), the live database was found to contain 251 rows, not 275 -- a 24-row gap that prompted a dedicated investigation (see the handover and unit-classification-sources documents for the full trail).

4.2. The investigation considered and ruled out several candidate explanations, including a suspected-but-unconfirmed silently-failed UBC insert (see item 3.7 above), before the real explanation emerged during the reclassification work itself: the original "275" figure, and the 251-row count checked against it, were both counting a mix of true top-level faculties and nested sub-units (schools, colleges, departments) as if they were peers. Some institutions' seed pages captured only top-level faculties; others (e.g. Waterloo, whose seed "mixes core Faculties and constituent Schools, both captured," per Section 1's original methodology note) captured both levels flattened into one list. Neither 275 nor 251 was ever a clean count of the same thing.

4.3. The reclassification pass (v1.21 schema: unit_level, parent_unit_id) resolved this by separating every row into faculty (true top-level, the project's actual unit of analysis), sub_unit (nested under a specific faculty, with parent_unit_id recorded), or excluded (real but not a faculty -- administrative bodies, historic affiliated colleges, and similar). This also surfaced and corrected a small number of genuine gaps missed by the original discovery passes (e.g. Dalhousie was missing Agriculture, Medicine, and Open Learning & Career Development; Universite Laval was missing FAAAD and FESP; Calgary's 13 rows all pointed to University Calendar pages instead of real faculty websites, and one whole faculty -- School of Public Policy -- had never been captured at all).

4.4. The reconciled total is 259 rows, of which 193 are true top-level faculties (stated as 191 in v1.1; see item 4.6). Full per-institution source citations for the reclassification work are in u15data_unit_classification_sources_v1.1.md.

4.5. The practical lesson, worth carrying forward: a raw row count is only meaningful once the unit-of-analysis is unambiguous. Future discovery or audit passes on this project should classify unit_level at the time of insertion, not after the fact, to avoid a repeat of this exact ambiguity.

4.6. Correction (v1.2, 2026-11-08). The 191 faculties stated above was an addition slip: the per-institution faculty counts (Alberta 17, Dalhousie 13, McMaster 6, Queen's 7, McGill 11, Calgary 14, Laval 17, Montréal 13, Manitoba 15, Ottawa 10, Saskatchewan 17, Toronto 17, Waterloo 6, Western 12, UBC 18) add to 193, and the live database holds 193. The split of the remaining 66 rows is 47 sub_unit and 19 excluded in the database; the 41 and 27 in v1.1 could not be traced to any source. The database is authoritative.

6. Faculty-directory discovery and checking, 2026-11-08 (new in v1.2)

6.1. Aim. For each of the 193 faculty-level units, find the page that lists its faculty members (the directory page), so that the people scraper (see u15data_people_collection_design) can later read it. The unit's home page and its directory page remain two separate fields (home_page_url, directory_url), as in item 1.4.

6.2. Method. discover_faculty_directories (v1.2 for the first full run) fetches each faculty's home page, saves the raw HTML, keeps links whose text or address look like a staff directory (English and French keywords), and probes 16 common directory paths inside the faculty's own section of the site. summarize_faculty_candidates (v1.0) scores and ranks the candidates and labels each faculty EASY, CHECK or WORK. check_faculty_directories (v1.3) then visits the best candidates and the address already on file and records, as counts only (no names, no emails, no page content), whether the page loads, looks like a list of people, is paginated, or is probably built by JavaScript, and proposes one address per faculty.

6.3. Results of the first full run (run folder discovery_output/runs/2026-11-08_run_v1.2/): of 193 faculties, 154 produced a result file, 36 were recorded as blocked by robots.txt and 3 failed to fetch (Laval FAAAD, Montréal Médecine, UBC Land and Food Systems). The checker run discovery_output/checks/2026-11-08_141658/ checked 318 addresses and wrote proposal rows for 157 faculties, 14 of them without a proposed address. Of the 15 faculties that already had a directory address on file and were checked, 6 redirect elsewhere and are probably stale.

6.4. Flaws found in the first scripts, and corrections. Stated plainly so later readers know what the first run did: a. The robots.txt reader was Python's built-in one, which ignores wildcards, longest-match and Crawl-delay (confirmed on the server's Python 3.12.3). b. Discovery treated an unreadable robots.txt as permission, and the checker followed redirects before checking the final site's robots.txt (one request went to an address that robots.txt disallows). c. Requests to one site were spaced 1 to 2 seconds apart whatever the site's Crawl-delay (McGill asks 5 seconds, Queen's 10). d. A 2 minute 16 second stall in the checker was probably a robots.txt read with no time limit. e. Results: the first checker version mislabelled an address on file that redirects to the proposed address, proposed an address for a faculty with no people on it, and over-flagged needs_person; all fixed in checker v1.3. f. The first version also waited a fixed 2 seconds after every request. It was replaced by rotation across sites, so waiting for one site never stops work on another.

6.5. Robots audit of 2026-11-08. audit_robots_v1.0.py (run folder discovery_output/robots_audit/2026-11-08_153739/) fetched robots.txt once per site, saved the text, and re-tested every address the discovery and checker runs had requested against the correct rules. Findings: of the 36 faculties recorded as blocked, 14 (Ottawa 9, Waterloo 5) are allowed under the correct reader and are being re-run; 22 remain, 19 where robots.txt returns 403 or cannot be reached (University of Alberta 17, McMaster Health Sciences, Toronto Arts & Science) and 3 with a real Disallow: / (McMaster Humanities, Laval Faculté des sciences de l'administration, Rotman). These 22 stay in the database with their addresses and go to a manual route; nothing is deleted. A first prediction that McGill's robots.txt had been violated was wrong: the wildcard rule belongs to the archive.org_bot group; what McGill asks of other agents is Crawl-delay: 5, which the first run did not honor. A prediction that all 36 blocks were false was also wrong (14 of 36 were). Whether the 403s depend on our user agent or on the server's address is not yet known.

6.6. Rebuilt scripts (committed ee511a4, 2026-11-08): u15data_robots_v1_1.py (standards-following reader, unreadable robots.txt counts as disallowed, text saved), u15data_scheduler_v1_0.py (rotation across sites with per-site not-before times, retry later, Retry-After honored), discover_faculty_directories_v1.3.py (each redirect step robots-checked; each faculty's result file states what happened, including the reason for blocked or failed; everything in a time-stamped run folder), summarize_faculty_candidates_v1.1.py (reads that status; --run-dir, --unit-ids-file), check_faculty_directories_v1.4.py (per-hop robots checks, hops column, robots text saved). All were tested offline against local test servers (normal pages, 404, unreadable robots.txt, a disallowed page, a redirect into a disallowed area, retries) and have been run on real sites only in the re-run described next.

6.7. Re-run of 2026-11-08 afternoon, finished. 18 faculties (Ottawa 9, Waterloo 5, Laval FAAAD, Montréal Médecine, UBC Land and Food Systems, UBC Medicine) through discovery v1.3, run folder discovery_output/runs/2026-11-08_155845_discovery_v1.3/, 15:58 to 16:39 UTC (Ottawa and Waterloo have no Crawl-delay and ran at the 2-second floor; Ottawa faculties took 2 to 4 minutes each). Result: 16 faculties loaded; Montréal Médecine and UBC Medicine were recorded as blocked because their robots.txt could not be read. Summary v1.1: EASY 11, CHECK 5, WORK 2 (Ottawa 4 EASY and 5 CHECK; Waterloo 5 EASY; Laval 1 EASY; UBC Land and Food 1 EASY; Montréal and UBC Medicine WORK).

6.8. Checker v1.4 on that summary (run folder discovery_output/checks/2026-11-08_164614/, 16:46 to 16:53 UTC): 30 addresses, 16 faculties. Seven faculties received a LIKELY_DIRECTORY proposal needing no person: Ottawa Common Law, Ottawa Social Sciences and all five Waterloo faculties (Waterloo Arts to be looked at first: its address ends in faculty-and-staff-resources). Eight are PARTIAL_OR_UNSURE and need a person, and some proposals are clearly wrong or too broad: Ottawa Arts proposes a news article; Ottawa Education, Health Sciences and Science all propose the same university-wide page about-us/faculty-staff; Laval FAAAD proposes a dean's-office page. Montréal Médecine, UBC Medicine and UBC Land and Food Systems have no proposal.

6.9. A weakness found and fixed. UBC Land and Food Systems' best address was marked ROBOTS_DISALLOWED in that checker run although the site's robots.txt had been read normally (HTTP 200, only /wp-admin/ and two bot rules disallowed) in the discovery run 30 minutes earlier. The saved snapshot of the checker run shows "Connection aborted: Remote end closed connection without response": one dropped connection, kept by the robots module for the whole run. Robots module v1.2 now retries a failed read (3 attempts, waits of 3 and 10 seconds) and re-reads a site that stayed unreadable after 10 minutes if asked about it again; scheduler v1.1, discovery v1.4 and checker v1.5 use it (committed cd86a8c). A later curl test (3 tries each, 4 seconds apart) found that both UBC hosts (www.med.ubc.ca and www.landfood.ubc.ca) now reset every connection (curl exit 56), and that www.med.umontreal.ca no longer resolves (curl exit 6). Land and Food Systems had answered at 15:58 and 16:12 UTC, so the change happened after our runs; the cause is not known and nothing was done to get around it. Both UBC faculties are on the retry list (ids 181 and 179, no earlier than the next day, one robots.txt request per host first); Montréal Médecine needs a correct address from a person.

6.10. Import into the database (2026-11-08 18:34 UTC, import_directory_results_v1.1.py, dry run first, report in discovery_output/imports/2026-11-08_183404/; committed d5986ee). Loaded: both checker runs (318 and 30 address rows), every other address that the two discovery runs found (590 and 33 rows, status proposed), 1 on-file address nobody had checked (proposed), and 158 robots.txt snapshots. Result in faculty_directory_pages: 972 rows for 166 of the 193 faculties (ai_likely 212, ai_unsure 110, ai_rejected 22, manual_needed 4, proposed 624), 972 event rows, 8 collection_runs rows. These count addresses, not faculties. The status rule is in the architecture document, Section 5.14. The 27 faculties with no row are those with no address at all (manual route and others); their list is in the pending-items document.

6.11. Still open: the correct address for Montréal Médecine; the UBC retry; the manual route for the 22 robots-blocked faculties and for faculties without any address; review of the proposals by people; the 17 on-file addresses that the public status page already counts as confirmed although unverified.

5. Version history

Version Date Change
1.0 2026-09-26 Initial version, written once all 15 institutions reached 275 confirmed faculty/school units with a home page each.
1.1 2026-09-27 Added Section 4, resolving the "275 vs 251" discrepancy discovered during the v1.21 unit-level reclassification pass. Updated Status line to reflect the reconciled total (191 true faculties out of 259 rows).
1.2 2026-11-08 Corrected the faculty count to 193 and the split to 193 / 47 / 19 (Section 4.6). Added Section 6: the 2026-11-08 faculty-directory discovery and checking, the flaws found in the first scripts, the robots audit and its findings, the rebuilt scripts and the 18-faculty re-run.
1.3 2026-11-08 Section 6 completed: re-run results (6.7), checker results and proposals (6.8), the dropped-connection weakness and robots module v1.2 and the UBC connection resets (6.9), the import of 972 directory-page rows and 158 robots.txt snapshots (6.10), open items (6.11).