u15data Phase 1 Audit Log

u15data — Phase 1 Audit Log (Institution-Level Pass)

Version: 1.7 Status: Institution-level robots.txt pass fully resolved for all 15 institutions (2026-09-22). Subdomain-specific follow-up checks for 6 institutions completed 2026-09-26 (Section 4) — 1 flagged for human decision (McMaster), 1 connection failure needing retry (Calgary), 1 confirmed hard technical block (UBC). Loaded into the institutions table in u15data_db. This document is the narrative/methodology record; the database is the authoritative source for the data itself. Faculty-level fallback pass and Terms of Use review not yet started. Last updated: 2026-09-26

1. Methodology

1.1. Checks are run from a local machine, not the uOttawa server or any institutional infrastructure, to avoid IP-reputation risk to institutional systems (see architecture document Section 6.4).

1.2. robots.txt is fetched via curl.exe with a standard browser user-agent string, since some institutions’ CDN/WAF configurations (e.g., CloudFront) block requests carrying curl’s default user-agent, which is not itself evidence of bot-blocking intent.

1.3. Checked in batches of 5, results pasted back and logged here.

1.4. This pass is institution-level only (one robots.txt check per main domain). It does not yet confirm whether individual faculties/units sit on separate subdomains with their own robots.txt, or what each institution’s Terms of Use says — both are follow-up work, not covered by this table.

2. Findings

# Institution Domain checked robots.txt result Notable findings
1 University of Alberta ualberta.ca Retrieved — has_rules_permissive Blocks a few specific paths (homepage-archive, facilities, staging, one dean-test page). Nothing broadly blocking directory content.
2 University of British Columbia ubc.ca Retrieved — has_rules_restrictive Standard admin-path blocks, plus Disallow: /okanagan/facultystaff/bba — an explicit, named block on a faculty/staff path, scoped to Okanagan.
3 University of Calgary ucalgary.ca has_rules_permissive (resolved 2026-09-22) Standard Drupal robots.txt. /profiles/ confirmed as Drupal core’s default installation-profiles directory (verified against Drupal’s own source repository), not staff/user profile content — not directory-relevant.
4 Dalhousie University dal.ca Retrieved — has_rules_permissive Minimal — only blocks search-results query pages. No directory-relevant restriction found.
5 Université Laval ulaval.ca has_rules_permissive (resolved 2026-09-22) Same Drupal pattern as Calgary. /profiles/ confirmed as Drupal core boilerplate, not directory-relevant.
6 University of Manitoba umanitoba.ca has_rules_permissive (resolved 2026-09-22) Same Drupal /profiles/ pattern, confirmed as core boilerplate. Also lists dozens of per-faculty sitemap URLs (engineering, law, medicine, nursing, etc.) — useful for directory_url mapping.
7 McGill University mcgill.ca has_rules_restrictive (confirmed 2026-09-22) **Disallow: /*/people/* andDisallow: //events/— wildcard, sitewide blocks**, not scoped to one unit. **Confirmed with direct evidence**:mcgill.ca/newsroom/peopleis a real, live faculty-expert directory (name, title, department per person), sitting exactly under a/peoplepath — matching the blocked pattern precisely. Not a false alarm; a genuine, directory-relevant restriction. Crawl-delay: 5 for all other bots. McGill is a strong candidate for the manual collection pathway (architecture document Section 5.8) rather than automated. | | 8 | McMaster University | mcmaster.ca | Retrieved —has_rules_permissive| Older-style robots.txt (legacy/admin paths). No directory-specific restriction found. | | 9 | Université de Montréal | umontreal.ca | Retrieved —empty_file| File exists (HTTP 200) butDisallow:has no value, meaning no rules restrict anything. | | 10 | University of Ottawa | uottawa.ca |has_rules_permissive(resolved 2026-09-22) | Same Drupal/profiles/pattern, confirmed as core boilerplate. Also lists per-faculty sitemap URLs (Arts, Education, Engineering, Health Sciences, Law, Medicine, Science, Social Sciences) — useful fordirectory_urlmapping. | | 11 | Queen's University | queensu.ca | Retrieved —has_rules_permissive| Standard pattern, Crawl-delay: 10. Blocks/facultyrecruitment/(reads as recruitment info, not a staff directory). No directory-specific restriction found. | | 12 | University of Saskatchewan | usask.ca | Retrieved —has_rules_permissive| Minimal — blocks cgi-bin, test paths, sandbox/codebase environments only. No directory-relevant restriction found. | | 13 | University of Toronto | utoronto.ca |has_rules_permissive(resolved 2026-09-22) | Same Drupal/profiles/ambiguity as Calgary/Laval/Manitoba/Ottawa, now confirmed as core boilerplate. One sitemap URL listed (not per-faculty). | | 14 | University of Waterloo | uwaterloo.ca | Confirmed —no_file(HTTP 404) | No robots.txt exists at this domain. Distinct from Montréal'sempty_fileeven though both currently mean no stated restriction — see architecture document Section 5.1 for why the distinction is kept. | | 15 | Western University | uwo.ca |has_rules_restrictive— still unconfirmed at path level | **Disallow: /grad/faculty_staff/andDisallow: /hr/facultystaff/** — explicit, literal blocks on paths named "faculty_staff" / "facultystaff." **Checked 2026-09-22, inconclusive**: nothing is indexed by search engines at these specific paths — mildly consistent with real blocked content existing there (search engines don't index disallowed paths), but not confirmation, since Western's robots.txt clearly contains other stale entries from as far back as 2005–2006 and these two could be similarly abandoned. Did find a real, current, unblocked staff-coordinator directory atuwo.ca/westerndir/coordinators/` — not the path in question, but useful for a later central-directory pass at Western.

3. Follow-up items raised by findings so far

3.1. Resolved 2026-09-22: “/profiles/” confirmed, via Drupal’s own source code repository, as Drupal core’s default “installation profiles” directory — standard boilerplate present in Drupal’s default robots.txt template across many versions, unrelated to staff/user profile pages. Applied to all 5 affected institutions (Calgary, Laval, Manitoba, Toronto, Ottawa).

3.2. Resolved 2026-09-22: confirmed via direct evidence that McGill’s staff/faculty directory does live under a “people” URL path (mcgill.ca/newsroom/people). Disallow: /*/people/* is a genuine, sitewide, directory-relevant block. McGill is a strong candidate for the manual collection pathway (architecture document Section 5.8) rather than automated.

3.3. This table covers robots.txt only. Terms of Use review, and the faculty/unit-level pass (per-faculty directory URLs and their own possible robots.txt exceptions), are separate follow-up work not yet started.

3.4. Waterloo confirmed via HTTP status-code check (404 — no robots.txt file exists). Resolved; no further follow-up needed on this point.

3.5. Checked 2026-09-22, still inconclusive: Western’s faculty_staff / facultystaff path blocks are the strongest-worded restriction found in this pass, but a search pass could not confirm what’s actually at those paths — nothing is indexed there, which is mildly suggestive but not proof, given Western’s robots.txt contains other clearly stale entries from 2005–2006. Treating this the same as before pending firmer confirmation: a strong candidate for the manual collection pathway rather than automated, but not yet fully confirmed the way McGill now is.

4. Subdomain-specific follow-up checks (2026-09-26)

5.1. Context: while attempting to set automated_collection_allowed for institutions with a confirmed central directory, it became clear that Phase 1’s original robots.txt checks (Section 2) were run against each institution’s main domain (e.g. ualberta.ca), not the specific subdomain actually hosting the directory (e.g. apps.ualberta.ca) — robots.txt is per-domain, so the main-domain result does not establish permission for the directory subdomain itself. This section checks the actual directory subdomains directly, dated separately from the original Phase 1 pass since it happened four days later, on the server’s own clock.

5.2. Methodology: same as Section 1 — checks run from a local machine via curl.exe with a standard browser user-agent, one command per subdomain.

# Institution Subdomain checked Result Notes
1 University of Alberta apps.ualberta.ca has_rules_permissive Blocks /catalogue/archive, /catalogue/syllabus, /ezsend, /saml2, plus a few named SEO bots (MegaIndex.ru, DataForSeoBot, BLEXBot) entirely. Nothing blocking /directory.
2 University of Calgary contacts.ucalgary.ca Connection failed (curl exit code 000) Not a real HTTP response — 000 means the request never completed (DNS failure, connection refused, timeout, or TLS handshake failure; specific cause not determined). Distinct from both no_file (404) and empty_file (200, blank). Needs a retry or different diagnostic approach before concluding anything.
3 McMaster University directories.mcmaster.ca Rules exist; wildcard is permissive, but explicitly blocks named AI-company crawlers General User-agent: * only blocks /wp/wp-admin/. Separately, explicitly disallows a long list of named bots including ClaudeBot and anthropic-ai by name, plus GPTBot, Google-Extended, PerplexityBot, Bytespider, and others. Flagged for a human decision on automated_collection_allowed rather than auto-resolved — the wildcard technically permits an unnamed scraper, but the clear intent is to keep AI-company crawling out.
4 Dalhousie University directory.dal.ca no_file (404) No robots.txt at this subdomain. Does not change the explicit usage-terms prohibition already captured on this same page (architecture document Section 4.6).
5 University of British Columbia directory.ubc.ca Active bot-defense block Request did not reach robots.txt at all — returned a CAPTCHA/bot-defense challenge page (commercial WAF, “BOTDEFENSE”/“TSPD” markers), explicitly stating unusual/automated request volume was detected. A real technical barrier to automated access, independent of any robots.txt content. automated_collection_allowed set to no on this basis — not a judgment call, the site itself is actively refusing automated requests.
6 University of Waterloo iamtools.uwaterloo.ca no_file (404-style “Not Found”) No robots.txt at this subdomain. Still open whether the directory page itself is public or requires WatIAM login — the site’s own page bundles “Campus People Directory” and “Authenticated WatIAM Search” under the same link.

5.3. URLs newly confirmed during this pass: UBC’s actual directory access point is http://www.directory.ubc.ca/index.cfm (found via the it.ubc.ca service page, which explicitly states “Anyone is free to use the online directory” — distinct from Toronto’s login-gated equivalent). Waterloo’s is https://iamtools.uwaterloo.ca/directory.

5. Version history

Version Date Change
1.0 2026-09-22 Initial log — institution-level robots.txt findings for 10 of 15 institutions
1.1 2026-09-22 Added remaining 5 institutions (Queen’s, Saskatchewan, Toronto, Waterloo, Western); institution-level robots.txt pass now complete except Waterloo, pending a status-code follow-up check (Section 3.4); flagged Western’s explicit faculty_staff/facultystaff path blocks as the strongest restriction found
1.2 2026-09-22 Waterloo confirmed via HTTP status check — 404, no robots.txt file exists. Institution-level robots.txt pass now complete for all 15 institutions.
1.3 2026-09-22 Re-coded all 15 findings against the precise robots_txt_status values now defined in architecture document Section 5.1 (no_file / empty_file / has_rules_permissive / has_rules_restrictive), so Waterloo’s no_file and Montréal’s empty_file are recorded as distinct facts rather than both collapsing into “no restriction”
1.4 2026-09-22 Findings loaded into the institutions table in u15data_db (Section 5.1 of architecture document); this document now serves as the methodology/reasoning record rather than the primary data store; caught and corrected an accidental duplicate insert of 7 rows during loading
1.5 2026-09-22 Resolved the /profiles/ ambiguity (Section 3.1) for all 5 affected institutions (Calgary, Laval, Manitoba, Toronto, Ottawa): confirmed via Drupal’s own source repository as core CMS boilerplate, not staff/user profile content. All 5 re-coded to has_rules_permissive in the database and in this log. Institution-level robots.txt pass now fully resolved for all 15 institutions, no open items remain at this level.
1.6 2026-09-22 Confirmed McGill’s /*/people/* block with direct evidence — a real, live faculty directory exists at mcgill.ca/newsroom/people, matching the blocked pattern precisely. Checked Western’s faculty_staff/facultystaff blocks but could not confirm what’s actually at those paths; found a separate, unblocked staff-coordinator directory (uwo.ca/westerndir/coordinators/) useful for later central-directory work at Western, but the original question remains open.
1.7 2026-09-26 Added Section 4: subdomain-specific robots.txt checks for 6 institutions whose confirmed directory lives on a different subdomain than Phase 1 originally checked. Alberta confirmed permissive at the correct subdomain. UBC found to actively block automated requests with a bot-defense CAPTCHA challenge — automated_collection_allowed set to no. McMaster’s directory subdomain explicitly blocks named AI-company crawlers (including ClaudeBot and anthropic-ai by name) though its wildcard rule is technically permissive — flagged for a human decision rather than auto-resolved. Calgary’s check failed to connect entirely (curl 000), a genuinely different result from both no_file and empty_file, not yet explained. Dalhousie’s directory subdomain has no robots.txt (404). Waterloo’s directory subdomain has no robots.txt either; whether the directory page itself requires login remains unconfirmed. Newly confirmed URLs: UBC (www.directory.ubc.ca/index.cfm) and Waterloo (iamtools.uwaterloo.ca/directory).