Version: 1.7 Status: Institution-level robots.txt pass fully resolved
for all 15 institutions (2026-09-22). Subdomain-specific follow-up
checks for 6 institutions completed 2026-09-26 (Section 4) — 1 flagged
for human decision (McMaster), 1 connection failure needing retry
(Calgary), 1 confirmed hard technical block (UBC). Loaded into the
institutions table in u15data_db. This
document is the narrative/methodology record; the database is the
authoritative source for the data itself. Faculty-level fallback pass
and Terms of Use review not yet started. Last updated: 2026-09-26
1.1. Checks are run from a local machine, not the uOttawa server or any institutional infrastructure, to avoid IP-reputation risk to institutional systems (see architecture document Section 6.4).
1.2. robots.txt is fetched via curl.exe with a standard
browser user-agent string, since some institutions’ CDN/WAF
configurations (e.g., CloudFront) block requests carrying curl’s default
user-agent, which is not itself evidence of bot-blocking intent.
1.3. Checked in batches of 5, results pasted back and logged here.
1.4. This pass is institution-level only (one robots.txt check per main domain). It does not yet confirm whether individual faculties/units sit on separate subdomains with their own robots.txt, or what each institution’s Terms of Use says — both are follow-up work, not covered by this table.
| # | Institution | Domain checked | robots.txt result | Notable findings |
|---|---|---|---|---|
| 1 | University of Alberta | ualberta.ca | Retrieved — has_rules_permissive |
Blocks a few specific paths (homepage-archive, facilities, staging, one dean-test page). Nothing broadly blocking directory content. |
| 2 | University of British Columbia | ubc.ca | Retrieved — has_rules_restrictive |
Standard admin-path blocks, plus
Disallow: /okanagan/facultystaff/bba — an explicit, named
block on a faculty/staff path, scoped to Okanagan. |
| 3 | University of Calgary | ucalgary.ca | has_rules_permissive (resolved 2026-09-22) |
Standard Drupal robots.txt. /profiles/ confirmed as
Drupal core’s default installation-profiles directory (verified against
Drupal’s own source repository), not staff/user profile content — not
directory-relevant. |
| 4 | Dalhousie University | dal.ca | Retrieved — has_rules_permissive |
Minimal — only blocks search-results query pages. No directory-relevant restriction found. |
| 5 | Université Laval | ulaval.ca | has_rules_permissive (resolved 2026-09-22) |
Same Drupal pattern as Calgary. /profiles/ confirmed as
Drupal core boilerplate, not directory-relevant. |
| 6 | University of Manitoba | umanitoba.ca | has_rules_permissive (resolved 2026-09-22) |
Same Drupal /profiles/ pattern, confirmed as core
boilerplate. Also lists dozens of per-faculty sitemap URLs (engineering,
law, medicine, nursing, etc.) — useful for directory_url
mapping. |
| 7 | McGill University | mcgill.ca | has_rules_restrictive (confirmed 2026-09-22) |
**Disallow: /*/people/* andDisallow:
//events/— wildcard, sitewide blocks**, not scoped to one unit. **Confirmed with direct evidence**:mcgill.ca/newsroom/peopleis a real, live faculty-expert directory (name, title, department per person), sitting exactly under a/peoplepath — matching the blocked pattern precisely. Not a false alarm; a genuine, directory-relevant restriction. Crawl-delay: 5 for all other bots. McGill is a strong candidate for the manual collection pathway (architecture document Section 5.8) rather than automated. | | 8 | McMaster University | mcmaster.ca | Retrieved —has_rules_permissive| Older-style robots.txt (legacy/admin paths). No directory-specific restriction found. | | 9 | Université de Montréal | umontreal.ca | Retrieved —empty_file| File exists (HTTP 200) butDisallow:has no value, meaning no rules restrict anything. | | 10 | University of Ottawa | uottawa.ca |has_rules_permissive(resolved 2026-09-22) | Same Drupal/profiles/pattern, confirmed as core boilerplate. Also lists per-faculty sitemap URLs (Arts, Education, Engineering, Health Sciences, Law, Medicine, Science, Social Sciences) — useful fordirectory_urlmapping. | | 11 | Queen's University | queensu.ca | Retrieved —has_rules_permissive| Standard pattern, Crawl-delay: 10. Blocks/facultyrecruitment/(reads as recruitment info, not a staff directory). No directory-specific restriction found. | | 12 | University of Saskatchewan | usask.ca | Retrieved —has_rules_permissive| Minimal — blocks cgi-bin, test paths, sandbox/codebase environments only. No directory-relevant restriction found. | | 13 | University of Toronto | utoronto.ca |has_rules_permissive(resolved 2026-09-22) | Same Drupal/profiles/ambiguity as Calgary/Laval/Manitoba/Ottawa, now confirmed as core boilerplate. One sitemap URL listed (not per-faculty). | | 14 | University of Waterloo | uwaterloo.ca | Confirmed —no_file(HTTP 404) | No robots.txt exists at this domain. Distinct from Montréal'sempty_fileeven though both currently mean no stated restriction — see architecture document Section 5.1 for why the distinction is kept. | | 15 | Western University | uwo.ca |has_rules_restrictive— still unconfirmed at path level | **Disallow:
/grad/faculty_staff/andDisallow:
/hr/facultystaff/** — explicit, literal blocks on paths named "faculty_staff" / "facultystaff." **Checked 2026-09-22, inconclusive**: nothing is indexed by search engines at these specific paths — mildly consistent with real blocked content existing there (search engines don't index disallowed paths), but not confirmation, since Western's robots.txt clearly contains other stale entries from as far back as 2005–2006 and these two could be similarly abandoned. Did find a real, current, unblocked staff-coordinator directory atuwo.ca/westerndir/coordinators/`
— not the path in question, but useful for a later central-directory
pass at Western. |
3.1. Resolved 2026-09-22: “/profiles/” confirmed, via Drupal’s own source code repository, as Drupal core’s default “installation profiles” directory — standard boilerplate present in Drupal’s default robots.txt template across many versions, unrelated to staff/user profile pages. Applied to all 5 affected institutions (Calgary, Laval, Manitoba, Toronto, Ottawa).
3.2. Resolved 2026-09-22: confirmed via direct evidence that McGill’s
staff/faculty directory does live under a “people” URL path
(mcgill.ca/newsroom/people).
Disallow: /*/people/* is a genuine, sitewide,
directory-relevant block. McGill is a strong candidate for the manual
collection pathway (architecture document Section 5.8) rather than
automated.
3.3. This table covers robots.txt only. Terms of Use review, and the faculty/unit-level pass (per-faculty directory URLs and their own possible robots.txt exceptions), are separate follow-up work not yet started.
3.4. Waterloo confirmed via HTTP status-code check (404 — no robots.txt file exists). Resolved; no further follow-up needed on this point.
3.5. Checked 2026-09-22, still inconclusive: Western’s
faculty_staff / facultystaff path blocks are
the strongest-worded restriction found in this pass, but a search pass
could not confirm what’s actually at those paths — nothing is indexed
there, which is mildly suggestive but not proof, given Western’s
robots.txt contains other clearly stale entries from 2005–2006. Treating
this the same as before pending firmer confirmation: a strong candidate
for the manual collection pathway rather than automated, but not yet
fully confirmed the way McGill now is.
5.1. Context: while attempting to set
automated_collection_allowed for institutions with a
confirmed central directory, it became clear that Phase 1’s original
robots.txt checks (Section 2) were run against each institution’s main
domain (e.g. ualberta.ca), not the specific subdomain
actually hosting the directory (e.g. apps.ualberta.ca) —
robots.txt is per-domain, so the main-domain result does not establish
permission for the directory subdomain itself. This section checks the
actual directory subdomains directly, dated separately from the original
Phase 1 pass since it happened four days later, on the server’s own
clock.
5.2. Methodology: same as Section 1 — checks run from a local machine
via curl.exe with a standard browser user-agent, one
command per subdomain.
| # | Institution | Subdomain checked | Result | Notes |
|---|---|---|---|---|
| 1 | University of Alberta | apps.ualberta.ca | has_rules_permissive |
Blocks /catalogue/archive,
/catalogue/syllabus, /ezsend,
/saml2, plus a few named SEO bots (MegaIndex.ru,
DataForSeoBot, BLEXBot) entirely. Nothing blocking
/directory. |
| 2 | University of Calgary | contacts.ucalgary.ca | Connection failed (curl exit code
000) |
Not a real HTTP response — 000 means the request never
completed (DNS failure, connection refused, timeout, or TLS handshake
failure; specific cause not determined). Distinct from both
no_file (404) and empty_file (200, blank).
Needs a retry or different diagnostic approach before concluding
anything. |
| 3 | McMaster University | directories.mcmaster.ca | Rules exist; wildcard is permissive, but explicitly blocks named AI-company crawlers | General User-agent: * only blocks
/wp/wp-admin/. Separately, explicitly disallows a long list
of named bots including ClaudeBot and
anthropic-ai by name, plus GPTBot,
Google-Extended, PerplexityBot, Bytespider, and others. Flagged for a
human decision on automated_collection_allowed rather than
auto-resolved — the wildcard technically permits an unnamed scraper, but
the clear intent is to keep AI-company crawling out. |
| 4 | Dalhousie University | directory.dal.ca | no_file (404) |
No robots.txt at this subdomain. Does not change the explicit usage-terms prohibition already captured on this same page (architecture document Section 4.6). |
| 5 | University of British Columbia | directory.ubc.ca | Active bot-defense block | Request did not reach robots.txt at all — returned a
CAPTCHA/bot-defense challenge page (commercial WAF, “BOTDEFENSE”/“TSPD”
markers), explicitly stating unusual/automated request volume was
detected. A real technical barrier to automated access, independent of
any robots.txt content. automated_collection_allowed set to
no on this basis — not a judgment call, the site itself is
actively refusing automated requests. |
| 6 | University of Waterloo | iamtools.uwaterloo.ca | no_file (404-style “Not Found”) |
No robots.txt at this subdomain. Still open whether the directory page itself is public or requires WatIAM login — the site’s own page bundles “Campus People Directory” and “Authenticated WatIAM Search” under the same link. |
5.3. URLs newly confirmed during this pass: UBC’s actual directory
access point is http://www.directory.ubc.ca/index.cfm
(found via the it.ubc.ca service page, which explicitly
states “Anyone is free to use the online directory” — distinct from
Toronto’s login-gated equivalent). Waterloo’s is
https://iamtools.uwaterloo.ca/directory.
| Version | Date | Change |
|---|---|---|
| 1.0 | 2026-09-22 | Initial log — institution-level robots.txt findings for 10 of 15 institutions |
| 1.1 | 2026-09-22 | Added remaining 5 institutions (Queen’s, Saskatchewan, Toronto, Waterloo, Western); institution-level robots.txt pass now complete except Waterloo, pending a status-code follow-up check (Section 3.4); flagged Western’s explicit faculty_staff/facultystaff path blocks as the strongest restriction found |
| 1.2 | 2026-09-22 | Waterloo confirmed via HTTP status check — 404, no robots.txt file exists. Institution-level robots.txt pass now complete for all 15 institutions. |
| 1.3 | 2026-09-22 | Re-coded all 15 findings against the precise
robots_txt_status values now defined in architecture
document Section 5.1 (no_file / empty_file /
has_rules_permissive / has_rules_restrictive),
so Waterloo’s no_file and Montréal’s
empty_file are recorded as distinct facts rather than both
collapsing into “no restriction” |
| 1.4 | 2026-09-22 | Findings loaded into the institutions table in
u15data_db (Section 5.1 of architecture document); this
document now serves as the methodology/reasoning record rather than the
primary data store; caught and corrected an accidental duplicate insert
of 7 rows during loading |
| 1.5 | 2026-09-22 | Resolved the /profiles/ ambiguity (Section 3.1) for all
5 affected institutions (Calgary, Laval, Manitoba, Toronto, Ottawa):
confirmed via Drupal’s own source repository as core CMS boilerplate,
not staff/user profile content. All 5 re-coded to
has_rules_permissive in the database and in this log.
Institution-level robots.txt pass now fully resolved for all 15
institutions, no open items remain at this level. |
| 1.6 | 2026-09-22 | Confirmed McGill’s /*/people/* block with direct
evidence — a real, live faculty directory exists at
mcgill.ca/newsroom/people, matching the blocked pattern
precisely. Checked Western’s
faculty_staff/facultystaff blocks but could
not confirm what’s actually at those paths; found a separate, unblocked
staff-coordinator directory
(uwo.ca/westerndir/coordinators/) useful for later
central-directory work at Western, but the original question remains
open. |
| 1.7 | 2026-09-26 | Added Section 4: subdomain-specific robots.txt checks for 6
institutions whose confirmed directory lives on a different subdomain
than Phase 1 originally checked. Alberta confirmed permissive at the
correct subdomain. UBC found to actively block automated requests with a
bot-defense CAPTCHA challenge —
automated_collection_allowed set to no.
McMaster’s directory subdomain explicitly blocks named AI-company
crawlers (including ClaudeBot and anthropic-ai by name) though its
wildcard rule is technically permissive — flagged for a human decision
rather than auto-resolved. Calgary’s check failed to connect entirely
(curl 000), a genuinely different result from both
no_file and empty_file, not yet explained.
Dalhousie’s directory subdomain has no robots.txt (404). Waterloo’s
directory subdomain has no robots.txt either; whether the directory page
itself requires login remains unconfirmed. Newly confirmed URLs: UBC
(www.directory.ubc.ca/index.cfm) and Waterloo
(iamtools.uwaterloo.ca/directory). |