bengarves.com/work/update-robots-txt
Update robots.txt File
[CVR-2231] Replace DotCom robots.txt Contents
From Ben Garves’s work at Udemy. Re-created from the original document, with people’s names, internal links and non-public numbers removed or generalized.
TaskCVR-2231
- Company
- Udemy
- Created
- Resolved
- Status
- Done
- Priority
- P1 - High
- Story points
- 1
- Epic
- 2026 robots.txt Updates (CVR-2230)
- Project
- FE Home Squad
- Sprint
- CVR - Sprint 52
- Requirement source
- Engineering
A one-point task to replace udemy.com’s robots.txt with a revised version that lets Baidu crawl at a limited rate, lets LinkedIn’s crawler reach completion certificates that the catch-all rules block, and fixes the faceted-URL wildcard while removing redundant rules.
- Technical SEO
- robots.txt
- Crawl efficiency
Acceptance criteria
Replace current udemy.com robots.txt file contents with this:
#2026-03-16: Allow Baidu but limit crawl frequency
User-agent: Baiduspider
Crawl-delay: 10
#Allow LinkedIn only to see completion certificates
User-agent: LinkedInBot
Allow: /certificate/
#2026-03-16: Fix faceted URL wildcard, remove redundancies.
User-Agent: *
Disallow: /404/
Disallow: /*?/
Disallow: /api-2.0/
Disallow: /api/
Disallow: /buy/
Disallow: /cart/
Disallow: /category/
Disallow: /certificate/
Disallow: /collection/
Disallow: /course/reviews-popup/
Disallow: /curated/
Disallow: /dashboard/
Disallow: /draft/Note
The exported ticket shows the file only through Disallow: /draft/. Any rules after that line are not reproduced here.