bengarves.com/work/noindex-faceted-urls
Page-Level noindex for Non-Pagination Faceted URLs
[SEO-38] Topic/Cat/Subcat: Set robots meta of noindex for faceted URLs
From Ben Garves’s work at Udemy. Re-created from the original document, with people’s names, internal links and non-public numbers removed or generalized.
StorySEO-38
- Company
- Udemy
- Created
- Resolved
- Status
- Done
- Priority
- P2 - Medium
- Story points
- 3
- Epic
- Crawl Efficiency Cleanup (SEO-4)
- Project
- SEO Squad
- Sprint
- SEO Sprint 2
- Cloned from
- SEO-33, the same change for course landing pages (Done)
- Requirement source
- Engineering
A story to add a robots noindex tag to Topic, Category and Subcategory URLs that carry non-pagination parameters, such as UTM or coupon codes, so they stop competing with the original page in Google, while URLs whose only parameter is pagination stay indexable.
- Technical SEO
- Indexing
- Faceted navigation
- Crawl efficiency
User story
As an SEO, I’d like pages with non-pagination facets in their URL to defer their canonical to the original URL so that Google doesn’t view them as competitors of the original URL.
Subset: Topic/Category/Subcategory pages
Background
Google has indexed a number of dotcom URLs with non-pagination facets (e.g. UTM codes, coupon codes, etc.). This ticket seeks to roll out functionality that will automatically defer the canonical of the page if the URL has a facet that isn’t for pagination. This is for Topic, Category, and Subcategory pages.
html
<meta name="robots" content="noindex" />Acceptance criteria
- IF the URL has a facet AND the facet is not for pagination, add noindex
- IF the URL has a facet AND the only facet is for pagination, allow the page to index
- IF the URL has multiple facets AND one is for pagination, add noindex
Examples
Below are examples of indexed pages with URL facets that aren’t pagination:
- https://www.udemy.com/join/login-popup/?locale=en_US&response_type=html&next=https%3A%2F%2Fwww.udemy.com%2F
- https://www.udemy.com/organization/global-login/email?ref=ufb_header&locale=ja_JP
- https://www.udemy.com/?matchtype=e
- https://www.udemy.com/affiliate/?locale=pt_BR
- https://www.udemy.com/course/es-curso-de-certificacion-profesional-en-ingenieria-de-ia/?couponCode=JAN_FREE_03
- https://www.udemy.com/course/bootcamp-de-devops-a-mlops-transicion-hacia-la-ingenieria-p/?couponCode=JAN_FREE_03
- https://www.udemy.com/?persist_locale&locale=de_DE
This isn’t a comprehensive list, but here are many examples of facets we commonly see added to Udemy URLs:
| Facet type | Parameters |
|---|---|
| Search, Filters and Sorting | topic, product, date_filter, taught_courses, price, q |
| Couponing | couponCode |
| Localization | Language, persist_locale |
| Experimentation | test |
| UTM Codes | utm_term, utm_source, utm_medium, utm_content, utm_campaign, utm_tactic, utm_audience |
| Other Attribution | gclid, xref, ref, src, srsltid, campaigntype |
| Unknown | priority, portfolio, matchtype, gbraid, gad_source, gad_campaignad, dtcode, path, pmtag, size, sl, subscription-id, amp, boType, course_id, data_scope, funnel, mx_pg, placement, product_type |