bengarves.com/work/ai-overviews
AI Innovation - Overviews
Project One Pager: AI Overviews on CLPs and Topic Pages
From Ben Garves’s work at Udemy. Re-created from the original document, with people’s names, internal links and non-public numbers removed or generalized.
Project brief
- Company
- Udemy
- Format
- Project one-pager
- Product
- Ben Garves
- Teams
- Product, DS/ML, Engineering, Design, SEO
- Prioritization
- RICE
- Page types
- Topic pages, course landing pages (CLPs)
Proposes LLM-generated AI overviews in the header of Topic pages and course landing pages (CLPs) to help learners judge fit and navigate, and to give search and AI crawlers indexable content that dynamic and lazy-loaded sections hide.
- AI
- Content modules
- Indexing
- Technical SEO
Background
AI Overviews (also called AI Summaries) are concise, LLM-generated summaries that distill complex information into actionable insights for users. These overviews leverage natural language processing to synthesize content from multiple sources (including course descriptions, reviews, curriculum details, and topic information) into digestible, structured summaries that appear prominently on the page.
The intent behind adding these summaries to pages would be to:
- Help a user confirm they’ve landed on the content they want
- Help a user understand more about the value proposition of Udemy and/or a specific course
- Help a user navigate on-site to relevant content if they deem the course or topic they’ve landed on to be not an ideal match
Who is the user?
- Topic page user: Users completing discovery and requiring wayfinding to the ideal topic or course. In the current experience, they are:
- Overwhelmed by choices (hundreds to thousands of courses per topic)
- Unclear where to start based on skill level, which is why they’re searching for general topic queries and not specific solutions
- Uncertain about the time commitment learning about the topic will require
- May need to scroll through many courses to fully understand the landscape of the topic they’re exploring
- Course page user: Users close to a purchase decision, doing final comparison between courses and/or providers. In the current experience they are:
- Overwhelmed by exceedingly long, keyword-optimized course descriptions
- Having difficulty extracting key outcomes from the detailed curriculum
- Looking for certainty on course difficulty and prerequisites
- Want to read many reviews to get a balanced perspective
- In fear of choosing the wrong course for their level and goals
- Search engine and AI crawlers: Google, Bing, ChatGPT, Claude, etc., which are actively exploring and evaluating content for quality and relevance to their audiences.
- Need access to more informational content which we’re lazy loading or loading dynamically and isn’t crawlable
- Can benefit from clear, structured summaries to parse and reference in query responses
Why is solving these problems big enough to matter?
Udemy has a large volume of Topic pages and CLPs available to Google that are being under-indexed.
While it’s understandable that the index rate of a CLP isn’t high, given the frequency of courses covering similar topics, providing more indexable content on these pages can help Google differentiate nuances between them and aid visitors with:
- Topic pages: Wayfinding to their ideal CLP by providing:
- Context about the topic
- A summary of the course landscape for the topic
- Emphasizing highly-rated courses
- Similar topics if the current one isn’t the ideal fit
- CLPs: Ease in evaluating a course before making a purchase by providing:
- A summary of coverage
- Details about what they can expect to learn
- A summary of the course’s reviews
- A summary of the course’s level of difficulty and prerequisites
Note
We say “providing more indexable content on these pages” because dynamic and lazy-loaded content isn’t crawlable. Summarizing this in a statically-loaded AI overview will surface details about the page’s deeper content without harming our ability to provide a quality, conversion rate-optimized page experience to users.
What is the hypothesis?
- CLPs
- By providing crawlers with an indexable page summary, platforms will be able to better differentiate between courses that cover similar topics.
- By providing users with a page summary, users will be able to make a streamlined purchase decision on a page with information overload.
- Topic pages
- By providing crawlers with an indexable page summary, platforms will be able to see more contextual and informational content on the page, aiding them in prioritizing these pages over competitors.
- By providing users with a page summary, they’ll be able to make better decisions about wayfinding to the topic page or CLP that best fits their learning need.
- Both
- Decreased pogo-sticking will result in higher rankings for these pages in search results, as well as better citation rates and references in AI platforms.
What are the success criteria?
| Page type | Metric | Goal |
|---|---|---|
| Both | Index rate | Increase |
| Both | Google CTR | Increase |
| Both | Ranking | Improve (move up in position) |
What are we building?
AI-generated summaries of page content in the header/lede of Topic pages and CLPs.
Topic pages
Provide:
- An intro to the topic
- Alternative topics or prefilters
- Suggested starting points
- Breakdown of skill level typically required
- Breakdown of time commitment range
- List of careers/jobs that utilize the skill/tool
(full-size image)
(full-size image)
(full-size image)
(full-size image)CLPs
Provide:
- A summary of the course
- Who the course is best for and less ideal for
- Top outcomes of completing the course
- Time commitment
- Module count
- Learner level
- Prerequisites
- Summary of reviews
(full-size image)
(full-size image)
(full-size image)
(full-size image)What are the risks?
| Risk | About | Mitigation |
|---|---|---|
| Content quality | LLMs aren’t impervious to hallucinations and creating content of average quality. | Ground what we generate in what’s already available on the page. Complete multiple rounds of prompt modification to ensure delivered content is high quality. QA a statistically-significant amount of summaries. |
| Public perception | Public opinion on AI usage has mixed sentiment. | Be up front about usage. Make this content available to users but small enough to navigate past if it’s unwanted. Collect feedback on whether the experience is helpful to users. |
Content quality
- About
- LLMs aren’t impervious to hallucinations and creating content of average quality.
- Mitigation
- Ground what we generate in what’s already available on the page. Complete multiple rounds of prompt modification to ensure delivered content is high quality. QA a statistically-significant amount of summaries.
Public perception
- About
- Public opinion on AI usage has mixed sentiment.
- Mitigation
- Be up front about usage. Make this content available to users but small enough to navigate past if it’s unwanted. Collect feedback on whether the experience is helpful to users.
How will we go to market with this launch?
This launch will start with a mix of high-quality Topic pages and CLPs (60 total) that are:
- Not indexed by Google despite significant search volume (10 of each page type)
- Receiving traffic from Google, but a very small percentage of market share (10 of each page type)
- Receiving significant traffic from Google (10 of each page type)
This test will take 3 months to mature from the time of launch, but will provide us with a confirmation of the hypothesis that AI overviews will increase indexing rates, CTR, and average position. A positive result will lead to developing the data pipeline for automation and global rollout to all CLPs and Topic pages.