Skip to content
Ben GarvesLinkedIn
Menu

bengarves.com/work/tool-exploration

Tool Exploration

[CVR-2177] [SPIKE] Log File Analyzer Research, Access, and Pipeline Planning

From Ben Garves’s work at Udemy. Re-created from the original document, with people’s names, internal links and non-public numbers removed or generalized.

SpikeCVR-2177

Company
Udemy
Created
Resolved
Status
Done
Type
Task (spike)
Priority
P2 - Medium
Story points
3
Parent epic
Log File Analyzer Implementation Exploration (CVR-2175)
Project
FE Home Squad
Sprint
CVR - Sprint 51
Requirement source
Engineering

A research spike to find where Udemy’s log files live, get access to them, and plan and estimate a pipeline (60 days of history, then daily uploads, with alerts when delivery fails) for feeding them into the OnCrawl Log File Analyzer.

  • Technical SEO
  • Log file analysis
  • Data pipelines
  • Research spike

Notes from refinement

  • These logs may be Cloudflare logs, showing requests; we need to look at the OnCrawl documentation.
    • Looks like we already have CF logs in Databricks (data lake).
    • An example SQL query pulling CF log data already exists.
  • For the WordPress sites (probably all subdomains), we may need to reach out separately.
  • We should prioritize CF, and if this solution doesn’t catch all, we can reassess.

Tasks

  • Determine where log files live for these domains/subdomains:
    • udemy.com
    • blog.udemy.com
    • Bonus if they’re included, but not required: teach.udemy.com, business.udemy.com, community.udemy.com, events.udemy.com, support.udemy.com
  • Request access to those platforms
  • Read through OnCrawl documentation:
  • Plan and estimate a pipeline for delivering log files to OnCrawl
    • 60 days of historic data in initial upload
    • Daily uploads moving forward
  • Plan and estimate a monitoring and alert process for situations where log file delivery fails