Tool Onboarding

While this work doesn’t directly impact Udemy’s reach, it implements a tool that gives us visibility into AI bot behavior on our content, which we can use to more accurately understand and measure the impact of work we do.

For Udemy’s SEO initiatives, log file analysis is a critical capability that will allow us to understand how search engine and LLM crawlers interact with our website, giving us a window into how users discover our content through organic search and organic AI interactions.

We currently have no visibility into:

The BrightEdge/OnCrawl Log File Analyzer is an enterprise SEO/AI tool that processes server log files to provide us with insights into crawler behavior. By implementing this tool, we’d gain the ability to:

• Cross-reference what Google crawls vs. what we think is crawlable
• Identify crawl budget waste and optimization opportunities
• Detect server errors affecting search bots that may not appear in other monitoring tools
• Validate that technical SEO changes improve crawler accessibility
• Analyze organic traffic patterns at the server level

• Crawl budget optimization
  • Large websites can waste significant crawl budget on low-value pages (pagination, filters, session IDs)
  • Google may never reach important new content if crawlers get stuck in inefficient site areas
• Hidden technical issues
  • Visibility on the “truth” about what bots experience vs. what we see in testing
  • Can identify: server errors (500s, timeouts) that only affect bots; redirect chains invisible to crawl tools; pages returning 200 status but empty content; mobile vs. desktop crawl discrepancies; duration of crawl when JS is being executed
• Competitive advantage
  • Competitors likely don’t have log analysis capability
  • Provides unique insights unavailable in Search Console or analytics platforms
  • Allows proactive SEO strategy vs. reactive strategy (problem-solving)

By implementing OnCrawl Log File Analyzer, we will gain visibility into search engine crawler behavior and organic traffic patterns that will allow us to identify and fix technical SEO issues, optimize crawl efficiency, and ultimately improve organic search performance.

• Log files successfully uploaded and parsed by OnCrawl
• Parsing correctly identifies Googlebot hits, along with other primary LLM bots (Perplexity, OpenAI, etc.)
• FTP/upload automation functioning reliably
• Data appears correctly in OnCrawl dashboard within 24 hours of log generation
• SEO team trained on tool usage and interpretation

Identify log sources

• Determine whether logs needed from CDN (e.g., Cloudflare, Akamai), load balancer, or origin servers
• Confirm logs capture first user/bot interaction point
• Identify if multiple log sources needed (mobile, desktop, subdomains)
  • Coverage should include, at a minimum: udemy.com, blog.udemy.com, teach.udemy.com, business.udemy.com, community.udemy.com, events.udemy.com, support.udemy.com

Verify log coverage

• Ensure logs cover all site sections to be monitored
• Check for subdomain coverage if needed

Establish access

• Work with DevOps/IT to obtain credentials/access to log storage location
• Document log file storage locations (e.g., /var/log/nginx/)
• Confirm ability to access and download sample files

Log format validation

• Obtain sample log files (1–2 days of logs)
• Verify format compatibility
• Verify log contains required fields:
  • Query path or full URL
  • Date and timestamp
  • User Agent
  • Status code
  • Scheme (http/https) or port (80/443) if site uses HTTPS
  • vhost if multiple subdomains
  • Referer (recommended for organic traffic)
  • Client IP (recommended for Googlebot verification)
  • Response size (recommended)
  • Response time (recommended)
• Check for Googlebot and organic visits in sample files
• Document server type (Apache, Nginx, IIS, etc.)

Activate Log Monitoring in OnCrawl

• Confirm OnCrawl Log Monitoring subscription is active
• Navigate to project > “ADD DATA SOURCES” > “ACTIVATE LOGS MONITORING”
• Complete configuration steps:
  • Step 1: “Configure your needs” — Define what to monitor
  • Step 2: “Logs completeness” — Confirm log file contents
• Use completed checklist from Phase 1 to answer parsing questions

Determine upload method

Automatic upload (preferred if using supported platforms)

• Best for: AWS S3, Google Cloud Storage, Akamai, Azure Blob
• Contact OnCrawl to set up secure connector
• Configure connector with:
  • Storage location credentials
  • Folder paths for log files
  • Upload schedule (daily recommended)

Manual upload (via FTPS)

• Best for: Server-based logs or unsupported storage platforms
• Set up FTPS access:
  • Server: ftp.oncrawl.com
  • Username: OnCrawl account username
  • Password: Set in workspace settings > FTP Access
  • Ports: 21 (connection), 10090–10999 (passive mode)
  • Security: FTPS with TLS 1.2 (Explicit FTP over TLS)
• Configure network/firewall:
  • Open FTP ports on firewall
  • Ensure FTPS (not plain FTP) is allowed
• Set up FTP client (FileZilla recommended)
• Test connection and locate project folder

Log file preparation

• Compress log files to supported formats: .zip, .gz, or .7z (note: 7z PPMd compression NOT supported)
• If required by security policies, filter log files to include only:
  • Bot hits (e.g., Googlebot)
  • Organic search traffic
  • Remove irrelevant entries (internal traffic, other bots, etc.)

Upload and parsing

Initial upload

• Upload 2–7 days of sample log files
• For automatic upload: Monitor connector to confirm files received
• For manual upload:
  • Connect via FTPS
  • Navigate to project folder
  • Upload compressed log files
  • Confirm upload completion

Configure parsing

• Return to OnCrawl > Add Data Sources > Log Files tab
• Confirm OnCrawl detected uploaded files
• Click “I’VE UPLOADED ALL FILES”
• Review parsing results (Step 4: “Check logs format”):
• Address errors:
  • Target: “Everything seems OK” status
  • Fix any issues in: the “Issues” section, “HTTP/HTTPS analysis” and “Subdomain analysis”
• Resolve warnings:
  • “No SEO visits detected” — Verify organic traffic exists in logs
  • “No bot hits detected” — Verify Googlebot visits exist; may need to disable IP validation if using cache
  • “High parse error rate” — Investigate format inconsistencies
• Manual parsing configuration (if needed):
  • Switch to “Manual” in “Configure logs parser”
  • Map log elements to correct field types using dropdowns
  • Advanced settings: disable “Check Google IP” if using cache server; set timezone if not in logs
• Iterate: Click “CHECK LOGS FORMAT” after each change
• Finalize: Click “CONFIGURATION IS OK” when parse succeeds and warnings resolved

Automation and monitoring

• Automate daily uploads
• Set up monitoring

SEO team training

• Conduct training session on:
  • OnCrawl platform navigation
  • Log analysis reports and metrics
  • How to interpret crawl data
  • Cross-analysis with crawl data (if applicable)
  • Use cases and workflows

• Log Monitoring Setup Guide — Step-by-step setup documentation
• Log File Coverage — Preference for 60-day history and daily uploads
• OnCrawl Date Ranges — Details about how date ranges are handled in the OnCrawl UX/UI
• Log File Upload Tutorial — Details on OnCrawl log file upload process
• Log File Upload Frequency — OnCrawl details on upload frequency