What Are Log Recordsdata?
Log recordsdata are paperwork that document each request made to your server, whether or not attributable to an individual interacting together with your web site or a search engine bot crawling it (i.e., discovering your pages).
Log recordsdata can show necessary particulars about:
- The time of the request
- The IP tackle making the request
- Which bot crawled your web site (like Googlebot or ChatGPT bot)
- The kind of useful resource being accessed (like a web page or picture)
Right here’s what a log file can seem like:
Servers typically store log files for a limited time based on your settings, relevant regulatory requirements, and business needs.
What Is Log File Analysis?
Log file analysis is the process of downloading and auditing your site’s log files to proactively identify bugs, crawling issues, and other technical SEO problems.
Analyzing log files can show how Google and other search engines interact with a site. And reveal crawl errors that can affect your visibility in search results.
For example, log file analysis can reveal 404 errors that happen when a page no longer exists. Which prevents both users and bots from accessing the content.
Identifying any issues with your log files can help you start the process of fixing them.
What Is Log File Analysis Used for in SEO?
Log file analysis shows you how bots crawl your site, and you can use this information to improve your site’s crawlability—and ultimately your SEO performance.
For example, analysis of log files helps to:
- Discover which pages search engine bots crawl the most and least
- Find out if search crawlers can access your most important pages
- See if there are low-value pages that are wasting your crawl budget (i.e., the time and resources search engines will spend on crawling before moving on)
- Detect technical issues like HTTP status code errors (like “error 404 page not found”) and broken redirects that prevent search engines (and users) from accessing your content
- Uncover URLs with slow page speed, which can negatively impact your performance in search rankings
- Identify orphan pages (i.e., pages with no internal links pointing to them) that search engines may miss
- Track spikes or drops in crawl frequency that may signal other technical problems
- Inform AI SEO strategies by analyzing how AI bots interact with your site
Being able to see how AI bots interact with your site is especially important if you’re eager to understand what conversations relevant to your brand users are having in tools like ChatGPT and Perplexity.
Dan Hinckley, Board Member and Co-Founder at Go Fish Digital, explains this well in a LinkedIn post:
“Your log files can tell you how ChatGPT and Claude are engaging with your site on behalf of users. The screenshot below highlights how during a 30-day window, the ChatGPT-User agent hit this site 48,000+ times across nearly 7,000 unique URLs.”
Plus, doing a log file analysis can flag issues that you might otherwise miss. For example, Ivan Vislavskiy, CEO and Co-Founder of Comrade Digital Marketing Agency, performed a log file analysis for a mid-sized ecommerce site.
The site was experiencing a gradual decline in traffic. Despite no major site changes or any obvious errors. So, Ivan turned to log files to see if he could spot the reason for declining traffic.
“The logs showed that Googlebot was hitting redirect chains and dead-end URLs tied to out-of-stock product variants, something the client’s CMS didn’t expose clearly. These issues were eating up crawl budget and signaling instability.”
Ivan and his team implemented proper canonical tags and cleaned up legacy redirects.
They also blocked URLs with parameters at the end like “?ref=123” using the robots.txt file (a file that tells bots which parts of your site to crawl and which to avoid).
“Within two months, crawl efficiency improved and Googlebot shifted focus to evergreen category pages. Organic traffic stabilized, then grew by 15%.”
How to Analyze Log Files
Now that you know some of the benefits of doing log file analysis for SEO, let’s look at how to do it.
You’ll need:
- Your website’s server log files
- Access to a log file analyzer (we’ll show you how to analyze Googlebot using Semrush’s Log File Analyzer)
1. Access Log Files
Access your website’s log files by downloading them from your server.
Some hosting platforms (like Hostinger) have a built-in file manager where you can find and download your log files.
Here’s how to do it.
From your dashboard or control panel, look for a folder named “file management,” “files,” “file manager,” or something similar.
Here’s what that folder looks like on Hostinger:
Just open the folder, find your log files (typically in the “.logs” folder), and download the files you need. Files from the past 30 days are a good start.
Alternatively, your developer or IT specialist can access the server and download the files through a file transfer protocol (FTP) client like FileZilla.
Once you’ve downloaded your log files, it’s time to analyze them.
2. Analyze Log Files for Crawler Activity
Seeing how Googlebot crawls your site helps you see which pages search engines prioritize and where potential issues can affect your site in search results, including AI Overviews.
To analyze your log files, make sure your files are unarchived (extracted from their folder). And ensure they’re in one of these formats:
- Combined Log Format
- W3C Extended
- Amazon Classic Load Balancer
- Kinsta
Then, drag and drop your files into the Log File Analyzer. Then click “Start Log File Analyzer.”
Once your results are ready, you’ll see a chart showing Googlebot activity over the past 30 days.
Monitor this chart to find any unusual spikes or drops in activity. These can indicate changes in how search engines crawl your site or highlight problems you need to fix.
To the right of the chart, you’ll also see a breakdown of:
- HTTP status codes: These codes show whether search engines and users can successfully access your site’s pages. For example, too many 4xx errors might indicate broken links or missing pages that you should fix.
- File types crawled: Knowing how much time search engine bots spend crawling different file types shows how search engines interact with your content. This helps you identify if they’re spending too much time on unnecessary resources (e.g., JavaScript) instead of prioritizing important content (e.g., HTML).
Scroll down to “Hits by Pages” for more specific insights. This report will show you:
- Which pages and folders search engine bots crawl most often
- How frequently search engine bots crawl those pages
- HTTP errors like 404s
Sort the table by “Crawl Frequency” to see how Google allocates your crawl budget.
Or click the “Inconsistent status codes” button to see URL paths with inconsistent status codes.
For example, a path switching between a 404 status code (meaning a page can’t be found) and a 301 status code (a permanent redirect) could signal a misconfigured redirect.
Pay particular attention to your most important pages. And use the insights you gain about them to make adjustments that can improve your performance in search results.
How to Stop Googlebot from Crawling Irrelevant Resources
Optimizing which resources Googlebot crawls helps make the most of your crawl budget. This helps make sure your key content gets more attention. You can’t control exactly what Googlebot spends its time on—but you can influence it.
For example, instead of letting Googlebot spend time on empty category pages, you can block those types of pages. So it can focus on your most valuable and relevant content.
You can try to prevent Googlebot from crawling irrelevant pages by:
- Adding a rule to your robots.txt to block crawling of unnecessary pages
- Using canonical tags (a line of code that signals the primary version of a webpage) to tell Google which page version is the main one, preventing Googlebot from crawling duplicates
- Removing or updating low-value content to ensure Googlebot focuses on your most important pages
Prioritize Site Crawlability
Taking proactive steps to make sure your site is optimized for crawlability can help your site appear in answers to your user’s queries. Whether that’s in traditional search results, AI Overviews, or chatbot responses.
To optimize your site, conduct a technical SEO audit using Semrush’s Site Audit tool.
First, open the tool and configure the settings by following our configuration guide. (Or stick with the default settings.)
Once your report is ready, you’ll see an overview page that highlights your site’s most important technical SEO issues and areas for improvement.
Head to the “Issues” tab and select “Crawlability” to see issues affecting your site’s crawlability. Many of the potential issues here are ones log file analysis can flag.
Then, select “AI Search” to see issues that might prevent you from ranking in AI Overviews specifically.
If you don’t know what an issue means or how to address it, click “Why and how to fix it” to learn more.
Run a site audit like this every month. And iron out any issues that pop up, either by yourself or by working with a developer.
As you make optimizations, keep an eye on your log files to see how fixing your crawlability issues impacts how Googlebot crawls your site.
Try Semrush today for free to use tools like Site Audit to optimize your website’s crawlability.
