How to Find and Fix 404 Errors and Broken URLs at Scale (Without Hurting SEO)

What Is a 404 Error (and a Broken Link)?

A 404 error is an HTTP status code that means "Not Found." It's the server's way of saying: "I'm reachable, but there is no resource at this URL." A broken URL (or broken link) is a link that points to a URL that returns an error instead of a valid page, most commonly a 404.

HTTP status codes are grouped into categories:

4xx codes generally indicate a problem with the request (often the URL), while 2xx means success and 3xx means redirection.

It's useful to distinguish between:

• A URL that returns 404 but is never linked from anywhere meaningful. This can be harmless background noise (e.g., bots probing random paths).
• A broken link, where an actual link on your site (or another site) points to that 404 URL. Broken links are the ones that hurt users and waste crawl budget.

A 410 Gone is similar to a 404, but more explicit: it tells crawlers the resource was intentionally removed and won't be coming back. This can help search engines drop it from their index a bit faster.

A soft 404 is different again: it's when a page looks like a "not found" page to users, but the server returns a 200 OK (or another non-404 code). Search engines treat it as a 404 in practice, but it can confuse crawling and indexing. Soft 404s and how to fix them need special handling.

Common examples include a mistyped URL (e.g., /produt/widget instead of /product/widget) or a deleted article that no longer exists but is still linked from old content.

Why 404 Errors and Broken URLs Matter for SEO and UX

404 errors and broken links matter because they waste crawl budget, break internal linking, and frustrate users at key moments.

Search engines allocate a crawl budget to each site: a rough limit on how many URLs they'll crawl in a given period. When crawlers repeatedly hit dead-end 404 pages instead of valid content, some of that budget is wasted. On very large sites, this can mean fewer important pages are crawled and refreshed as often as they should be.

Broken internal links also damage internal link equity. Links between your own pages help search engines understand which pages are important and how topics relate. If many internal links point to 404s, that equity is effectively lost instead of being passed to live, valuable URLs.

From a user experience perspective, broken links are jarring. They increase:

• Bounce rate: users hit a 404 and leave instead of exploring further.
• Abandonment in funnels: if a link in a checkout or signup flow is broken, users may never complete the process.
• Support load: users encountering errors may contact support or complain.

For example, imagine you remove a high-traffic product page without redirecting it. Users coming from search or internal category pages land on a 404 instead of a relevant product, leading to lost revenue and wasted ad spend. Contrast that with intentionally retired content, like an old campaign landing page that's no longer relevant; in that case, a well-handled 404 or 410 is acceptable.

When 404s Are a Problem vs When They're Fine

Not all 404s are harmful. They're a standard part of the web and are expected in situations like:

• Content intentionally removed with no suitable replacement.
• Old, low-value URLs with no traffic or backlinks.
• Random, spammy, or malformed URLs generated by bots.

404s become a problem when:

• Important URLs with traffic or backlinks return 404.
• Internal navigation or templates contain broken links.
• Large volumes of previously valid URLs start returning 404 after a migration.
• Key conversion paths (e.g., cart, account, pricing pages) contain broken steps.

The goal is not to eradicate every 404, but to identify and fix the ones that affect users, SEO performance, and business outcomes.

Common Causes of 404s and Broken URLs (Especially After Site Changes)

Large spikes in 404s usually trace back to a few repeatable patterns, especially after migrations or rebuilds. Recognizing these patterns helps you design scalable fixes instead of chasing individual URLs.

Common causes include:

• CMS/platform migrations (e.g., ASP → WordPress)
  Old URLs often use query strings or file extensions (/products.asp?id=123) while new platforms use clean permalinks (/products/123/). If you don't map and redirect these patterns, every legacy URL becomes a 404.
  Typical fix hint: pattern-based 301 redirects and updating internal links.
• Deleted or moved pages without redirects
  Content editors remove or move pages ("archive last year's sale pages") but forget redirects. Any internal or external link to those URLs now breaks.
  Typical fix hint: redirect to the best equivalent page or mark as 410 if there's no replacement.
• URL structure changes
  For example, changing /category/page to /page/ or adding/removing language or date segments. If navigation and sitemaps are updated but old URLs aren't redirected, search engines and external links hit 404s.
  Typical fix hint: pattern rules to map old structures to new ones.
• Typos in internal links and navigation
  A missing character in a menu link (/prodcts/), incorrect case on case-sensitive servers, or copy-pasted URLs with stray spaces can create thousands of broken links when the typo sits in a shared template.
  Typical fix hint: fix the template or component once, then re-crawl.
• External links to outdated URLs
  Other sites still link to URLs you've removed or changed. You can't edit their links, so users arrive on 404s.
  Typical fix hint: add redirects for high-value legacy URLs, or serve a helpful 404 page.
• JavaScript-generated URLs
  URLs stored in JS variables, widgets, or tracking scripts (for example, a script that constructs product URLs based on IDs) can keep pointing to old paths after a rebuild. These don't always show up in simple HTML scans.
  Typical fix hint: search code repositories and JS bundles for old patterns and update logic or centralize URL generation.
• Caching/CDN misconfigurations
  Edge caches may store an incorrect 404 or 200 response and serve it widely. In some cases, a page might return 404 once, then be cached and served as 200 with "not found" content, causing soft 404 issues.
  Typical fix hint: correct cache rules and ensure the right status code is sent and not incorrectly cached.
• Redirect chains and loops ending in 404s
  Complex redirect logic can chain several hops and eventually land on a dead URL. Users may see a 404 after multiple redirects, and crawlers may give up early.
  Typical fix hint: simplify chains and ensure final targets are valid.

A typical mini-case: after rebuilding a large ASP-based site into WordPress, you deploy with new pretty permalinks but no redirect mapping. Overnight, 30,000+ legacy URLs from search, bookmarks, and internal links now 404. The fix isn't editing 30,000 links manually; it's defining a small set of rules that convert /products.asp?id=123 into /products/123/, updating key templates, and validating with a full crawl.

How to Find 404 Errors and Broken Links on Your Site

To fix 404s at scale, you first need a complete, de-duplicated view of both error URLs and the links that point to them. This requires combining several data sources rather than relying on a single report.

A key distinction:

• 404 URLs: the endpoints that return a 4xx status.
• Broken links: the pages that link to those endpoints and cause users or crawlers to hit them.

You need both to understand what to fix and where.

Using Google Search Console

In Google Search Console (GSC), the "Pages" (or similar) report lists URLs that Google tried to crawl and that returned 404 or soft 404 responses. You can:

• Filter for Not found (404) and Soft 404.
• Export the list of affected URLs as CSV or Google Sheets.
• See example referring sources (e.g., sitemaps) for some URLs.

Limitations:

• GSC only shows URLs Google has tried to crawl.
• It doesn't give a complete view of all internal links or all external referrers.
• Data is sampled and may lag behind recent changes.

Treat GSC as one input, especially useful for understanding which 404s matter to Google.

Using Analytics

Your analytics platform can show where real users hit 404 pages:

• Ensure your 404 template fires a distinct pageview (e.g., /404/ or a unique title like "Page not found").
• Build a report that includes: 404 page path or title; the requested URL (often stored in a query parameter or event); the referrer (previous page or source).

This tells you which broken URLs actually receive traffic and from where, helping prioritize fixes. Example useful columns: requested_url, referrer_url, sessions, bounce_rate.

Using Crawlers

Desktop or server-based crawlers can systematically scan your site and report 4xx responses with inlinks:

• Configure the crawler to start from your homepage or sitemap.
• Let it crawl all internal URLs.
• Filter results to 4xx status codes.
• For each 4xx URL, inspect the Inlinks/Linked From panel to see which internal pages contain the broken links.

This is crucial for internal link cleanup, because it shows exactly which templates, menus, or content blocks are generating broken links.

Using Server Logs

Server access logs record every request to your site, including: requested URL; status code (e.g., 404); referrer (if present); user agent and timestamp. For very large sites, log analysis helps you identify 404s that crawlers and users are actually hitting, see long-tail legacy URLs that no longer appear in sitemaps or internal links, and spot automated attacks or spammy requests you can safely ignore. You can process logs with log-analysis tools or scripts to aggregate 404 occurrences and referrers.

Combining Data Sources into a Single View

For at-scale remediation, combine exports from GSC, analytics, crawlers, and logs into a unified spreadsheet or database. A simple working schema might look like:

Basic workflow: (1) Export GSC 404 URLs, analytics 404 hits (with requested URL + referrer), crawler 4xx URLs with inlinks, and optionally log-derived 404s. (2) Combine into one table keyed by Broken URL. (3) Deduplicate rows per broken URL, aggregating metrics. (4) Tag each row as internal vs external broken link and by pattern group. (5) Use this master list for prioritization and pattern-based fixes.

How to Fix 404 Errors and Broken URLs (By Type of Issue)

Once you've identified your 404s and broken links, the next step is deciding what to do with each case. Not every error should be redirected; some should remain 404 or be marked as 410.

Broadly, you have three main options:

• 301 redirect: when there's a clear, relevant replacement or new location.
• 410 Gone: when content is permanently removed with no replacement.
• Leave as 404: when the URL is junk, spam, or otherwise not worth preserving.

When to 301 Redirect

Use a 301 (permanent) redirect when: the content has moved to a new URL; you've updated a blog post and consolidated it under a new slug; you've changed URL structure. Examples: old product URL /products.asp?id=123 → redirect to /products/123/; old blog post /seo-tips-2018/ → redirect to /seo-tips/ containing updated content. In addition to adding redirects, update internal links to point directly to the new URL. Redirects are a safety net, but relying on them long-term adds latency and complexity.

When to Use 410 Gone

Use a 410 status when: content is intentionally retired and has no suitable replacement; you want search engines to drop the URL from their index more decisively; the URL had some visibility but is no longer relevant. Example: /campaign/summer-2019/ → 410 Gone. You typically don't need to use 410 for every removed URL; reserve it for cases where you explicitly want to signal permanent removal.

When to Leave as 404

Leaving a URL as a 404 is appropriate when the URL is clearly spammy or auto-generated by bots, never had real content, or is an old test/staging URL that was accidentally crawled. In these cases, adding redirects or custom handling is unnecessary overhead.

Updating Internal Links vs Relying on Redirects

Redirects are essential, especially after migrations, but internal links should ideally point directly to live URLs. This reduces redirect chains and server load, improves user experience, and gives search engines a cleaner internal link graph. Priority order: (1) Fix broken links in navigation, templates, and shared components. (2) Update links in high-traffic content and key conversion paths. (3) Use redirects to catch remaining legacy requests and external links you can't edit.

A Simple Decision Flow (Text Form)

For each broken URL: (1) Does a closely equivalent page exist? Yes → 301 redirect and update internal links. No → step 2. (2) Was the content valuable or does it have traffic/backlinks? Yes → consider redirecting to the most relevant category, hub, or updated resource. No → step 3. (3) Is the URL junk or spam? Yes → leave as 404. No → return 410 Gone if you explicitly want it removed from the index; otherwise, 404 is fine.

Fixing Broken Internal Links at Scale

Fixing a handful of broken links is easy; fixing tens of thousands across a large site requires a structured, pattern-based workflow. The key is to correct the sources (templates, components, database records) rather than editing individual pages.

Step 1: Export Internal 4xx URLs with Inlinks

Use a crawler to crawl your entire site (or key sections), filter to internal URLs returning 4xx, and export a report with: Broken URL, Status code, "Linked from" URLs (inlinks), Anchor text and link location if available. This gives you a raw list of broken URLs and the internal pages that reference them.

Step 2: Classify by Pattern

Look for recurring patterns: old file extensions (.asp, .php, .aspx, .html), old directory structures (/old-section/, /v1/, /en/), legacy query parameters (?id=123, ?cat=shoes), specific slug changes. Create a simple table:

Group URLs under these patterns so you can design a small set of fixes that cover many URLs.

Step 3: Decide: Update Links, Add Redirects, or Both

For each pattern: Update internal links when the target content still exists at a new URL and broken URLs appear in navigation or commonly visited content. Add redirect rules when there's a clear mapping from old to new URLs or external sites/bookmarks still use the old URLs. Often the answer is both: update internal links and add redirects to catch external traffic and long-tail references.

Step 4: Implement Bulk Changes

Depending on your stack:

• CMS/global navigation updates — Update menus, header/footer links, and shared content blocks in one place.
• Database search-and-replace — For content in a database, use controlled search-and-replace. Pseudo-SQL example:

UPDATE content_table
SET body_html = REPLACE(body_html, 'href="/products.asp?id=', 'href="/products/')
WHERE body_html LIKE '%/products.asp?id=%';

Always test on a staging copy and back up before running.

• Template and component fixes — Update template logic so it constructs the correct URL pattern.
• Pattern-based redirect rules — Pseudo-htaccess example:

RewriteCond %{QUERY_STRING} ^id=([0-9]+)$
RewriteRule ^products\.asp$ /products/%1/ [R=301,L]

This converts /products.asp?id=123 to /products/123/ for any ID.

Handling Broken Links in JavaScript

Broken internal links aren't always visible in HTML. They can be in JS variables, widgets that construct URLs, or tracking scripts. To find and fix: (1) Search code repositories for old patterns in .js and template files. (2) Inspect built JS bundles or source modules. (3) Centralize URL configuration in a config file or constants module. (4) Update and redeploy.

Step 5: Re-Crawl to Validate

After implementing bulk changes: re-run a full crawl; compare the new 4xx report to the old one (internal 4xx count should drop; previously broken patterns should resolve); spot-check key patterns and high-priority pages. If some broken links remain, refine patterns and repeat.

Fixing Broken External Links at Scale

Broken external links are links on your site that point to other sites' 4xx or 5xx pages. They don't directly waste your crawl budget, but they hurt user experience and can weaken the perceived quality of your content.

Identify: Use a crawler configured to follow and check external links, report external URLs returning 4xx or 5xx, and include the internal pages where those links appear. Export: external target URL, status code, internal source URL, anchor text.

Decide: Remove, Update, or Replace. Update to a new URL when the external site moved the content. Replace with an alternative when the original is gone but similar content exists elsewhere. Remove the link when the resource is no longer necessary, it's a minor reference in an old low-traffic post, or no suitable replacement exists.

Handle template vs one-off: A broken link in a footer or sidebar appears on thousands of pages — fix in the template. For one-off content links, prioritize posts with significant traffic or backlinks; for low-traffic legacy posts, removing broken links without replacements may be acceptable.

Handling Legacy URLs After a Site Migration or Rebuild

Site migrations and rebuilds are the most common cause of massive 404 spikes. The core challenge is ensuring that thousands (or millions) of old URLs either redirect appropriately or return an intentional 404/410.

Step 1: Inventory Old URLs

Before or immediately after migration, gather: old XML sitemaps, exports from the previous CMS, historical server logs (top-requested URLs), GSC reports for indexed and 404 URLs, third-party backlink tools. Combine into a master list of old URLs.

Step 2: Map Old URL Patterns to New Ones

Look for systematic relationships, e.g. /products.asp?id=123 → /products/123/, /category/page.asp → /category/page/, /blog/article-title.aspx → /blog/article-title/. Create a mapping table:

Step 3: Create Pattern-Based Redirect Rules

Implement redirect rules. Pseudo-htaccess example:

# Map /products.asp?id=123 to /products/123/
RewriteCond %{THE_REQUEST} \s/+products\.asp\?id=([0-9]+)\s [NC]
RewriteRule ^products\.asp$ /products/%1/ [R=301,L]

# Map /category/page.asp to /category/page/
RewriteRule ^category/(.+)\.asp$ /category/$1/ [R=301,L]

Test on staging with a sample set of old URLs to ensure correct redirects, no loops or chains, and no accidental catch of unrelated URLs.

Step 4: Handle Obsolete Content

Not every old URL deserves a redirect. For legacy content with no modern equivalent, return 404 or 410. For retired product lines with no replacements, 410 can help clean up the index. Avoid blanket redirects to the homepage; they confuse users and are treated as soft 404s.

Step 5: Update Internal Links

Even with redirects in place, update internal links to point directly to new URLs: run search-and-replace for old patterns in database and templates; update navigation, sitemaps, and internal cross-links; ensure XML sitemaps only contain live, canonical URLs.

Dealing with Soft 404s and Misconfigured 404 Responses

Soft 404s and misconfigured status codes can undermine your efforts even when your 404 page "looks" correct.

What Is a Soft 404?

A soft 404 occurs when a page appears to be a "not found" page but the server returns a 200 OK or another non-404 code. Search engines treat these as errors: they waste crawl budget, pollute the index with low-quality error pages, and obscure real issues in reports.

How to Detect Soft 404s

Use Google Search Console (Soft 404 category); configure a crawler to request known non-existent URLs and check returned status and content; or use curl or browser dev tools to inspect HTTP status and headers for non-existent URLs.

Caching Edge Case: 404 Once, 200 Later

A subtle failure: (1) First request to a non-existent URL hits origin, which correctly returns 404. (2) The caching layer caches the response but incorrectly stores or serves it as 200. (3) Subsequent requests receive cached 200 with "page not found" content. From the outside it looks like the URL exists; search engines may treat it as a soft 404. To test: request a made-up URL twice and compare status; if the first is 404 and the second is 200, your cache is misconfigured.

Example curl test:

curl -I https://www.example.com/this-page-should-not-exist-12345
curl -I https://www.example.com/this-page-should-not-exist-12345

Check the HTTP/ status line in both responses.

Fix Strategies

Ensure custom 404 templates return a true 404 (application or server config should set status to 404 when rendering the 404 template). Configure caching/CDN correctly: don't rewrite status codes when caching; avoid caching 404 responses as 200; if you cache 404s, cache them with the correct 404 status. Test systematically with crawlers and scripted tests; re-check after configuration changes or deployments.

How to Prioritize Which 404s to Fix First

Large sites often have thousands of 404s. A simple prioritization framework helps you focus on what matters most.

Key factors: Traffic (sessions/pageviews); Backlinks (inbound links from other sites); Conversion impact (part of key conversion path?); Internal navigation and templates (menus, footers, category pages, sitemaps).

Simple scoring: Assign each factor 0–3. Traffic: 0 = none, 1 = low, 2 = medium, 3 = high. Backlinks: 0 = none, 1 = few low-value, 2 = several or some strong, 3 = many or high-authority. Conversion: 0 = not in path, 1 = minor, 2 = supporting, 3 = critical. Priority score = T + B + C (0–9). Higher scores get fixed first.

URLs scoring 7–9: fix immediately (redirect + update links). 4–6: fix in next batch. 1–3: fix opportunistically. 0: usually safe to ignore or leave as 404. Deprioritize obviously junk or spammy URLs regardless of score.

How to Monitor and Prevent 404 Errors Over Time

Cleaning up 404s once is not enough; sites evolve and new errors appear. Ongoing monitoring prevents another backlog.

Scheduled crawls: Monthly — crawl entire site or key sections; review new 4xx/5xx, redirect chains and loops, internal link count changes. Quarterly — deep crawl including external links; review templates and navigation.

GSC and Analytics alerts: In GSC, monitor spikes in 404 and soft 404 reports; set email notifications for coverage changes. In Analytics, track 404 pageviews over time; set alerts for sudden increases in 404 hits from key referrers or campaigns.

Pre-deployment QA: Crawl staging; check for 4xx in navigation and key flows. Manually test signups, checkouts, and key funnels. Validate sitemaps list only live, canonical URLs; remove old or deprecated URLs before deployment.

Keep sitemaps and content practices clean: Update XML sitemaps when content is added, moved, or removed; remove URLs that are permanently 404/410. Train editors to use CMS link pickers, avoid staging URLs, and update links when content is moved or consolidated.

About the Author

Shakur Abdirahman

Technical SEO Specialist

Shakur is a Technical SEO Specialist with expertise in large-scale website migrations, redirect management, and technical SEO optimization. He helps businesses preserve search rankings and maintain crawl efficiency during complex site changes.

LinkedIn