Category: Blog

Part of my mission in launching this blog is to provide my target audience, mainly marketers and small businesses who are looking for digital marketing tactics and strategies to grow their sites, with actionable and useful content that they can implement on their own. If you want to receive the best and exclusive content straight to your inbox, do your online business a favor by signing up to my newsletter.

  • Same Brands, Different Sources: Why Winning AI Search Is Engine-Specific

    Same Brands, Different Sources: Why Winning AI Search Is Engine-Specific

    I pulled 1,648 sources that AI engines cited when they recommended products. Then I checked something that surprised me. The engines mostly agree on which brands to name. But they build those same recommendations from completely different kinds of sources. One engine reads YouTube. Another reads review sites. Same answer, different receipts.

    The Setup

    I looked at three buying categories: project management software, mattresses, and VPNs. I ran product recommendation prompts through three engines: Google AI Overviews, Perplexity, and Gemini. Then I did two separate counts.

    I ran two separate counts:

    • Share of voice. Which brands each engine recommended, and how often.
    • Source mix. The source behind every recommendation. I sorted all 1,648 citations into five types: YouTube and video, review and comparison sites, editorial “best-of” articles, Reddit and forums, and brand-owned sites.

    Two honest notes before the numbers:

    • ChatGPT is left out of the source cut. When I call it through the API, it rarely returns linked source URLs, so I cannot count what I cannot see.
    • Gemini’s citations were resolved first. They arrive as redirect URLs, not real domains, so I resolved each one to the real site before counting. The Gemini numbers reflect actual sites, not Google’s wrapper links.

    This is the fourth study in the series. It builds on two things I found earlier. In where AI gets its recommendations, I found that 79% of AI’s cited sources are third-party, not brand-owned. In who AI recommends, I found the SEO leader often loses in AI. This study connects them. It shows the mechanics underneath.

    The Engines Agree on Who to Recommend

    Here is the part that made me stop and re-run the count.

    The engines largely agree on the brands. When I lined up the recommendations, each major brand landed with a roughly even share of voice across engines. Call it about 20% each for the top names in a category. AI Overviews, Perplexity, and Gemini were not fighting over who to name. They were reading from the same short list of winners.

    The four AI engines recommend the same brands at a roughly even share of voice

    That tracks with what I found in who AI recommends. The brands that win tend to win everywhere. Reputation carries across engines. If the market already treats you as a top option, most engines will too.

    So if you only measured share of voice, you would think GEO is one game. Get recommended, and you get recommended everywhere. Simple.

    It is not simple. Because the agreement stops the moment you ask why.

    But They Cite Completely Different Sources

    Same brands. Different reasons. Here is the source mix for each engine.

    Source typeAI OverviewsPerplexityGemini
    YouTube / video29%6%3%
    Review & comparison sites27%47%32%
    Editorial “best-of” articles11%18%17%
    Reddit / forums7%1%3%
    Brand-owned sites11%6%10%
    How each AI engine sources its answers: AI Overviews leans on YouTube, Perplexity on reviews

    Read across any row and the gaps jump out. YouTube is 29% of AI Overviews sources and 6% of Perplexity sources. Review sites are 47% of Perplexity and 27% of AI Overviews. Reddit matters a little to AI Overviews and almost nothing to Perplexity.

    The same brand can get recommended by AI Overviews because of a video, and by Perplexity because of a review roundup. The recommendation looks identical to the buyer. The path to earning it is not.

    That changes how you should think about GEO work. You are not optimizing for one machine. You are feeding different diets to different engines.

    AI Overviews Runs on YouTube

    Almost one in three AI Overviews sources is a video. 29%. No other engine comes close.

    This lines up with something obvious once you see it. Google owns YouTube. Google surfaces video in a lot of places already. So when AI Overviews assembles a product answer, video is right there in the pantry.

    The kind of video that pulls weight depends on the category:

    CategoryVideo that earns the citation
    Project management softwareA walkthrough showing the tool in use
    MattressesAn unboxing or a firmness test
    VPNsA speed test on camera

    If you have been treating YouTube as a nice-to-have, this flips it. For AI Overviews, video is a primary ingredient. A brand with strong video coverage and a brand with none can look very different to this one engine, even when everything else is equal.

    AI Overviews also leans on review sites at 27% and pulls from Reddit at 7%, more than the other two engines. So the recipe is video plus reviews plus a little community talk. I wrote more about the mechanics in how to rank in Google AI Overviews.

    Perplexity Runs on Reviews and Comparisons

    Perplexity is the mirror image. It barely touches video at 6%. It leans hardest on review and comparison sites at 47%. Nearly half of everything it cites.

    So Perplexity is reading G2, Capterra, comparison articles, and “X vs Y” pages. It wants structured, side-by-side, third-party judgment. It wants the pages that already do the comparing.

    Editorial “best-of” articles are its second biggest source at 18%. Brand-owned sites are low at 6%, and Reddit is almost nothing at 1%.

    Gemini sits closer to Perplexity than to AI Overviews. Review sites are its top source at 32%. Editorial best-of is 17%. Video is only 3%. Brand sites are 10%. So both Perplexity and Gemini want the same thing: third-party review and comparison coverage, not your homepage and not a video.

    That is the practical split. AI Overviews wants video. Perplexity and Gemini want reviews. One brand, two different reasons to get named.

    What This Means for Your GEO Strategy

    The headline for me is this. Share of voice tells you if you are winning. Source mix tells you how to win, and it is different per engine. You cannot run one playbook and expect all three engines to pick you up.

    So split the work by engine. Here is how I do it.

    The per-engine playbook: feed AI Overviews video, feed Perplexity and Gemini reviews

    If You Want to Win AI Overviews

    Feed the engine what it eats. Video first.

    • Build real YouTube coverage. Product walkthroughs, comparisons, honest demos. Not one polished ad, but a body of useful video.
    • Get into best-of listicles. Editorial roundups still matter here at 11%.
    • Show up on Reddit. It is a small slice at 7%, but AI Overviews uses it more than the other engines do. Real threads, real answers, not spam.

    If your category has thin video coverage, that is your opening. Video is where AI Overviews is hungriest and where many brands are weakest.

    If You Want to Win Perplexity or Gemini

    Feed these engines review and comparison coverage.

    • Get listed and reviewed on the big third-party sites. For software, that means G2 and Capterra. For consumer products, the review outlets that own your category.
    • Earn comparison coverage. “Best X” and “X vs Y” pages are what these engines pull from most.
    • Support the editorial best-of articles that rank in your space. Both engines lean on them at 17% and 18%.

    Video will barely move these two. Spending your whole budget on YouTube and expecting Perplexity to notice is a mismatch. Match the spend to the diet.

    If you want the full framework behind all of this, my AI search optimization pillar walks through the method start to finish, and how to show up in ChatGPT covers the engine I had to leave out of this source cut.

    How to Find Your Own Per-Engine Gaps

    You do not have to guess. You can measure where you show up, engine by engine, and see which source types are carrying your competitors.

    Here is the process I use.

    1. Pick your core buying prompts. The real questions your buyers type. “Best project management software for agencies.” “Most durable mattress for side sleepers.” Whatever fits your category.
    2. Run each prompt through AI Overviews, Perplexity, and Gemini separately. Do not blend the results. The whole point is the difference between them.
    3. For every recommendation, log the source. Note the type: video, review site, best-of article, forum, or brand page.
    4. Compare your source footprint to the winners. If competitors own the YouTube sources in AI Overviews and you have none, that is your AI Overviews gap. If they own the G2 and comparison coverage in Perplexity and you do not, that is your Perplexity gap.

    Two gaps, two fixes. That is the value of splitting by engine instead of averaging everything into one score.

    If you want this run for you, done properly with the redirect resolving and the source classification handled, that is exactly what my AI Search Visibility service does. I measure where you show up per engine, find the source gaps, and hand you the work that closes them.

    FAQ

    Do all AI engines use the same sources?

    No. In my study they agreed on the brands but split hard on sources. AI Overviews pulled 29% of its sources from YouTube. Perplexity pulled 47% from review and comparison sites and only 6% from video. Same recommendations, different source types behind them.

    Which matters more for AI visibility, YouTube or review sites?

    It depends on the engine you care about. YouTube matters most for Google AI Overviews, where video is 29% of sources. Review and comparison sites matter most for Perplexity at 47% and Gemini at 32%. If you serve buyers across all three, you need both.

    How do I know which engine my buyers use?

    Ask them, and check your analytics for referral traffic from each engine. Then weight your work toward the engines that actually send you buyers. If most of your traffic comes through Perplexity or Gemini, invest in review and comparison coverage first. If AI Overviews drives your category, build video.

  • What Happened to Long Tail Pro? The Rise and Fall of a Keyword Research Classic

    What Happened to Long Tail Pro? The Rise and Fall of a Keyword Research Classic

    If you did SEO or built niche sites in the 2010s, you probably remember Long Tail Pro. For a while there, it was the keyword research tool. It found the low-competition, long-tail keywords that small sites could actually rank for, and a whole generation of bloggers built their businesses on it.

    So when I went to check the tool recently and found the website dead, I wanted to know what actually happened. The short version? A tool that once earned close to a million dollars a year was sold, resold, and eventually mismanaged into the ground. The longer version is a genuinely useful cautionary tale, and its founder has been surprisingly open about the whole thing.

    Here’s the full story of what happened to Long Tail Pro, plus the keyword tools I’d point you to now that it’s gone.

    What Was Long Tail Pro?

    Long Tail Pro was a keyword research tool built by Spencer Haws, the founder of Niche Pursuits. Back in 2010, Spencer was a business banker at Wells Fargo who happened to be building dozens of niche websites on the side. The problem he kept running into: existing tools like Market Samurai only let you research one seed keyword at a time.

    He wanted something faster. Specifically, a tool where he could drop in 10, 20, or 30 seed keywords across different niches and have it spit out thousands of keyword ideas at once, complete with the metrics he needed to decide whether a keyword was worth targeting. When he couldn’t find one, he hired a developer and had it built himself.

    Once it matured, a few things set it apart:

    • Bulk long-tail keyword research. Feed it multiple seed keywords and it returned hundreds of low-competition, long-tail suggestions with search volume attached.
    • A keyword competitiveness score. Its signature “Long Tail Platinum” feature scored how hard a keyword would be to rank for, so you weren’t guessing.
    • SERP metrics for every result. It pulled the SEO metrics of the pages already ranking, so you could judge whether you had a real shot.
    • Rank tracking. You could track the keywords you’d targeted and watch your positions over time.

    If that sounds a lot like the workflow behind modern keyword research tools, that’s because Long Tail Pro helped popularize it. It was ahead of its time when it launched.

    The Rise: From Side Project to Seven Figures

    The first version launched to Spencer’s small email list (around 1,500 people) at the tail end of 2010, plus a Warrior Forum special offer. It made about $2,500 in its first month. Not life-changing, but enough to prove people wanted it.

    Then came a rough patch that nearly killed the tool before it got going. The original developer went quiet, and when Spencer asked for the source code, he was told he didn’t actually own it, and buying it would cost another $15,000. He had a real decision to make: pay up, shut down, or rebuild from scratch. He chose to rebuild, spent about six months on a second version, handed refunds and free upgrades to early customers, and relaunched.

    That’s when it took off. It started as a one-time desktop purchase (around $47, later $97), then Spencer introduced a recurring subscription for the keyword competitiveness feature. A series of big affiliate launches, run with marketer Mark Thompson, poured fuel on the fire: roughly $250,000 in sales over about ten days in 2013, and around $500,000 from a single launch in 2015.

    By 2015, Long Tail Pro was doing just under $1 million in revenue with more than $600,000 in profit, and Spencer owned 100% of it. By 2016 he was netting over $50,000 a month. For a tool that started as a way to scratch his own itch, that’s a remarkable run.

    What Happened to Long Tail Pro?

    This is where the story turns. Long Tail Pro didn’t die because the product was bad or because nobody wanted keyword tools anymore. It died because it changed hands one too many times and landed with an owner who let it rot. Spencer has told this story himself in two videos on the Niche Pursuits channel, and the timeline goes like this.

    Spencer Haws walks through how he built and sold Long Tail Pro.

    The First Sale to Wired Investors (2016)

    By 2016, competitors were multiplying, Google was tightening access to its keyword data, and Long Tail Pro was still a desktop application in a world moving online. Spencer decided he’d rather sell than take on the huge investment of rebuilding it as a web app.

    He sold to a small private equity group called Wired Investors for $1.8 million. He actually kept a 20% stake because he still believed in the upside. And for about a year, that faith looked justified. Wired Investors did the hard work of moving Long Tail Pro from desktop to an online tool and shipped a bunch of new features.

    The Second Sale and the Beginning of the End (2018)

    Then Wired Investors got overextended. According to Spencer, they acquired more than 20 companies in about two years, then decided to liquidate everything and dissolve the business. At the end of 2018, they sold Long Tail Pro again, this time to a new owner. That’s when things really started going downhill.

    The new owner’s whole plan was to acquire a bunch of businesses and take the group public. The trouble, in Spencer’s telling, is that he seemed far more excited about going public than about actually running any of the tools he’d bought. The leadership team was bloated with management layers that managed nothing, and nobody was ever really put in charge of growing or even maintaining Long Tail Pro.

    Death by Mismanagement

    Within 12 months of that second sale, it was obvious the new owner didn’t know how to run the business. Spencer, who still owned a sliver of the company, hopped on multiple consulting calls to spell out exactly what he’d do to grow it. His advice was never followed. Long Tail Pro was, in his words, left on a deserted island while the owner hoped it would somehow rescue itself.

    Sales slid. Support went dark. Customers who reached out to the Long Tail Pro help desk got no response, and some of them started emailing Spencer directly, even though he hadn’t run the company in years. The financial side unraveled too. Wired Investors had sold with seller financing, meaning they were still owed multiple six figures, and the new owner couldn’t make the payments because every business he’d acquired was failing at the same time.

    By the time Spencer published his “They Just Killed My $1.8 Million Startup” video in mid-2024, the Long Tail Pro website had been down for over two months. From what he understood, the owner could no longer afford the hosting bill and owed money to multiple people, now including the hosting company. Attorneys were involved and legal proceedings were underway.

    Spencer Haws on how Long Tail Pro was mismanaged into the ground.

    Where Things Stand Now

    As of this writing, Long Tail Pro is effectively gone. Visit longtailpro.com today and you won’t find a keyword tool at all. The domain now points to a completely unrelated business, and the old product pages return errors. There’s no working signup, no support, and no sign anyone is reviving it.

    Spencer’s own verdict is blunt: Long Tail Pro died through mismanagement. Someone spent seven figures on a healthy, profitable software company and then let it die a slow, painful death. If you’re a former customer wondering why your tool stopped working, that’s your answer. It wasn’t you, and it wasn’t the product.

    The Lesson Hiding in Long Tail Pro’s Story

    There’s something worth sitting with here, especially if you rely on software to run your business. A tool can be profitable, well-loved, and genuinely useful, and still disappear because of decisions made in a boardroom you’ll never see. The people who felt it first were the customers, who lost access with no warning and nowhere to turn.

    The practical takeaway: don’t build your entire workflow around a single tool, and keep your keyword data exportable. If a tool is central to how you work, it’s worth knowing you could switch without losing everything. Which brings me to what you should actually use now.

    The Best Long Tail Pro Alternatives Still Going Strong

    Here’s the part that stings. While Long Tail Pro was being neglected, the market for affordable keyword tools kept thriving. Plenty of small, well-run tools do what Long Tail Pro used to do, and several of them do it better. These are the ones I’d recommend. (A quick note: some of the links below are affiliate links, so I may earn a commission at no extra cost to you. I only recommend tools I’d actually use.)

    LowFruits

    If you want the closest spiritual successor to Long Tail Pro, start here. LowFruits is built specifically to find low-competition, long-tail keywords that smaller sites can rank for, which is exactly the job Long Tail Pro was famous for. It analyzes the SERP and flags weak spots (like forums and low-authority pages ranking on page one) so you know where you’ve got a real opening. Notably, Spencer himself named LowFruits as one of the small keyword tools doing well today.

    KWFinder by Mangools

    KWFinder is probably the most direct like-for-like replacement. It’s beginner-friendly, affordable, and laser-focused on long-tail keywords with a clear difficulty score, which is the feature most Long Tail Pro users cared about. It comes bundled with the rest of the Mangools suite (SERPWatcher for rank tracking, SERPChecker for SERP analysis), so you get a lot of the old Long Tail Pro workflow in one place.

    Serpstat

    Ready to move up to a full SEO platform without enterprise pricing? Serpstat is my pick. It handles keyword research, competitor analysis, rank tracking, and site audits in one dashboard. It’s the tool my own content writing campaigns lean on, and it scales well as your needs grow.

    SE Ranking

    SE Ranking is another excellent all-in-one option that’s still very affordable. Its keyword research database is deep, its rank tracker is one of the best in its price range, and it’s a genuinely comfortable step up if you’ve outgrown a single-purpose keyword tool.

    Semrush

    When you’re ready for the industry heavyweight, Semrush is the standard. It’s more than a keyword tool, it’s a complete SEO and marketing suite. It’s overkill if all you want is long-tail keywords, but if you’re doing this professionally or for clients, it’s hard to beat the depth of its data.

    Keyword Revealer

    Rounding out the list, Keyword Revealer is a budget-friendly tool built around finding profitable, low-competition keywords. I use it for its bulk keyword checker, especially for pulling questions out of the “People Also Ask” boxes to build out FAQ sections. It has the same low-competition-hunting DNA that made Long Tail Pro popular.

    Worth mentioning too: Spencer specifically praised KeySearch and Keyword Chef as small, well-run tools thriving in the space Long Tail Pro left behind. I don’t use those two personally, but they’re proof the category is alive and well.

    Not sure how to actually put any of these to work? My guide on how to do keyword research for SEO walks through the whole process, and if you want to understand the type of keywords Long Tail Pro specialized in, start with my primer on long-tail keywords.

    Frequently Asked Questions

    Is Long Tail Pro still available in 2026?

    No. The Long Tail Pro website is down and the domain no longer hosts the tool. There’s no working signup or support, so for all practical purposes the tool is defunct.

    What happened to Long Tail Pro?

    Founder Spencer Haws sold it to Wired Investors in 2016 for $1.8 million. Wired later sold it again at the end of 2018 to a new owner who mismanaged it, defaulted on the seller financing, and eventually couldn’t even keep the website’s hosting paid. By 2024 the site was down and legal proceedings were underway.

    How much did Long Tail Pro sell for?

    Spencer Haws sold Long Tail Pro to Wired Investors in 2016 for $1.8 million, keeping a small ownership stake at the time.

    What is the best Long Tail Pro alternative?

    For long-tail keyword hunting specifically, LowFruits and KWFinder are the closest replacements. If you want a full SEO platform, Serpstat and SE Ranking are excellent affordable options, and Semrush is the top choice for professionals who need the deepest data.

    Can I get a refund or recover my Long Tail Pro subscription?

    Unfortunately, there’s no active support to reach. Since the business appears to be insolvent and in legal proceedings, the realistic move is to cancel any recurring charge through your bank or card provider and migrate to one of the alternatives above.

    Final Thoughts

    Long Tail Pro deserved a better ending. It was a great tool built by someone who cared, and it helped a lot of people (myself included) learn how to find keywords worth chasing. Its downfall had nothing to do with the software and everything to do with who ended up holding it.

    The good news is that the job Long Tail Pro did so well is now handled by tools that are cheaper, better, and actively maintained. If you’ve been hanging on to Long Tail Pro out of loyalty or habit, it’s time to move on. Pick one of the alternatives above, and if you want help putting it to work, my SEO content writing guide will show you how to turn those keywords into content that ranks.

  • Who AI Actually Recommends (And Why It’s Not the Brand That Ranks #1)

    Who AI Actually Recommends (And Why It’s Not the Brand That Ranks #1)

    I've now run two studies on this. The first showed that ranking well in Google doesn't get you recommended by AI, the two ladders barely lined up. The second showed why: 79% of the sources AI cites to build a recommendation are third parties, not your own website.

    Both studies left one question sitting there unanswered. If the brand that ranks #1 isn't the one AI recommends, then who is?

    So I picked three categories, took the single most SEO-dominant brand in each, and asked four AI engines who they'd actually recommend. The famous name lost all three times.

    The Setup

    I chose categories where a household name owns Google, then measured whether that dominance survives in AI:

    • Project management software, focal brand monday.com (the one running Super Bowl-adjacent ad budgets).
    • Mattresses, focal brand Casper (the brand that basically invented the bed-in-a-box category).
    • VPNs, focal brand NordVPN (the most-advertised VPN on the planet).

    Each category had the focal brand plus four real competitors. I pulled every brand's U.S. organic traffic from DataforSEO to build the SEO ladder, then ran 7 buyer-style prompts ("best project management software," "best mattress for back pain," "best VPN for streaming," and so on) across ChatGPT, Google AI Overviews, Perplexity, and Gemini, five times each, and measured share of voice: out of all the brand recommendations AI made, what percentage went to each brand.

    Non-determinism is real, so nothing here rests on a single answer. Every prompt ran five times per engine and the results are the aggregate.

    The Finding: The Famous Name Lost Every Time

    Here is the whole study in one picture. Left side is where each brand ranks in Google. Right side is where AI actually recommends it.

    Rank flip across three categories: the SEO leader drops 3 to 4 places in AI

    Every focal brand, the biggest SEO name in its category, fell:

    • monday.com: #2 in Google, dead last in AI.
    • Casper: #1 in Google, dead last in AI.
    • NordVPN: #1 in Google by a mile, 4th of 5 in AI.

    Meanwhile the brands with the smallest search footprints climbed. Two of them took the #1 AI spot outright. Let me show you each category, because the three tell slightly different versions of the same story, and the differences matter.

    Mattresses: A Near-Total Inversion

    This one is the cleanest. The SEO ladder and the AI ladder are almost perfect mirror images.

    Mattress AI leaderboard: Nectar 28%, Saatva 26%, Tempur-Pedic 22%, Purple 21%, Casper 4%
    Brand Google traffic SEO rank AI share AI rank
    Nectar $330K 5, last 28% 1
    Saatva $370K 4 26% 2
    Tempur-Pedic $473K 3 22% 3
    Purple $819K 2 21% 4
    Casper $1.11M 1 4% 5, last

    Casper has the most organic traffic in the category by a wide margin. In AI, it showed up in exactly one of seven prompts, once, on a single engine. It is functionally invisible when someone asks ChatGPT or Perplexity for a mattress. Nectar, the weakest brand in the group on Google, wins the AI recommendation outright.

    If you only watched your Google rankings, you'd think Casper was crushing it. In the place buyers are increasingly asking, it has almost no presence at all.

    VPNs: The Great Flattening

    VPNs behaved differently, and that's exactly why I'm glad this category was in the study.

    VPN AI leaderboard: ExpressVPN 21%, Surfshark 21%, Proton VPN 21%, NordVPN 20%, CyberGhost 19%
    Brand Google traffic SEO rank AI share AI rank
    ExpressVPN $324K 3 21% 1
    Surfshark $221K 4 21% 2
    Proton VPN $370K 2 21% 3
    NordVPN $1.28M 1 20% 4
    CyberGhost $37K 5, last 19% 5

    NordVPN outspends and out-ranks this entire field. It has 3.5 times the organic traffic of the next brand and roughly 80,000 ranking keywords. In AI, all of that collapses into a dead heat. Every VPN sits within two points of every other one, and NordVPN lands 4th.

    Nobody got destroyed here the way Casper did. But look at what happened to the advantage. NordVPN spent years and a fortune building a search lead that AI simply ignores. It treats the category as a five-way tie and hands the top slots to smaller brands. A commanding SEO position bought NordVPN nothing in the AI answer.

    Project Management: The Nuance

    The PM category is the one that keeps me honest, because it contains the study's one real exception.

    Project management AI leaderboard: Asana 21%, ClickUp 21%, Trello 21%, Notion 20%, monday.com 17%
    Brand Google traffic SEO rank AI share AI rank
    Asana $777K 1 21% 1
    ClickUp $157K 4 21% 2
    Trello $206K 3 21% 3
    Notion $144K 5, last 20% 4
    monday.com $328K 2 17% 5, last

    Two things are happening. First, my focal brand monday.com, #2 in Google, came in dead last in AI, right on script. ClickUp and Notion, both weaker in search, beat it.

    Second, and this is the honest exception: Asana ranks #1 in Google and holds #1 in AI. So SEO dominance can carry into AI. But look at the condition. Asana isn't merely strong in search, it's the outright leader by a wide margin, and it has the deep third-party review presence to match. Being #2, like monday, offered no protection at all. The takeaway isn't "rankings never transfer." It's "only true category dominance transfers, and even then it's the third-party coverage riding alongside it doing the work, not the ranking itself."

    Why This Keeps Happening

    Put the three categories together and the mechanism from my second study explains all of it.

    Google ranks your page. You can win it largely on your own property, with your content, your links, your technical health. That's why Casper, monday, and NordVPN dominate search: they've each poured resources into their own websites for years.

    AI doesn't rank your page. It assembles a recommendation from what the rest of the web says about you, the "best of" listicles, the review labs, the Reddit threads, the YouTube round-ups. And on that battlefield, none of these famous brands is winning. Nectar and Saatva get the mattress-review love. ExpressVPN and Surfshark own the VPN affiliate ecosystem. ClickUp and Notion get talked about constantly. The famous brands optimized the one surface AI mostly ignores, their own site, and under-invested in the surfaces AI actually reads.

    That's the whole thing. AI recommendations track third-party reputation, and third-party reputation is not the same asset as search rankings. You can lead one and lose the other, and most brands don't even know there are now two scoreboards.

    What This Means For Your Brand

    If you're the biggest name in your category, do not assume you're safe in AI. Casper was the biggest name and finished last. Go check.

    If you're not the biggest name, this is the best news you'll read this quarter. The AI recommendation isn't locked up by whoever has the fattest SEO budget. Nectar and ClickUp prove a smaller brand can win it outright, by being the one reviewers and communities actually talk about.

    Either way, the work is the same, and it's off your website:

    • Find out where you actually stand across ChatGPT, AI Overviews, Perplexity, and Gemini. Your Google rank tells you nothing about this.
    • Earn the third-party coverage that AI reads, the category review sites, the "best of" listicles, the relevant subreddits and YouTube channels.
    • Track it as its own scoreboard, separate from rankings, because the two now move independently.

    That's exactly what my AI Search Visibility service does: measure where you show up across every major engine, then earn the placements that move you up. If you're the category's biggest name and you're not sure whether AI recommends you, that's the first thing I'd check, and it's a quick thing to find out.

    FAQ

    Does ranking #1 on Google mean AI will recommend me?

    No. Across three categories, the most SEO-dominant brand lost in AI every time, twice finishing dead last. The only brand that held its #1 spot in both was an outright category leader with deep third-party review coverage, and even there it's the coverage doing the work, not the ranking.

    Why does AI recommend smaller brands over famous ones?

    Because AI builds recommendations from third-party sources, review sites, editorial "best of" lists, forums, and video, rather than from a brand's own website. Smaller brands that earn strong review coverage can out-perform famous brands that rely on their search rankings. In this study, the lowest-traffic mattress brand won its category's AI recommendation outright.

    How do I find out if AI recommends my brand?

    Run your key buyer prompts across ChatGPT, Google AI Overviews, Perplexity, and Gemini, several times each to account for non-determinism, and measure your share of voice against competitors. That measurement is the first step of the AI Search Visibility service.

    Is SEO still worth it if AI ignores my rankings?

    Yes. SEO still drives search traffic and keeps you eligible to be cited. But it's now one of two separate jobs. Ranking wins Google. Earning third-party coverage wins the AI recommendation. Treating them as the same task is why so many strong-ranking brands are invisible in AI.

  • Where AI Actually Gets Its Recommendations (Hint: Not Your Website)

    Where AI Actually Gets Its Recommendations (Hint: Not Your Website)

    In an earlier study, I showed that good SEO doesn’t guarantee AI visibility. Brands that dominate Google can still get left out when buyers ask ChatGPT. That raised the obvious question: if ranking your own pages isn’t what earns the recommendation, what is?

    So I went and looked at the receipts.

    I analyzed 1,648 sources that AI actually cited when recommending products, across three engines and three popular categories. The pattern is blunt: AI barely recommends you based on your own website. It recommends you based on what everyone else says about you.

    The Headline: 79% of AI’s Sources Are Third Parties

    Here’s the whole study in one chart.

    Source mix: 79% of AI citations are third-party, 21% brand-owned
    Source mix: 79% of AI citations are third-party, 21% brand-owned

    Of every source AI pulled from to build a recommendation, 79% were third parties, media articles, review sites, Reddit, YouTube, and only 21% were the brand’s own website. And that 21% flatters brands, because a big chunk of it isn’t product pages at all. It’s companies’ own listicle blogs (more on that below).

    Put simply: for roughly every seven sources AI uses to decide what to recommend, only one is the brand’s own site.

    The Method

    I kept this reproducible on purpose:

    • Three popular categories where people genuinely ask AI for recommendations: project management software, mattresses, and VPNs. Each has a different third-party ecosystem, so the finding isn’t a quirk of one niche.
    • 21 buyer-style prompts (“best project management software,” “best mattress for back pain,” “best VPN for streaming,” and so on).
    • All four major engines: ChatGPT, Google AI Overviews, Perplexity, and Gemini. Each prompt was run several times per engine.
    • Instead of tracking brand mentions this time, I captured every source URL each answer cited, then categorized the domains: the brand’s own site, editorial media, review sites, community forums, YouTube, retailers, or other.

    Two honest limits. ChatGPT returned answers but almost no linked sources through its API, so it couldn’t contribute citations. Gemini wrapped its citations in redirect URLs, so I resolved those to their real domains (580 of 584 resolved) before counting. The 79% figure is across AI Overviews, Perplexity, and Gemini.

    What Kinds of Sources Win

    Break the citations down by type and the story sharpens:

    • Editorial media, 31%. “Best of” articles from TechRadar, CNET, Forbes, PCMag, and the like. The single biggest bucket.
    • Review sites and test labs, 21%. NapLab, Sleepopolis, Mattress Nerd, security.org, dedicated reviewers.
    • The brand’s own site, 21%. And often the blog, not the product page.
    • YouTube, 12%. Video reviews and explainers, the single most-cited domain in the whole study.
    • Reddit and forums, 6%. Real people comparing options.
    • Everything else (info sites, retailers) made up the rest.

    Editorial, review, and community sources together dwarf owned content. That’s the machinery behind an AI recommendation, and none of it is your homepage.

    The More Consumer the Category, the Less Your Site Matters

    The overall average hides a sharp gradient. I broke the third-party share out by category.

    Third-party citation share by category: project management 49%, mattresses 90%, VPNs 93%
    Third-party citation share by category: project management 49%, mattresses 90%, VPNs 93%
    • Mattresses: 90% third-party. The names you’d expect, Casper, Purple, Saatva, and Nectar, still came up, but almost entirely through review labs and “best of” articles. Their own sites were nearly invisible in the AI answers.
    • VPNs: 93% third-party. Even more lopsided. The household names, NordVPN, ExpressVPN, and Surfshark, surfaced through affiliate reviews and Reddit threads far more than through their own pages. This category runs on other people’s recommendations.
    • Project management software: 49% third-party. The one category where “brand” citations were close to half. But look closer and most of those aren’t product pages. They’re SaaS companies publishing their own “best project management tools” listicles (Paymo, Zapier, Wrike, Toggl). Even the biggest names, Notion, Asana, and monday, mostly showed up inside other people’s roundups, not on the strength of their own pages.

    So the true rate of “AI cites the actual product page” is even lower than 21% across the board. In consumer categories, it rounds to almost nothing.

    The Sources AI Actually Cites

    This is the part you can act on. AI’s recommendations don’t come from a random long tail. They concentrate in a short, nameable list of places.

    Top sources AI cites: YouTube, TechRadar, Reddit, NapLab, CNET, security.org, and more
    Top sources AI cites: YouTube, TechRadar, Reddit, NapLab, CNET, security.org, and more

    YouTube was the most-cited source in the entire study. After it came TechRadar, Reddit, NapLab, CNET, security.org, Mattress Nerd, PCMag, Forbes, and Sleep Foundation. If you’re in one of these categories and you’re not present on those sources, you are not in the conversation AI is having with your buyers.

    This is a different to-do list than SEO gives you. It isn’t “optimize your title tags.” It’s “get reviewed, get listed, get talked about on the sources AI trusts.”

    Why This Happens

    Google and AI answer different questions, and this study is the proof.

    Google ranks your page against a query. It rewards your on-page work, your links, your technical health. So you can win Google largely on your own property.

    An AI assistant isn’t ranking your page. It’s assembling a recommendation, and it builds that recommendation from what it can find said about you across the open web: the listicles you’re included in, the reviews you’ve earned, the comparisons people write, the threads where real users vouch for you. Your own site is one voice in that chorus, and a quiet one.

    That’s why the first study came out the way it did. You can rank beautifully and still lose in AI, because the thing AI reads to make its pick lives mostly on other people’s domains.

    What This Means for Your Brand

    If you want to show up when buyers ask AI, the work moves off your website:

    • Earn placements on the sources AI cites. Get into the “best of” listicles, onto the review sites, into the YouTube round-ups for your category. That is the lever.
    • Treat reviews and third-party mentions as an AI-visibility channel, not just reputation management. They’re literally what AI reads.
    • Don’t mistake ranking for being recommended. They’re now two separate jobs, and this is the one most brands aren’t doing on purpose.

    None of this means SEO is dead. Your site still has to be crawlable, credible, and clear. But your on-page work is table stakes now, not the finish line. The recommendation is won out in the wider web.

    That off-site, get-mentioned-and-cited work is exactly what my AI Search Visibility service is built to do, measure where you show up across every engine, then earn the placements that move you. If your rankings aren’t turning into AI recommendations, this is why, and it’s fixable.

    FAQ

    How does AI decide what to recommend?

    It assembles an answer from sources across the web, weighted heavily toward third parties: editorial “best of” articles, review sites, community discussion like Reddit, and video. In this study, 79% of cited sources were third parties rather than the recommended brand’s own website.

    Does my own website still matter for AI search?

    Yes, but less than most people assume. Your site needs to be crawlable and credible to stay eligible, but it made up only about a fifth of the sources AI cited, and much of that was blog content, not product pages. The recommendation is won mostly off your site.

    What is answer engine optimization (AEO)?

    Answer engine optimization is the practice of getting your brand named and cited in AI-generated answers rather than only ranking in traditional search results. It’s sometimes called generative engine optimization (GEO). Based on this data, a big part of it is earning third-party mentions, not just optimizing your own pages.

    How do I get cited by AI?

    Get present on the sources AI actually pulls from in your category: the review sites, the “best of” listicles, the relevant subreddits, and YouTube. Then measure whether it’s working across ChatGPT, AI Overviews, Perplexity, and Gemini. That measurement and outreach is what the AI Search Visibility service handles.

  • Does Good SEO Get You Recommended by AI? I Tested It on Four Email Tools

    Does Good SEO Get You Recommended by AI? I Tested It on Four Email Tools

    There’s a comfortable assumption behind a lot of marketing budgets right now. If we already rank well in Google, we’ll show up when people ask ChatGPT too. The SEO is done, so the AI visibility tags along for free.

    I didn’t want to assume it. I wanted to measure it.

    So I ran a small experiment. I took four email marketing platforms, ranked them by how well they perform in traditional Google search, then measured how often each one actually gets recommended across the big AI answer engines. Same brands, two scoreboards: SEO on one side, AI search visibility on the other.

    If good SEO reliably turned into AI visibility, the two scoreboards would line up. They didn’t. They nearly inverted. Here’s the full test, the data behind it, and what it means if you’re trying to get found when buyers ask AI.

    Why SEO and AI Visibility Are Not the Same Thing

    Search is splitting into two habits. People still Google. But more and more, they also ask ChatGPT, read Google’s AI Overviews, or run a question through Perplexity or Gemini and take the answer at face value.

    That creates a new question most brands haven’t measured:

    • Google ranking tells you where your page sits for a keyword.
    • AI visibility tells you whether an AI recommends your brand when someone asks.

    Those sound like the same thing. The work of optimizing for the second one even has its own names now, answer engine optimization (AEO) and generative engine optimization (GEO). But nobody had shown me hard data on whether the first buys you the second. So I tested it, and I picked a category where the answer would be clear.

    The Test: How I Measured SEO Against AI Visibility

    This is the part that makes the finding trustworthy, so I’ll be specific about how it ran.

    The Four Brands

    I used four email marketing platforms, chosen because they sit at clearly different levels of SEO strength:

    • Mailchimp, the category giant that everyone already knows.
    • MailerLite, the strongest of the three challengers in Google.
    • Moosend, a mid-tier player in search.
    • EmailOctopus, the smallest search footprint of the group.

    The Rule I Set First

    I ranked the four on SEO metrics before I ran a single AI query. The order was locked in based on Google performance alone, using organic traffic value and page-one keyword counts. That matters. It means I couldn’t pick the story after seeing the AI results.

    The Engines and Prompts

    • I measured all four across the four engines that carry the most weight today: ChatGPT, Google AI Overviews, Perplexity, and Gemini.
    • I used 12 buyer-intent prompts, the kind people actually type: “best email marketing software,” “best tool for a small business,” “cheapest option,” and so on.
    • I ran every prompt five times per engine and averaged the results, because AI answers shift from one run to the next. A single query tells you almost nothing.

    How I Scored It

    For each answer, I recorded which brands got named and which got cited with a link. Then I rolled it up into share of voice: each brand’s slice of all the mentions across every prompt and engine. Higher share of voice means AI put that brand forward more often. This is the same method behind my AI Search Visibility service, pointed at a public test instead of a client.

    Step 1: The Google Rankings Were Not Close

    Ranked by estimated organic traffic and page-one keywords, the four sat in a clear order:

    • Mailchimp was in a different league. Around $745K in estimated monthly search traffic value and roughly 12,600 keywords on page one. It laps the field.
    • MailerLite was a clear second, with strong page-one presence and about 1,150 keywords up top.
    • Moosend came third. It ranks for plenty of terms, but most of them sit on page three and deeper.
    • EmailOctopus was last and smallest, with a fraction of the group’s search footprint.

    MailerLite had roughly seven times the search presence of EmailOctopus. In Google terms, it wasn’t a fair fight. If SEO decided AI visibility, MailerLite should have crushed the two smaller brands in the AI answers too.

    Then I asked AI, and the fight changed completely.

    Step 2: Then I Asked AI, and the Order Flipped

    Here’s the AI share of voice next to the SEO ranking. The two columns are the whole story.

    Brand SEO rank (Google) AI rank (share of voice)
    Mailchimp 1 1 (28%)
    MailerLite 2 4 (21%)
    Moosend 3 2 (26%)
    EmailOctopus 4 3 (25%)

    Mailchimp held the top spot. Below it, everything reshuffled, and it reshuffled in exactly the wrong direction for anyone who thinks SEO decides this.

    Google rank versus AI share of voice for four email platforms; the strongest challenger in SEO finished last in AI.
    Google rank versus AI share of voice. The lines that cross are the story.

    MailerLite Ranked #2 on Google and Finished Last in AI

    The strongest challenger in search came dead last of the four in AI answers. Second in Google, fourth in AI. All that ranking strength did not carry over.

    EmailOctopus Punched Far Above Its SEO Weight

    The weakest platform in Google, with about a seventh of MailerLite’s search footprint, got recommended more often by AI. On paper it should have been an afterthought. In the answers, it wasn’t.

    What the AI Answers Actually Showed

    Report heatmap showing where a brand is named on each AI engine, prompt by prompt.
    A piece of the report you get: exactly where you are named, prompt by prompt, on every engine.

    Share of voice is the summary. The individual answers are where it gets concrete, and a few of them made the pattern impossible to ignore.

    • On “best email marketing software,” the biggest query in the set, ChatGPT and Perplexity both left MailerLite out completely. They named Mailchimp, Moosend, and EmailOctopus, and skipped the one challenger with the strongest Google rankings. Only two of the four engines mentioned MailerLite at all for that query.
    • On “best email marketing service for ecommerce,” three of the four engines did the same thing. ChatGPT, AI Overviews, and Perplexity each named EmailOctopus, the weakest platform in search, and none of them named MailerLite. Gemini was the only engine that included it.
    • The citations told the same story. When an engine backed a brand with a source link, Moosend got cited seven times across the test, more than Mailchimp’s four. MailerLite, second in Google, was cited twice. EmailOctopus was named plenty but rarely linked, which is its own gap to close.

    The platform with the second-best SEO in the group kept getting left out of the exact answers it should have owned, while weaker-ranked competitors got named and linked in its place.

    There’s one more detail worth sitting with. MailerLite showed up in all 12 prompts somewhere. The models clearly knew it existed. They just almost never put it forward as the answer. It was in the room the whole time and rarely got picked. That gap between “known” and “recommended” is the whole problem in one brand.

    So Does Good SEO Mean Good AI Visibility?

    Share of voice bars for the four email platforms across the AI engines.
    Straight from the report: each brand’s share of voice across all four AI engines.

    On its own, no.

    The only place the two rankings agreed was the very top, and Mailchimp is a special case worth calling out.

    The Only Agreement Was at the Top, and That’s Brand, Not SEO

    Mailchimp is the name half the market already knows. Its brand sits in the training data, in the reviews, in every “best email tools” listicle. That kind of dominance shows up everywhere, in Google and in AI, and it says more about scale and reputation than about any single ranking. Take the giant out of the picture, and the link between ranking position and AI visibility falls apart.

    Google and AI Answer Different Questions

    • Google ranks your page against a search. It rewards relevance, links, technical health, and your position for a keyword.
    • AI isn’t ranking your page. It’s assembling a recommendation, and it builds that from what it can find said about you across the web: third-party lists, reviews, comparisons, and the words other people use to describe your product.

    Ranking well means your page is strong. It doesn’t mean the wider web talks about you like an answer. AI leans on the second thing, which is why content that earns mentions and citations matters as much as on-page ranking now.

    Being Present Is Not Being Preferred

    MailerLite is the whole lesson. It ranks well. It gets crawled. AI knows it’s there. And it still isn’t the name handed to the buyer. Ranking earned it visibility. It didn’t earn it the recommendation.

    What This Means for Your Brand

    Ranking in Google is necessary. This test is a blunt reminder that it isn’t sufficient. A few takeaways if you’re serious about AI search:

    • Don’t treat your rankings as proof of AI visibility. For most brands, the two don’t move together, and the gap is invisible until you measure it.
    • Measure both, separately. Your Google positions and your AI share of voice are different scoreboards. Track them as such.
    • If you’re not the category giant, do the AI work on purpose. Brand fame carries the leader into both channels. Everyone else has to earn AI visibility deliberately.

    How to Check Your Own AI Visibility

    AI visibility report scorecard tiles.
    The report’s scorecard: your AI visibility scored at a glance, the same view every client gets.

    You can’t fix what you haven’t measured. The starting point is a baseline: where your brand actually shows up across ChatGPT, AI Overviews, Perplexity, and Gemini, and where competitors are getting recommended instead of you. That’s exactly what my AI Search Visibility service does, using the same method you just read. If you’d rather start with the fundamentals, a technical and content SEO audit is still the foundation everything else sits on.

    The Honest Limits of This Test

    I’d rather you trust the direction than oversell the certainty, so here are the caveats:

    • This is one category over one month. A different niche could behave differently.
    • AI answers are non-deterministic. They shift from run to run, which is why every prompt ran five times and got averaged.
    • Share of voice is directional, not a fixed ranking. It shows a trend, not a guarantee.

    It’s a direction, not a law of physics. But the direction here is hard to miss.

    FAQ

    Does SEO still matter for AI search?

    Yes. Strong SEO gets your pages crawled, indexed, and treated as credible, which keeps you eligible to appear in AI answers. This test shows it isn’t enough on its own, but it’s still the foundation. Think of good SEO as necessary but not sufficient.

    What is answer engine optimization (AEO)?

    Answer engine optimization is the practice of getting your brand named and cited in AI-generated answers, from ChatGPT to Google’s AI Overviews, rather than only ranking in the traditional list of blue links. It’s sometimes called generative engine optimization (GEO).

    Why does a lower-ranking brand show up more in AI?

    Usually because the wider web talks about it more in the contexts AI pulls from: third-party best-of lists, reviews, and comparisons. AI builds recommendations from what’s said about a brand across the web, not only from who ranks highest for a keyword.

    How do I find out where my brand stands in AI search?

    Run a baseline across the major AI engines for the questions your buyers actually ask, and compare your share of voice to your competitors. That’s what the AI Search Visibility service measures, and it’s the fastest way to see the gap between your Google rankings and your AI visibility.

  • Free Screaming Frog Alternative: I Tested LibreCrawl on 4 Real Sites (Where It Wins, Where It Breaks)

    Free Screaming Frog Alternative: I Tested LibreCrawl on 4 Real Sites (Where It Wins, Where It Breaks)

    The best free Screaming Frog alternative I have tested is LibreCrawl, an open-source crawler that does most of what a £259-a-year license does, for nothing. But “most” is carrying weight in that sentence. I installed it, drove it from Claude Code through its API, and pointed it at four real websites across three different platforms. Two things surprised me. On my own site, the free tool caught a structured-data error that Screaming Frog got wrong. On two client stores, it invented hundreds of server errors that did not exist.

    So this is not a “10 best free SEO crawlers” list. It is one tool, four real crawls, and a straight answer to the only question that matters: can a free crawler replace the paid one you are already paying for?

    What LibreCrawl is, and why I tested it against Screaming Frog

    LibreCrawl vs Screaming Frog in one line

    Screaming Frog SEO Spider is the desktop crawler most technical SEOs already run. It costs £259 per year. LibreCrawl is a free, open-source, web-based crawler that pulls the same kind of data: status codes, titles, meta descriptions, headings, canonicals, the internal link graph, images, and structured data. One is the polished industry default. The other is a community project you host yourself.

    How I ran the test (driven from code, not just clicking)

    I did not just open the LibreCrawl interface and watch a progress bar. I drove it programmatically through its REST API, the same crawl-then-reason loop I described in my Screaming Frog MCP write-up. Start a crawl, poll it, pull the per-page data as structured output, and analyze it. That matters here, because driving both tools the same way is the only way to compare them fairly rather than comparing my patience with two different dashboards.

    The test setup: four sites, three platforms, one honest method

    The four sites, and why they are a fair spread

    I crawled my own WordPress site, a Shopify furniture store, a Shopify fashion store, and a PrestaShop electronics retailer. Three platforms, three very different technical setups, one open-source crawler. The client sites stay anonymous; the numbers do not.

    Site Platform Pages crawled Clean (200)
    My own site WordPress 1,185 34%
    Furniture store Shopify 155 (sample) 100%
    Fashion store Shopify 1,038 93%
    Electronics retailer PrestaShop 1,192 55%

    Polite settings, and verifying every finding

    I ran every crawl gently: one request at a time, with a two-second delay between pages. This was not a brute-force hammering. And when a result looked dramatic, I checked it by hand with a single request, rather than trusting the crawler’s word for it. That verification step is where the most interesting finding of the whole test came from.

    Where the free tool won: it caught a schema error Screaming Frog missed

    Screaming Frog reported zero structured data; LibreCrawl found it on 1,156 of 1,185 pages
    Same site, same day’s markup. One crawler said zero structured data. The other found it on 1,156 of 1,185 pages.

    Screaming Frog said “zero structured data.” It was wrong.

    When I audited my own site with Screaming Frog earlier, the crawl reported zero structured data across the entire site. On a site competing for AI Overview citations, that reads like a five-alarm fire. Except it was not true. My schema framework had been switched on the whole time. Screaming Frog had surfaced a problem that did not exist, and a less careful audit would have sent me “fixing” something that was already working.

    LibreCrawl detected schema on 1,156 of 1,185 pages

    I ran the free tool over the same site. It correctly parsed the JSON-LD schema and reported it present on 1,156 of the 1,185 pages it crawled. The 29 pages it flagged as missing schema genuinely had none. This was not a one-site fluke either. Across all four sites, LibreCrawl’s structured-data detection held between 79 and 100 percent, and every spot-check I ran confirmed it was reading the markup correctly.

    Why this is the finding that matters for AI Overviews

    Structured data is one of the levers that decides whether your pages get pulled into AI Overviews and other generative answers. A false “zero” pushes you in exactly the wrong direction: you either waste a day implementing schema you already have, or you rip out working markup chasing a ghost. On the single highest-stakes check in the audit, the free tool was the more accurate one. That is not the result I expected to write.

    Where it was a near-perfect match: redirects

    The default-settings trap: LibreCrawl is blind to redirects

    Here is the catch that will bite you if you do not know it. By default, LibreCrawl follows redirects and records the final destination. A link pointing at a 301 gets logged as the 200 it eventually lands on. So on its first pass over my site, it reported zero redirects, on a site I know is full of them. Screaming Frog reports every redirect as its own hop, out of the box. LibreCrawl hides them unless you tell it not to.

    194 versus 197, once you run the second pass

    So I ran it again with redirect-following turned off. This time it saw them: 194 redirecting URLs, against the 197 Screaming Frog found on the same site. That is agreement within two percent, from two independent crawlers, which is about as much validation as you can ask for. The cost is that you have to run two separate crawls to get what Screaming Frog hands you in one, and the second pass discovers fewer pages because it stops at each redirect instead of following it through. The data is accurate. The workflow is clunkier.

    Where it broke: it invented 503 server errors that did not exist

    Crawler-induced false server errors by platform, all verified returning 200
    Crawler-induced false server errors by platform. Every one returned a clean 200 when I requested it by hand.

    A Shopify fashion store: 70 fake Cloudflare challenges

    This is the part that would have embarrassed me in front of a client. On the Shopify fashion store, the crawl reported 70 pages returning 503 and 429 errors. They were not broken pages. They were Cloudflare bot-challenge responses, served because the crawler tripped the store’s protection even at one request every two seconds.

    A PrestaShop electronics retailer: 503 fake 500s

    The electronics store was worse. The crawl flagged 503 pages returning HTTP 500, more than 40 percent of the crawl. That is a catastrophic-looking result that would set off every alarm in a status report.

    I verified every one with a single request, and they all returned 200

    So I did the thing every audit should do and I checked. I requested those exact “broken” URLs by hand, one at a time. Every single one returned a clean 200. The pages were fine. The crawler had provoked the errors itself, by requesting product pages faster than a fragile server, or a defensive one, wanted to answer. The errors were real HTTP responses. They were not real problems.

    The pattern: the more protected the store, the more noise

    Put the four sites in a row and the rule is obvious. My WordPress site, with no aggressive protection, returned mostly real errors that were worth fixing. The protected and fragile stores returned storms of fake ones. The more defensive the platform, the more garbage LibreCrawl pours into your issue list. The takeaway is simple and non-negotiable: on any e-commerce store, filter the 5xx pages and re-check them before you trust a single count.

    The noise tax: 8,000 “issues” that are not 8,000 problems

    What is noise, and there is a lot of it

    LibreCrawl threw between roughly 4,000 and 8,400 raw issues per full-site crawl. The electronics store alone showed 7,830. That number is almost meaningless until you clean it. A huge slice is the fake 5xx pages from the section above. Another huge slice is correct but not actionable: noindexed archive pages flagged for “missing meta description,” login and cart pages flagged for missing canonicals, the normal furniture of a CMS that the tool dutifully reports as problems.

    What is actually worth fixing

    Strip the noise and the real, actionable list is small and consistent across every site: images missing alt text (a template-level issue almost everywhere), slow response times, overlong or too-short titles, and genuinely missing meta descriptions. Screaming Frog does this triage for you and hands back a curated list. LibreCrawl hands you everything and makes the filtering your job. That filtering pass is not optional. It is the difference between an audit and a panic.

    The engineering reality of “free”

    The bugs and limits I hit

    Free has a price, it is just not on the invoice. Getting clean crawls out of LibreCrawl meant working around real rough edges. A session-cookie bug had it polling empty sessions and reporting zero pages until I fixed how it held state. Its settings persist in a shared database in a way that leaks between runs. The stores rate-limited it. And when I ran two large crawls at once, the server ran out of memory and crashed, taking a half-finished crawl down with it.

    Free in dollars, not in hours

    None of that is fatal. All of it is fixable, and a more careful setup avoids most of it. But it is time, and time is the thing a £259 license quietly buys back. Screaming Frog’s polish is not a luxury; it is hours you do not spend debugging your crawler instead of auditing your site.

    So can a free tool replace your £259 Screaming Frog license?

    Decision table: when to use LibreCrawl versus Screaming Frog
    The short version: free for the work you verify yourself, paid for the work someone else is counting on.

    The honest answer is: for some of your work, yes, and you should know which.

    Use LibreCrawl when

    Reach for the free tool for internal sweeps, quick structured-data checks, and budget projects where you are going to read and filter the data yourself anyway. It is genuinely good at the core job, it is more accurate on schema than I expected, and the price is unbeatable. If you are technical enough to drive it and skeptical enough to verify it, it earns its place.

    Stick with Screaming Frog when

    Keep the license for client-billed audits, for protected or fragile e-commerce stores where the false-error problem is worst, for redirect-heavy migrations where you need the full picture in one pass, and for any job where curated, trustworthy output is the product you are selling. The polish, the reliability, and the clean list are what you are paying for, and on client work they are worth it.

    That is the line. Free for the work you check yourself. Paid for the work someone else is counting on. I went in expecting the free tool to be a toy and the paid one to be untouchable. The truth was more useful than that, and on the one finding that mattered most, the free tool was the one that got it right.

    FAQ

    Is LibreCrawl a good free alternative to Screaming Frog?

    Yes, for internal and budget work. LibreCrawl captures the same core technical SEO data as Screaming Frog and is free and open-source. It is noisier and needs more verification, so it suits practitioners who will filter the data themselves rather than hand it straight to a client.

    Is LibreCrawl as accurate as Screaming Frog?

    On structured data it was more accurate in my test, correctly detecting schema that Screaming Frog reported as missing. On redirects it matched Screaming Frog within two percent, but only after a second crawl with redirect-following disabled. Out of the box it hides redirects, so accuracy depends on configuring it correctly.

    Why does LibreCrawl report server errors that are not real?

    Because its crawler can trip bot protection and overload fragile servers, which then return 503, 429, or 500 responses. In my test, hundreds of these “errors” returned a clean 200 when I requested them by hand. Always re-check 5xx pages before trusting the count.

    Is LibreCrawl really free?

    Yes. It is open-source and free to run. The real cost is your time: setup, working around bugs, and filtering a much noisier issue list than a paid tool produces.