Excite: The $1 Billion Search Engine Mistake
Why Excite Said No
In the spring of 1997, Vinod Khosla arranged a demonstration. Khosla, the Sun Microsystems co-founder who had joined Kleiner Perkins, sat on the board of Excite, then one of the most trafficked destinations on the internet. He had been shown a search engine developed by two Stanford graduate students, Larry Page and Sergey Brin, that ranked results using something called BackRub, an algorithm that measured the authority of web pages by counting and weighting inbound links. The Excite engineers ran a side-by-side test against their own search index. The results were not close. BackRub returned markedly more relevant pages. Excite’s chief executive reviewed the data and declined the offer. His reasoning, reported later, was candid: “Why on earth would we move to your algorithm? I want people to stay on my site. I make money when people stay on my site.”
This moment captures the central economic architecture of modern search. The Excite executive was not wrong about his incentives. He was operating a portal, a property that monetized attention through banner advertisements sold on a cost-per-thousand-impressions basis. Every page view generated inventory; every extra click through his directory generated another opportunity to display a display ad. A search engine that instantly delivered users to the most relevant external destination, sending them away as quickly as possible, would cannibalize his revenue. Better search was economically irrational for the incumbent.
Page and Brin faced a choice that defined the company they would build. They could optimize for the prevailing business model, padding their results with portal features, encouraging stickiness, and selling banner space. Or they could optimize for relevance, trusting that a sufficiently superior product would spread through word-of-mouth, capturing market share without marketing spend, and that some yet-undiscovered mechanism would eventually extract value from the intent of users who arrived with specific questions. They chose the second path. The structural consequence was a monopoly built on an orthogonal economic logic: monetizing departure rather than detention.
The Fee Is the Intent
The divergence between Google and the portals was not merely technical; it was a different theory of value capture. The portal model treated the web as a bundle of content to be corralled. Users entered through a homepage and navigated through channels, news, shopping, finance, each page loaded with display advertising. The economic unit was the impression, the eyeball held captive. Search, within this framework, was a loss leader, a utility to keep users inside the garden.
Google inverted this. The critical insight was that a user typing a query into a search box had already performed the most valuable act in advertising: they had expressed intent. A visitor to a portal’s finance page might be browsing; a user searching for “mortgage rates 30 year fixed” was preparing to transact. By delivering this high-intent traffic directly to the merchant or publisher who could satisfy the query, Google positioned itself as a broker of commercial intent rather than a seller of attention.
This alignment created a bidding mechanism that the portal model could not replicate. Advertisers did not pay for the possibility that a user might notice their banner; they paid for the certainty that a user had asked a question their product answered. The cost-per-click model meant Google only captured revenue when it successfully matched a buyer and seller, taking a toll on the transaction rather than rent on the real estate. The margins were extraordinary because the infrastructure cost of serving a text link was trivial compared to the value of a qualified lead. The gross margins implied by early internal documents, described by those who saw them as a “cash gusher”, reflected this arbitrage between the cost of indexing the web and the value of commercial intent flowing across it.
Yet this model carried a hidden structural constraint. To maintain the viral adoption that built the monopoly, Google had to remain the fastest path from question to answer. Any friction, splash pages, required registration, cluttered interfaces, would break the spell. The search result page had to be clean, white, austere. The algorithm had to favor the most relevant external destination, even if that destination was a competitor. The monopoly depended on neutrality.
The Crawl Budget Window
Most analyses of Google’s dominance focus on PageRank, the algorithmic insight that treated links as citations. This was indeed crucial. But an underappreciated structural advantage was temporal. Google captured the web at a specific moment when it was still possible to index the entire corpus with the resources available to graduate students.
Between 1993 and 1996, the web grew from 130 sites to more than 600,000: 723 percent annual growth rate sustained for four years. In 1996, when Page and Brin began crawling from Stanford’s servers, the web was small enough that a research project could build a complete copy. By 1998 or 1999, the cost to replicate this feat would have required tens to hundreds of millions of dollars in server infrastructure, bandwidth, and storage. The window to build a comprehensive index at low cost lasted approximately eighteen months.
This timing was not luck; it was exploitation of a capital barrier that closed behind them. Once Google possessed the complete index, it possessed a moat that could not be crossed without venture capital deployed at scale. More importantly, the completeness of the index created a compounding advantage. A more comprehensive index returned better results; better results attracted more users; more users generated more query data to refine the ranking; more traffic convinced webmasters to ensure their sites were crawlable, expanding the index further. The crawl budget, the computational resources allocated to discovering and indexing pages, became a strategic weapon. Competitors who started later faced not only higher capital costs but a network effect: the best data for training a ranking algorithm was the query-click behavior of the largest user base.
The Citation Graph
PageRank itself was elegant because it imported a mature academic concept into a chaotic new medium. Citations in scholarly literature indicate authority; a paper cited by many other papers carries more weight than an uncited one. Applying this to the hyperlink structure of the web treated each link as a vote, weighted by the authority of the voter. This solved a problem that plagued keyword-based engines like AltaVista and Excite, which could be gamed by stuffing pages with search terms.
The durability of this advantage came from the alignment between the mathematical model and the economic reality of the early web. Links were hard to fake at scale; they required another webmaster to endorse your page. The algorithm improved as the web grew more complex because the link graph became richer. This created a compounding effect distinct from the data moat: the structure of the web itself made the ranking system more accurate over time.
However, the algorithm’s power rested on an assumption about the architecture of information. PageRank assumes that authority flows through links, that the web is a graph of interconnected documents, and that the best answer to a query is the document most endorsed by this graph. This assumption held true while the web remained a collection of hyperlinked pages. It begins to fracture when content moves into walled gardens, when platforms replace websites, or when the primary interface shifts from clicking links to consuming generated summaries.
Sending Them Away to Keep Them
The defining tension of Google’s business has always been the conflict between relevance and revenue extraction. The search engine’s utility depends on sending users to external sites as quickly as possible. Its financial performance, however, creates constant pressure to delay that departure, o keep users on Google properties where ad inventory can be served, to answer questions directly in snippets that prevent clicks, to verticalize into travel, shopping, and local services where Google can capture the transaction margin rather than merely the advertising toll.
In the early years, this tension was resolved by the necessity of growth. The viral adoption curve required purity of product. Excite’s rejection of BackRub demonstrated what happened when portals prioritized monetization over relevance: they trained users to expect mediocrity. Google’s clean interface and fast results created a habit that became a monopoly. By the time the company held 90 percent market share in the United States, the habit was entrenched enough that Google could begin relaxing the constraint, inserting more ads above the fold, displaying knowledge panels that answered queries without clicks, and prioritizing Google Shopping over organic results.
Yet the monopoly remains fragile precisely because the original constraint was economic, not merely aesthetic. If Google degrades relevance too far, if it becomes obvious that the engine is optimizing for advertiser yield rather than user utility, the viral adoption that built the moat can reverse. Users do not pay to switch search engines; the cost of departure is zero. The monopoly persists because the algorithmic advantage, compounded over two decades, still produces superior results for most queries. But the margin for error has narrowed. Every pixel devoted to keeping a user on Google.com rather than sending them to the destination site erodes the structural foundation of the business.
When the Index Fractures
The most serious threat to Google’s model is not a competitor with a better algorithm but a fundamental shift in how information is organized and consumed. The search business rests on five implicit assumptions: that users prefer external destinations to walled gardens; that the web remains open and crawlable; that advertisers will continue to buy intent through keyword auctions; that link-based authority remains a reliable signal of quality; and that regulatory action will not force structural separation of the index from the ad business.
The first two assumptions are already cracking. Publishers, realizing that Google captures the advertising value while they bear the content creation cost, are retreating behind paywalls and proprietary platforms. Social media platforms, apps, and closed messaging systems have fragmented the web into indexable and non-indexable zones. If high-quality content moves into spaces Google cannot crawl, whether because of technical barriers or commercial restrictions, the comprehensiveness that justifies the monopoly evaporates.
More profoundly, artificial intelligence threatens to obsolete the link-click interface itself. If users receive satisfactory answers from large language models without visiting websites, the entire economic architecture collapses. Google can no longer broker intent if the user never leaves to transact. The company recognizes this: the shift from “ten blue links” to AI-generated overviews represents an attempt to capture the user within Google’s environment, to become the destination rather than the conduit. But this directly contradicts the original economic logic. The portal strategy that Excite pursued in 1997 and Google rejected is now, ironically, Google’s defensive posture against AI disruption.
The market has noticed this contradiction. Despite generating more net income than any other American company, surpassing Apple, Microsoft, ExxonMobil, and Berkshire Hathaway, Google’s equity trades at a significant discount to its technology peers, a gap representing roughly one trillion dollars in market capitalization. Investors are pricing in the possibility that the intent-arbitrage model, which required sending users away, cannot survive a transition to interfaces that keep users inside.
Ambition as a Cost Function
The resolution of these tensions has been shaped by the specific governance structure and psychological orientation of the founders. Larry Page, born in 1973 to computer science professors at Michigan State University, and Sergey Brin, born the same year in Moscow to a mathematician father, brought to the company a particular blend of technical absolutism and long-term ambition. Page’s stated goal was not merely to build a successful company but to “make the world better,” a framing that functioned as an economic constraint rather than mere public relations. It prevented the short-term optimization for page views that destroyed Excite’s competitive position.
This governance approach, treating the business as a vehicle for world-changing impact rather than profit maximization, created economic optionality. It justified the massive infrastructure investments required to index the growing web when venture capitalists might have preferred profitability. It permitted the development of projects like autonomous vehicles and artificial intelligence labs that had no immediate revenue connection to search. In the short term, this reduced returns; in the long term, it created a portfolio of bets that might define the post-search era.
However, as the company matured and faced public market pressures, this constraint relaxed. The founders’ declining operational involvement and the rise of professional management introduced the standard optimizations of mature monopolies: margin expansion, revenue per user growth, and defensive moat-building through vertical integration. The economic variable that had prevented Google from becoming a portal, Page and Brin’s specific refusal to optimize for stickiness, has weakened. The company now faces the same structural logic that destroyed Excite, and it must solve it without the benefit of the low-cost indexing window that built its original moat.
The Margin of the Arbitrage
Google’s dominance represents a specific historical conjuncture: a moment when the web was small enough to index cheaply but large enough to matter, combined with a business model that monetized intent rather than attention, executed by founders who rejected the portal economics that trapped their competitors. The monopoly persists because the compounding advantages of data, infrastructure, and algorithmic refinement created barriers that no rival has crossed.
Yet the fragility is equally specific. The model depends on the continued existence of an open, crawlable web that users wish to explore through ranked links. If the interface shifts to generated answers, or if content fragments into inaccessible silos, the economic engine stalls. The company’s current attempt to become the answer rather than the index, to keep users inside rather than send them away, risks validating the Excite executive’s logic from 1997, sacrificing the relevance that built the monopoly for the retention that sustains quarterly revenue.
The trillion-dollar discount on Google’s equity reflects this uncertainty. The market is weighing whether the highest net income in American corporate history represents a durable toll on global commerce or the final extraction of value from a model that is approaching obsolescence. The answer depends on whether the constraint that defined the company’s origin, the economic necessity of sending users away, can be abandoned without destroying the utility that made users arrive in the first place.
The Crawl Arms Race and the Cost of Comprehensiveness
The defensibility of Google’s index arose not merely from the eighteen-month window when crawling was cheap, but from the subsequent decade when crawling became astronomically expensive. In 1998, when Page and Brin incorporated the company and began serving queries from a garage in Menlo Park, the web contained roughly 2.4 million sites. By 2000, it contained 17 million. The infrastructure required to store, index, and serve this expanding corpus grew non-linearly; each order of magnitude increase in web pages required not just linearly more storage, but exponentially more computation to rank, de-duplicate, and update. Google’s competitors in the portal era, Yahoo, Excite, Lycos, viewed search as a feature, not a core business, and allocated capital accordingly. They spent on content licensing, sports partnerships, and email storage. Google spent on custom server racks, fiber optic networks, and data centers built near cheap hydroelectric power.
This capital allocation decision, invisible to users clicking on the clean white page, created a secondary moat. By 2003, Google had indexed over 3 billion web documents, a corpus that required an estimated 100,000 servers to process. The cost to replicate this infrastructure had risen from the hundreds of thousands of dollars available to Stanford graduate students to the billions of dollars required for global data center footprints. More importantly, the crawl itself had become a strategic weapon. The frequency with which Googlebot visited a site, the “crawl budget”, determined how quickly fresh content appeared in search results. Publishers optimized their site architecture to ensure Google’s spiders could navigate efficiently, creating a feedback loop: the more comprehensive the index, the more traffic publishers received; the more traffic at stake, the more effort publishers expended to ensure crawlability. This dynamic made the index self-reinforcing in a way that brute-force capital could not easily replicate. A competitor with $10 billion could build servers, but they could not buy the decade of crawl history that taught the algorithm which pages changed frequently and which remained static.
Yet this infrastructure advantage carried a structural fragility. The cost of maintaining comprehensiveness rose indefinitely while the marginal value of each additional page declined. By 2010, the “long tail” of obscure queries generated negligible advertising value; indexing the 500 billionth page contributed almost nothing to revenue but required proportional storage and computation. Google continued the arms race because the alternative, allowing corners of the web to go unindexed, created vulnerability. If Microsoft’s Bing could claim a more recent index of travel sites or local businesses, it could chip away at monopoly share in high-value verticals. The result was a permanent state of high fixed-cost investment, a treadmill that required billions in annual capital expenditure simply to maintain parity. The “cash gusher” from search advertising funded this infrastructure, but it also created a cost structure that made any decline in advertising yield catastrophic. Unlike software businesses with marginal costs near zero, Google’s search business carried the economic weight of a utility, assive, ongoing infrastructure spend that could not be dialed back without degrading the product.
Quality Score as Temporary Truce
The resolution of the relevance-versus-monetization tension in Google’s early years was not accidental; it was engineered through a specific auction mechanism that temporarily aligned advertiser incentives with user utility. When Google introduced AdWords in 2000, it faced the same corrupting pressure that had destroyed Excite. The obvious move, selling the top results to the highest bidder, would have degraded the organic quality that drove viral adoption. Users would quickly learn that the first link was an advertisement, trust would erode, and the monopoly would unravel.
The solution, implemented by economist Hal Varian and engineer Eric Veach, was the Quality Score algorithm. Advertisers bid on keywords, but their position and cost-per-click depended on both their bid and their click-through rate. A highly relevant ad that users actually clicked cost less per click than a irrelevant ad with a higher bid. This created a Darwinian pressure: advertisers who matched intent precisely paid less and received better placement; advertisers who tried to buy irrelevant traffic paid premiums that made their campaigns unprofitable. The economic genius was that Google’s revenue maximization, total ad dollars, depended on aggregate click-through rates. By weighting relevance heavily in the auction, Google ensured that monetization enhanced rather than degraded the user experience. The ads were often indistinguishable from organic results in utility; a user searching for “digital camera” who saw a link to Amazon’s camera section received value whether the link was paid or organic.
This truce held for roughly a decade. However, as growth targets intensified and the company faced quarterly earnings pressure, the weighting in the Quality Score algorithm shifted. The number of ads displayed above organic results grew from two to three to four. Shopping results, initially a neutral comparison of merchant feeds, evolved into paid placements. The “yellow box” that once clearly delineated advertisements blended into the white background. Each incremental change was justified through A/B testing that measured short-term revenue against short-term user engagement. But the cumulative effect was a slow betrayal of the original constraint. By 2019, for high-commercial-intent queries like “best CRM software” or “personal injury lawyer,” the entire viewport above the fold on a standard laptop screen consisted of paid placements and Google-owned properties. The organic results, the neutral index that built the monopoly, had been pushed below the scroll line. The Excite executive’s dilemma had been solved not by refusing to send users away, but by ensuring that every destination, paid or organic, generated toll revenue.
From Broker to Principal
The final erosion of the original economic logic came through vertical integration. In the portal era, Yahoo had attempted to be the destination for all activities, ail, news, shopping, finance. Google had defined itself against this by being the conduit. Yet starting in the early 2010s, Google began constructing its own properties for the most valuable query categories. When a user searches for a flight from New York to London, they now see a Google Flights interface embedded directly in the results page, powered by Google’s own booking partnerships. For local restaurants, Google Reviews and reservation buttons replace links to OpenTable or Yelp. For product searches, Google Shopping displays a carousel of merchant listings, ften paid, but increasingly including Google’s own fulfillment options.
This transformation represents the ultimate resolution of the 1997 Excite dilemma. Google no longer merely captures value by sending users to external sites; it captures value by keeping them within Google’s environment for the entire transaction. The economic rationale is impeccable: why settle for a $2 click when you can capture a $50 booking commission or a percentage of merchandise sold? But the structural consequence is that Google has become the portal it once defeated. The neutrality that justified the monopoly, the claim to be an unbiased index of the open web, has been replaced by a curated ecosystem where Google is simultaneously the search engine, the advertiser, and increasingly the merchant.
The fragility of this position is that it invites regulatory scrutiny and competitive attack. When Google was a neutral broker, it could claim safe harbor provisions; it was merely organizing information, not creating it. As it becomes a principal in transactions, selling flights, recommending restaurants, shipping products, it assumes liability and invites comparison to closed platforms like Amazon. The advertiser relationships that sustained the auction model become strained when Google competes directly with the merchants buying those ads. A hotel chain bidding on “Manhattan hotel” now finds itself paying for placement against Google’s own booking engine, a conflict of interest that recalls the portal economics of the 1990s.
The Dissolution of Commercial Intent
The most profound threat to the model, however, comes not from vertical integration but from the disintermediation of the click itself. The entire economic architecture rests on the moment of departure: a user types a query, scans the results, and clicks a link, thereby signaling to an advertiser that this is a qualified lead worth paying for. Large language models threaten to collapse this chain. When a user asks an AI assistant for the “best wireless router for a 3,000 square foot home with mesh capability,” and the AI synthesizes a specific product recommendation based on aggregated reviews, the user has no need to visit a comparison site, a retailer, or a manufacturer’s page. The intent is satisfied without a click.
Google recognizes this threat because it has already observed the trend in “zero-click” searches. For years, the company has displayed “featured snippets”, boxed answers extracted from websites, that answer factual queries directly on the results page. This served user convenience while destroying the business models of the publishers whose content was excerpted. A recipe site might see 100,000 impressions for “chocolate chip cookies” but receive only 10,000 clicks because Google displayed the ingredients and instructions directly. The publishers could not complain; blocking Google meant losing the remaining traffic. But the economic logic was clear: Google captured the value of the content without sending the user to the content creator.
AI-generated answers accelerate this to its logical conclusion. If a user asks “how do I fix a leaky faucet” and receives a step-by-step guide generated by an LLM trained on plumbing forums and DIY sites, the informational sites that once monetized this traffic through display ads and affiliate links receive nothing. Worse, the advertiser who once paid $3 per click for “emergency plumber” now finds that the user has already attempted the repair based on AI advice, or has been directed to a Google Local Services ad that bypasses the organic web entirely. The intent, the moment of commercial need that generated the $200 billion in search revenue, dissolves into a conversation with an oracle that needs no index.
Google’s response, the deployment of AI Overviews, attempts to split the difference. The company must provide generated answers to compete with ChatGPT and Perplexity, yet every accurate answer reduces the click volume that advertisers pay for. The company is attempting to introduce new ad formats within the AI interface, sponsored recommendations within the generated text, but these lack the precision of keyword intent. A user who asked for a router recommendation and receives a sponsored mention of Linksys has not clicked to compare prices; the advertiser has paid for an impression disguised as advice, the very model of banner advertising that proved inferior to search intent in the first place.
The trillion-dollar discount reflects this irreconcilable tension. The market is pricing in the probability that Google cannot simultaneously maintain the monopoly on search and transition to an AI answer engine. The former requires sending users away; the latter requires keeping them. The infrastructure moat, the billions spent on crawling and indexing, becomes worthless if users no longer need to visit websites. The algorithmic advantage of PageRank becomes obsolete if authority is determined not by links but by training data weights. And the orthogonal business model, the monetization of intent through departure, collapses when departure becomes unnecessary.
If the AI transition proceeds, Google faces a choice it has not confronted since 1997: it can optimize for the new interface, becoming a destination that keeps users inside its walls, sacrificing the neutrality that defined its brand and inviting the regulatory and competitive fates that destroyed the portals. Or it can cling to the link-based index, watching as users migrate to AI assistants that provide faster satisfaction. Either path leads to the erosion of the cash gusher. The constraint that defined the company’s origin, the necessity of sending users away, cannot be abandoned without destroying the utility that created the monopoly; yet maintaining that constraint in an age of generated answers may be equally fatal. The Excite executive’s rejection of BackRub was wrong about the future, but he understood something Page and Brin are now confronting: when you make money by keeping people, you cannot survive by letting them leave.
The irony is architectural. In 1997, Excite’s failure to acquire BackRub stemmed from a capital structure that required immediate monetization of page views; the company had raised venture capital on a portal model and could not tolerate the revenue dip that superior search would cause. Google’s current predicament inverts this trap. The company possesses nearly $200 billion in cash reserves and generates quarterly profits that dwarf the GDP of most nations, yet it faces the same structural coercion. Every percentage point of search market share lost to a conversational AI interface represents not merely a revenue decline but an obsolescence event for the crawl infrastructure that constitutes the majority of its capital expenditure. The data centers in Council Bluffs, the subsea cables traversing the Pacific, the global fleet of custom tensor processing units, hese assets depreciate on schedules that assume decades of search query volume. If users begin asking ChatGPT or Claude for product recommendations instead of typing keywords into a search bar, those investments become stranded assets, as economically productive as railroad tracks leading to a flooded mine.
Consider the specific mechanics of local search, a vertical that generates approximately $20 billion in annual high-margin revenue. A user searching for “emergency dentist open Saturday” currently triggers an auction among dental practices bidding on that intent, with Google taking $15–$40 per click. In an AI-mediated interaction, the user receives a specific practice recommendation based on aggregated review sentiment, insurance compatibility, and real-time appointment availability drawn from API integrations. The transaction occurs without a click; the dentist pays nothing to Google; the $40 of economic rent vanishes into the cost of training the model. Google can attempt to insert itself as the booking agent, taking a percentage of the procedure cost, but this requires sales relationships with individual dental practices: high-touch, low-margin business indistinguishable from the yellow pages it once disrupted. The orthogonal elegance dissolves; Google becomes the portal, the intermediary it was designed to destroy.
This transition exposes the fragility of the cultural constraint that originally prevented such regression. Page and Brin’s refusal to optimize for stickiness in 1998 was not merely a product decision but a theory of corporate governance: the founders possessed voting control sufficient to sacrifice quarterly earnings for long-term structural position. As institutional investors have accumulated influence and the founders have receded from operational roles, the pressure to demonstrate growth through vertical integration has eroded that constraint. The company now faces the same quarterly scrutiny that corrupted Excite’s algorithm in 1997, but with the added burden of maintaining infrastructure designed for a web that may cease to exist in its current form. The crawl budget, once the moat, has become the millstone; the index, once the asset, has become the liability. The executive who rejected BackRub failed to see that sending users away could create a larger business than keeping them, but he correctly diagnosed that once you commit to keeping them, the quality of what you keep them for must justify the imprisonment. Google’s AI answers must now be sufficiently superior to the open web to justify the walls it is constructing around its garden: alls that, once complete, will make it indistinguishable from the portal it replaced.