In an age of LLMs, is it time to reconsider human-edited web directories?
Back in the early-to-mid '90s, one of the main ways of finding anything on the web was to browse through a web directory.
These directories generally had a list of categories on their front page. News/Sport/Entertainment/Arts/Technology/Fashion/etc.
Each of those categories had subcategories, and sub-subcategories that you clicked through until you got to a list of websites. These lists were maintained by actual humans.
Typically, these directories also had a limited web search that would crawl through the pages of websites listed in the directory.
Lycos, Excite, AltaVista, and of course Yahoo all were originally web directories of this sort.
By the late '90s, the standard narrative goes, the web got too big to index websites manually.
Google promised the world its algorithms would weed out the spam automatically.
And for a time, it worked.
But then SEO and SEM became a multi-billion-dollar industry. The spambots proliferated. Google itself began promoting its own content and advertisers above search results.
And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.
My question is, if a lot of the web is turning to crap, do we even want to search the entire web anymore?
Do we really want to search every single website on the web?
Or just those that aren't filled with LLM-generated SEO spam?
Or just those that don't feature 200 tracking scripts, and passive-aggressive privacy warnings, and paywalls, and popovers, and newsletters, and increasingly obnoxious banner ads, and dark patterns to prevent you cancelling your "free trial" subscription?
At some point, does it become more desirable to go back to search engines that only crawl pages on human-curated lists of trustworthy, quality websites?
And is it time to begin considering what a modern version of those early web directories might look like?
npub13sc58cl5pze5s5srdqdlndmg6nh2tu3sg6tryt2wsgw9d5yzfg2qtkwx88 (npub13sc…wx88) #tech #google #web #internet #LLM #LLMs #enshittification #technology #search #SearchEngines #SEO #SEM
AJ Sadauskas /
npub1s8…5fgvf
2024-03-05 03:37:11
Author Public Key
npub1s8hcaap75txgxanxkqqu0x8mv9ew6nhyu5q3yjh8dn83hxgr8mrqs5fgvfPublished at
2024-03-05 03:37:11Event JSON
{
"id": "1bd3e36f04abcb827d70a7aabf8daba3e69ef68b45346795f27f2e18bcbdc589",
"pubkey": "81ef8ef43ea2cc837666b001c798fb6172ed4ee4e501124ae76ccf1b99033ec6",
"created_at": 1709609831,
"kind": 1,
"tags": [
[
"p",
"8c3143e3f408b3485203681bf9b768d4eea5f2304696322d4e821c56d0824a14",
"wss://relay.mostr.pub"
],
[
"t",
"llm"
],
[
"t",
"llms"
],
[
"t",
"technology"
],
[
"t",
"search"
],
[
"t",
"searchengines"
],
[
"t",
"seo"
],
[
"t",
"sem"
],
[
"t",
"tech"
],
[
"t",
"google"
],
[
"t",
"web"
],
[
"t",
"internet"
],
[
"t",
"enshittification"
],
[
"proxy",
"https://aus.social/users/ajsadauskas/statuses/112040989944870233",
"activitypub"
]
],
"content": "In an age of LLMs, is it time to reconsider human-edited web directories?\n\nBack in the early-to-mid '90s, one of the main ways of finding anything on the web was to browse through a web directory.\n\nThese directories generally had a list of categories on their front page. News/Sport/Entertainment/Arts/Technology/Fashion/etc.\n\nEach of those categories had subcategories, and sub-subcategories that you clicked through until you got to a list of websites. These lists were maintained by actual humans.\n\nTypically, these directories also had a limited web search that would crawl through the pages of websites listed in the directory.\n\nLycos, Excite, AltaVista, and of course Yahoo all were originally web directories of this sort.\n\nBy the late '90s, the standard narrative goes, the web got too big to index websites manually.\n\nGoogle promised the world its algorithms would weed out the spam automatically.\n\nAnd for a time, it worked.\n\nBut then SEO and SEM became a multi-billion-dollar industry. The spambots proliferated. Google itself began promoting its own content and advertisers above search results.\n\nAnd now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.\n\nMy question is, if a lot of the web is turning to crap, do we even want to search the entire web anymore?\n\nDo we really want to search every single website on the web? \n\nOr just those that aren't filled with LLM-generated SEO spam? \n\nOr just those that don't feature 200 tracking scripts, and passive-aggressive privacy warnings, and paywalls, and popovers, and newsletters, and increasingly obnoxious banner ads, and dark patterns to prevent you cancelling your \"free trial\" subscription?\n\nAt some point, does it become more desirable to go back to search engines that only crawl pages on human-curated lists of trustworthy, quality websites?\n\nAnd is it time to begin considering what a modern version of those early web directories might look like?\n\nnostr:npub13sc58cl5pze5s5srdqdlndmg6nh2tu3sg6tryt2wsgw9d5yzfg2qtkwx88 #tech #google #web #internet #LLM #LLMs #enshittification #technology #search #SearchEngines #SEO #SEM",
"sig": "16b10031d3e76de81c72e26d7efe11e08cdca73a126a377e3baea36b1382aecf3c4ff7322fdb23f15c6872c999baed53342da2ea5a48317190907f24c4d69273"
}