@bathalumang_peppa@lemmy.world to Technology@lemmy.worldEnglish • 1 year agoApple considered switching to DuckDuckGo from Google for Safari - Bloomberg Newswww.reuters.comexternal-linkmessage-square246fedilinkarrow-up11.17Karrow-down122cross-posted to: apple@lemdro.id
arrow-up11.15Karrow-down1external-linkApple considered switching to DuckDuckGo from Google for Safari - Bloomberg Newswww.reuters.com@bathalumang_peppa@lemmy.world to Technology@lemmy.worldEnglish • 1 year agomessage-square246fedilinkcross-posted to: apple@lemdro.id
minus-square@loki@lemmy.mllinkfedilinkEnglish2•1 year agoI can’t find any information about their search engine crawler. Isn’t it standard for search engines to label their crawlers or something?
minus-square@darth_helmet@sh.itjust.workslinkfedilinkEnglish2•1 year agoIt’s conventional to do this, but a user agent string is entirely up to the client, and robots.txt is just a suggestion. So for the best results, you probably want to mock Google’s crawler because it’s suicidal to block that if you want search traffic.
minus-squareAtemulinkfedilinkEnglish1•1 year agohttps://help.kagi.com/kagi/search-details/search-sources.html
I can’t find any information about their search engine crawler. Isn’t it standard for search engines to label their crawlers or something?
It’s conventional to do this, but a user agent string is entirely up to the client, and robots.txt is just a suggestion.
So for the best results, you probably want to mock Google’s crawler because it’s suicidal to block that if you want search traffic.
https://help.kagi.com/kagi/search-details/search-sources.html