web crawlers

常见例句

Once the Sitemap XML is completed, you can make it available to Web crawlers by submitting a URL to the sitemap to those sites that support the protocol.
完成 Sitemap XML 文件后，便可以向支持这一协议的站点提交站点地图的 URL，这样 web 爬虫程序便可以使用该文件了。
Sitemaps 0.90 is a simple and intuitive way for webmasters to provide the right level of information to Web crawlers so that they can efficiently crawl a Web site.
Sitemaps 0.90为 Web 站点管理员提供了简单而且直观的方法，使他们可以向 web 爬虫程序提供正确的信息，从而使爬虫程序能够高效地爬行某个 Web 站点。
Also known as "invisible web", the term "deep web" refers to a vast repository of underlying content, such as documents in online databases that general-purpose web crawlers cannot reach.
“深网”也被称做“隐形网”，指的是网络中存储数量浩繁的基本信息的空间，例如存储于在线数据库中的文件，一般用途的网络搜索器是找不到这些文件的。

"Think algorithms and models and different software and Web crawlers, " says Christine Elliott, an American Express spokesperson.
FORBES: Magazine Article
Industry specific web crawlers or vulnerability discovery bots, network scans, and resource exhaustion attacks against web servers are rampant.
FORBES: Clean That Network Traffic
Through the use of proprietary Web crawlers, this news organization had breached the firewall to provide its readers with an upper hand.
FORBES: Data Security: A Shareholder Protection Imperative