LisanBot

LisanBot is the web crawler run by Lisan (LISAN AI - FZE, Dubai). It identifies itself on every request as LisanBot/1.0 (+https://lisan.com/bot), and it fetches from one fixed address, makhzan-bot-5jhjvuzuldqgm.centralus.cloudapp.azure.com (20.109.207.167), whose reverse lookup names that host. A crawler that uses this name from another address is not ours.

What it reads

Public web pages only: company websites (home, about, team, products, contact), public tender and procurement portals, public company registers and exchange notices. It reads one page at a time from a site, two seconds apart, or the delay your robots.txt asks for, up to ten seconds. It never signs in, never creates an account, never fills in a form and never gets round a check that asks it to prove it is a person. A page that asks for any of those is recorded as closed and left alone for seven days; after three such answers in a row it stays away for ninety.

robots.txt

Honoured. LisanBot reads yours before any other page on your site and again every 24 hours. It follows the group for LisanBot if there is one, otherwise *. A Disallow is a refusal, and the page is never fetched. A Crawl-delay is honoured up to ten seconds.

What it keeps

A copy of each page as it was received, its address, the time, and the response headers. From these Lisan builds records of companies and tenders for use inside Lisan's products. Lisan does not republish your pages. A page is read again when it is likely to have changed: after about 90 days for companies in the UAE, Saudi Arabia, Qatar, Kuwait, Bahrain and Oman, and 180 elsewhere, and less often when nothing has moved.

To keep it off your site

Write to bot@lisan.com with the domain name. The site is excluded within seven days and what was kept of it is removed. A Disallow for LisanBot in your robots.txt does the same for future reads, within 24 hours. If you think LisanBot has read something it should not have, write to the same address and say which pages.