points by cmeacham98 1 year ago

I work for Amazon, but not directly on web crawling.

Based on the internal information I have been able to gather, it is highly unlikely this is actually Amazon. Amazonbot is supposed to respect robots.txt and should always come from an Amazon-owned IP address (You can see verification steps here: https://developer.amazon.com/en/amazonbot).

I've forwarded this internally just in case there is some crazy internal team I'm not aware of pulling this stunt, but I would strongly suggest the author treats this traffic as malicious and lying about its user agent.

xena 1 year ago

Randomly selected IPs from my logs show that 80% of them have the matching that forward confirming reverse DNS domain. The most aggressive ones were from the amazonbot domain.

Believe what you want though. Search for `xeiaso.net` in ticketing if you want proof.

  • petee 1 year ago

    Reverse DNS doesn't mean much, they can set it to anything; can you forward match them to any amazon domain?

    • xena 1 year ago

      It's forward confirming reverse DNS. I assumed that everyone does that by default.

      • petee 1 year ago

        What everyone does by default doesn't matter really here, it's that an IP owner/user can literally set the reverse to any arbitrary domain regardless if the actual domain has a record for that IP. What matters is both match, thats all I meant

        • nalllar 1 year ago

          xena said "forward confirming reverse" twice which means rdns and then resolving that forward to confirm it matches.

          • petee 1 year ago

            I don't know if it was edited or I missed it in the first post but you're right.

            I'd still be surprised if an Amazon domain resolved to a residential IP

  • guardiangod 1 year ago

    So you said the IPs are residential IP, but their reverse DNS points to a amazonbot domain? Does that even make sense?