Hey, Magento Buddies!!
Your Robots.txt file is what tells the search engines which pages to access and index on your website. This is used mainly to avoid overloading your site with requests. Deep dive into How to Configure Robots.txt in Magento 2?
Complete Tutorial at: How to Configure Robots.txt in Magento 2?