But, optimizing your robots.txt file remains to be an search engine optimization greatest apply because it lets crawlers know which pages aren’t price indexing.