Class RobotsTxtFilter

java.lang.Object
ch.tocco.nice2.dms.impl.publish.RobotsTxtFilter
All Implemented Interfaces:
javax.servlet.Filter

@Component public class RobotsTxtFilter extends Object implements javax.servlet.Filter
This filter delivers a robots.txt. Paths which mustn't be crawled and indexed by search engine bots can be contributed by implementing a RobotsTxtDisallowedPathsProvider. -------------------------------------------------------------------- Content for non-production systems, as they should never be indexed by search engines: User-agent: * [... additional user agent line for each contributed additional user agent ...] Disallow: / -------------------------------------------------------------------- Content for production systems with disallowed content: User-agent: * [... additional user agent line for each contributed additional user agent ...] Disallow: /disallowed-path/1 Disallow: /another/disallowed/path Sitemap: {domain}/sitemap.xml -------------------------------------------------------------------- Content for production systems without disallowed content: User-agent: * [... additional user agent line for each contributed additional user agent ...] Allow: / Sitemap: {domain}/sitemap.xml --------------------------------------------------------------------
  • Field Details

  • Constructor Details

  • Method Details

    • setAdditionalUserAgents

      @Autowired(required=false) public void setAdditionalUserAgents(List<String> additionalUserAgents)
      Contribution for additional user agents which must be named explicitly (e.g. AdsBot-Google, Googlebot, Googlebot-Image to enable crawling for Ad bots which ignore "*")
    • setDisallowedPathsProviders

      @Autowired(required=false) public void setDisallowedPathsProviders(List<RobotsTxtDisallowedPathsProvider> disallowedPathsProviders)
    • doFilter

      public void doFilter(javax.servlet.ServletRequest request, javax.servlet.ServletResponse response, javax.servlet.FilterChain chain) throws IOException, javax.servlet.ServletException
      Specified by:
      doFilter in interface javax.servlet.Filter
      Throws:
      IOException
      javax.servlet.ServletException