Skip to content
Snippets Groups Projects

crawl-idiv-website

  • Clone with SSH
  • Clone with HTTPS
  • Embed
  • Share
    The snippet can be accessed without any authentication.
    Authored by Dirk Sarpe

    crawls the idiv website to warm the cache (e.g. after truncating the search tables)

    Edited
    crawl-idiv-website.sh 294 B
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11 
    #!/usr/bin/env bash
wget --recursive --level 7 --reject \
"*.mp4","*.MP4",\
"*.pdf","*.PDF",\
"*.jpg","*.JPG","*.jpeg","*.JPEG",\
"*.svg","*.SVG",\
"*.png","*.PNG",\
"*.xlsx","*.XLSX","*.xls","*.XLS",\
"*.docx","*.DOCX","*.doc","*.DOC",\
"*.pptx","*.PPTX","*.ppt","*.PPT" \
https://www.idiv.de
    0% or .
    You are about to add 0 people to the discussion. Proceed with caution.
    Please register or to comment