Batch download files from a list of URLs

This guide provides a comprehensive walkthrough on how to create a Bash script for batch downloading files from a list of URLs. This script will rely on wget, a free utility for non-interactive download of files from the web.

Prerequisites

Ensure that wget is installed on your system. If not, you can install it using the package manager of your distribution. For example, on Debian-based distributions:

sudo apt-get update
sudo apt-get install wget

Step 1: Script Initialization

Start by creating a new bash script file:

touch batch_downloader.sh

Open the file in your preferred text editor. The first line of any bash script should be the shebang (#!/bin/bash), which tells the system the script should be executed as a bash script:

#!/bin/bash

Step 2: Accepting User Input

We will allow users to provide custom options and settings, specifically:

1. URL List File: A file containing the list of URLs to download files from.
2. Output Directory: Directory to save the downloaded files.

Here’s how to add this feature to your script:

read -p "Enter the file path containing URLs: " url_file
read -p "Enter the output directory: " output_directory

Step 3: Input Validation

Validate that the URL list file exists and is readable, and the output directory is writable:

if [[ ! -f "$url_file" ]]; then
  echo "URL file does not exist."
  exit 1
fi

if [[ ! -d "$output_directory" ]]; then
  echo "Output directory does not exist."
  exit 1
fi

Step 4: The Download Process

Create a loop that goes through all URLs in the file and uses wget to download them:

while read -r url; do
  wget -P "$output_directory" "$url"
done < "$url_file"

Step 5: Handling Errors

To handle download errors, capture wget‘s exit status:

while read -r url; do
  wget -P "$output_directory" "$url"

  if [[ $? -ne 0 ]]; then
    echo "An error occurred while downloading $url."
  fi
done < "$url_file"

Full Script

Here is the full bash script:

#!/bin/bash

# Prompt user for input
read -p "Enter the file path containing URLs: " url_file
read -p "Enter the output directory: " output_directory

# Validate input
if [[ ! -f "$url_file" ]]; then
  echo "URL file does not exist."
  exit 1
fi

if [[ ! -d "$output_directory" ]]; then
  echo "Output directory does not exist."
  exit 1
fi

# Download files
while read -r url; do
  wget -P "$output_directory" "$url"

  if [[ $? -ne 0 ]]; then
    echo "An error occurred while downloading $url."
  fi
done < "$url_file"

To run the script, use:

bash batch_downloader.sh

Or make the script executable:

chmod +x batch_downloader.sh

Then run:

./batch_downloader.sh

Considerations

This script doesn’t handle URLs with spaces. If your URL file contains such URLs, you may need to modify the script accordingly.

wget can be customized with a wide range of options. For instance, you could add options to limit the download rate, retry failed downloads, or enable quiet mode. This script uses the basic form of the wget command, and can be expanded to include more complex features, such as logging, verbose output, parallel downloads, and more.

Remember to always respect the terms of service of the server you are downloading from and don’t overload servers with too many simultaneous connections.