In this guide we are going to show you how you can use the wget command in Linux, we will also give you 12 useful wget commands. In the end, you will have knowledge of every wget command and you will be able to use it to download files from the world wide web.
What is the Wget Command?
Wget is the computer tool which was made by the GNU Project. You may use it as you like for retrieving content and files from different web servers. The name comes from a combination of World Wide Web and the word get. It will support downloads from FTP, SFTP, HTTP, and HTTPS.
Wget was made in portable C and usable on any Unix system. It’s also possible to implement it on Mac OS X, Microsoft Windows, AmigaOS and other popular platforms.
How to Install Wget?
For this wget command example, we are going to be using Ubuntu 16.04. However, the syntax should work on every linux distribution.
Begin by accessing your server with SSH. Run the command below to install wget on your system:
apt-get install wget
After the setup is done, you should be prepared to use it. Also, knowing simple SSH commands could cause things to be easier.
Wget Command Examples
In order to begin, we are going to show you 12 wget command examples which you may use for everyday tasks. Make sure you know tat you could also call this function via scripts or cron jobs.
Using Wget Command to Download Single Files
The simplest wget command example is a download of a single file, then storing it on your current working directory. For example, you could download the latest version of WordPress by executing the command below:
wget https://wordpress.org/latest.zip
You should receive the following output:
--2018-02-23 12:53:10-- https://wordpress.org/latest.zip Resolving wordpress.org (wordpress.org)... 198.143.164.252 Connecting to wordpress.org (wordpress.org)|198.143.164.252|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 9332728 (8,9M) [application/zip] Saving to: 'latest.zip' latest.zip 100%[===================>] 8,90M 6,88MB/s in 1,3s 2018-02-23 12:53:14 (6,88 MB/s) - 'latest.zip' saved [9332728/9332728]
This guide will show you how to use the wget command in Linux. We’ll also provide 12 useful wget command examples. By the end, you’ll know all about the wget command and will be able to use it to download files from the World Wide Web.
What is the Wget Command?
Wget is a computer tool created by the GNU Project. You can use it to retrieve content and files from various web servers. The name is a combination of World Wide Web and the word get. It supports downloads via FTP, SFTP, HTTP, and HTTPS.
Wget is created in portable C and usable on any Unix system. It’s also possible to implement on Mac OS X, Microsoft Windows, AmigaOS and other popular platforms.
How to Install Wget?
For this wget command demonstration, we’ll be using Ubuntu 16.04. But the syntax will work on any other Linux distribution too.
Firstly, access your server via SSH. Execute the following command to install wget on your system:
apt-get install wget
Once the setup finishes, you’ll be ready to use it. Also, the knowledge of basic SSH commands can make things easier.
Wget Command Examples
To get you started, we’ll provide 12 wget command examples that you can use for everyday tasks. Keep in mind that you may also call this function from scripts and cron jobs!
Using Wget Command to Download Single Files
One of the most basic wget command examples is downloading a single file and storing it on your current working directory. For example, you may get the latest version of WordPress by using the following:
wget https://wordpress.org/latest.zip
Here is the output that you will see:
--2018-02-23 12:53:10-- https://wordpress.org/latest.zip Resolving wordpress.org (wordpress.org)... 198.143.164.252 Connecting to wordpress.org (wordpress.org)|198.143.164.252|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 9332728 (8,9M) [application/zip] Saving to: 'latest.zip' latest.zip 100%[===================>] 8,90M 6,88MB/s in 1,3s 2018-02-23 12:53:14 (6,88 MB/s) - 'latest.zip' saved [9332728/9332728]
In the output above you will see that a file called latest.zip is downloaded in the current working directory. You should also see some additional information, like the download progress, speed, size, time, and date.
Using Wget Command to Download Multiple Files
We could take wget usage one step further and download several files at the same time. In order to do that, we are going to have to create a text document, then place a download URL there. In the next example , we are going to get the latest versions of WordPress, joomla, and Drupal. Enter the command below:
nano example.txt
This should make an example.txt file, then open a text editor interface. Paste the links below there:
https://wordpress.org/latest.zip https://downloads.joomla.org/cms/joomla3/3-8-5/Joomla_3-8-5-Stable-Full_Package.zip https://ftp.drupal.org/files/projects/drupal-8.4.5.zip
After it’s finished, you may use –I to retrieve all the files located in your example text file:
wget -i example.txt
Wait until the procedure is done, then you should have the installations of the three most popular content management systems (CMS for short.)
Using Wget Command to Get Files Under Different Names
In the next wget example, we are going to be saving a file using a different name with the help of –O option:
wget -O wordpress-install.zip https://wordpress.org/latest.zip
In our example, the downloaded resource is going to be called and saved as wordpress-install.zip instead of it’s original name.
Using Wget Command to Save Files in Specified Directory
You could use wget to place a file in a different directory using the –P function:
wget -P documents/archives/ https://wordpress.org/latest.zip
The file you are getting using this syntax is going to show up in the documents/archives/ folder.
Using Wget Command to Limit Download Speed
Via wget, you could limit the speed of your downloads, which can be useful when getting huge files and stop it from using every bit of your bandwidth. In this wget example we will limit the bandwidth to 500k:
wget --limit-rate=500k https://wordpress.org/latest.zip
Using Wget Command to Set Retry Attempts
Internet connection problems might appear and interrupt your download. To deal with this issue, we simply increase the retry attempts using the –tries function:
wget -tries=100 https://wordpress.org/latest.zip
Using Wget Command to Download in Background
For really big files, you can take advantage of the –b function. This will cause for your content to be downloaded in the background.
wget -b http://example.com/beefy-file.tar.gz
A wget-log is going to show up in your working directory, though you could use it to check your download’s progress and status. The command below is going to do the trick:
tail -f wget-log
Using Wget Command to Download via FTP
The command could be used with FTP, however you will have to specify the username and password as shown in the next example:
wget --ftp-user=YOUR_USERNAME --ftp-password=YOUR_PASSWORD ftp://example.com/something.tar
Using Wget Command to Continue Interrupted Downloads
The download could be interrupted if you lose your internet connection or experience a power outage, which can happen pretty often when retrieving big files. Instead of starting over, you can just proceed the download using the –c function:
wget -c https://example/very-big-file.zip
If you continue without using the –c function, the new file is going to have .1 added at the end since it’s already there.
Using Wget Command to Retrieve Whole Websites
You could also use wget to retrieve the content of an entire site. This will let you see it locally without an internet connection. Here is an example:
wget --mirror --convert-links --page-requisites --no-parent -P documents/websites/ https://some-website.com
Let’s look at the ingredients of the wget above us:
–mirror | This causes for your download to be recursive. |
–convert-links | Every link is going to be converted for proper offline usage. |
–page-requisites | The following will include all necessary files such as CSS, JS, and images. |
–no-parent | This will make sure that every directory above the hierarchy is not retrieved. |
-P documents/websites/ | This should tell the content to go to the specified directory. |
After the procedure is done, you should be able to open the downloaded website locally and locate all of the files in the documents/websites/ directory.
Using Wget Command to Locate Broken Links
Let’s attempt something more advanced. We will use wget to find all of the broken URLs which display an 404 error on a particular website. Begin by running the following:
wget -o wget-log -r -l 5 --spider http://example.com
- –o Gathers output into a file for later use.
- –l Specifies the recursion level.
- –r Makes the download recursive.
- –spider Sets wget to spider mode.
We could now investigate the wget-log file to look for a list of broken links. This is the command to do it:
grep -B 2 '404' wget-log | grep "http" | cut -d " " -f 4 | sort -u
Using Wget Command to Download Numbered Files
In case you have files or images numbered in a particular list, you could download them if you’d like to, using the command below:
wget http://example.com/images/{1..50}.jpg
Conclusion
By finishing this tutorial you learn several uses for the wget command, you could now use it to get a single file or several. And maybe you’ve learned a couple of advanced uses, like downloading an entire website or looking for broken URLs.