shell linux怎么从文件中取出要访问的url，然后批量去curlLinuxCurlcommandisveryam

Linux Curl command is very amazing. It’s very simple command which is use to send or get data from and to any server. Server would be any server like end point URL, ftp endpoint, etc.

In this tutorial we will go over how to read file line by line and then perform curl operation to get HTTP response code for each HTTP URL.

Flow would look like this:

Create file crunchify.txt
Add 5 URLs including http:// as part of URL
Create curl command and read crunchify.txt file
Perform curl operation to get HTTP response code
Print HTTP Response code

Let’s get started:

Step-1

Go to Mac Terminal if you are running this script on Macbook or open bash shell on Linux/Unix terminal

Step-2

Go to ~/<username>/Downloads/
Create file crunchify.txt
Put below URLs into file and save file using command vi crunchify.txt to create file and :wq to save file.

https://crunchify.com
http://google.com
https://www.facebook.com
http://crunchify.me
https://crunchify.com:8080

Step-3

Execute below command to get result.

for URL in `cat crunchify.txt`; do echo $URL; curl -m 10 -s -I $1 "$URL" | grep HTTP/1.1 |  awk {'print $2'}; done

-m: Maximum time in seconds that you allow the whole operation to take. This is useful for preventing your batch jobs from hanging for hours due to any network issue
-s: show error message
-I: Show document info only
awk: matches the pattern and prints result

Result:

bash-3.2$ for URL in `cat crunchify.txt`; do echo $URL; curl -m 10 -s -I $1 "$URL" | grep HTTP/1.1 |  awk {'print $2'}; done
https://crunchify.com
200
http://google.com
301
https://www.facebook.com
200
http://crunchify.me
405
https://crunchify.com:8080

If you want to see detailed result and response just try adding -v into curl and you will see detailed verbose result in command prompt. Here is an updated command.

for URL in `cat crunchify.txt`; do echo $URL; curl -v -m 10 -s -I $1 "$URL" | grep HTTP/1.1 |  awk {'print $2'}; done

And that’s it, above command will print HTTP response code for each URL.

Above command and tutorial works well if you have any of below questions too:

Use bash to read a file and then execute a command
Run a command on each line in a file
bash – Execute a command once per line of piped input?
HowTo : Read a file Line By Line
Shell Script to execute a command on every line of a file

Join the Discussion

Share & leave us some comments on what you think about this topic or if you like to add something.

shell linux怎么从文件中取出要访问的url，然后批量去curl