Tip of the Trade: cURL

by Carla Schroder

The "client for URLs," or cURL utility, is a deceptively simple download and copying utility that recognizes a host of transfer protocols. Its capabilities expand with the user's familiarity.

Discuss this article in the ServerWatch discussion forum

The "client for URLs," or cURL utility, is one of those deceptively simple, handy little programs. The more you dig into it, the more you learn about what it can do. cURL is a download and copying utility that recognizes a host of transfer protocols — HTTP, HTTPS, FTP, FTPS, TFTP, GopherR, Telnet, DICT, LDAP, and FILE. It also supports, and I quote from the cURL Web site:

"SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos ...), file transfer resume, proxy tunneling and a busload of other useful tricks."

Some folks compare cURL to wget, but they're really different beasts. wget is a mirroring tool and file downloader. cURL can do all manner of clever tricks in addition to merely downloading files. Its simplest invocation fetches Web pages, as this snippet shows:

$ curl http://www.serverwatch.com/ 

<b style="font-size:10pt;"></a
href="http://cp.jupiterweb.com/index.php/3681_default">Winners announced June 5!
</a></b></p>--> <P><span class=header>New on
class=topic>Greening Your Data Center — You May Have No
Choice</span></a><br><span class=body>[August 22,

As you can see, cURL fetches the raw page code. This could be useful if you don't want to use the "View page source" button in a graphical Web browser. But cURL can perform even more useful tasks, such as measuring download performance on a Web site:

$ $ curl -w 'nPreXfer time:t%{time_pretransfer}nStartXfer
time:t%{time_starttransfer}nnTotal time:t%{time_total}n' -o /dev/null -s
PreXfer time: 0.138 StartXfer time: 0.286 Total time: 0.286

This shows that the file transfer starts almost as soon as the user connects, which is a good thing. You can fetch server banners and see what different sites are running, or see what your own server is revealing to the world:

$ curl -I www.serverwatch.comHTTP/1.1 200 OK
Date: Thu, 23 Aug 2007 20:12:03 GMT
Server: Apache
Content-Type: application/x-httpd-php

That's a good banner. It doesn't give away too much information. Visit cURL for downloads and documentation and to learn hundreds of nifty ways to use cURL.

This article was originally published on Monday Aug 27th 2007
Mobile Site | Full Site