diff options
Diffstat (limited to 'plugins/FTPFileYM/curl/docs/libcurl/libcurl-tutorial.3')
| -rw-r--r-- | plugins/FTPFileYM/curl/docs/libcurl/libcurl-tutorial.3 | 1369 |
1 files changed, 1369 insertions, 0 deletions
diff --git a/plugins/FTPFileYM/curl/docs/libcurl/libcurl-tutorial.3 b/plugins/FTPFileYM/curl/docs/libcurl/libcurl-tutorial.3 new file mode 100644 index 0000000000..1cca23f335 --- /dev/null +++ b/plugins/FTPFileYM/curl/docs/libcurl/libcurl-tutorial.3 @@ -0,0 +1,1369 @@ +.\" ************************************************************************** +.\" * _ _ ____ _ +.\" * Project ___| | | | _ \| | +.\" * / __| | | | |_) | | +.\" * | (__| |_| | _ <| |___ +.\" * \___|\___/|_| \_\_____| +.\" * +.\" * Copyright (C) 1998 - 2011, Daniel Stenberg, <daniel@haxx.se>, et al. +.\" * +.\" * This software is licensed as described in the file COPYING, which +.\" * you should have received as part of this distribution. The terms +.\" * are also available at http://curl.haxx.se/docs/copyright.html. +.\" * +.\" * You may opt to use, copy, modify, merge, publish, distribute and/or sell +.\" * copies of the Software, and permit persons to whom the Software is +.\" * furnished to do so, under the terms of the COPYING file. +.\" * +.\" * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY +.\" * KIND, either express or implied. +.\" * +.\" ************************************************************************** +.\" +.TH libcurl-tutorial 3 "4 Mar 2009" "libcurl" "libcurl programming" +.SH NAME +libcurl-tutorial \- libcurl programming tutorial +.SH "Objective" +This document attempts to describe the general principles and some basic +approaches to consider when programming with libcurl. The text will focus +mainly on the C interface but might apply fairly well on other interfaces as +well as they usually follow the C one pretty closely. + +This document will refer to 'the user' as the person writing the source code +that uses libcurl. That would probably be you or someone in your position. +What will be generally referred to as 'the program' will be the collected +source code that you write that is using libcurl for transfers. The program +is outside libcurl and libcurl is outside of the program. + +To get more details on all options and functions described herein, please +refer to their respective man pages. + +.SH "Building" +There are many different ways to build C programs. This chapter will assume a +UNIX-style build process. If you use a different build system, you can still +read this to get general information that may apply to your environment as +well. +.IP "Compiling the Program" +Your compiler needs to know where the libcurl headers are located. Therefore +you must set your compiler's include path to point to the directory where you +installed them. The 'curl-config'[3] tool can be used to get this information: + +$ curl-config --cflags + +.IP "Linking the Program with libcurl" +When having compiled the program, you need to link your object files to create +a single executable. For that to succeed, you need to link with libcurl and +possibly also with other libraries that libcurl itself depends on. Like the +OpenSSL libraries, but even some standard OS libraries may be needed on the +command line. To figure out which flags to use, once again the 'curl-config' +tool comes to the rescue: + +$ curl-config --libs + +.IP "SSL or Not" +libcurl can be built and customized in many ways. One of the things that +varies from different libraries and builds is the support for SSL-based +transfers, like HTTPS and FTPS. If a supported SSL library was detected +properly at build-time, libcurl will be built with SSL support. To figure out +if an installed libcurl has been built with SSL support enabled, use +\&'curl-config' like this: + +$ curl-config --feature + +And if SSL is supported, the keyword 'SSL' will be written to stdout, +possibly together with a few other features that could be either on or off on +for different libcurls. + +See also the "Features libcurl Provides" further down. +.IP "autoconf macro" +When you write your configure script to detect libcurl and setup variables +accordingly, we offer a prewritten macro that probably does everything you +need in this area. See docs/libcurl/libcurl.m4 file - it includes docs on how +to use it. + +.SH "Portable Code in a Portable World" +The people behind libcurl have put a considerable effort to make libcurl work +on a large amount of different operating systems and environments. + +You program libcurl the same way on all platforms that libcurl runs on. There +are only very few minor considerations that differ. If you just make sure to +write your code portable enough, you may very well create yourself a very +portable program. libcurl shouldn't stop you from that. + +.SH "Global Preparation" +The program must initialize some of the libcurl functionality globally. That +means it should be done exactly once, no matter how many times you intend to +use the library. Once for your program's entire life time. This is done using + + curl_global_init() + +and it takes one parameter which is a bit pattern that tells libcurl what to +initialize. Using \fICURL_GLOBAL_ALL\fP will make it initialize all known +internal sub modules, and might be a good default option. The current two bits +that are specified are: +.RS +.IP "CURL_GLOBAL_WIN32" +which only does anything on Windows machines. When used on +a Windows machine, it'll make libcurl initialize the win32 socket +stuff. Without having that initialized properly, your program cannot use +sockets properly. You should only do this once for each application, so if +your program already does this or of another library in use does it, you +should not tell libcurl to do this as well. +.IP CURL_GLOBAL_SSL +which only does anything on libcurls compiled and built SSL-enabled. On these +systems, this will make libcurl initialize the SSL library properly for this +application. This only needs to be done once for each application so if your +program or another library already does this, this bit should not be needed. +.RE + +libcurl has a default protection mechanism that detects if +\fIcurl_global_init(3)\fP hasn't been called by the time +\fIcurl_easy_perform(3)\fP is called and if that is the case, libcurl runs the +function itself with a guessed bit pattern. Please note that depending solely +on this is not considered nice nor very good. + +When the program no longer uses libcurl, it should call +\fIcurl_global_cleanup(3)\fP, which is the opposite of the init call. It will +then do the reversed operations to cleanup the resources the +\fIcurl_global_init(3)\fP call initialized. + +Repeated calls to \fIcurl_global_init(3)\fP and \fIcurl_global_cleanup(3)\fP +should be avoided. They should only be called once each. + +.SH "Features libcurl Provides" +It is considered best-practice to determine libcurl features at run-time +rather than at build-time (if possible of course). By calling +\fIcurl_version_info(3)\fP and checking out the details of the returned +struct, your program can figure out exactly what the currently running libcurl +supports. + +.SH "Handle the Easy libcurl" +libcurl first introduced the so called easy interface. All operations in the +easy interface are prefixed with 'curl_easy'. + +Recent libcurl versions also offer the multi interface. More about that +interface, what it is targeted for and how to use it is detailed in a separate +chapter further down. You still need to understand the easy interface first, +so please continue reading for better understanding. + +To use the easy interface, you must first create yourself an easy handle. You +need one handle for each easy session you want to perform. Basically, you +should use one handle for every thread you plan to use for transferring. You +must never share the same handle in multiple threads. + +Get an easy handle with + + easyhandle = curl_easy_init(); + +It returns an easy handle. Using that you proceed to the next step: setting +up your preferred actions. A handle is just a logic entity for the upcoming +transfer or series of transfers. + +You set properties and options for this handle using +\fIcurl_easy_setopt(3)\fP. They control how the subsequent transfer or +transfers will be made. Options remain set in the handle until set again to +something different. Alas, multiple requests using the same handle will use +the same options. + +Many of the options you set in libcurl are "strings", pointers to data +terminated with a zero byte. When you set strings with +\fIcurl_easy_setopt(3)\fP, libcurl makes its own copy so that they don't +need to be kept around in your application after being set[4]. + +One of the most basic properties to set in the handle is the URL. You set +your preferred URL to transfer with CURLOPT_URL in a manner similar to: + +.nf + curl_easy_setopt(handle, CURLOPT_URL, "http://domain.com/"); +.fi + +Let's assume for a while that you want to receive data as the URL identifies a +remote resource you want to get here. Since you write a sort of application +that needs this transfer, I assume that you would like to get the data passed +to you directly instead of simply getting it passed to stdout. So, you write +your own function that matches this prototype: + + size_t write_data(void *buffer, size_t size, size_t nmemb, void *userp); + +You tell libcurl to pass all data to this function by issuing a function +similar to this: + + curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, write_data); + +You can control what data your callback function gets in the fourth argument +by setting another property: + + curl_easy_setopt(easyhandle, CURLOPT_WRITEDATA, &internal_struct); + +Using that property, you can easily pass local data between your application +and the function that gets invoked by libcurl. libcurl itself won't touch the +data you pass with \fICURLOPT_WRITEDATA\fP. + +libcurl offers its own default internal callback that will take care of the data +if you don't set the callback with \fICURLOPT_WRITEFUNCTION\fP. It will then +simply output the received data to stdout. You can have the default callback +write the data to a different file handle by passing a 'FILE *' to a file +opened for writing with the \fICURLOPT_WRITEDATA\fP option. + +Now, we need to take a step back and have a deep breath. Here's one of those +rare platform-dependent nitpicks. Did you spot it? On some platforms[2], +libcurl won't be able to operate on files opened by the program. Thus, if you +use the default callback and pass in an open file with +\fICURLOPT_WRITEDATA\fP, it will crash. You should therefore avoid this to +make your program run fine virtually everywhere. + +(\fICURLOPT_WRITEDATA\fP was formerly known as \fICURLOPT_FILE\fP. Both names +still work and do the same thing). + +If you're using libcurl as a win32 DLL, you MUST use the +\fICURLOPT_WRITEFUNCTION\fP if you set \fICURLOPT_WRITEDATA\fP - or you will +experience crashes. + +There are of course many more options you can set, and we'll get back to a few +of them later. Let's instead continue to the actual transfer: + + success = curl_easy_perform(easyhandle); + +\fIcurl_easy_perform(3)\fP will connect to the remote site, do the necessary +commands and receive the transfer. Whenever it receives data, it calls the +callback function we previously set. The function may get one byte at a time, +or it may get many kilobytes at once. libcurl delivers as much as possible as +often as possible. Your callback function should return the number of bytes it +\&"took care of". If that is not the exact same amount of bytes that was +passed to it, libcurl will abort the operation and return with an error code. + +When the transfer is complete, the function returns a return code that informs +you if it succeeded in its mission or not. If a return code isn't enough for +you, you can use the CURLOPT_ERRORBUFFER to point libcurl to a buffer of yours +where it'll store a human readable error message as well. + +If you then want to transfer another file, the handle is ready to be used +again. Mind you, it is even preferred that you re-use an existing handle if +you intend to make another transfer. libcurl will then attempt to re-use the +previous connection. + +For some protocols, downloading a file can involve a complicated process of +logging in, setting the transfer mode, changing the current directory and +finally transferring the file data. libcurl takes care of all that +complication for you. Given simply the URL to a file, libcurl will take care +of all the details needed to get the file moved from one machine to another. + +.SH "Multi-threading Issues" +The first basic rule is that you must \fBnever\fP simultaneously share a +libcurl handle (be it easy or multi or whatever) between multiple +threads. Only use one handle in one thread at any time. You can pass the +handles around among threads, but you must never use a single handle from more +than one thread at any given time. + +libcurl is completely thread safe, except for two issues: signals and SSL/TLS +handlers. Signals are used for timing out name resolves (during DNS lookup) - +when built without c-ares support and not on Windows. + +If you are accessing HTTPS or FTPS URLs in a multi-threaded manner, you are +then of course using the underlying SSL library multi-threaded and those libs +might have their own requirements on this issue. Basically, you need to +provide one or two functions to allow it to function properly. For all +details, see this: + +OpenSSL + + http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION + +GnuTLS + + http://www.gnu.org/software/gnutls/manual/html_node/Multi_002dthreaded-applications.html + +NSS + + is claimed to be thread-safe already without anything required. + +PolarSSL + + Required actions unknown. + +yassl + + Required actions unknown. + +axTLS + + Required actions unknown. + +When using multiple threads you should set the CURLOPT_NOSIGNAL option to 1 +for all handles. Everything will or might work fine except that timeouts are +not honored during the DNS lookup - which you can work around by building +libcurl with c-ares support. c-ares is a library that provides asynchronous +name resolves. On some platforms, libcurl simply will not function properly +multi-threaded unless this option is set. + +Also, note that CURLOPT_DNS_USE_GLOBAL_CACHE is not thread-safe. + +.SH "When It Doesn't Work" +There will always be times when the transfer fails for some reason. You might +have set the wrong libcurl option or misunderstood what the libcurl option +actually does, or the remote server might return non-standard replies that +confuse the library which then confuses your program. + +There's one golden rule when these things occur: set the CURLOPT_VERBOSE +option to 1. It'll cause the library to spew out the entire protocol +details it sends, some internal info and some received protocol data as well +(especially when using FTP). If you're using HTTP, adding the headers in the +received output to study is also a clever way to get a better understanding +why the server behaves the way it does. Include headers in the normal body +output with CURLOPT_HEADER set 1. + +Of course, there are bugs left. We need to know about them to be able +to fix them, so we're quite dependent on your bug reports! When you do report +suspected bugs in libcurl, please include as many details as you possibly can: a +protocol dump that CURLOPT_VERBOSE produces, library version, as much as +possible of your code that uses libcurl, operating system name and version, +compiler name and version etc. + +If CURLOPT_VERBOSE is not enough, you increase the level of debug data your +application receive by using the CURLOPT_DEBUGFUNCTION. + +Getting some in-depth knowledge about the protocols involved is never wrong, +and if you're trying to do funny things, you might very well understand +libcurl and how to use it better if you study the appropriate RFC documents +at least briefly. + +.SH "Upload Data to a Remote Site" +libcurl tries to keep a protocol independent approach to most transfers, thus +uploading to a remote FTP site is very similar to uploading data to a HTTP +server with a PUT request. + +Of course, first you either create an easy handle or you re-use one existing +one. Then you set the URL to operate on just like before. This is the remote +URL, that we now will upload. + +Since we write an application, we most likely want libcurl to get the upload +data by asking us for it. To make it do that, we set the read callback and +the custom pointer libcurl will pass to our read callback. The read callback +should have a prototype similar to: + + size_t function(char *bufptr, size_t size, size_t nitems, void *userp); + +Where bufptr is the pointer to a buffer we fill in with data to upload and +size*nitems is the size of the buffer and therefore also the maximum amount +of data we can return to libcurl in this call. The 'userp' pointer is the +custom pointer we set to point to a struct of ours to pass private data +between the application and the callback. + + curl_easy_setopt(easyhandle, CURLOPT_READFUNCTION, read_function); + + curl_easy_setopt(easyhandle, CURLOPT_READDATA, &filedata); + +Tell libcurl that we want to upload: + + curl_easy_setopt(easyhandle, CURLOPT_UPLOAD, 1L); + +A few protocols won't behave properly when uploads are done without any prior +knowledge of the expected file size. So, set the upload file size using the +CURLOPT_INFILESIZE_LARGE for all known file sizes like this[1]: + +.nf + /* in this example, file_size must be an curl_off_t variable */ + curl_easy_setopt(easyhandle, CURLOPT_INFILESIZE_LARGE, file_size); +.fi + +When you call \fIcurl_easy_perform(3)\fP this time, it'll perform all the +necessary operations and when it has invoked the upload it'll call your +supplied callback to get the data to upload. The program should return as much +data as possible in every invoke, as that is likely to make the upload perform +as fast as possible. The callback should return the number of bytes it wrote +in the buffer. Returning 0 will signal the end of the upload. + +.SH "Passwords" +Many protocols use or even require that user name and password are provided +to be able to download or upload the data of your choice. libcurl offers +several ways to specify them. + +Most protocols support that you specify the name and password in the URL +itself. libcurl will detect this and use them accordingly. This is written +like this: + + protocol://user:password@example.com/path/ + +If you need any odd letters in your user name or password, you should enter +them URL encoded, as %XX where XX is a two-digit hexadecimal number. + +libcurl also provides options to set various passwords. The user name and +password as shown embedded in the URL can instead get set with the +CURLOPT_USERPWD option. The argument passed to libcurl should be a char * to +a string in the format "user:password". In a manner like this: + + curl_easy_setopt(easyhandle, CURLOPT_USERPWD, "myname:thesecret"); + +Another case where name and password might be needed at times, is for those +users who need to authenticate themselves to a proxy they use. libcurl offers +another option for this, the CURLOPT_PROXYUSERPWD. It is used quite similar +to the CURLOPT_USERPWD option like this: + + curl_easy_setopt(easyhandle, CURLOPT_PROXYUSERPWD, "myname:thesecret"); + +There's a long time UNIX "standard" way of storing ftp user names and +passwords, namely in the $HOME/.netrc file. The file should be made private +so that only the user may read it (see also the "Security Considerations" +chapter), as it might contain the password in plain text. libcurl has the +ability to use this file to figure out what set of user name and password to +use for a particular host. As an extension to the normal functionality, +libcurl also supports this file for non-FTP protocols such as HTTP. To make +curl use this file, use the CURLOPT_NETRC option: + + curl_easy_setopt(easyhandle, CURLOPT_NETRC, 1L); + +And a very basic example of how such a .netrc file may look like: + +.nf + machine myhost.mydomain.com + login userlogin + password secretword +.fi + +All these examples have been cases where the password has been optional, or +at least you could leave it out and have libcurl attempt to do its job +without it. There are times when the password isn't optional, like when +you're using an SSL private key for secure transfers. + +To pass the known private key password to libcurl: + + curl_easy_setopt(easyhandle, CURLOPT_KEYPASSWD, "keypassword"); + +.SH "HTTP Authentication" +The previous chapter showed how to set user name and password for getting +URLs that require authentication. When using the HTTP protocol, there are +many different ways a client can provide those credentials to the server and +you can control which way libcurl will (attempt to) use them. The default HTTP +authentication method is called 'Basic', which is sending the name and +password in clear-text in the HTTP request, base64-encoded. This is insecure. + +At the time of this writing, libcurl can be built to use: Basic, Digest, NTLM, +Negotiate, GSS-Negotiate and SPNEGO. You can tell libcurl which one to use +with CURLOPT_HTTPAUTH as in: + + curl_easy_setopt(easyhandle, CURLOPT_HTTPAUTH, CURLAUTH_DIGEST); + +And when you send authentication to a proxy, you can also set authentication +type the same way but instead with CURLOPT_PROXYAUTH: + + curl_easy_setopt(easyhandle, CURLOPT_PROXYAUTH, CURLAUTH_NTLM); + +Both these options allow you to set multiple types (by ORing them together), +to make libcurl pick the most secure one out of the types the server/proxy +claims to support. This method does however add a round-trip since libcurl +must first ask the server what it supports: + + curl_easy_setopt(easyhandle, CURLOPT_HTTPAUTH, + CURLAUTH_DIGEST|CURLAUTH_BASIC); + +For convenience, you can use the 'CURLAUTH_ANY' define (instead of a list +with specific types) which allows libcurl to use whatever method it wants. + +When asking for multiple types, libcurl will pick the available one it +considers "best" in its own internal order of preference. + +.SH "HTTP POSTing" +We get many questions regarding how to issue HTTP POSTs with libcurl the +proper way. This chapter will thus include examples using both different +versions of HTTP POST that libcurl supports. + +The first version is the simple POST, the most common version, that most HTML +pages using the <form> tag uses. We provide a pointer to the data and tell +libcurl to post it all to the remote site: + +.nf + char *data="name=daniel&project=curl"; + curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDS, data); + curl_easy_setopt(easyhandle, CURLOPT_URL, "http://posthere.com/"); + + curl_easy_perform(easyhandle); /* post away! */ +.fi + +Simple enough, huh? Since you set the POST options with the +CURLOPT_POSTFIELDS, this automatically switches the handle to use POST in the +upcoming request. + +Ok, so what if you want to post binary data that also requires you to set the +Content-Type: header of the post? Well, binary posts prevent libcurl from +being able to do strlen() on the data to figure out the size, so therefore we +must tell libcurl the size of the post data. Setting headers in libcurl +requests are done in a generic way, by building a list of our own headers and +then passing that list to libcurl. + +.nf + struct curl_slist *headers=NULL; + headers = curl_slist_append(headers, "Content-Type: text/xml"); + + /* post binary data */ + curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDS, binaryptr); + + /* set the size of the postfields data */ + curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDSIZE, 23L); + + /* pass our list of custom made headers */ + curl_easy_setopt(easyhandle, CURLOPT_HTTPHEADER, headers); + + curl_easy_perform(easyhandle); /* post away! */ + + curl_slist_free_all(headers); /* free the header list */ +.fi + +While the simple examples above cover the majority of all cases where HTTP +POST operations are required, they don't do multi-part formposts. Multi-part +formposts were introduced as a better way to post (possibly large) binary data +and were first documented in the RFC1867 (updated in RFC2388). They're called +multi-part because they're built by a chain of parts, each part being a single +unit of data. Each part has its own name and contents. You can in fact create +and post a multi-part formpost with the regular libcurl POST support described +above, but that would require that you build a formpost yourself and provide +to libcurl. To make that easier, libcurl provides \fIcurl_formadd(3)\fP. Using +this function, you add parts to the form. When you're done adding parts, you +post the whole form. + +The following example sets two simple text parts with plain textual contents, +and then a file with binary contents and uploads the whole thing. + +.nf + struct curl_httppost *post=NULL; + struct curl_httppost *last=NULL; + curl_formadd(&post, &last, + CURLFORM_COPYNAME, "name", + CURLFORM_COPYCONTENTS, "daniel", CURLFORM_END); + curl_formadd(&post, &last, + CURLFORM_COPYNAME, "project", + CURLFORM_COPYCONTENTS, "curl", CURLFORM_END); + curl_formadd(&post, &last, + CURLFORM_COPYNAME, "logotype-image", + CURLFORM_FILECONTENT, "curl.png", CURLFORM_END); + + /* Set the form info */ + curl_easy_setopt(easyhandle, CURLOPT_HTTPPOST, post); + + curl_easy_perform(easyhandle); /* post away! */ + + /* free the post data again */ + curl_formfree(post); +.fi + +Multipart formposts are chains of parts using MIME-style separators and +headers. It means that each one of these separate parts get a few headers set +that describe the individual content-type, size etc. To enable your +application to handicraft this formpost even more, libcurl allows you to +supply your own set of custom headers to such an individual form part. You can +of course supply headers to as many parts as you like, but this little example +will show how you set headers to one specific part when you add that to the +post handle: + +.nf + struct curl_slist *headers=NULL; + headers = curl_slist_append(headers, "Content-Type: text/xml"); + + curl_formadd(&post, &last, + CURLFORM_COPYNAME, "logotype-image", + CURLFORM_FILECONTENT, "curl.xml", + CURLFORM_CONTENTHEADER, headers, + CURLFORM_END); + + curl_easy_perform(easyhandle); /* post away! */ + + curl_formfree(post); /* free post */ + curl_slist_free_all(headers); /* free custom header list */ +.fi + +Since all options on an easyhandle are "sticky", they remain the same until +changed even if you do call \fIcurl_easy_perform(3)\fP, you may need to tell +curl to go back to a plain GET request if you intend to do one as your +next request. You force an easyhandle to go back to GET by using the +CURLOPT_HTTPGET option: + + curl_easy_setopt(easyhandle, CURLOPT_HTTPGET, 1L); + +Just setting CURLOPT_POSTFIELDS to "" or NULL will *not* stop libcurl from +doing a POST. It will just make it POST without any data to send! + +.SH "Showing Progress" + +For historical and traditional reasons, libcurl has a built-in progress meter +that can be switched on and then makes it present a progress meter in your +terminal. + +Switch on the progress meter by, oddly enough, setting CURLOPT_NOPROGRESS to +zero. This option is set to 1 by default. + +For most applications however, the built-in progress meter is useless and +what instead is interesting is the ability to specify a progress +callback. The function pointer you pass to libcurl will then be called on +irregular intervals with information about the current transfer. + +Set the progress callback by using CURLOPT_PROGRESSFUNCTION. And pass a +pointer to a function that matches this prototype: + +.nf + int progress_callback(void *clientp, + double dltotal, + double dlnow, + double ultotal, + double ulnow); +.fi + +If any of the input arguments is unknown, a 0 will be passed. The first +argument, the 'clientp' is the pointer you pass to libcurl with +CURLOPT_PROGRESSDATA. libcurl won't touch it. + +.SH "libcurl with C++" + +There's basically only one thing to keep in mind when using C++ instead of C +when interfacing libcurl: + +The callbacks CANNOT be non-static class member functions + +Example C++ code: + +.nf +class AClass { + static size_t write_data(void *ptr, size_t size, size_t nmemb, + void *ourpointer) + { + /* do what you want with the data */ + } + } +.fi + +.SH "Proxies" + +What "proxy" means according to Merriam-Webster: "a person authorized to act +for another" but also "the agency, function, or office of a deputy who acts as +a substitute for another". + +Proxies are exceedingly common these days. Companies often only offer Internet +access to employees through their proxies. Network clients or user-agents ask +the proxy for documents, the proxy does the actual request and then it returns +them. + +libcurl supports SOCKS and HTTP proxies. When a given URL is wanted, libcurl +will ask the proxy for it instead of trying to connect to the actual host +identified in the URL. + +If you're using a SOCKS proxy, you may find that libcurl doesn't quite support +all operations through it. + +For HTTP proxies: the fact that the proxy is a HTTP proxy puts certain +restrictions on what can actually happen. A requested URL that might not be a +HTTP URL will be still be passed to the HTTP proxy to deliver back to +libcurl. This happens transparently, and an application may not need to +know. I say "may", because at times it is very important to understand that +all operations over a HTTP proxy use the HTTP protocol. For example, you +can't invoke your own custom FTP commands or even proper FTP directory +listings. + +.IP "Proxy Options" + +To tell libcurl to use a proxy at a given port number: + + curl_easy_setopt(easyhandle, CURLOPT_PROXY, "proxy-host.com:8080"); + +Some proxies require user authentication before allowing a request, and you +pass that information similar to this: + + curl_easy_setopt(easyhandle, CURLOPT_PROXYUSERPWD, "user:password"); + +If you want to, you can specify the host name only in the CURLOPT_PROXY +option, and set the port number separately with CURLOPT_PROXYPORT. + +Tell libcurl what kind of proxy it is with CURLOPT_PROXYTYPE (if not, it will +default to assume a HTTP proxy): + + curl_easy_setopt(easyhandle, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS4); + +.IP "Environment Variables" + +libcurl automatically checks and uses a set of environment variables to know +what proxies to use for certain protocols. The names of the variables are +following an ancient de facto standard and are built up as "[protocol]_proxy" +(note the lower casing). Which makes the variable \&'http_proxy' checked for a +name of a proxy to use when the input URL is HTTP. Following the same rule, +the variable named 'ftp_proxy' is checked for FTP URLs. Again, the proxies are +always HTTP proxies, the different names of the variables simply allows +different HTTP proxies to be used. + +The proxy environment variable contents should be in the format +\&"[protocol://][user:password@]machine[:port]". Where the protocol:// part is +simply ignored if present (so http://proxy and bluerk://proxy will do the +same) and the optional port number specifies on which port the proxy operates +on the host. If not specified, the internal default port number will be used +and that is most likely *not* the one you would like it to be. + +There are two special environment variables. 'all_proxy' is what sets proxy +for any URL in case the protocol specific variable wasn't set, and +\&'no_proxy' defines a list of hosts that should not use a proxy even though a +variable may say so. If 'no_proxy' is a plain asterisk ("*") it matches all +hosts. + +To explicitly disable libcurl's checking for and using the proxy environment +variables, set the proxy name to "" - an empty string - with CURLOPT_PROXY. +.IP "SSL and Proxies" + +SSL is for secure point-to-point connections. This involves strong encryption +and similar things, which effectively makes it impossible for a proxy to +operate as a "man in between" which the proxy's task is, as previously +discussed. Instead, the only way to have SSL work over a HTTP proxy is to ask +the proxy to tunnel trough everything without being able to check or fiddle +with the traffic. + +Opening an SSL connection over a HTTP proxy is therefor a matter of asking the +proxy for a straight connection to the target host on a specified port. This +is made with the HTTP request CONNECT. ("please mr proxy, connect me to that +remote host"). + +Because of the nature of this operation, where the proxy has no idea what kind +of data that is passed in and out through this tunnel, this breaks some of the +very few advantages that come from using a proxy, such as caching. Many +organizations prevent this kind of tunneling to other destination port numbers +than 443 (which is the default HTTPS port number). + +.IP "Tunneling Through Proxy" +As explained above, tunneling is required for SSL to work and often even +restricted to the operation intended for SSL; HTTPS. + +This is however not the only time proxy-tunneling might offer benefits to +you or your application. + +As tunneling opens a direct connection from your application to the remote +machine, it suddenly also re-introduces the ability to do non-HTTP +operations over a HTTP proxy. You can in fact use things such as FTP +upload or FTP custom commands this way. + +Again, this is often prevented by the administrators of proxies and is +rarely allowed. + +Tell libcurl to use proxy tunneling like this: + + curl_easy_setopt(easyhandle, CURLOPT_HTTPPROXYTUNNEL, 1L); + +In fact, there might even be times when you want to do plain HTTP +operations using a tunnel like this, as it then enables you to operate on +the remote server instead of asking the proxy to do so. libcurl will not +stand in the way for such innovative actions either! + +.IP "Proxy Auto-Config" + +Netscape first came up with this. It is basically a web page (usually using a +\&.pac extension) with a Javascript that when executed by the browser with the +requested URL as input, returns information to the browser on how to connect +to the URL. The returned information might be "DIRECT" (which means no proxy +should be used), "PROXY host:port" (to tell the browser where the proxy for +this particular URL is) or "SOCKS host:port" (to direct the browser to a SOCKS +proxy). + +libcurl has no means to interpret or evaluate Javascript and thus it doesn't +support this. If you get yourself in a position where you face this nasty +invention, the following advice have been mentioned and used in the past: + +- Depending on the Javascript complexity, write up a script that translates it +to another language and execute that. + +- Read the Javascript code and rewrite the same logic in another language. + +- Implement a Javascript interpreter; people have successfully used the +Mozilla Javascript engine in the past. + +- Ask your admins to stop this, for a static proxy setup or similar. + +.SH "Persistence Is The Way to Happiness" + +Re-cycling the same easy handle several times when doing multiple requests is +the way to go. + +After each single \fIcurl_easy_perform(3)\fP operation, libcurl will keep the +connection alive and open. A subsequent request using the same easy handle to +the same host might just be able to use the already open connection! This +reduces network impact a lot. + +Even if the connection is dropped, all connections involving SSL to the same +host again, will benefit from libcurl's session ID cache that drastically +reduces re-connection time. + +FTP connections that are kept alive save a lot of time, as the command- +response round-trips are skipped, and also you don't risk getting blocked +without permission to login again like on many FTP servers only allowing N +persons to be logged in at the same time. + +libcurl caches DNS name resolving results, to make lookups of a previously +looked up name a lot faster. + +Other interesting details that improve performance for subsequent requests +may also be added in the future. + +Each easy handle will attempt to keep the last few connections alive for a +while in case they are to be used again. You can set the size of this "cache" +with the CURLOPT_MAXCONNECTS option. Default is 5. There is very seldom any +point in changing this value, and if you think of changing this it is often +just a matter of thinking again. + +To force your upcoming request to not use an already existing connection (it +will even close one first if there happens to be one alive to the same host +you're about to operate on), you can do that by setting CURLOPT_FRESH_CONNECT +to 1. In a similar spirit, you can also forbid the upcoming request to be +"lying" around and possibly get re-used after the request by setting +CURLOPT_FORBID_REUSE to 1. + +.SH "HTTP Headers Used by libcurl" +When you use libcurl to do HTTP requests, it'll pass along a series of headers +automatically. It might be good for you to know and understand these. You +can replace or remove them by using the CURLOPT_HTTPHEADER option. + +.IP "Host" +This header is required by HTTP 1.1 and even many 1.0 servers and should be +the name of the server we want to talk to. This includes the port number if +anything but default. + +.IP "Accept" +\&"*/*". + +.IP "Expect" +When doing POST requests, libcurl sets this header to \&"100-continue" to ask +the server for an "OK" message before it proceeds with sending the data part +of the post. If the POSTed data amount is deemed "small", libcurl will not use +this header. + +.SH "Customizing Operations" +There is an ongoing development today where more and more protocols are built +upon HTTP for transport. This has obvious benefits as HTTP is a tested and +reliable protocol that is widely deployed and has excellent proxy-support. + +When you use one of these protocols, and even when doing other kinds of +programming you may need to change the traditional HTTP (or FTP or...) +manners. You may need to change words, headers or various data. + +libcurl is your friend here too. + +.IP CUSTOMREQUEST +If just changing the actual HTTP request keyword is what you want, like when +GET, HEAD or POST is not good enough for you, CURLOPT_CUSTOMREQUEST is there +for you. It is very simple to use: + + curl_easy_setopt(easyhandle, CURLOPT_CUSTOMREQUEST, "MYOWNREQUEST"); + +When using the custom request, you change the request keyword of the actual +request you are performing. Thus, by default you make a GET request but you can +also make a POST operation (as described before) and then replace the POST +keyword if you want to. You're the boss. + +.IP "Modify Headers" +HTTP-like protocols pass a series of headers to the server when doing the +request, and you're free to pass any amount of extra headers that you +think fit. Adding headers is this easy: + +.nf + struct curl_slist *headers=NULL; /* init to NULL is important */ + + headers = curl_slist_append(headers, "Hey-server-hey: how are you?"); + headers = curl_slist_append(headers, "X-silly-content: yes"); + + /* pass our list of custom made headers */ + curl_easy_setopt(easyhandle, CURLOPT_HTTPHEADER, headers); + + curl_easy_perform(easyhandle); /* transfer http */ + + curl_slist_free_all(headers); /* free the header list */ +.fi + +\&... and if you think some of the internally generated headers, such as +Accept: or Host: don't contain the data you want them to contain, you can +replace them by simply setting them too: + +.nf + headers = curl_slist_append(headers, "Accept: Agent-007"); + headers = curl_slist_append(headers, "Host: munged.host.line"); +.fi + +.IP "Delete Headers" +If you replace an existing header with one with no contents, you will prevent +the header from being sent. For instance, if you want to completely prevent the +\&"Accept:" header from being sent, you can disable it with code similar to this: + + headers = curl_slist_append(headers, "Accept:"); + +Both replacing and canceling internal headers should be done with careful +consideration and you should be aware that you may violate the HTTP protocol +when doing so. + +.IP "Enforcing chunked transfer-encoding" + +By making sure a request uses the custom header "Transfer-Encoding: chunked" +when doing a non-GET HTTP operation, libcurl will switch over to "chunked" +upload, even though the size of the data to upload might be known. By default, +libcurl usually switches over to chunked upload automatically if the upload +data size is unknown. + +.IP "HTTP Version" + +All HTTP requests includes the version number to tell the server which version +we support. libcurl speaks HTTP 1.1 by default. Some very old servers don't +like getting 1.1-requests and when dealing with stubborn old things like that, +you can tell libcurl to use 1.0 instead by doing something like this: + + curl_easy_setopt(easyhandle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0); + +.IP "FTP Custom Commands" + +Not all protocols are HTTP-like, and thus the above may not help you when +you want to make, for example, your FTP transfers to behave differently. + +Sending custom commands to a FTP server means that you need to send the +commands exactly as the FTP server expects them (RFC959 is a good guide +here), and you can only use commands that work on the control-connection +alone. All kinds of commands that require data interchange and thus need +a data-connection must be left to libcurl's own judgement. Also be aware +that libcurl will do its very best to change directory to the target +directory before doing any transfer, so if you change directory (with CWD +or similar) you might confuse libcurl and then it might not attempt to +transfer the file in the correct remote directory. + +A little example that deletes a given file before an operation: + +.nf + headers = curl_slist_append(headers, "DELE file-to-remove"); + + /* pass the list of custom commands to the handle */ + curl_easy_setopt(easyhandle, CURLOPT_QUOTE, headers); + + curl_easy_perform(easyhandle); /* transfer ftp data! */ + + curl_slist_free_all(headers); /* free the header list */ +.fi + +If you would instead want this operation (or chain of operations) to happen +_after_ the data transfer took place the option to \fIcurl_easy_setopt(3)\fP +would instead be called CURLOPT_POSTQUOTE and used the exact same way. + +The custom FTP command will be issued to the server in the same order they are +added to the list, and if a command gets an error code returned back from the +server, no more commands will be issued and libcurl will bail out with an +error code (CURLE_QUOTE_ERROR). Note that if you use CURLOPT_QUOTE to send +commands before a transfer, no transfer will actually take place when a quote +command has failed. + +If you set the CURLOPT_HEADER to 1, you will tell libcurl to get +information about the target file and output "headers" about it. The headers +will be in "HTTP-style", looking like they do in HTTP. + +The option to enable headers or to run custom FTP commands may be useful to +combine with CURLOPT_NOBODY. If this option is set, no actual file content +transfer will be performed. + +.IP "FTP Custom CUSTOMREQUEST" +If you do want to list the contents of a FTP directory using your own defined FTP +command, CURLOPT_CUSTOMREQUEST will do just that. "NLST" is the default one +for listing directories but you're free to pass in your idea of a good +alternative. + +.SH "Cookies Without Chocolate Chips" +In the HTTP sense, a cookie is a name with an associated value. A server sends +the name and value to the client, and expects it to get sent back on every +subsequent request to the server that matches the particular conditions +set. The conditions include that the domain name and path match and that the +cookie hasn't become too old. + +In real-world cases, servers send new cookies to replace existing ones to +update them. Server use cookies to "track" users and to keep "sessions". + +Cookies are sent from server to clients with the header Set-Cookie: and +they're sent from clients to servers with the Cookie: header. + +To just send whatever cookie you want to a server, you can use CURLOPT_COOKIE +to set a cookie string like this: + + curl_easy_setopt(easyhandle, CURLOPT_COOKIE, "name1=var1; name2=var2;"); + +In many cases, that is not enough. You might want to dynamically save +whatever cookies the remote server passes to you, and make sure those cookies +are then used accordingly on later requests. + +One way to do this, is to save all headers you receive in a plain file and +when you make a request, you tell libcurl to read the previous headers to +figure out which cookies to use. Set the header file to read cookies from with +CURLOPT_COOKIEFILE. + +The CURLOPT_COOKIEFILE option also automatically enables the cookie parser in +libcurl. Until the cookie parser is enabled, libcurl will not parse or +understand incoming cookies and they will just be ignored. However, when the +parser is enabled the cookies will be understood and the cookies will be kept +in memory and used properly in subsequent requests when the same handle is +used. Many times this is enough, and you may not have to save the cookies to +disk at all. Note that the file you specify to CURLOPT_COOKIEFILE doesn't have +to exist to enable the parser, so a common way to just enable the parser and +not read any cookies is to use the name of a file you know doesn't exist. + +If you would rather use existing cookies that you've previously received with +your Netscape or Mozilla browsers, you can make libcurl use that cookie file +as input. The CURLOPT_COOKIEFILE is used for that too, as libcurl will +automatically find out what kind of file it is and act accordingly. + +Perhaps the most advanced cookie operation libcurl offers, is saving the +entire internal cookie state back into a Netscape/Mozilla formatted cookie +file. We call that the cookie-jar. When you set a file name with +CURLOPT_COOKIEJAR, that file name will be created and all received cookies +will be stored in it when \fIcurl_easy_cleanup(3)\fP is called. This enables +cookies to get passed on properly between multiple handles without any +information getting lost. + +.SH "FTP Peculiarities We Need" + +FTP transfers use a second TCP/IP connection for the data transfer. This is +usually a fact you can forget and ignore but at times this fact will come +back to haunt you. libcurl offers several different ways to customize how the +second connection is being made. + +libcurl can either connect to the server a second time or tell the server to +connect back to it. The first option is the default and it is also what works +best for all the people behind firewalls, NATs or IP-masquerading setups. +libcurl then tells the server to open up a new port and wait for a second +connection. This is by default attempted with EPSV first, and if that doesn't +work it tries PASV instead. (EPSV is an extension to the original FTP spec +and does not exist nor work on all FTP servers.) + +You can prevent libcurl from first trying the EPSV command by setting +CURLOPT_FTP_USE_EPSV to zero. + +In some cases, you will prefer to have the server connect back to you for the +second connection. This might be when the server is perhaps behind a firewall +or something and only allows connections on a single port. libcurl then +informs the remote server which IP address and port number to connect to. +This is made with the CURLOPT_FTPPORT option. If you set it to "-", libcurl +will use your system's "default IP address". If you want to use a particular +IP, you can set the full IP address, a host name to resolve to an IP address +or even a local network interface name that libcurl will get the IP address +from. + +When doing the "PORT" approach, libcurl will attempt to use the EPRT and the +LPRT before trying PORT, as they work with more protocols. You can disable +this behavior by setting CURLOPT_FTP_USE_EPRT to zero. + +.SH "Headers Equal Fun" + +Some protocols provide "headers", meta-data separated from the normal +data. These headers are by default not included in the normal data stream, +but you can make them appear in the data stream by setting CURLOPT_HEADER to +1. + +What might be even more useful, is libcurl's ability to separate the headers +from the data and thus make the callbacks differ. You can for example set a +different pointer to pass to the ordinary write callback by setting +CURLOPT_WRITEHEADER. + +Or, you can set an entirely separate function to receive the headers, by +using CURLOPT_HEADERFUNCTION. + +The headers are passed to the callback function one by one, and you can +depend on that fact. It makes it easier for you to add custom header parsers +etc. + +\&"Headers" for FTP transfers equal all the FTP server responses. They aren't +actually true headers, but in this case we pretend they are! ;-) + +.SH "Post Transfer Information" + + [ curl_easy_getinfo ] + +.SH "Security Considerations" + +The libcurl project takes security seriously. The library is written with +caution and precautions are taken to mitigate many kinds of risks encountered +while operating with potentially malicious servers on the Internet. It is a +powerful library, however, which allows application writers to make trade offs +between ease of writing and exposure to potential risky operations. If +used the right way, you can use libcurl to transfer data pretty safely. + +Many applications are used in closed networks where users and servers +can be trusted, but many others are used on arbitrary servers and are fed +input from potentially untrusted users. Following is a discussion about +some risks in the ways in which applications commonly use libcurl and +potential mitigations of those risks. It is by no means comprehensive, but +shows classes of attacks that robust applications should consider. The +Common Weakness Enumeration project at http://cwe.mitre.org/ is a good +reference for many of these and similar types of weaknesses of which +application writers should be aware. + +.IP "Command Lines" +If you use a command line tool (such as curl) that uses libcurl, and you give +options to the tool on the command line those options can very likely get read +by other users of your system when they use 'ps' or other tools to list +currently running processes. + +To avoid this problem, never feed sensitive things to programs using command +line options. Write them to a protected file and use the \-K option to +avoid this. + +.IP ".netrc" +\&.netrc is a pretty handy file/feature that allows you to login quickly and +automatically to frequently visited sites. The file contains passwords in +clear text and is a real security risk. In some cases, your .netrc is also +stored in a home directory that is NFS mounted or used on another network +based file system, so the clear text password will fly through your network +every time anyone reads that file! + +To avoid this problem, don't use .netrc files and never store passwords in +plain text anywhere. + +.IP "Clear Text Passwords" +Many of the protocols libcurl supports send name and password unencrypted as +clear text (HTTP Basic authentication, FTP, TELNET etc). It is very easy for +anyone on your network or a network nearby yours to just fire up a network +analyzer tool and eavesdrop on your passwords. Don't let the fact that HTTP +Basic uses base64 encoded passwords fool you. They may not look readable at a +first glance, but they very easily "deciphered" by anyone within seconds. + +To avoid this problem, use HTTP authentication methods or other protocols that +don't let snoopers see your password: HTTP with Digest, NTLM or GSS +authentication, HTTPS, FTPS, SCP, SFTP and FTP-Kerberos are a few examples. + +.IP "Redirects" +The CURLOPT_FOLLOWLOCATION option automatically follows HTTP redirects sent +by a remote server. These redirects can refer to any kind of URL, not just +HTTP. A redirect to a file: URL would cause the libcurl to read (or write) +arbitrary files from the local filesystem. If the application returns +the data back to the user (as would happen in some kinds of CGI scripts), +an attacker could leverage this to read otherwise forbidden data (e.g. +file://localhost/etc/passwd). + +If authentication credentials are stored in the ~/.netrc file, or Kerberos +is in use, any other URL type (not just file:) that requires +authentication is also at risk. A redirect such as +ftp://some-internal-server/private-file would then return data even when +the server is password protected. + +In the same way, if an unencrypted SSH private key has been configured for +the user running the libcurl application, SCP: or SFTP: URLs could access +password or private-key protected resources, +e.g. sftp://user@some-internal-server/etc/passwd + +The CURLOPT_REDIR_PROTOCOLS and CURLOPT_NETRC options can be used to +mitigate against this kind of attack. + +A redirect can also specify a location available only on the machine running +libcurl, including servers hidden behind a firewall from the attacker. +e.g. http://127.0.0.1/ or http://intranet/delete-stuff.cgi?delete=all or +tftp://bootp-server/pc-config-data + +Apps can mitigate against this by disabling CURLOPT_FOLLOWLOCATION and +handling redirects itself, sanitizing URLs as necessary. Alternately, an +app could leave CURLOPT_FOLLOWLOCATION enabled but set CURLOPT_REDIR_PROTOCOLS +and install a CURLOPT_OPENSOCKETFUNCTION callback function in which addresses +are sanitized before use. + +.IP "Private Resources" +A user who can control the DNS server of a domain being passed in within +a URL can change the address of the host to a local, private address +which the libcurl application will then use. e.g. The innocuous URL +http://fuzzybunnies.example.com/ could actually resolve to the IP address +of a server behind a firewall, such as 127.0.0.1 or 10.1.2.3 +Apps can mitigate against this by setting a CURLOPT_OPENSOCKETFUNCTION +and checking the address before a connection. + +All the malicious scenarios regarding redirected URLs apply just as well +to non-redirected URLs, if the user is allowed to specify an arbitrary URL +that could point to a private resource. For example, a web app providing +a translation service might happily translate file://localhost/etc/passwd +and display the result. Apps can mitigate against this with the +CURLOPT_PROTOCOLS option as well as by similar mitigation techniques for +redirections. + +A malicious FTP server could in response to the PASV command return an +IP address and port number for a server local to the app running libcurl +but behind a firewall. Apps can mitigate against this by using the +CURLOPT_FTP_SKIP_PASV_IP option or CURLOPT_FTPPORT. + +.IP Uploads +When uploading, a redirect can cause a local (or remote) file to be +overwritten. Apps must not allow any unsanitized URL to be passed in +for uploads. Also, CURLOPT_FOLLOWLOCATION should not be used on uploads. +Instead, the app should handle redirects itself, sanitizing each URL first. + +.IP Authentication +Use of CURLOPT_UNRESTRICTED_AUTH could cause authentication information to +be sent to an unknown second server. Apps can mitigate against this +by disabling CURLOPT_FOLLOWLOCATION and handling redirects itself, +sanitizing where necessary. + +Use of the CURLAUTH_ANY option to CURLOPT_HTTPAUTH could result in user +name and password being sent in clear text to an HTTP server. Instead, +use CURLAUTH_ANYSAFE which ensures that the password is encrypted over +the network, or else fail the request. + +Use of the CURLUSESSL_TRY option to CURLOPT_USE_SSL could result in user +name and password being sent in clear text to an FTP server. Instead, +use CURLUSESSL_CONTROL to ensure that an encrypted connection is used or +else fail the request. + +.IP Cookies +If cookies are enabled and cached, then a user could craft a URL which +performs some malicious action to a site whose authentication is already +stored in a cookie. e.g. http://mail.example.com/delete-stuff.cgi?delete=all +Apps can mitigate against this by disabling cookies or clearing them +between requests. + +.IP "Dangerous URLs" +SCP URLs can contain raw commands within the scp: URL, which is a side effect +of how the SCP protocol is designed. e.g. +scp://user:pass@host/a;date >/tmp/test; +Apps must not allow unsanitized SCP: URLs to be passed in for downloads. + +.IP "Denial of Service" +A malicious server could cause libcurl to effectively hang by sending +a trickle of data through, or even no data at all but just keeping the TCP +connection open. This could result in a denial-of-service attack. The +CURLOPT_TIMEOUT and/or CURLOPT_LOW_SPEED_LIMIT options can be used to +mitigate against this. + +A malicious server could cause libcurl to effectively hang by starting to +send data, then severing the connection without cleanly closing the +TCP connection. The app could install a CURLOPT_SOCKOPTFUNCTION callback +function and set the TCP SO_KEEPALIVE option to mitigate against this. +Setting one of the timeout options would also work against this attack. + +A malicious server could cause libcurl to download an infinite amount of +data, potentially causing all of memory or disk to be filled. Setting +the CURLOPT_MAXFILESIZE_LARGE option is not sufficient to guard against this. +Instead, the app should monitor the amount of data received within the +write or progress callback and abort once the limit is reached. + +A malicious HTTP server could cause an infinite redirection loop, causing a +denial-of-service. This can be mitigated by using the CURLOPT_MAXREDIRS +option. + +.IP "Arbitrary Headers" +User-supplied data must be sanitized when used in options like +CURLOPT_USERAGENT, CURLOPT_HTTPHEADER, CURLOPT_POSTFIELDS and others that +are used to generate structured data. Characters like embedded carriage +returns or ampersands could allow the user to create additional headers or +fields that could cause malicious transactions. + +.IP "Server-supplied Names" +A server can supply data which the application may, in some cases, use as +a file name. The curl command-line tool does this with --remote-header-name, +using the Content-disposition: header to generate a file name. An application +could also use CURLINFO_EFFECTIVE_URL to generate a file name from a +server-supplied redirect URL. Special care must be taken to sanitize such +names to avoid the possibility of a malicious server supplying one like +"/etc/passwd", "\autoexec.bat" or even ".bashrc". + +.IP "Server Certificates" +A secure application should never use the CURLOPT_SSL_VERIFYPEER option to +disable certificate validation. There are numerous attacks that are enabled +by apps that fail to properly validate server TLS/SSL certificates, +thus enabling a malicious server to spoof a legitimate one. HTTPS without +validated certificates is potentially as insecure as a plain HTTP connection. + +.IP "Showing What You Do" +On a related issue, be aware that even in situations like when you have +problems with libcurl and ask someone for help, everything you reveal in order +to get best possible help might also impose certain security related +risks. Host names, user names, paths, operating system specifics, etc (not to +mention passwords of course) may in fact be used by intruders to gain +additional information of a potential target. + +To avoid this problem, you must of course use your common sense. Often, you +can just edit out the sensitive data or just search/replace your true +information with faked data. + +.SH "Multiple Transfers Using the multi Interface" + +The easy interface as described in detail in this document is a synchronous +interface that transfers one file at a time and doesn't return until it is +done. + +The multi interface, on the other hand, allows your program to transfer +multiple files in both directions at the same time, without forcing you +to use multiple threads. The name might make it seem that the multi +interface is for multi-threaded programs, but the truth is almost the +reverse. The multi interface can allow a single-threaded application +to perform the same kinds of multiple, simultaneous transfers that +multi-threaded programs can perform. It allows many of the benefits +of multi-threaded transfers without the complexity of managing and +synchronizing many threads. + +To use this interface, you are better off if you first understand the basics +of how to use the easy interface. The multi interface is simply a way to make +multiple transfers at the same time by adding up multiple easy handles into +a "multi stack". + +You create the easy handles you want and you set all the options just like you +have been told above, and then you create a multi handle with +\fIcurl_multi_init(3)\fP and add all those easy handles to that multi handle +with \fIcurl_multi_add_handle(3)\fP. + +When you've added the handles you have for the moment (you can still add new +ones at any time), you start the transfers by calling +\fIcurl_multi_perform(3)\fP. + +\fIcurl_multi_perform(3)\fP is asynchronous. It will only execute as little as +possible and then return back control to your program. It is designed to never +block. + +The best usage of this interface is when you do a select() on all possible +file descriptors or sockets to know when to call libcurl again. This also +makes it easy for you to wait and respond to actions on your own application's +sockets/handles. You figure out what to select() for by using +\fIcurl_multi_fdset(3)\fP, that fills in a set of fd_set variables for you +with the particular file descriptors libcurl uses for the moment. + +When you then call select(), it'll return when one of the file handles signal +action and you then call \fIcurl_multi_perform(3)\fP to allow libcurl to do +what it wants to do. Take note that libcurl does also feature some time-out +code so we advise you to never use very long timeouts on select() before you +call \fIcurl_multi_perform(3)\fP, which thus should be called unconditionally +every now and then even if none of its file descriptors have signaled +ready. Another precaution you should use: always call +\fIcurl_multi_fdset(3)\fP immediately before the select() call since the +current set of file descriptors may change when calling a curl function. + +If you want to stop the transfer of one of the easy handles in the stack, you +can use \fIcurl_multi_remove_handle(3)\fP to remove individual easy +handles. Remember that easy handles should be \fIcurl_easy_cleanup(3)\fPed. + +When a transfer within the multi stack has finished, the counter of running +transfers (as filled in by \fIcurl_multi_perform(3)\fP) will decrease. When +the number reaches zero, all transfers are done. + +\fIcurl_multi_info_read(3)\fP can be used to get information about completed +transfers. It then returns the CURLcode for each easy transfer, to allow you +to figure out success on each individual transfer. + +.SH "SSL, Certificates and Other Tricks" + + [ seeding, passwords, keys, certificates, ENGINE, ca certs ] + +.SH "Sharing Data Between Easy Handles" +You can share some data between easy handles when the easy interface is used, +and some data is share automatically when you use the multi interface. + +When you add easy handles to a multi handle, these easy handles will +automatically share a lot of the data that otherwise would be kept on a +per-easy handle basis when the easy interface is used. + +The DNS cache is shared between handles within a multi handle, making +subsequent name resolvings faster and the connection pool that is kept to +better allow persistent connections and connection re-use is shared. If you're +using the easy interface, you can still share these between specific easy +handles by using the share interface, see \fIlibcurl-share(3)\fP. + +Some things are never shared automatically, not within multi handles, like for +example cookies so the only way to share that is with the share interface. +.SH "Footnotes" + +.IP "[1]" +libcurl 7.10.3 and later have the ability to switch over to chunked +Transfer-Encoding in cases where HTTP uploads are done with data of an unknown +size. +.IP "[2]" +This happens on Windows machines when libcurl is built and used as a +DLL. However, you can still do this on Windows if you link with a static +library. +.IP "[3]" +The curl-config tool is generated at build-time (on UNIX-like systems) and +should be installed with the 'make install' or similar instruction that +installs the library, header files, man pages etc. +.IP "[4]" +This behavior was different in versions before 7.17.0, where strings had to +remain valid past the end of the \fIcurl_easy_setopt(3)\fP call. |