libcurl extracted to the separate lib

git-svn-id: http://svn.miranda-ng.org/main/trunk@14683 1316c22d-e87f-b044-9b9b-93d7a3e3ba9c

1 files changed, 471 insertions, 0 deletions

diff --git a/libs/libcurl/docs/INTERNALS b/libs/libcurl/docs/INTERNALS
new file mode 100644
index 0000000000..66e11a46b7
--- /dev/null
+++ b/libs/libcurl/docs/INTERNALS
@@ -0,0 +1,471 @@
+                                  _   _ ____  _
+                              ___| | | |  _ \| |
+                             / __| | | | |_) | |
+                            | (__| |_| |  _ <| |___
+                             \___|\___/|_| \_\_____|
+
+INTERNALS
+
+ The project is split in two. The library and the client. The client part uses
+ the library, but the library is designed to allow other applications to use
+ it.
+
+ The largest amount of code and complexity is in the library part.
+
+GIT
+===
+ All changes to the sources are committed to the git repository as soon as
+ they're somewhat verified to work. Changes shall be committed as independently
+ as possible so that individual changes can be easier spotted and tracked
+ afterwards.
+
+ Tagging shall be used extensively, and by the time we release new archives we
+ should tag the sources with a name similar to the released version number.
+
+Portability
+===========
+
+ We write curl and libcurl to compile with C89 compilers.  On 32bit and up
+ machines. Most of libcurl assumes more or less POSIX compliance but that's
+ not a requirement.
+
+ We write libcurl to build and work with lots of third party tools, and we
+ want it to remain functional and buildable with these and later versions
+ (older versions may still work but is not what we work hard to maintain):
+
+ OpenSSL      0.9.6
+ GnuTLS       1.2
+ zlib         1.1.4
+ libssh2      0.16
+ c-ares       1.6.0
+ libidn       0.4.1
+ cyassl       2.0.0
+ openldap     2.0
+ MIT krb5 lib 1.2.4
+ qsossl       V5R3M0
+ NSS          3.12.x
+ axTLS        1.2.7
+ Heimdal      ?
+
+ On systems where configure runs, we aim at working on them all - if they have
+ a suitable C compiler. On systems that don't run configure, we strive to keep
+ curl running fine on:
+
+ Windows      98
+ AS/400       V5R3M0
+ Symbian      9.1
+ Windows CE   ?
+ TPF          ?
+
+ When writing code (mostly for generating stuff included in release tarballs)
+ we use a few "build tools" and we make sure that we remain functional with
+ these versions:
+
+ GNU Libtool  1.4.2
+ GNU Autoconf 2.57
+ GNU Automake 1.7 (we currently avoid 1.10 due to Solaris-related bugs)
+ GNU M4       1.4
+ perl         5.004
+ roffit       0.5
+ groff        ? (any version that supports "groff -Tps -man [in] [out]")
+ ps2pdf (gs)  ?
+
+Windows vs Unix
+===============
+
+ There are a few differences in how to program curl the unix way compared to
+ the Windows way. The four perhaps most notable details are:
+
+ 1. Different function names for socket operations.
+
+   In curl, this is solved with defines and macros, so that the source looks
+   the same at all places except for the header file that defines them. The
+   macros in use are sclose(), sread() and swrite().
+
+ 2. Windows requires a couple of init calls for the socket stuff.
+
+   That's taken care of by the curl_global_init() call, but if other libs also
+   do it etc there might be reasons for applications to alter that behaviour.
+
+ 3. The file descriptors for network communication and file operations are
+    not easily interchangeable as in unix.
+
+   We avoid this by not trying any funny tricks on file descriptors.
+
+ 4. When writing data to stdout, Windows makes end-of-lines the DOS way, thus
+    destroying binary data, although you do want that conversion if it is
+    text coming through... (sigh)
+
+   We set stdout to binary under windows
+
+ Inside the source code, We make an effort to avoid '#ifdef [Your OS]'. All
+ conditionals that deal with features *should* instead be in the format
+ '#ifdef HAVE_THAT_WEIRD_FUNCTION'. Since Windows can't run configure scripts,
+ we maintain a curl_config-win32.h file in lib directory that is supposed to
+ look exactly as a curl_config.h file would have looked like on a Windows
+ machine!
+
+ Generally speaking: always remember that this will be compiled on dozens of
+ operating systems. Don't walk on the edge.
+
+Library
+=======
+
+ (See LIBCURL-STRUCTS for a separate document describing all major internal
+ structs and their purposes.)
+
+ There are plenty of entry points to the library, namely each publicly defined
+ function that libcurl offers to applications. All of those functions are
+ rather small and easy-to-follow. All the ones prefixed with 'curl_easy' are
+ put in the lib/easy.c file.
+
+ curl_global_init_() and curl_global_cleanup() should be called by the
+ application to initialize and clean up global stuff in the library. As of
+ today, it can handle the global SSL initing if SSL is enabled and it can init
+ the socket layer on windows machines. libcurl itself has no "global" scope.
+
+ All printf()-style functions use the supplied clones in lib/mprintf.c. This
+ makes sure we stay absolutely platform independent.
+
+ curl_easy_init() allocates an internal struct and makes some initializations.
+ The returned handle does not reveal internals. This is the 'SessionHandle'
+ struct which works as an "anchor" struct for all curl_easy functions. All
+ connections performed will get connect-specific data allocated that should be
+ used for things related to particular connections/requests.
+
+ curl_easy_setopt() takes three arguments, where the option stuff must be
+ passed in pairs: the parameter-ID and the parameter-value. The list of
+ options is documented in the man page. This function mainly sets things in
+ the 'SessionHandle' struct.
+
+ curl_easy_perform() is just a wrapper function that makes use of the multi
+ API.  It basically curl_multi_init(), curl_multi_add_handle(),
+ curl_multi_wait(), and curl_multi_perform() until the transfer is done and
+ then returns.
+
+ Some of the most important key functions in url.c are called from multi.c
+ when certain key steps are to be made in the transfer operation.
+
+ o Curl_connect()
+
+   Analyzes the URL, it separates the different components and connects to the
+   remote host. This may involve using a proxy and/or using SSL. The
+   Curl_resolv() function in lib/hostip.c is used for looking up host names
+   (it does then use the proper underlying method, which may vary between
+   platforms and builds).
+
+   When Curl_connect is done, we are connected to the remote site. Then it is
+   time to tell the server to get a document/file. Curl_do() arranges this.
+
+   This function makes sure there's an allocated and initiated 'connectdata'
+   struct that is used for this particular connection only (although there may
+   be several requests performed on the same connect). A bunch of things are
+   inited/inherited from the SessionHandle struct.
+
+ o Curl_do()
+
+   Curl_do() makes sure the proper protocol-specific function is called. The
+   functions are named after the protocols they handle.
+
+   The protocol-specific functions of course deal with protocol-specific
+   negotiations and setup. They have access to the Curl_sendf() (from
+   lib/sendf.c) function to send printf-style formatted data to the remote
+   host and when they're ready to make the actual file transfer they call the
+   Curl_Transfer() function (in lib/transfer.c) to setup the transfer and
+   returns.
+
+   If this DO function fails and the connection is being re-used, libcurl will
+   then close this connection, setup a new connection and re-issue the DO
+   request on that. This is because there is no way to be perfectly sure that
+   we have discovered a dead connection before the DO function and thus we
+   might wrongly be re-using a connection that was closed by the remote peer.
+
+   Some time during the DO function, the Curl_setup_transfer() function must
+   be called with some basic info about the upcoming transfer: what socket(s)
+   to read/write and the expected file transfer sizes (if known).
+
+ o Curl_readwrite()
+
+   Called during the transfer of the actual protocol payload.
+
+   During transfer, the progress functions in lib/progress.c are called at a
+   frequent interval (or at the user's choice, a specified callback might get
+   called). The speedcheck functions in lib/speedcheck.c are also used to
+   verify that the transfer is as fast as required.
+
+ o Curl_done()
+
+   Called after a transfer is done. This function takes care of everything
+   that has to be done after a transfer. This function attempts to leave
+   matters in a state so that Curl_do() should be possible to call again on
+   the same connection (in a persistent connection case). It might also soon
+   be closed with Curl_disconnect().
+
+ o Curl_disconnect()
+
+   When doing normal connections and transfers, no one ever tries to close any
+   connections so this is not normally called when curl_easy_perform() is
+   used. This function is only used when we are certain that no more transfers
+   is going to be made on the connection. It can be also closed by force, or
+   it can be called to make sure that libcurl doesn't keep too many
+   connections alive at the same time.
+
+   This function cleans up all resources that are associated with a single
+   connection.
+
+
+ HTTP(S)
+
+ HTTP offers a lot and is the protocol in curl that uses the most lines of
+ code. There is a special file (lib/formdata.c) that offers all the multipart
+ post functions.
+
+ base64-functions for user+password stuff (and more) is in (lib/base64.c) and
+ all functions for parsing and sending cookies are found in (lib/cookie.c).
+
+ HTTPS uses in almost every means the same procedure as HTTP, with only two
+ exceptions: the connect procedure is different and the function used to read
+ or write from the socket is different, although the latter fact is hidden in
+ the source by the use of Curl_read() for reading and Curl_write() for writing
+ data to the remote server.
+
+ http_chunks.c contains functions that understands HTTP 1.1 chunked transfer
+ encoding.
+
+ An interesting detail with the HTTP(S) request, is the Curl_add_buffer()
+ series of functions we use. They append data to one single buffer, and when
+ the building is done the entire request is sent off in one single write. This
+ is done this way to overcome problems with flawed firewalls and lame servers.
+
+ FTP
+
+ The Curl_if2ip() function can be used for getting the IP number of a
+ specified network interface, and it resides in lib/if2ip.c.
+
+ Curl_ftpsendf() is used for sending FTP commands to the remote server. It was
+ made a separate function to prevent us programmers from forgetting that they
+ must be CRLF terminated. They must also be sent in one single write() to make
+ firewalls and similar happy.
+
+ Kerberos
+
+ The kerberos support is mainly in lib/krb4.c and lib/security.c.
+
+ TELNET
+
+ Telnet is implemented in lib/telnet.c.
+
+ FILE
+
+ The file:// protocol is dealt with in lib/file.c.
+
+ LDAP
+
+ Everything LDAP is in lib/ldap.c and lib/openldap.c
+
+ GENERAL
+
+ URL encoding and decoding, called escaping and unescaping in the source code,
+ is found in lib/escape.c.
+
+ While transferring data in Transfer() a few functions might get used.
+ curl_getdate() in lib/parsedate.c is for HTTP date comparisons (and more).
+
+ lib/getenv.c offers curl_getenv() which is for reading environment variables
+ in a neat platform independent way. That's used in the client, but also in
+ lib/url.c when checking the proxy environment variables. Note that contrary
+ to the normal unix getenv(), this returns an allocated buffer that must be
+ free()ed after use.
+
+ lib/netrc.c holds the .netrc parser
+
+ lib/timeval.c features replacement functions for systems that don't have
+ gettimeofday() and a few support functions for timeval conversions.
+
+ A function named curl_version() that returns the full curl version string is
+ found in lib/version.c.
+
+Persistent Connections
+======================
+
+ The persistent connection support in libcurl requires some considerations on
+ how to do things inside of the library.
+
+ o The 'SessionHandle' struct returned in the curl_easy_init() call must never
+   hold connection-oriented data. It is meant to hold the root data as well as
+   all the options etc that the library-user may choose.
+ o The 'SessionHandle' struct holds the "connection cache" (an array of
+   pointers to 'connectdata' structs).
+ o This enables the 'curl handle' to be reused on subsequent transfers.
+ o When libcurl is told to perform a transfer, it first checks for an already
+   existing connection in the cache that we can use. Otherwise it creates a
+   new one and adds that the cache. If the cache is full already when a new
+   conncetion is added added, it will first close the oldest unused one.
+ o When the transfer operation is complete, the connection is left
+   open. Particular options may tell libcurl not to, and protocols may signal
+   closure on connections and then they won't be kept open of course.
+ o When curl_easy_cleanup() is called, we close all still opened connections,
+   unless of course the multi interface "owns" the connections.
+
+ The curl handle must be re-used in order for the persistent connections to
+ work.
+
+multi interface/non-blocking
+============================
+
+ The multi interface is a non-blocking interface to the library. To make that
+ interface work as good as possible, no low-level functions within libcurl
+ must be written to work in a blocking manner. (There are still a few spots
+ violating this rule.)
+
+ One of the primary reasons we introduced c-ares support was to allow the name
+ resolve phase to be perfectly non-blocking as well.
+
+ The FTP and the SFTP/SCP protocols are examples of how we adapt and adjust
+ the code to allow non-blocking operations even on multi-stage command-
+ response protocols. They are built around state machines that return when
+ they would otherwise block waiting for data.  The DICT, LDAP and TELNET
+ protocols are crappy examples and they are subject for rewrite in the future
+ to better fit the libcurl protocol family.
+
+SSL libraries
+=============
+
+ Originally libcurl supported SSLeay for SSL/TLS transports, but that was then
+ extended to its successor OpenSSL but has since also been extended to several
+ other SSL/TLS libraries and we expect and hope to further extend the support
+ in future libcurl versions.
+
+ To deal with this internally in the best way possible, we have a generic SSL
+ function API as provided by the sslgen.[ch] system, and they are the only SSL
+ functions we must use from within libcurl. sslgen is then crafted to use the
+ appropriate lower-level function calls to whatever SSL library that is in
+ use.
+
+Library Symbols
+===============
+
+ All symbols used internally in libcurl must use a 'Curl_' prefix if they're
+ used in more than a single file. Single-file symbols must be made static.
+ Public ("exported") symbols must use a 'curl_' prefix. (There are exceptions,
+ but they are to be changed to follow this pattern in future versions.) Public
+ API functions are marked with CURL_EXTERN in the public header files so that
+ all others can be hidden on platforms where this is possible.
+
+Return Codes and Informationals
+===============================
+
+ I've made things simple. Almost every function in libcurl returns a CURLcode,
+ that must be CURLE_OK if everything is OK or otherwise a suitable error code
+ as the curl/curl.h include file defines. The very spot that detects an error
+ must use the Curl_failf() function to set the human-readable error
+ description.
+
+ In aiding the user to understand what's happening and to debug curl usage, we
+ must supply a fair amount of informational messages by using the Curl_infof()
+ function. Those messages are only displayed when the user explicitly asks for
+ them. They are best used when revealing information that isn't otherwise
+ obvious.
+
+API/ABI
+=======
+
+ We make an effort to not export or show internals or how internals work, as
+ that makes it easier to keep a solid API/ABI over time. See docs/libcurl/ABI
+ for our promise to users.
+
+Client
+======
+
+ main() resides in src/tool_main.c.
+
+ src/tool_hugehelp.c is automatically generated by the mkhelp.pl perl script
+ to display the complete "manual" and the src/tool_urlglob.c file holds the
+ functions used for the URL-"globbing" support. Globbing in the sense that the
+ {} and [] expansion stuff is there.
+
+ The client mostly messes around to setup its 'config' struct properly, then
+ it calls the curl_easy_*() functions of the library and when it gets back
+ control after the curl_easy_perform() it cleans up the library, checks status
+ and exits.
+
+ When the operation is done, the ourWriteOut() function in src/writeout.c may
+ be called to report about the operation. That function is using the
+ curl_easy_getinfo() function to extract useful information from the curl
+ session.
+
+ It may loop and do all this several times if many URLs were specified on the
+ command line or config file.
+
+Memory Debugging
+================
+
+ The file lib/memdebug.c contains debug-versions of a few functions. Functions
+ such as malloc, free, fopen, fclose, etc that somehow deal with resources
+ that might give us problems if we "leak" them. The functions in the memdebug
+ system do nothing fancy, they do their normal function and then log
+ information about what they just did. The logged data can then be analyzed
+ after a complete session,
+
+ memanalyze.pl is the perl script present in tests/ that analyzes a log file
+ generated by the memory tracking system. It detects if resources are
+ allocated but never freed and other kinds of errors related to resource
+ management.
+
+ Internally, definition of preprocessor symbol DEBUGBUILD restricts code which
+ is only compiled for debug enabled builds. And symbol CURLDEBUG is used to
+ differentiate code which is _only_ used for memory tracking/debugging.
+
+ Use -DCURLDEBUG when compiling to enable memory debugging, this is also
+ switched on by running configure with --enable-curldebug. Use -DDEBUGBUILD
+ when compiling to enable a debug build or run configure with --enable-debug.
+
+ curl --version will list 'Debug' feature for debug enabled builds, and
+ will list 'TrackMemory' feature for curl debug memory tracking capable
+ builds. These features are independent and can be controlled when running
+ the configure script. When --enable-debug is given both features will be
+ enabled, unless some restriction prevents memory tracking from being used.
+
+Test Suite
+==========
+
+ The test suite is placed in its own subdirectory directly off the root in the
+ curl archive tree, and it contains a bunch of scripts and a lot of test case
+ data.
+
+ The main test script is runtests.pl that will invoke test servers like
+ httpserver.pl and ftpserver.pl before all the test cases are performed. The
+ test suite currently only runs on unix-like platforms.
+
+ You'll find a description of the test suite in the tests/README file, and the
+ test case data files in the tests/FILEFORMAT file.
+
+ The test suite automatically detects if curl was built with the memory
+ debugging enabled, and if it was it will detect memory leaks, too.
+
+Building Releases
+=================
+
+ There's no magic to this. When you consider everything stable enough to be
+ released, do this:
+
+   1. Tag the source code accordingly.
+
+   2. run the 'maketgz' script (using 'make distcheck' will give you a pretty
+      good view on the status of the current sources). maketgz requires a
+      version number and creates the release archive. maketgz uses 'make dist'
+      for the actual archive building, why you need to fill in the Makefile.am
+      files properly for which files that should be included in the release
+      archives.
+
+   3. When that's complete, sign the output files.
+
+   4. Upload
+
+   5. Update web site and changelog on site
+
+   6. Send announcement to the mailing lists
+
+ NOTE: you must have curl checked out from git to be able to do a proper
+ release build. The release tarballs do not have everything setup in order to
+ do releases properly.