diff options
Diffstat (limited to 'libs/libcurl/docs/INTERNALS')
| -rw-r--r-- | libs/libcurl/docs/INTERNALS | 471 |
1 files changed, 471 insertions, 0 deletions
diff --git a/libs/libcurl/docs/INTERNALS b/libs/libcurl/docs/INTERNALS new file mode 100644 index 0000000000..66e11a46b7 --- /dev/null +++ b/libs/libcurl/docs/INTERNALS @@ -0,0 +1,471 @@ + _ _ ____ _ + ___| | | | _ \| | + / __| | | | |_) | | + | (__| |_| | _ <| |___ + \___|\___/|_| \_\_____| + +INTERNALS + + The project is split in two. The library and the client. The client part uses + the library, but the library is designed to allow other applications to use + it. + + The largest amount of code and complexity is in the library part. + +GIT +=== + All changes to the sources are committed to the git repository as soon as + they're somewhat verified to work. Changes shall be committed as independently + as possible so that individual changes can be easier spotted and tracked + afterwards. + + Tagging shall be used extensively, and by the time we release new archives we + should tag the sources with a name similar to the released version number. + +Portability +=========== + + We write curl and libcurl to compile with C89 compilers. On 32bit and up + machines. Most of libcurl assumes more or less POSIX compliance but that's + not a requirement. + + We write libcurl to build and work with lots of third party tools, and we + want it to remain functional and buildable with these and later versions + (older versions may still work but is not what we work hard to maintain): + + OpenSSL 0.9.6 + GnuTLS 1.2 + zlib 1.1.4 + libssh2 0.16 + c-ares 1.6.0 + libidn 0.4.1 + cyassl 2.0.0 + openldap 2.0 + MIT krb5 lib 1.2.4 + qsossl V5R3M0 + NSS 3.12.x + axTLS 1.2.7 + Heimdal ? + + On systems where configure runs, we aim at working on them all - if they have + a suitable C compiler. On systems that don't run configure, we strive to keep + curl running fine on: + + Windows 98 + AS/400 V5R3M0 + Symbian 9.1 + Windows CE ? + TPF ? + + When writing code (mostly for generating stuff included in release tarballs) + we use a few "build tools" and we make sure that we remain functional with + these versions: + + GNU Libtool 1.4.2 + GNU Autoconf 2.57 + GNU Automake 1.7 (we currently avoid 1.10 due to Solaris-related bugs) + GNU M4 1.4 + perl 5.004 + roffit 0.5 + groff ? (any version that supports "groff -Tps -man [in] [out]") + ps2pdf (gs) ? + +Windows vs Unix +=============== + + There are a few differences in how to program curl the unix way compared to + the Windows way. The four perhaps most notable details are: + + 1. Different function names for socket operations. + + In curl, this is solved with defines and macros, so that the source looks + the same at all places except for the header file that defines them. The + macros in use are sclose(), sread() and swrite(). + + 2. Windows requires a couple of init calls for the socket stuff. + + That's taken care of by the curl_global_init() call, but if other libs also + do it etc there might be reasons for applications to alter that behaviour. + + 3. The file descriptors for network communication and file operations are + not easily interchangeable as in unix. + + We avoid this by not trying any funny tricks on file descriptors. + + 4. When writing data to stdout, Windows makes end-of-lines the DOS way, thus + destroying binary data, although you do want that conversion if it is + text coming through... (sigh) + + We set stdout to binary under windows + + Inside the source code, We make an effort to avoid '#ifdef [Your OS]'. All + conditionals that deal with features *should* instead be in the format + '#ifdef HAVE_THAT_WEIRD_FUNCTION'. Since Windows can't run configure scripts, + we maintain a curl_config-win32.h file in lib directory that is supposed to + look exactly as a curl_config.h file would have looked like on a Windows + machine! + + Generally speaking: always remember that this will be compiled on dozens of + operating systems. Don't walk on the edge. + +Library +======= + + (See LIBCURL-STRUCTS for a separate document describing all major internal + structs and their purposes.) + + There are plenty of entry points to the library, namely each publicly defined + function that libcurl offers to applications. All of those functions are + rather small and easy-to-follow. All the ones prefixed with 'curl_easy' are + put in the lib/easy.c file. + + curl_global_init_() and curl_global_cleanup() should be called by the + application to initialize and clean up global stuff in the library. As of + today, it can handle the global SSL initing if SSL is enabled and it can init + the socket layer on windows machines. libcurl itself has no "global" scope. + + All printf()-style functions use the supplied clones in lib/mprintf.c. This + makes sure we stay absolutely platform independent. + + curl_easy_init() allocates an internal struct and makes some initializations. + The returned handle does not reveal internals. This is the 'SessionHandle' + struct which works as an "anchor" struct for all curl_easy functions. All + connections performed will get connect-specific data allocated that should be + used for things related to particular connections/requests. + + curl_easy_setopt() takes three arguments, where the option stuff must be + passed in pairs: the parameter-ID and the parameter-value. The list of + options is documented in the man page. This function mainly sets things in + the 'SessionHandle' struct. + + curl_easy_perform() is just a wrapper function that makes use of the multi + API. It basically curl_multi_init(), curl_multi_add_handle(), + curl_multi_wait(), and curl_multi_perform() until the transfer is done and + then returns. + + Some of the most important key functions in url.c are called from multi.c + when certain key steps are to be made in the transfer operation. + + o Curl_connect() + + Analyzes the URL, it separates the different components and connects to the + remote host. This may involve using a proxy and/or using SSL. The + Curl_resolv() function in lib/hostip.c is used for looking up host names + (it does then use the proper underlying method, which may vary between + platforms and builds). + + When Curl_connect is done, we are connected to the remote site. Then it is + time to tell the server to get a document/file. Curl_do() arranges this. + + This function makes sure there's an allocated and initiated 'connectdata' + struct that is used for this particular connection only (although there may + be several requests performed on the same connect). A bunch of things are + inited/inherited from the SessionHandle struct. + + o Curl_do() + + Curl_do() makes sure the proper protocol-specific function is called. The + functions are named after the protocols they handle. + + The protocol-specific functions of course deal with protocol-specific + negotiations and setup. They have access to the Curl_sendf() (from + lib/sendf.c) function to send printf-style formatted data to the remote + host and when they're ready to make the actual file transfer they call the + Curl_Transfer() function (in lib/transfer.c) to setup the transfer and + returns. + + If this DO function fails and the connection is being re-used, libcurl will + then close this connection, setup a new connection and re-issue the DO + request on that. This is because there is no way to be perfectly sure that + we have discovered a dead connection before the DO function and thus we + might wrongly be re-using a connection that was closed by the remote peer. + + Some time during the DO function, the Curl_setup_transfer() function must + be called with some basic info about the upcoming transfer: what socket(s) + to read/write and the expected file transfer sizes (if known). + + o Curl_readwrite() + + Called during the transfer of the actual protocol payload. + + During transfer, the progress functions in lib/progress.c are called at a + frequent interval (or at the user's choice, a specified callback might get + called). The speedcheck functions in lib/speedcheck.c are also used to + verify that the transfer is as fast as required. + + o Curl_done() + + Called after a transfer is done. This function takes care of everything + that has to be done after a transfer. This function attempts to leave + matters in a state so that Curl_do() should be possible to call again on + the same connection (in a persistent connection case). It might also soon + be closed with Curl_disconnect(). + + o Curl_disconnect() + + When doing normal connections and transfers, no one ever tries to close any + connections so this is not normally called when curl_easy_perform() is + used. This function is only used when we are certain that no more transfers + is going to be made on the connection. It can be also closed by force, or + it can be called to make sure that libcurl doesn't keep too many + connections alive at the same time. + + This function cleans up all resources that are associated with a single + connection. + + + HTTP(S) + + HTTP offers a lot and is the protocol in curl that uses the most lines of + code. There is a special file (lib/formdata.c) that offers all the multipart + post functions. + + base64-functions for user+password stuff (and more) is in (lib/base64.c) and + all functions for parsing and sending cookies are found in (lib/cookie.c). + + HTTPS uses in almost every means the same procedure as HTTP, with only two + exceptions: the connect procedure is different and the function used to read + or write from the socket is different, although the latter fact is hidden in + the source by the use of Curl_read() for reading and Curl_write() for writing + data to the remote server. + + http_chunks.c contains functions that understands HTTP 1.1 chunked transfer + encoding. + + An interesting detail with the HTTP(S) request, is the Curl_add_buffer() + series of functions we use. They append data to one single buffer, and when + the building is done the entire request is sent off in one single write. This + is done this way to overcome problems with flawed firewalls and lame servers. + + FTP + + The Curl_if2ip() function can be used for getting the IP number of a + specified network interface, and it resides in lib/if2ip.c. + + Curl_ftpsendf() is used for sending FTP commands to the remote server. It was + made a separate function to prevent us programmers from forgetting that they + must be CRLF terminated. They must also be sent in one single write() to make + firewalls and similar happy. + + Kerberos + + The kerberos support is mainly in lib/krb4.c and lib/security.c. + + TELNET + + Telnet is implemented in lib/telnet.c. + + FILE + + The file:// protocol is dealt with in lib/file.c. + + LDAP + + Everything LDAP is in lib/ldap.c and lib/openldap.c + + GENERAL + + URL encoding and decoding, called escaping and unescaping in the source code, + is found in lib/escape.c. + + While transferring data in Transfer() a few functions might get used. + curl_getdate() in lib/parsedate.c is for HTTP date comparisons (and more). + + lib/getenv.c offers curl_getenv() which is for reading environment variables + in a neat platform independent way. That's used in the client, but also in + lib/url.c when checking the proxy environment variables. Note that contrary + to the normal unix getenv(), this returns an allocated buffer that must be + free()ed after use. + + lib/netrc.c holds the .netrc parser + + lib/timeval.c features replacement functions for systems that don't have + gettimeofday() and a few support functions for timeval conversions. + + A function named curl_version() that returns the full curl version string is + found in lib/version.c. + +Persistent Connections +====================== + + The persistent connection support in libcurl requires some considerations on + how to do things inside of the library. + + o The 'SessionHandle' struct returned in the curl_easy_init() call must never + hold connection-oriented data. It is meant to hold the root data as well as + all the options etc that the library-user may choose. + o The 'SessionHandle' struct holds the "connection cache" (an array of + pointers to 'connectdata' structs). + o This enables the 'curl handle' to be reused on subsequent transfers. + o When libcurl is told to perform a transfer, it first checks for an already + existing connection in the cache that we can use. Otherwise it creates a + new one and adds that the cache. If the cache is full already when a new + conncetion is added added, it will first close the oldest unused one. + o When the transfer operation is complete, the connection is left + open. Particular options may tell libcurl not to, and protocols may signal + closure on connections and then they won't be kept open of course. + o When curl_easy_cleanup() is called, we close all still opened connections, + unless of course the multi interface "owns" the connections. + + The curl handle must be re-used in order for the persistent connections to + work. + +multi interface/non-blocking +============================ + + The multi interface is a non-blocking interface to the library. To make that + interface work as good as possible, no low-level functions within libcurl + must be written to work in a blocking manner. (There are still a few spots + violating this rule.) + + One of the primary reasons we introduced c-ares support was to allow the name + resolve phase to be perfectly non-blocking as well. + + The FTP and the SFTP/SCP protocols are examples of how we adapt and adjust + the code to allow non-blocking operations even on multi-stage command- + response protocols. They are built around state machines that return when + they would otherwise block waiting for data. The DICT, LDAP and TELNET + protocols are crappy examples and they are subject for rewrite in the future + to better fit the libcurl protocol family. + +SSL libraries +============= + + Originally libcurl supported SSLeay for SSL/TLS transports, but that was then + extended to its successor OpenSSL but has since also been extended to several + other SSL/TLS libraries and we expect and hope to further extend the support + in future libcurl versions. + + To deal with this internally in the best way possible, we have a generic SSL + function API as provided by the sslgen.[ch] system, and they are the only SSL + functions we must use from within libcurl. sslgen is then crafted to use the + appropriate lower-level function calls to whatever SSL library that is in + use. + +Library Symbols +=============== + + All symbols used internally in libcurl must use a 'Curl_' prefix if they're + used in more than a single file. Single-file symbols must be made static. + Public ("exported") symbols must use a 'curl_' prefix. (There are exceptions, + but they are to be changed to follow this pattern in future versions.) Public + API functions are marked with CURL_EXTERN in the public header files so that + all others can be hidden on platforms where this is possible. + +Return Codes and Informationals +=============================== + + I've made things simple. Almost every function in libcurl returns a CURLcode, + that must be CURLE_OK if everything is OK or otherwise a suitable error code + as the curl/curl.h include file defines. The very spot that detects an error + must use the Curl_failf() function to set the human-readable error + description. + + In aiding the user to understand what's happening and to debug curl usage, we + must supply a fair amount of informational messages by using the Curl_infof() + function. Those messages are only displayed when the user explicitly asks for + them. They are best used when revealing information that isn't otherwise + obvious. + +API/ABI +======= + + We make an effort to not export or show internals or how internals work, as + that makes it easier to keep a solid API/ABI over time. See docs/libcurl/ABI + for our promise to users. + +Client +====== + + main() resides in src/tool_main.c. + + src/tool_hugehelp.c is automatically generated by the mkhelp.pl perl script + to display the complete "manual" and the src/tool_urlglob.c file holds the + functions used for the URL-"globbing" support. Globbing in the sense that the + {} and [] expansion stuff is there. + + The client mostly messes around to setup its 'config' struct properly, then + it calls the curl_easy_*() functions of the library and when it gets back + control after the curl_easy_perform() it cleans up the library, checks status + and exits. + + When the operation is done, the ourWriteOut() function in src/writeout.c may + be called to report about the operation. That function is using the + curl_easy_getinfo() function to extract useful information from the curl + session. + + It may loop and do all this several times if many URLs were specified on the + command line or config file. + +Memory Debugging +================ + + The file lib/memdebug.c contains debug-versions of a few functions. Functions + such as malloc, free, fopen, fclose, etc that somehow deal with resources + that might give us problems if we "leak" them. The functions in the memdebug + system do nothing fancy, they do their normal function and then log + information about what they just did. The logged data can then be analyzed + after a complete session, + + memanalyze.pl is the perl script present in tests/ that analyzes a log file + generated by the memory tracking system. It detects if resources are + allocated but never freed and other kinds of errors related to resource + management. + + Internally, definition of preprocessor symbol DEBUGBUILD restricts code which + is only compiled for debug enabled builds. And symbol CURLDEBUG is used to + differentiate code which is _only_ used for memory tracking/debugging. + + Use -DCURLDEBUG when compiling to enable memory debugging, this is also + switched on by running configure with --enable-curldebug. Use -DDEBUGBUILD + when compiling to enable a debug build or run configure with --enable-debug. + + curl --version will list 'Debug' feature for debug enabled builds, and + will list 'TrackMemory' feature for curl debug memory tracking capable + builds. These features are independent and can be controlled when running + the configure script. When --enable-debug is given both features will be + enabled, unless some restriction prevents memory tracking from being used. + +Test Suite +========== + + The test suite is placed in its own subdirectory directly off the root in the + curl archive tree, and it contains a bunch of scripts and a lot of test case + data. + + The main test script is runtests.pl that will invoke test servers like + httpserver.pl and ftpserver.pl before all the test cases are performed. The + test suite currently only runs on unix-like platforms. + + You'll find a description of the test suite in the tests/README file, and the + test case data files in the tests/FILEFORMAT file. + + The test suite automatically detects if curl was built with the memory + debugging enabled, and if it was it will detect memory leaks, too. + +Building Releases +================= + + There's no magic to this. When you consider everything stable enough to be + released, do this: + + 1. Tag the source code accordingly. + + 2. run the 'maketgz' script (using 'make distcheck' will give you a pretty + good view on the status of the current sources). maketgz requires a + version number and creates the release archive. maketgz uses 'make dist' + for the actual archive building, why you need to fill in the Makefile.am + files properly for which files that should be included in the release + archives. + + 3. When that's complete, sign the output files. + + 4. Upload + + 5. Update web site and changelog on site + + 6. Send announcement to the mailing lists + + NOTE: you must have curl checked out from git to be able to do a proper + release build. The release tarballs do not have everything setup in order to + do releases properly. |