diff options
Diffstat (limited to 'protocols/Tox/toxcore/docs')
| -rw-r--r-- | protocols/Tox/toxcore/docs/Group-Chats.md | 71 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/Hardening.txt | 60 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/Hardening_docs.txt | 30 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/Prevent_Tracking.txt | 158 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/TCP_Network.txt | 154 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/TODO | 62 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/Tox_middle_level_network_protocol.txt | 120 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/av_api.md | 194 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/updates/Crypto.md | 54 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/updates/DHT.md | 105 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/updates/Spam-Prevention.md | 12 | ||||
| -rw-r--r-- | protocols/Tox/toxcore/docs/updates/Symmetric-NAT-Transversal.md | 43 |
12 files changed, 1063 insertions, 0 deletions
diff --git a/protocols/Tox/toxcore/docs/Group-Chats.md b/protocols/Tox/toxcore/docs/Group-Chats.md new file mode 100644 index 0000000000..5e00b550cc --- /dev/null +++ b/protocols/Tox/toxcore/docs/Group-Chats.md @@ -0,0 +1,71 @@ +Massive public group chats. + +Note that not all this document has been implemented: only private (invite only) group chats are currently implemented. + +Everyone generates a short term public private key pair right before joining +the chat. + +Note that for public group chats it is impossible to protect the chat from +being spied on by a very dedicated attacker, encryption is therefor used as a +form of spam/access control. + +## Joining the chats + + +## Protocol + + +Node format: +See DHT, currently uses the IPv6 Node_format. + +Get nodes (Request): +Packet contents: +``` +[char with a value of 48][Bob's (The receiver's) Public key (client_id) (32 bytes))][Alice's (The sender's) Public key (client_id) (32 bytes)][Random nonce (24 bytes)][Encrypted with the nonce, private key of the sender and public key of the receiver:[char with a value of 48][random 8 byte (ping_id)] +``` +Valid replies: a send_nodes packet + +Send_nodes (response): +``` +[char with a value of 48][Bob's (The receiver's) Public key (client_id) (32 bytes))][Alice's (The sender's) Public key (client_id) (32 bytes)][Random nonce (24 bytes)][Encrypted with the nonce, private key of the sender and public key of the receiver:[char with a value of 49][random 8 byte (ping_id)][Nodes in node format, length=40 * (number of nodes (maximum of 6 nodes)) bytes]] +``` + +Broadcast packet: +``` +[char with a value of 48][Bob's (The receiver's) Public key (client_id) (32 bytes))][Alice's (The sender's) Public key (client_id) (32 bytes)][nonce][Encrypted with the nonce, private key of the sender and public key of the receiver:[char with a value of 50][Data to send to everyone]] +``` + + +Data to send to everyone: +TODO: signing and spam control + permissions. +[client_id of sender][uint32_t message number][char with a value representing id of message][data] + +Note: the message number is increased by 1 for each sent message. + +message ids: +0 - ping +sent every ~60 seconds by every peer. +No data. + +16 - new_peer +Tell everyone about a new peer in the chat. +[uint8_t public_key[public_key_len]] + +17 - ban_peer +Ban a peer +[uint8_t public_key[public_key_len]] + +18 - topic change +[uint8_t topic[topiclen]] + +48 - name change +[uint8_t name[namelen]] + +49 - status change +[uint8_t (status id)] + +64 - chat message +[uint8_t message[messagelen]] + +65 - action (/me) +[uint8_t message[messagelen]] diff --git a/protocols/Tox/toxcore/docs/Hardening.txt b/protocols/Tox/toxcore/docs/Hardening.txt new file mode 100644 index 0000000000..50ccfd53e0 --- /dev/null +++ b/protocols/Tox/toxcore/docs/Hardening.txt @@ -0,0 +1,60 @@ +Currently an attacker with sufficient resources could launch a large scale +denial of service type attack by flooding the Tox network with a bunch of nodes +that do not act like real nodes to prevent people from finding each other. + +Due to the design of Tox, this is the worst thing an attacker can do to disrupt +the network. + +This solution's goal is to make these denial of service attack very very hard +to accomplish. + +For the network to work every Tox node must: +1. Respond to ping requests. +2. Respond to get node requests with the ids of nodes closest to a queried id +(It is assumed each nodes know at least the 32 nodes closest to them.) +3. Properly send crypto request packets to their intended destination. + +Currently the only thing a node needs to do to be part of the network is +respond correctly to ping requests. + +The only people we really trust on the network are the nodes in our friends +list. + + +The behavior of each Tox node is easily predictable. This means that it possible +for Tox nodes to test the nodes that they are connected to to see if they +behave like normal Tox nodes and only send nodes that are confirmed to behave +like real Tox nodes as part of send node replies when other nodes query them. + +If correctly done, this means that to poison the network an attacker can only +infiltrate the network if his "fake" nodes behave exactly like real nodes +completely defeating the purpose of the attack. Of course nodes must be +rechecked regularly to defeat an attack where someone floods the network with +many good nodes then suddenly turns them all bad. + +This also prevents someone from accidentally killing the tox network with a bad +implementation of the protocol. + +Implementation ideas (In Progress): + +1. Use our friends to check if the nodes in our close list are good. + +EX: If our friend queries a node close to us and it correctly returns our +ip/port and then sends a crypto request packet to it and it routes it correctly +to us then it is good. + +Problems with this: People don't always have at least one online friend. + +2. Pick random nodes (add ourselves some random (fake) friends to increase the +pool of available nodes) and make then send requests to other nodes, the +response is then relayed back to us and compared to how the node should have +behaved. If the node is found to be behaving correctly, it is set as trusted. +Only trusted nodes are sent in send node packets, that is unless the exact node +being queried for in the getnode packet is present, it will be sent in the +sendnode packet even if it is not trusted. + +The hypothesis is that if to be part of the network nodes have to behave +correctly it should prevent disruption from nodes that behave incorrectly. + +(This idea is currently being implemented in the harden branch.) +... diff --git a/protocols/Tox/toxcore/docs/Hardening_docs.txt b/protocols/Tox/toxcore/docs/Hardening_docs.txt new file mode 100644 index 0000000000..53a6103418 --- /dev/null +++ b/protocols/Tox/toxcore/docs/Hardening_docs.txt @@ -0,0 +1,30 @@ +Hardening request packets are sent as crypto request packets (see crypto docs.) +NOTE: currently only get nodes requests are tested in the code which is why +there is only one test (more will be added soon.) + +All hardening requests must contain exactly 768 bytes of data. (The data sent +must be padded with zeros if it is smaller than that.) + +1. Get the information (IP_port, client_id) of the node we want to test. +2. Find a couple random nodes that is not that node (one for each test.) +3. Send crypto request packets to each of these random nodes with the data being: + +[byte with value: 02 (get nodes test request)][struct Node_format (the node to +test.)][client_id(32 bytes) the id to query the node with.][padding] + +4. The random node receives a packet. +-The packet is a get nodes test request: + send a get_node request to that node with the id to query in the request. + when a send_node response is received, send the following response to the + person who sent us the get nodes test request packet: + [byte with value: 03 (get nodes test response)][client_id(32 bytes): + the id of the tested node][The list of nodes it responded with in IPv6 + Node format (struct Node_Format)] + PROTIP: (get node requests and response contain an encrypted part that you + can use to store information so that you don't + have to store in your memory where/if to send back the response from the + send node) + +5. Receive the test responses. +-If the test(s) pass (the nodes behave in a satisfactory manner), make these +nodes have priority over those who don't pass the test(s). diff --git a/protocols/Tox/toxcore/docs/Prevent_Tracking.txt b/protocols/Tox/toxcore/docs/Prevent_Tracking.txt new file mode 100644 index 0000000000..d170103f0b --- /dev/null +++ b/protocols/Tox/toxcore/docs/Prevent_Tracking.txt @@ -0,0 +1,158 @@ +Current privacy issues with the Tox DHT: + +1. It makes tracking people across different IPs very easy. +Solution: Have each new DHT use a temporary public/private key pair not related +to the long term public/private key pair. + +2. Metadata on which key is friends to which can be collected (The hardening +makes this somewhat harder by introducing a bunch of random traffic but not +impossible.). +Solution: If no long term keys were used in the DHT it would solve this +problem. (possibly knowing which ip is connected to which is much less +precious.) + + +So, it seems all our privacy problems are solved if we can manage to make every +node in the DHT have a keypair that is not related to the long term keys and is +generated every session. + + +So, every node in the DHT now has a temporary keypair not related to their real +long term one. + +But, how do people find themselves then? We have to add a way for people to +tell their friends what their DHT public key is. We also have to somehow make +it so people can send/receive friend requests. This has to be done without +non-friends being able to find out where a node is. + +The solution: Onion routing + enable the storage of some small amount of data +on DHT nodes. + + +Alice and bob are friends. Before joining the DHT they generate temporary +session keypairs to be used for the DHT instead of their long term keys. + +Bob finds a bunch of random nodes then picks 3 random working ones (A, B, C). + +Bob gets the known working node with an id closest to his real one from his list (D) + +Bob then creates an onion (the packet will go through A, B, C and will end up at D) +announce request packet with his real public key, ping_id as zeros and +searching for his real public key. + +Bob will announce response packets and will recursively send onion announce request +packets to closer and closer nodes until he finds the ones closest to his real public key. + +Once he has done this, he will send some onion announce request packets with the right +ping_id previously received from the node when he queried it to announce himself to the node. + +The nodes he announces himself to keep the information to send onion packets to that node in +memory. + +Alice meanwhile searches for the nodes closest to Bobs real id using a temporary keypair and +announce request packets. She does this until she finds nodes that respond with a ping_id of zero. + +She sends data to route request packet with information telling Bob her temporary id in the DHT +(or a friend request if she is not friends with him). + +Bob finds her by using her temporary id and they connect to each other. + + +NOTE: crypto_box is used for all the asymmetric encryption and crypto_secretbox is used for all +the symmetric. Also every DHT node have a random symmetric key which they use to encrypt the stuff +in normal get node request that is used to encrypt stuff in the following. + +Onion packet (request): + +initial (sent from us to node A): + +[uint8_t packet id (128)][nonce] +[our temp DHT public key]encrypted with our temp DHT private key and the pub key of Node A and the nonce:[ +[IP_Port of node B][a random public key]encrypted with the random private key and the pub key of Node B and the nonce:[ +[IP_Port of node C][a random public key]encrypted with the random private key and the pub key of Node C and the nonce:[ +[IP_Port of node D][data to send to Node D]]]] + +(sent from node A to node B): + +[uint8_t packet id (129)][nonce] +[a random public key]encrypted with the random private key and the pub key of Node B and the nonce:[ +[IP_Port of node C][a random public key]encrypted with the random private key and the pub key of Node C and the nonce:[ +[IP_Port of node D][data to send to Node D]]][nonce (for the following symmetric encryption)]encrypted with temp symmetric key of Node A: [IP_Port (of us)] + +(sent from node B to node C): + +[uint8_t packet id (130)][nonce] +[a random public key]encrypted with the random private key and the pub key of Node C and the nonce:[ +[IP_Port of node D][data to send to Node D]][nonce (for the following symmetric encryption)] +encrypted with temp symmetric key of Node B:[IP_Port (of Node A)[nonce (for the following symmetric encryption)] +encrypted with temp symmetric key of Node A: [IP_Port (of us)]] + +(sent from node C to node D): +[data to send to Node D][nonce (for the following symmetric encryption)]encrypted with temp symmetric key of Node C: +[IP_Port (of Node B)[nonce (for the following symmetric encryption)] +encrypted with temp symmetric key of Node B:[IP_Port (of Node A)[nonce (for the following symmetric encryption)] +encrypted with temp symmetric key of Node A: [IP_Port (of us)]]] + +Data sent to Node D: + +announce request packet: +[uint8_t packet id (131)][nonce][our real long term public key or a temporary one (see next)] +encrypted (with our real long term private key if we want to announce ourselves, a temporary one if we are searching for friends) and the pub key of Node D and the nonce: +[[(32 bytes) ping_id][client id we are searching for][public key that we want those sending back data packets to use.][data to send back in response(fixed size)]] + +(if the ping id is zero, respond with a announce response packet) +(If the ping id matches the one the node sent in the announce response and the public key matches the one being searched for, +add the part used to send data to our list (if the list is full make it replace the furthest entry)) + +data to route request packet: +[uint8_t packet id (133)][public key of destination node][nonce][temporary just generated public key] +encrypted with that temporary private key and the nonce and the public key from the announce response packet of the destination node:[data] +(if Node D contains the ret data for the node, it sends the stuff in this packet as a data to route response packet to the right node) + +The data in the previous packet is in format: [real public key of sender] +encrypted with real private key of the sender, the nonce in the data packet and +the real public key of the receiver:[[uint8_t id][data (optional)]] + +Data sent to us: +announce response packet: +[uint8_t packet id (132)][data to send back in response(fixed size)][nonce] +encrypted with the DHT private key of Node D, the public key in the request and the nonce:[[uint8_t is_stored] +[(32 bytes) ping_id if is_stored is 0, public key that must be used to send data packets if is_stored is not 0][Node_Format * (maximum of 8)]] +(if the is_stored is not 0, it means the information to reach the client id we are searching for is stored on this node) + +data to route response packet: +[uint8_t packet id (134)][nonce][temporary just generated public key] +encrypted with that temporary private key, the nonce and the public key from the announce response packet of the destination node:[data] + + +Onion packet (response): + +initial (sent from node D to node C): + +[uint8_t packet id (140)][nonce (for the following symmetric encryption)]encrypted with temp symmetric key of Node C: +[IP_Port (of Node B)[nonce (for the following symmetric encryption)] +encrypted with temp symmetric key of Node B:[IP_Port (of Node A)[nonce (for the following symmetric encryption)] +encrypted with temp symmetric key of Node A: [IP_Port (of us)]]][data to send back] + +(sent from node C to node B): + +[uint8_t packet id (141)][nonce (for the following symmetric encryption)] +encrypted with temp symmetric key of Node B:[IP_Port (of Node A)[nonce (for the following symmetric encryption)] +encrypted with temp symmetric key of Node A: [IP_Port (of us)]][data to send back] + +(sent from node B to node A): + +[uint8_t packet id (142)][nonce (for the following symmetric encryption)] +encrypted with temp symmetric key of Node A: [IP_Port (of us)][data to send back] + +(sent from node A to us): + +[data to send back] + + +Data packets: + +To tell our friend what our DHT public key is so that he can connect to us we send a data packet +with id 156 and the data being:[uint64_t (in network byte order) no_replay, the packet will only be +accepted if this number is bigger than the last one received] [our dht public key][Node_Format * ( +maximum of 8) nodes closest to us so that the friend can find us faster] diff --git a/protocols/Tox/toxcore/docs/TCP_Network.txt b/protocols/Tox/toxcore/docs/TCP_Network.txt new file mode 100644 index 0000000000..4bc2d433ac --- /dev/null +++ b/protocols/Tox/toxcore/docs/TCP_Network.txt @@ -0,0 +1,154 @@ +It has come to our attention that to achieve decent market penetration Tox +must work behind ALL internet connections, may they be behind enterprise NATs +or any other bad network conditions. + +The people who have issues with the UDP direct connection approach seem to be a +small minority though it is hard to estimate how many. + +This means that routing their packets using good nodes on the network will +probably not take a huge toll on the network and will assure that people +can use Tox regardless of the quality of their internet connection. + + +How it's going to work: +1. Alice, a Tox client on a TCP only network generates a temporary public key +and connects to a bootstrap node. + +2. Using the bootstrap node she finds and connects to a couple (exact number +to be determined later) number of random nodes that have TCP relay support. + +3. She uses the onion through the TCP relay connections to send friend requests +or tell online friends which TCP nodes she is connected to and her temporary +public key. + +4. Bob receives an onion packet from Alice telling him which nodes she is +connected to. Bob connects to these nodes and establishes a routed connection +with alice using that temporary public key. + +5. That connection is used by both to transmit encrypted Messenger and A/V +packets. + +6. If one of the nodes shuts down while it is currently routing traffic, Alice +and bob just switch to one of the other nodes they are both connected to. + + +Detailed implementation details: + +There are two distinct parts for TCP relays, the client part and the server +part. + +The server acts as the actual relay. Servers must have fully forwarded TCP +ports (NAT-PMP and uPNP can help here). The first port the server will try +binding to is 443 followed by port 3389 and possibly some others. Onion packets +can be sent/received through the TCP servers. + + +Server: + +The public/private key pair the TCP server uses is the same one he uses for the +DHT. + +all crypto for communication with the server uses the crypto_box() function of +NaCl. + +TCP doesn't have packets so what we will refer to as packets are sent this way: +[[uint16_t (length of data)][data]] + +So if you would inspect the TCP stream you would see: +[[uint16_t (length of data)][data]][[uint16_t (length of +data)][data]][[uint16_t (length of data)][data]] + +Note that both handshake packets don't have this format (the length for them is +always the same so we don't need to specify it.) + +When the client connects to the server, he sends this packet: +[public key of client (32 bytes)][nonce for the encrypted data [24 +bytes]][encrypted with the private key of the client and public key of the +server and the nonce:[public key (32 bytes) and][base nonce we want the server +to use to encrypt the packets sent to us (24 bytes)]] + +The server responds with: +[nonce for the encrypted data [24 bytes]][encrypted with the public key of the +client and private key of the server and the nonce:[public key (32 bytes) +and][base nonce we want the client to use to encrypt the packets sent to us (24 +bytes)]] + +All packets to the server are end to end encrypted with the information +received +(and sent) in the handshake. + +(first packet is encrypted with the base nonce the private key for which the +client sent the server the public key and the public key we sent to the client, +the next with base nonce + 1...) + +The connection is set to an unconfirmed state until a packet is received and +decrypted correctly using the information in the handshake. + +each packet sent to/from the server has an id (the first byte of the plain text +data of the packet.) + +ids 0 to 15 are reserved for special packets, ids 16 to 255 are used to denote +who we want the data to be routed to/who the packet is from. + +special ids and packets: +0 - Routing request. +[uint8_t id (0)][public key (32 bytes)] +1 - Routing request response. +[uint8_t id (1)][uint8_t (rpid) 0 if refused, packet id if accepted][public key +(32 bytes)] +2 - Connect notification: +[uint8_t id (2)][uint8_t (packet id of connection that got connected)] +3 - Disconnect notification: +[uint8_t id (3)][uint8_t (packet id of connection that got disconnected)] +4 - ping packet +[uint8_t id (4)][uint64_t ping_id (0 is invalid)] +5 - ping response (pong) +[uint8_t id (5)][uint64_t ping_id (0 is invalid)] +6 - OOB send +[uint8_t id (6)][destination public key (32 bytes)][data] +7 - OOB recv +[uint8_t id (7)][senders public key (32 bytes)][data] +8 - onion packet (same format as initial onion packet (See: Prevent +tracking.txt) but packet id is 8 instead of 128) +9 - onion packet response (same format as onion packet with id 142 but id is 9 +instead.) + +The rest of the special ids are reserved for possible future usage. + +If the server receives a routing request he stores server side that the client +wants to connect to the person with that public key and sends back a Routing +request response with the rpid along with the public key sent in the request. + +If for some reason the server must refuse the routing request (too many) he +sends the response with a rpid of 0. + +If the person who the client wants to connect to is also online and wants to +connect to the client a connect notification is sent to both with the +appropriate packet id. + +If either one disconnects, a disconnect notification is sent to the other with +appropriate packet id. + +If a client sends a disconnect notification, the entry on the server for that +routed connection is cleared and a disconnect notification is sent to the peer +(if he was online) + +If the server receives an onion packet he handles it the same as he would if it +was one received normally via UDP, he must also assure himself that any +responses must be sent to the proper client. + +Ping responses must have the same ping_id as the request. + +If the server receives a ping packet he must respond with a ping response. + +The server will send a ping packet to clients every 30 seconds, they have 30 +seconds to respond, if they don't the connection is deleted. + +OOB send packets will be sent to the peer connected to the TCP server with the +destination public key as a OOB recv packet. The client sending this packet has +no way of knowing if the packet reached its destination. + + +Client: + +Implementation details coming soon. diff --git a/protocols/Tox/toxcore/docs/TODO b/protocols/Tox/toxcore/docs/TODO new file mode 100644 index 0000000000..dea513b25d --- /dev/null +++ b/protocols/Tox/toxcore/docs/TODO @@ -0,0 +1,62 @@ +TODO list. + +[IN PROGRESS] Add what is left to do to the TODO list. + +Networking: + [NOT STARTED] UPnP port forwarding. + [NOT STARTED] NAT-PMP port forwarding. + +DHT: + [ALMOST DONE] Metadata collection prevention. (docs/Prevent_Tracking.txt) + [IN PROGRESS] Hardening against attacks. + [IN PROGRESS] Optimizing the code. + +Lossless UDP: + [DONE] Increase data send/receive rates. + +[IN PROGRESS] Massive IRC like group chats (text only) + [DONE] Networking base. + [NOT STARTED] Syncing chat state between clients (nicknames, list of who is in chat, etc...) + [NOT STARTED] Make clients sign their messages so that peers can't modify them. + +[IN PROGRESS] Audio/Video + [DONE] encoding/streaming/decoding + [DONE] Call initiation + [DONE] Encryption + [IN PROGRESS] Auditing. + [NEEDS TESTING] Video packet splitting. + [IN PROGRESS] Prevent audio skew (seems to be easily solvable client side.) + [IN PROGRESS] Group chats. + +Friend_requests.c: + [NOT STARTED] What happens when a friend request is received needs to be changed. + [NOT STARTED] Add multiple nospam functionality. + +[DONE] File transfers +[NOT STARTED] Offline messaging +[NOT STARTED] Friends list syncing +[IN PROGRESS] IPV6 support + [DONE] Networking + [DONE] DHT + Messenger + [NOT STARTED] Group chats (They work with IPv6 but some things need to be tested.) + + +[IN PROGRESS] Make toxcore thread safe. + +[NOT STARTED] Make the core save/datafile portable across client versions/different processor architectures. + +[MOSTLY DONE] A way for people to connect to people on Tox if they are behind a bad NAT that +blocks UDP (or is just unpunchable) (docs/TCP_Network.txt) + +[NEEDS TESTING] Make the save made with tox_save_encrypted(...) harder to brute force. +See: (https://github.com/jencka/ProjectTox-libtoxdata) + +[IN PROGRESS] GUI (no official one chosen yet, a list of promising ones follows) +https://github.com/notsecure/uTox +https://github.com/naxuroqa/Venom +https://github.com/Impyy/Toxy +https://github.com/lehitoskin/blight +https://github.com/nurupo/ProjectTox-Qt-GUI +https://github.com/Astonex/Antox + +[NOT STARTED] Security audit from professionals diff --git a/protocols/Tox/toxcore/docs/Tox_middle_level_network_protocol.txt b/protocols/Tox/toxcore/docs/Tox_middle_level_network_protocol.txt new file mode 100644 index 0000000000..a84132b6ca --- /dev/null +++ b/protocols/Tox/toxcore/docs/Tox_middle_level_network_protocol.txt @@ -0,0 +1,120 @@ +The TCP client and TCP server part are in a state that can be considered +feature complete. Why doesn't Tox support TCP yet even if those parts are +complete? + +The answer is that a way to ensure a smooth switchover between the TCP and UDP +needs to be added. If Tox first connects to the other user using TCP but then +due to pure chance manages to connect using the faster direct UDP connection +Tox must switch seamlessly from the TCP to the UDP connection without there +being any data loss or the other user going offline and then back online. The +transition must be seamless whatever both connected users are doing be it +transferring files or simply chatting together. + +Possible evil/bad or simply TCP relays going offline must not impact the +connection between both clients. + +Typically Tox will use more than one TCP relay to connect to other peers for +maximum connection stability which means there must be a way for Tox to take +advantage of multiple relays in a way that the user will never be aware if one +of them goes offline/tries to slow down the connection/decides to corrupt +packets/etc.. + +To accomplish this Tox needs something between the low level protocol (TCP) and +high level Tox messaging protocol hence the name middle level. + +The plan is to move some functionality from lossless_UDP to a higher level: +more specifically the functionality for detecting which packets a peer is +missing and the ability to request and send them again. lossless UDP uses plain +text packets to request missing packets from the other peer while Tox is +currently designed to kill the connection if any packet tampering is detected. +This works very well when connecting directly with someone because if the +attacker can modify packets it means he can kill your connection anyways. With +TCP relays however that is not the case as such the packets used to request +missing packets must be encrypted. If it is detected that a packet has been +tampered, the connection must stay intact while the evil relay must be +disconnected from and replaced with a good relay, the behavior must be the same +as if the relay had just suddenly gone online. Of course something to protect +from evil "friends" framing relays must also be implemented. + +Detailed implementation details: + +cookie request packet: +[uint8_t 24][Senders DHT Public key (32 bytes)][Random nonce (24 +bytes)][Encrypted message containing: [Senders real public key (32 +bytes)][padding (32 bytes)][uint64_t number (must be sent +back untouched in cookie response)]] +Encrypted message is encrypted with sender DHT private key, recievers DHT +public key and the nonce. + +cookie response packet: +[uint8_t 25][Random nonce (24 bytes)][Encrypted message containing: +[Cookie][uint64_t number (that was sent in the request)]] +Encrypted message is encrypted with sender DHT private key, recievers DHT +public key and the nonce. + +The Cookie should be basically: +[nonce][encrypted data:[uint64_t time][Senders real public key (32 +bytes)][Senders dht public key (32 bytes)]] + +Handshake packet: +[uint8_t 26][Cookie][nonce][Encrypted message containing: [random 24 bytes base +nonce][session public key of the peer (32 bytes)][sha512 hash of the entire +Cookie sitting outside the encrypted part][Other Cookie (used by the other to +respond to the handshake packet)]] + +The handshake packet is encrypted using the real private key of the sender, the +real private key of the receiver and the nonce. + + +Alice wants to connect to bob. + +Alice sends a cookie request packet to bob and gets a cookie response back. + +Alice then generates a nonce and a temporary public/private keypair. + +Alice then takes that nonce and just generated private key, the obtained +cookie, creates a new cookie and puts them in a handshake packet which she +sends to bob. + +Bob gets the handshake packet, accepts the connection request, then generates a +nonce and a temporary public/private keypair and sends a handshake packet back +with this just generated information and with the cookie field being the Other +Cookie contained in the received handshake. + +Both then use these temporary keys to generate the session key with which every +data packet sent and received will be encrypted and decrypted. The nonce sent +in the handshake will be used to encrypt the first data packet sent, the nonce ++ 1 the second, the nonce + 2 the third and so on. + +Data packets: + +[uint8_t 27][uint16_t (in network byte order) the last 2 bytes of the nonce +used to encrypt this][encrypted with the session key and a nonce:[plain data]] + +Plain data in the data packets: + +[uint32_t our recvbuffers buffer_start, (highest packet number handled + +1)][uint32_t packet number if lossless, our sendbuffer buffer_end if +lossy][data] + +data ids: +0: padding (skipped until we hit a non zero (data id) byte) +1: packet request packet (lossy packet) +2: connection kill packet (lossy packet) (tells the other that the connection is over) +... +16+: reserved for Messenger usage (lossless packets). +192+: reserved for Messenger usage (lossy packets). +255: reserved for Messenger usage (lossless packet) + +packet request packet: [uint8_t (1)][uint8_t num][uint8_t num][uint8_t +num]...[uint8_t num] + +the list of nums are a list of packet numbers the other is requesting. +to get the real packet numbers from this list take the recvbuffers buffer_start +from the packet, substract 1 to it and put it in packet_num then start from the +beggining of the num list: if num is zero, add 255 to packet_num then do the +next num. if num isn't zero, add its value to packet_num, note that the other +has requested we send this packet again to them then continue to the next num in +the list. + + diff --git a/protocols/Tox/toxcore/docs/av_api.md b/protocols/Tox/toxcore/docs/av_api.md new file mode 100644 index 0000000000..2f536ade8b --- /dev/null +++ b/protocols/Tox/toxcore/docs/av_api.md @@ -0,0 +1,194 @@ +#A/V API reference + +##Take toxmsi/phone.c as a reference + +###Initialization: + +``` +phone_t* initPhone(uint16_t _listen_port, uint16_t _send_port); +``` + +function initializes sample phone. _listen_port and _send_port are variables only meant +for local testing. You will not have to do anything regarding to that since +everything will be started within a mesenger. + + +Phone requires one msi session and two rtp sessions ( one for audio and one for +video ). + +``` +msi_session_t* msi_init_session( void* _core_handler, const uint8_t* _user_agent ); +``` + +initializes msi session. +Params: + +``` +void* _core_handler - pointer to an object handling networking, +const uint8_t* _user_agent - string describing phone client version. +``` + +Return value: +msi_session_t* - pointer to a newly created msi session handler. + +###msi_session_t reference: + +How to handle msi session: +Controlling is done via callbacks and action handlers. +First register callbacks for every state/action received and make sure +NOT TO PLACE SOMETHING LIKE LOOPS THAT TAKES A LOT OF TIME TO EXECUTE; every callback is being called +directly from event loop. You can find examples in phone.c. + +Register callbacks: +``` +void msi_register_callback_call_started ( MCALLBACK ); +void msi_register_callback_call_canceled ( MCALLBACK ); +void msi_register_callback_call_rejected ( MCALLBACK ); +void msi_register_callback_call_ended ( MCALLBACK ); + +void msi_register_callback_recv_invite ( MCALLBACK ); +void msi_register_callback_recv_ringing ( MCALLBACK ); +void msi_register_callback_recv_starting ( MCALLBACK ); +void msi_register_callback_recv_ending ( MCALLBACK ); +void msi_register_callback_recv_error ( MCALLBACK ); + +void msi_register_callback_requ_timeout ( MCALLBACK ); +``` + +MCALLBACK is defined as: void (*callback) (void* _arg) +msi_session_t* handler is being thrown as \_arg so you can use that and \_agent_handler to get to your own phone handler +directly from callback. + + +Actions: + +``` +int msi_invite ( msi_session_t* _session, call_type _call_type, uint32_t _timeoutms ); +``` + +Sends call invite. Before calling/sending invite msi_session_t::_friend_id is needed to be set or else +it will not work. _call_type is type of the call ( Audio/Video ) and _timeoutms is how long +will poll wait until request is terminated. + +``` +int msi_hangup ( msi_session_t* _session ); +``` +Hangs up active call + +``` +int msi_answer ( msi_session_t* _session, call_type _call_type ); +``` +Answer incomming call. _call_type set's callee call type. + +``` +int msi_cancel ( msi_session_t* _session ); +``` +Cancel current request. + +``` +int msi_reject ( msi_session_t* _session ); +``` +Reject incomming call. + + +###Now for rtp: + +You will need 2 sessions; one for audio one for video. +You start them with: +``` +rtp_session_t* rtp_init_session ( int _max_users, int _multi_session ); +``` + +Params: +``` +int _max_users - max users. -1 if undefined +int _multi_session - any positive number means uses multi session; -1 if not. +``` + +Return value: +``` +rtp_session_t* - pointer to a newly created rtp session handler. +``` + +###How to handle rtp session: +Take a look at +``` +void* phone_handle_media_transport_poll ( void* _hmtc_args_p ) in phone.c +``` +on example. Basically what you do is just receive a message via: +``` +struct rtp_msg_s* rtp_recv_msg ( rtp_session_t* _session ); +``` + +and then you use payload within the rtp_msg_s struct. Don't forget to deallocate it with: +void rtp_free_msg ( rtp_session_t* _session, struct rtp_msg_s* _msg ); +Receiving should be thread safe so don't worry about that. + +When you capture and encode a payload you want to send it ( obviously ). + +first create a new message with: +``` +struct rtp_msg_s* rtp_msg_new ( rtp_session_t* _session, const uint8_t* _data, uint32_t _length ); +``` + +and then send it with: +``` +int rtp_send_msg ( rtp_session_t* _session, struct rtp_msg_s* _msg, void* _core_handler ); +``` + +_core_handler is the same network handler as in msi_session_s struct. + + +##A/V initialization: +``` +int init_receive_audio(codec_state *cs); +int init_receive_video(codec_state *cs); +Initialises the A/V decoders. On failure it will print the reason and return 0. On success it will return 1. + +int init_send_audio(codec_state *cs); +int init_send_video(codec_state *cs); +Initialises the A/V encoders. On failure it will print the reason and return 0. On success it will return 1. +init_send_audio will also let the user select an input device. init_send_video will determine the webcam's output codec and initialise the appropriate decoder. + +int video_encoder_refresh(codec_state *cs, int bps); +Reinitialises the video encoder with a new bitrate. ffmpeg does not expose the needed VP8 feature to change the bitrate on the fly, so this serves as a workaround. +In the future, VP8 should be used directly and ffmpeg should be dropped from the dependencies. +The variable bps is the required bitrate in bits per second. +``` + + +###A/V encoding/decoding: +``` +void *encode_video_thread(void *arg); +``` +Spawns the video encoding thread. The argument should hold a pointer to a codec_state. +This function should only be called if video encoding is supported (when init_send_video returns 1). +Each video frame gets encoded into a packet, which is sent via RTP. Every 60 frames a new bidirectional interframe is encoded. +``` +void *encode_audio_thread(void *arg); +``` +Spawns the audio encoding thread. The argument should hold a pointer to a codec_state. +This function should only be called if audio encoding is supported (when init_send_audio returns 1). +Audio frames are read from the selected audio capture device during intitialisation. This audio capturing can be rerouted to a different device on the fly. +Each audio frame is encoded into a packet, and sent via RTP. All audio frames have the same amount of samples, which is defined in AV_codec.h. +``` +int video_decoder_refresh(codec_state *cs, int width, int height); +``` +Sets the SDL window dimensions and creates a pixel buffer with the requested size. It also creates a scaling context, which will be used to convert the input image format to YUV420P. + +``` +void *decode_video_thread(void *arg); +``` +Spawns a video decoding thread. The argument should hold a pointer to a codec_state. The codec_state is assumed to contain a successfully initialised video decoder. +This function reads video packets and feeds them to the video decoder. If the video frame's resolution has changed, video_decoder_refresh() is called. Afterwards, the frame is displayed on the SDL window. +``` +void *decode_audio_thread(void *arg); +``` +Spawns an audio decoding thread. The argument should hold a pointer to a codec_state. The codec_state is assumed to contain a successfully initialised audio decoder. +All received audio packets are pushed into a jitter buffer and are reordered. If there is a missing packet, or a packet has arrived too late, it is treated as a lost packet and the audio decoder is informed of the packet loss. The audio decoder will then try to reconstruct the lost packet, based on information from previous packets. +Audio is played on the default OpenAL output device. + + +If you have any more qustions/bug reports/feature request contact the following users on the irc channel #tox-dev on irc.freenode.net: +For RTP and MSI: mannol +For audio and video: Martijnvdc diff --git a/protocols/Tox/toxcore/docs/updates/Crypto.md b/protocols/Tox/toxcore/docs/updates/Crypto.md new file mode 100644 index 0000000000..cee03f0aa9 --- /dev/null +++ b/protocols/Tox/toxcore/docs/updates/Crypto.md @@ -0,0 +1,54 @@ +Encryption library used: http://nacl.cr.yp.to/ + + +When running the program for the first time the crypto_box_keypair() function is used to +generate the users public-private key pair. (32 bytes each) + +The generated public key is set as the client_id of the peer. + +Adding a friend +--------------- + +Alice adds bob to her friends list by adding his 32 byte public key (client_id) to her friends list. +2 cases: +case 1: Alice adds Bobs public key and bob waits for Alice to attempt to connect to him. +case 2: Bob and Alice add their respective public keys to their friends list at the same time. + +case 1: +Alice sends a onion data (see: Prevent_tracking.txt) packet to bob with the encrypted part containing the friends request like so: +``` +[char with a value of 32][nospam number (4 bytes)][Message] +``` + +Ex message: hello bob it's me alice -_- add me pl0x. + +For more info on the nospam see: Spam_Prevention.txt + +Bob receives the request and decrypts the message using the function crypto_box_open() + +If the message decrypts successfully: +If Alice is already in Bobs friends list: case 2 +If Alice is not in Bob's friends list and the nospam is good: Bob is prompt to add Alice and is shown the message from her. +If Bobs accepts Alice's friends request he adds her public key to his friends list. + +case 2: +Bob and Alice both have the others public key in their friends list, they are ready for the next step: Connecting to an already added friend + +In the next step only crypto_box() is used for encryption and only crypto_box_open() for decryption (just like in the last step.) + + +Connecting to an already added friend +------------------------------------- + +see: Tox_middle_level_network_protocol.txt + +Crypto request packets +-------------------------------------- + +``` +[char with a value of 32][Bob's (The reciever's) Public key (client_id) (32 bytes))][Alice's (The sender's) Public key (client_id) (32 bytes)][Random nonce (24 bytes)][Encrypted message] +``` + +The encrypted message is encrypted with crypto_box() (using Bobs public key, Alice's private key and the nonce (randomly generated 24 bytes)) and is a message from Alice in which she tells Bob who she is. + +Each node can route the request to the receiver if they are connected to him. This is to bypass bad NATs. diff --git a/protocols/Tox/toxcore/docs/updates/DHT.md b/protocols/Tox/toxcore/docs/updates/DHT.md new file mode 100644 index 0000000000..17db70cecd --- /dev/null +++ b/protocols/Tox/toxcore/docs/updates/DHT.md @@ -0,0 +1,105 @@ +DHT protocol +============ + +Follows pretty much the principle of the torrent DHT: http://www.bittorrent.org/beps/bep_0005.html (READ IT) + +But: +Vastly simplified packet format and encryption. + +Boostrapping: +The first time you install the client we bootstrap it with a node. (bandwidth should not be a problem as the client only needs to be sent one reply.) + + +Basics +------ +(All the numbers here are just guesses and are probably not optimal values) + +client list: A list of node ids closest (mathematically see bittorrent doc) to ours matched with ip addresses + port number corresponding to that id and a timestamp containing the time or time since the client was successfully pinged. + +"friends" list: A list containing the node_ids of all our "friends" or clients we want to connect to. +Also contains the ip addresses + port + node_ids + timestamp(of last ping like in the client list) of the 8 clients closest (mathematically see bittorrent doc) to each "friend" + +One pinged lists: +-One for storing a list of ips along with their ping_ids and a timestamp for the ping requests +Entries in the pinged lists expire after 5 seconds. +If one of the lists becomes full, the expire rate reduces itself one second or the new ping takes the place of the oldest one. + + +Entries in client list and "friends" list expire after 300 seconds without ping response. +Each client stores a maximum of 32 entries in its client list. +Each client in the client list and "friends" list is pinged every 60 seconds. +Each client in the client list and "friends" list has a timestamp which denote the last time it was successfully pinged. +If the corresponding clients timestamp is more than 130 seconds old it is considered bad. +Send a get nodes request every 20 seconds to a random good node for each "friend" in our "friends" list. +Send a get nodes request every 20 seconds to a random good node in the client list. + + +When a client receives any request from another +----------------------------------------------- +-Respond to the request + -Ping request is replied to with with a ping response containing the same encrypted data + -Get nodes request is replied with a send nodes reply containing the same encrypted data and the good nodes from the client list and/or the "friends" list that are closest to the requested_node_id + +-If the requesting client is not in the client list: + -If there are no bad clients in the list and the list is full: + -If the id of the other client is closer (mathematically see bittorrent doc) than at least one of the clients in the list or our "friends" list: + -Send a ping request to the client. + -if not forget about the client. + + -If there are bad clients and/or the list isn't full: + -Send a ping request to the client + +When a client receives a response +--------------------------------- +-Ping response + -If the node was previously pinged with a matching ping_id (check in the corresponding pinged list.) + -If the node is in the client list the matching client's timestamp is set to current time. + -If the node is in the "friends" list the matching client's timestamp is set to current time for every occurrence. + -If the node is not in the client list: + -If the list isn't full, add it to the list. + -If the list is full, the furthest away (mathematically see bittorrent doc) bad client is replaced by the new one. + -If the list is filled with good nodes replace the furthest client with it only if it is closer than the replaced node. + -for each friend in the "friends" list: + -If that friend's client list isn't full, add that client to it + -If that friend's client list contains bad clients, replace the furthest one with that client. + -If that friend's client list contains only good clients + -If the client is closer to the friend than one of the other clients, it replaces the farthest one + -If not, nothing happens. + + -Send nodes + -If the ping_id matches what we sent previously (check in the corresponding pinged list.): + -Each node in the response is pinged. + + + + + +Protocol +-------- + +Node format: +``` +[uint8_t family (2 == IPv4, 10 == IPv6, 130 == TCP IPv4, 138 == TCP IPv6)][ip (in network byte order), length=4 bytes if ipv4, 16 bytes if ipv6][port (in network byte order), length=2 bytes][char array (node_id), length=32 bytes] +``` +see also: DHT.h (pack_nodes() and unpack_nodes()) + +Valid queries and Responses: + +Ping(Request and response): +``` +[byte with value: 00 for request, 01 for response][char array (client node_id), length=32 bytes][random 24 byte nonce][Encrypted with the nonce and private key of the sender: [1 byte type (0 for request, 1 for response)][random 8 byte (ping_id)]] +``` +ping_id = a random integer, the response must contain the exact same number as the request + + +Get nodes (Request): +Packet contents: +``` +[byte with value: 02][char array (client node_id), length=32 bytes][random 24 byte nonce][Encrypted with the nonce and private key of the sender:[char array: requested_node_id (node_id of which we want the ip), length=32 bytes][Sendback data (must be sent back unmodified by in the response), length=1 to NODES_ENCRYPTED_MESSAGE_LENGTH bytes]] +``` +Valid replies: a send_nodes packet + +Send_nodes (response (for all addresses)): +``` +[byte with value: 04][char array (client node_id), length=32 bytes][random 24 byte nonce][Encrypted with the nonce and private key of the sender:[uint8_t number of nodes in this packet][Nodes in node format, length=?? * (number of nodes (maximum of 8 nodes)) bytes][Sendback data, length=1 to NODES_ENCRYPTED_MESSAGE_LENGTH bytes]] +``` diff --git a/protocols/Tox/toxcore/docs/updates/Spam-Prevention.md b/protocols/Tox/toxcore/docs/updates/Spam-Prevention.md new file mode 100644 index 0000000000..a0713fd1e6 --- /dev/null +++ b/protocols/Tox/toxcore/docs/updates/Spam-Prevention.md @@ -0,0 +1,12 @@ + +Situation 1: +Someone randomly goes around the DHT sending friend requests to everyone. + +Prevented by: +Every friend address: +[client_id (32 bytes)][nospam number (4 bytes)][checksum (2 bytes)] +contains a number (nospam). + +The nospam in every friend request to that friend must be that number. + +If not it is rejected. diff --git a/protocols/Tox/toxcore/docs/updates/Symmetric-NAT-Transversal.md b/protocols/Tox/toxcore/docs/updates/Symmetric-NAT-Transversal.md new file mode 100644 index 0000000000..490382160f --- /dev/null +++ b/protocols/Tox/toxcore/docs/updates/Symmetric-NAT-Transversal.md @@ -0,0 +1,43 @@ +Notes: + +Friend requests need to be routed. + +The current DHT should be capable of punching all NATs except symmetric ones. + +###### + +Symmetric NAT hole punching: + +If we are not connected to the friend and if the DHT is queried and ips +returned for the friend are the same but the port is different, the friend is +assumed to be behind a symmetric NAT. + +Before attempting the procedure we first send a routed ping request to the +friend. This request is to be routed through the nodes who returned the ip of +the peer. + +As soon as we receive one routed ping request from the other peer, we respond +with a ping response. + +Ping request/response packet: +See: Crypto request packets in [[Crypto]] + +Message: +For the ping request: +[char with a value of 254][char with 0][8 byte random number] + +For the ping response: +[char with a value of 254][char with 1][8 byte random number (The same that was sent in the request)] + +As soon as we get a proper ping response from the other we run the different +ports returned by the DHT through our port guessing algorithm. + +###### + +Port guessing algorithm: + +Right now it just tries all the ports directly beside the known ports.(A better one is needed) + +###### + +We send DHT ping requests to all the guessed ports, only a couple at a time. |