What is Traceroute?
It is an application layer implementation to find the hops when a packet traverses to a destination.
What are the Protocols Used in Traceroute?
Traceroute works with combination of both ICMP and UDP. It mainly relies on ICMP Time-to-Live Exceeded (Type 11)
How it Works?
When you execute a traceroute command (ie traceroute www.yahoo.com), your machine sends out 3 UDP packets with a TTL (Time-to-Live) of 1. When those packets reach the next hop router, it will decrease the TTL to 0 and thus reject the packet. It will send an ICMP Time-to-Live Exceeded (Type 11), TTL equal 0 during transit (Code 0) back to your machine - with a source address of itself, therefore you now know the address of the first router in the path.
Next your machine will send 3 UDP packets with a TTL of 2, thus the first router that you already know passes the packets on to the next router after reducing the TTL by 1 to 1. The next router decreases the TTL to 0, thus rejecting the packet and sending the same ICMP Time-to-Live Exceeded with its address as the source back to your machine. Thus you now know the first 2 routers in the path.
This keeps going until you reach the destination. Since you are sending UDP packets with the destination address of the host you are concerned with, once it gets to the destination the UDP packet is wanting to connect to the port that you have sent as the destination port, since it is an uncommon port, it will most like be rejected with an ICMP Destination Unreachable (Type 3), Port Unreachable (Code 3). This ICMP message is sent back to your machine, which will understand this as being the last hop, therefore traceroute will exit, giving you the hops between you and the destination.
The UDP packet is sent on a high port, destined to another high port. On a Linux box, these ports were not the same, although usually in the 33000. The source port stayed the same throughout the session, however the destination port was increase by one for each packet sent out.
One note, traceroute actually sends 1 UDP packet of TTL, waits for the return ICMP message, sends the second UDP packet, waits, sends the third, waits, etc, etc, etc.
If during the session, you receive * * *, this could mean that that router in the path does not return ICMP messages, it returns messages with a TTL too small to reach your machine or a router with buggy software. After a * * * within the path, traceroute will still increment the TTL by 1, thus still continuing on in the path determination.
But what about the non unix implementation, ahhh.. Nothing but it uses fully ICMP messages for trace.. Instead of UDP it will use ICMP. One more twist, since it is UDP there is no port funda.. So the last hop will give a ICMP Reply.. Hope everthing is clear now..