Josep Sanjuas’ blog

As some of you may already know, one of my jobs is to maintain and develop for the CoMo network monitoring system. CoMo supports many input sniffers, such as bpf for FreeBSD systems, netflow, sflow, data from the network monitoring specialized DAG cards, and most notably libpcap.

In CoMo, every packet has a reception timestamp. How packets are stamped depends on the actual sniffer that receives the packet. For example, DAG cards have their own clocks to stamp each packet. In libpcap, it is the kernel (in my case, the linux kernel) that marks the packets according to the local time.

While developing for the upcoming CoMo 2.0 release, I added strict checks that packet stamps for each packet should monitonically increase - that is, no packet timestamp can be lower than the previous. I expected this to be a reasonable constraint to impose on sniffers. I was wrong: libpcap did in fact feed CoMo with what I believed to be unordered packets.

In order to confirm this issue, I wrote the following program:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

#include <pcap.h>
#include <err.h>
 
static void processpkt(u_char * data,
    const struct pcap_pkthdr * h,
    const u_char * buf)
{
    static double last_ts = 0;
    static int pkt_count = 0;
    double ts = h->ts.tv_sec +
        ((double) h->ts.tv_usec) / 1000000;
    if (last_ts == 0)
        last_ts = ts;
    if (ts < last_ts)
        printf("pkt #%d: ts decreased! %f to %f\n",
            pkt_count, last_ts, ts);
    pkt_count++;
    last_ts = ts;
}
 
int main(int argc, char **argv)
{
    char errbuf[1024];
    struct bpf_program prog;
    pcap_t *pcap = pcap_open_live(argv[1], 2048, 1, 0, errbuf);
    if (pcap == NULL || argv[1] == NULL)
        errx(1, "error opening interface %s", argv[1]);
    if (argv[2]) { /* we have a filter expression */
        int i = pcap_compile(pcap, &prog, argv[2], 0, 24);
    if (i < 0)
            errx(1, "error compiling filter expression "
                "`%s': %s\n", argv[2], pcap_geterr(pcap));
    i = pcap_setfilter(pcap, &prog);
        if (i < 0)
            errx(1, "error applying filter expression: %s\n",
                        pcap_geterr(pcap));
    }
    for (;;) {
        pcap_dispatch(pcap, 100, processpkt, NULL);
    }
}

The program receives two input parameters: the network interface to read from, and optionally the filter to apply to packets. For each packet that reaches the processpkt() function, it will check that packet timestamps increase, and output a warning if this is not the case.

Running this code yields the following results in our development box:

$gcc check_libpcap_timestamps.c -o check_libpcap_timestamps -lpcap
$sudo ./check_libpcap_timestamps eth0
pkt #23: ts decreased! 1184677973.006962 --> 1184677973.006952
pkt #28: ts decreased! 1184677973.007341 --> 1184677973.007328
pkt #38: ts decreased! 1184677973.007968 --> 1184677973.007952
pkt #43: ts decreased! 1184677973.008328 --> 1184677973.008322
pkt #48: ts decreased! 1184677973.008588 --> 1184677973.008577
pkt #53: ts decreased! 1184677973.008846 --> 1184677973.008830
(...)


Which confirms that packets returned by libpcap have non-increasing timestamps. Weird! Some joint research & problem debugging with Diego Amores revealed the following interesting threads, and especially this one that includes a post from Guy Harris to the tcpdump-workers mailing list which explains how libpcap gets the timestamps:

[Outgoing packets’ timestamps are] assigned whenever the packet is supplied, by the networking stack, to whatever piece of code time-stamps the packet; that’s before the first bit is even put onto the network.
(…)
[Incoming packets’ timestamps are] assigned whenever the packet is supplied, by the device driver or networking stack, to whatever piece of code time-stamps the packet; that’s after the lat bit is received.
In other words, the time stamps aren’t extremely precise measurements of when the first, or last, bit of the packet was put onto the network; the imprecision includes time spent passing the packet through the networking code and the driver code, as well as, possibly, through the networking card. It might also, for incoming packets, include interrupt latency.

The idea that incoming and outgoing packets are stamped this way made us think that maybe incoming packets have increasing stamps, and outgoing packets too, but when buffered together and returned by libpcap, there is no garantee that the timestamps increase. We confirmed this by doing the following:

$sudo ./check_libpcap_timestamps eth1 "src 192.168.0.1"
(no warnings, ctrl+c'd)
$sudo ./check_libpcap_timestamps eth1 "dst 192.168.0.1"
(no warnings, ctrl+c'd)
$sudo ./check_libpcap_timestamps eth1
(tons of warnings)

If we ask libpcap to process only either incoming or outgoing packets, timestamps are always increasing. As a conclusion, this has no impact on a passive network monitoring system that uses libpcap to capture packets, because a network interface will be used only to receive packets. However, in the case of CoMo, users may want to monitor active NICs, so I ended up relaxing the restriction that packet timestamps should increase, and merely issue a warning.