In a previous post, we introduced our network monitoring infrastructure goProbe/goQuery. On nearly 4000 hosts, the tool captures and stores metadata of packets flowing through global networks. But what good is all this data if you cannot make sense of it?

The true value of these records lies in what they tell you about your network in general and how its being used. To this end, we harness the capabilities of goQuery. They have greatly evolved with the latest iteration (v1.05 to v2.0), aiming to provide valuable insights "at a glance". This post will walk you through the most relevant steps and commands used in an in-depth traffic analysis.

For which interface do I have data?

goquery -list

      Iface    # of flows      Traffic                   From                  Until
  ---------    ----------    ---------    -------------------    -------------------
       eth0      276.64 k      1.92 GB    2016-02-04 10:11:18    2016-03-24 10:49:06
       eth1       14.49 k    263.64 MB    2016-02-04 10:11:18    2016-03-24 10:49:06
       eth2       14.49 k    263.64 MB    2016-02-04 10:11:18    2016-03-24 10:49:06
       eth3       14.49 k    353.02 MB    2016-02-04 10:11:18    2016-03-24 10:49:06
       eth4        0.00        0.00  B    2016-02-04 10:11:18    2016-03-24 10:49:06
       eth5       14.31 k    280.23 MB    2016-02-04 10:11:18    2016-03-24 10:49:06
   tun4_121       18.56 k     31.26 MB    2016-02-04 10:11:18    2016-03-07 15:28:38
   tun4_124       28.24 k     47.42 MB    2016-02-04 10:11:18    2016-03-24 10:49:06
   tun4_125       28.24 k     47.42 MB    2016-02-04 10:11:18    2016-03-24 10:49:06
   tun4_127       28.24 k     47.42 MB    2016-02-04 10:11:18    2016-03-24 10:49:06
   tun4_129       14.12 k     21.54 MB    2016-02-04 10:11:18    2016-03-24 10:49:06
   tun4_134        4.84 k      9.54 MB    2016-03-07 15:33:25    2016-03-24 10:49:06

      Total      456.67 k      3.25 GB

Find out on which interfaces goProbe collected metadata. The number of collected flows and the total traffic volume is indicated together with the data collection time frame.

Dive into the Data

Consider the following command:

goquery -i eth1 -f -1d -c 'snet=10.0.0.0/8' -n 5 sip,dip,dport,proto

Explanation:

  • -i eth1: I am interested in traffic that went over eth1...
  • -f -1d: ...in the last 24 hours (-f: first occurrence)...
  • -c "snet=...": ...and originated from my private IP range.
  • -n 5: Only show me the top 5 results...
  • sip,dip,dport,proto: ...and group them by source, destination IP, destination port and IP protocol.
goquery -i eth1 -f -1d -c 'snet=10.0.0.0/8' -n 5 sip,dip,dport,proto

                                     packets   packets            bytes     bytes
       sip       dip  dport  proto        in       out      %        in       out      %
  10.0.0.2  10.0.0.1      0    ESP   17.05 k   17.05 k  97.40   2.61 MB   2.62 MB  97.31
  10.0.0.1  10.0.0.2    500    ESP  439.00    440.00     2.51  68.88 kB  69.16 kB   2.51
  10.0.0.1  10.0.0.2    500    UDP   10.00     10.00     0.06   2.70 kB   3.46 kB   0.11
  10.0.0.2  10.0.0.1    500    UDP    6.00      6.00     0.03   2.00 kB   1.68 kB   0.07

                                     17.50 k   17.50 k          2.68 MB   2.69 MB

   Totals:                                     35.01 k                    5.37 MB

Timespan / Interface : [2016-03-23 11:12:57, 2016-03-24 11:14:06] / eth1
Sorted by            : accumulated data volume (sent and received)
Query stats          : 4.00   hits in 5ms

There's a lot going on here. We see that communication is exclusively restricted to two endpoints (10.0.0.{1,2}) and that the traffic completely comprises IPSec (VPN traffic): destination port 500 for key negotiation (IKE) and no destination port for the Encapsulating Security Payload (dport 0 and protocol ESP), e.g. the actual encrypted VPN traffic. The traffic totals also show that IKE only comprises a small portion of the exchanged packets.

You will notice that traffic volume/packets are now split into incoming and outgoing. This immediately points out whether the traffic was bi-directional or not. This in turn can help you to identify whether traffic was blocked by either your host or a particular endpoint.

Find the needle in the haystack

One of the most important parameters for targeted analyses is -c used for conditions on the data. goQuery supports a complete set of logical- and comparative-operators to allow you to drill down on the data to extract a very specific set of flows observed in the indicated time frame:

Base    Description            Other representations

COMPARATIVE
   =    equal to               eq, -eq, equals, ==, ===
  !=    not equal to           neq, -neq, ne, -ne
  <=    less or equal to       le, -le, leq, -leq
  >=    greater or equal to    ge, -ge, geq, -geq
   <    less than              less, l, -l, lt, -lt
   >    greater than           greater, g, -g, gt, -gt

LOGICAL
   !    unary negation         not
   &    and                    and, &&, *
   |    or                     or, ||, +

Precedence can be enforced by bracing condition chains appropriately, e.g.

! (( dport = 8080 | dport = 443 ) & proto = TCP )

This will negate the entire conditional and cause the dport conditions to be evaluated before the conjunction with proto.

Case Study

Disclaimer: for obvious reasons of confidentiality, all public IPs below were changed to randomly chosen ones.

During the analysis of a webserver in the DMZ of one of our customers, we noticed a peculiar traffic pattern. The picture shows how the webserver is connected to the internet and the customer's global WAN.

Customer Webserver

In this case study, the webserver's IP is 215.118.162.132. Two regional ISPs are connected on interfaces eth0 and eth2. The DMZ is located behind eth7.

When analyzing the traffic on the DMZ interface, we observed that the client producing the most traffic was 191.189.126.221. We wanted to find out how request and response were routed at the time, so the external interfaces were included in the analysis.

Thanks to added support for multi-interface queries and the finer separation of incoming and outgoing traffic all it took was "one query to rule them all":

goquery -i eth0,eth2,eth7 -f -5d -n 5 -c 'sip = 191.189.126.221 & dip = 215.118.162.132’ iface,sip,dip

                                          packets   packets             bytes      bytes
iface              sip              dip        in       out      %         in        out      %
 eth7  191.189.126.221  215.118.162.132  108.70 k    6.20 k  50.00  148.62 MB  413.83 kB  50.00
 eth2  191.189.126.221  215.118.162.132    0.00    108.70 k  47.30    0.00  B  148.62 MB  49.86
 eth0  191.189.126.221  215.118.162.132    6.20 k    0.00     2.70  413.83 kB    0.00  B   0.14

                                         114.90 k  114.90 k         149.03 MB  149.03 MB

  Totals:                                          229.80 k                    298.05 MB

Timespan / Interface : [2016-02-17 20:08:17, 2016-02-22 09:53:17] / eth0,eth2,eth7
Sorted by            : accumulated data volume (sent and received)
Query stats          : 3.00   hits in 290ms
Conditions:          : sip = 191.189.126.221 & dip = 215.118.162.132Add ‘iface’ column

The first row shows the bi-directional traffic for the DMZ. The other two rows show that requests came in via eth0 (ISP A) and responses were routed via eth2 (ISP B). There's a suitable explanation for this. The webserver's IP was announced via ISP A, while the default route of the customer host pointed to ISP B.

To gain a little more information about the connecting client, the -resolve flag was quite handy

goquery -i eth7 -f -5d -n 5 -c 'sip = 191.189.126.221’ -resolve sip,dip
                                                           sip              dip
                       221.126.189.191.dynamic.wless.ispa.tld.  215.118.162.132 ...

The reverse lookup by goQuery showed that this was a client from a residential WLAN network.

In summary, goQuery enabled us to understand complex routing decisions after the fact. Having access to the complete history of routing paths in your network proves to be a mighty foundation for traffic analyses and understanding the internet as a whole.

Start Analyzing

We just released a major version upgrade to both the goProbe and goQuery software. goProbe and goQuery are 100% open source. Go git them at open-ch/goProbe. We are looking forward to your contributions.

Significant improvements have been incorporated by Lorenz Breidenbach in the scope of an internship at Open Systems. Interested in contributing? Just drop us an email.

This system has been engineered in order to collect required information for a master's thesis in coorporation with the Distributed Computing Group at ETH Zurich. The resulting paper of Lennart Elsen, Fabian Kohn, Christian Decker and Roger Wattenhofer can be found here: goProbe: a Scalable Distributed Network Monitoring Solution. It is based on Lennart's thesis Optimized Distributed Network Traffic Monitoring in Global Networks.

Addendum: type goquery -help for a complete overview of the analytical features