It was an interesting week at SharkFest Europe 2017 this month. The Annual Sharkfest conference ran from 7th-10th November at the rather comfortable Palacio Estoril in Estoril, Portugal. Endace was there and our CTO, Dr. Stephen Donnelly, presented a session on packet capture meta-data.
This was the second Wireshark Europe event and was very well attended, attracting attendees from more than 30 countries. Congratulations to Janice and the team for an excellent event – and we look forward to hearing more about the inaugural Wireshark Asia in due course.
Stephen’s presentation, ‘Augmenting Packet Capture with Contextual Meta-Data: the What, Why & How’, was well received by the audience.
For those who couldn’t make SharkFest, here is a video of the presentation (if you’d like a copy of the full presentation please let us know)
Stephen outlined the importance of retaining context for packet capture files by pointing out that the oft-use line “Packets Don’t Lie” isn’t true if:
- You don’t know where they came from
- You don’t know if there was packet loss
- You don’t know if they’ve been filtered
- You don’t know if the time stamps are right
This becomes even important in environments where packet capture is happening in multiple places across a distributed network. Understanding where the packets came from, and what the state of the environment was like at the time, is crucial if you are to draw solid conclusions from examining the packet trace file.
The role of metadata, Stephen argues, is to provide this context. He went on to talk about some of the different types of packet capture metadata and what it can be useful for, outlining three main categories of metadata:
- Static metadata: data about things that do not change over time, such as the host name of the system that captured the packets, the speed of the link and so on.
- Dynamic metadata: data about environmental conditions that change over time – such as optical power levels or timing accuracy.
- Post-capture metadata: data such as user comments, flow information, statistics and annotations from analytics applications that process the captured packet data.
Stephen took a deep dive into three common formats for packet trace files – pcap, pcagng (now the default format in Wireshark) and Provenance™ and approach to writing metadata used in Endace’s Extensible Record Format (ERF) (which is also compatible with Wireshark). The presentation looked at what each offers in terms of recording packet capture metadata and how they go about associating it with packet trace files.
Provenance uses a different approach to writing metadata into packet capture files from either pcap or pcap ng. Provenace is designed to be able to record changing (dynamic data) that may change during the course of a packet capture. It works by writing a Provenance record into the ERF capture file once every second, as the diagram below shows.
One of the use cases for this is recording the accuracy of time stamping information over the course of a packet capture of high-frequency trade data. Under new MiFID 2 regulations which come into force in 2018, traders must record every trade and be able to demonstrate that the recorded trade data is timestamped accurately to a time-source that is synchronized to UTC with a maximum divergence of less than 100 microseconds. Provenance provides an easy way for them to record compliance with this regulatory obligation.
If you have an interesting use case for packet capture metadata (particularly post-capture metadata use cases), we’d love to hear more. Let us know. We see this as a fascinating area for further development.
SharkFest was an excellent opportunity for the Endace team to meet like-minded members of the Wireshark global community, including the original creator of the Wireshark Core Developers, Gerald Combs, and to share knowledge of the best practices in packet analysis.
We’re looking forward to seeing how SharkFest continues to grow in scale and influence, with three SharkFest events taking place in 2018, including the first-ever SharkFest Asia in Singapore.