Metdata Capture
By Rick Aguirre
This week’s topic is Data Capture and more specifically, Metadata Capture. In the previous blog we discussed Raw Packet Capture, this week we will discuss the differences in Packet and Metadata capture and the best use cases for Metadata Capture.
Metadata Capture is used to receive summary data from the network, including but not limited to NetFlow, IPFIX, SNMP and Syslog. Metadata has been used for years to provide network monitoring tools the necessary information data for Performance Monitoring, Security, Compliance and Business Analytics.
Today, one of the primary forces behind its rise in popularity is the ability to do real-time streaming analysis on the network to identify a performance issue or security breach. Advancements in machine learning now provide promise in the ability to predict performance issues or security breaches.
Raw packet data is very valuable for viewing the payload and being able to identify the root cause of an issue. Unfortunately, there is a vast amount of information to process even in a small to medium sized network. For real time holistic visibility, Packet Capture is a very difficult task and can be very costly. Fortunately, there is Metadata information available that can provide the real-time insight into the network to detect performance issues or security breaches. For certain trouble spots in the network, it is still essential to have access to historical packet data to be able to drill down and examine incidents and determine the cause and severity.
First let’s run through a standard implementation to understand the sources of data and how it can be aggregated, filtered and correlated to provide real time information for analysis.
Typically the source will be Flow records, SNMP, and/or Syslog from routers in the network. In some cases, it will be necessary to create Flow records from an auxiliary device with access to raw packets in the network, either through a span port on the router or via a tap in the target link(s). This raw data is then processed by the NetFlow generation device which aggregates the session traffic and creates NetFlow records for each distinct Source IP and Destination IP address. These records are then exported to the Metadata Capture instance in customer defined intervals, typically between 30 seconds and 5 minutes. Other data such as SNMP and Syslog provide status of the network elements for Bandwidth Utilization and CPU/Memory Usage.
The Metadata Capture device should be able to collect, aggregate and correlate the different types of records at a high performance. It is essential in high performance network to examine, enrich and export to applications in Performance Monitoring, Security, Compliance and Business Analytics at a rate in excess of 500,000 records per second. The device should also be able to eliminate duplicate Flow records prior to sending to the various monitoring and analytics tools.
The strength of Metadata is that it provides a high-level view of all network activities and this view can be captured and stored in a time series database. All subsequent traffic and various metrics including CPU and port utilization can be compared to the historical data. If there is an anomaly in comparison with historical data, an associated rule will invoke an automated response. This response is based on a workflow and can include automated actions with the various network elements.
Metadata is an effective tool in providing a real time holistic view of the network. In conjunction with a strategy to access raw packets in trouble spots in the network, an enterprise can create an efficient, cost effective network monitoring solution.
Rick Aguirre is a veteran of the telecommunications industry. He has a successful record of developing start-up companies that have emerging, industry-changing technologies. As the founder of Cirries Technologies, he has led his team to develop the fastest data extraction and aggregation tools which deliver the right data at the right time for any application. Cirries’ products can digest data from multiple sources and reduce it to the right format for real-time notification, storage, or application use to reveal real-time performance and security of any network. Rick’s passion outside of work is youth sports. In addition to coaching his children’s teams, he has coached Lacrosse in under resourced communities and has served on the Board for the North Texas Chapter of Positive Coaching Alliance.