Site maintenance: Ilka Miloucheva

Flexible flow measurement technologies

• Traffic measurement tools (Cflow, Cisco's NetFlow, FlowScan, RRDTool / MRTG, SNMP (including security issues), IPFIX meter.
• Flow measurement by different approaches
• NetFlow [NetFlow] - fine-grained flows (source and destination IP and port, protocol, TOS byte) considering headers of all (or sampled) packets and groups of packets
• Cisco’s aggregated NetFlow combining fine-grained flow information into coarser aggregates (e.g. based on source prefix or port numbers)
• Large Flow Identification by Estan and Varghese [EV 02] dynamically identifying and reporting only the large “flows” (elephants) at a router
• Sampled Charging proposed by Duffield et al [DLT 01] reduces the size of NetFlow traffic reports while still allowing the flexibly computing per customer aggregates of traffic for billing purposes.
• Header Export by sFlow protocol [sFlow]
• Traffic Cluster: Automatically identifying and measuring high volume traffic clusters without a priori knowledge [ESV 02] where traffic cluster to consist of all traffic that matches a specified set of values for certain header fields.
  

Sampling

• [DLT 02a] propose a sampling scheme that optimally controls the volume of the measurements, and the variance of unbiased usage estimates, while retaining usage detail down to the finest level of granularity.
• Optimal sampling scheme based on identification of thresholds for flows described in [DLT 01] is applicable to the problem of usage-sensitive pricing.
• Trajectory Sampling [DG 00], a method to sample a packet at either all links or no links of network.
  

Traffic flow classification

• A packet selection scheme for weighting flow construction in routers towards longer flows has recently been proposed in [EV 01a], [EV 01b], [EV 02].
• Elephants and mice phenomenon is known as one of the few invariants of Internet traffic [ZDPS 01].
• [PTBTSD 02] defined and studied metrics based on the temporal behaviour of elephants.
• Traffic cluster [ESV 02 ] in the framework of Sensilla Project
  

Flow counting

• Storage and processing problem arises by counting large numbers of distinct header patterns (flows) seen on a high speed link in a given time interval. A family of bitmap algorithms solving the flow counting problem using extremely small amounts of memory is proposed by [ECF 02].
• [IDGM 01] have shown that monitoring high speed links is feasible with current technologies given that packet traces are recorded in a flow based format.
• Monitoring of large flows – “heavy hitters” - [EV 02] introduces a paradigm shift by concentrating on measuring only large flows --- those above some threshold such as 0.1\% of the link capacity.
  

Automated flow predictions for real time applications

• [DLT 02b] develop a simple model that predicts both the export rate of flow packet-sampled flow statistics and the number of active flows
  

Temporal flow analysis

• [DG 01], [DG 00], [DGG 02] propose a method that allows the direct inference of traffic flows through a domain by observing the trajectories of a subset of all packets traversing the network.

Bibliography

• [CD 00] J. Cleary, S. Donnelly, I. Graham, A. McGregor, M. Pearson, Design Principles for Accurate Passive measurement, Passive and Active Measurement Workshop, Hamilton (New Zealand), April 2000
• [CPB 89] K. Claffy, G. Polyzos, and H.-W. Braun, "Internet traffic flow profiling," Tech. Rep. TR-CS93-328, University of California San Diego, November 1989
• [CPB 93] K.C. Claffy, G.C. Polyzos and H.W. Braun, Application of Sampling Methodologies to Network Traffic Characterization, ACM Sigcomm 1993
• [CBP 97] K.C. Claffy, H.W. Braun, G.C. Polyzos, A parameterizable methodology for Internet traffic flow profiling, IEEE JSAC 1997
• [DAG] The DAG project, http://dag.cs.waikato.ac.nz.
• [DG 01] N. Duffield and M. Grossglauser, Trajectory Sampling for Direct Traffic Observation, IEEE/ACM Trans. on Networking, June 2001.
• [DG 00] N. Duffield and M. Grossglauser, Trajectory Sampling for Direct Traffic Observation, ACM SIGCOMM 2000, Stockholm, Sweden, September 2000.
• [DGG 02] Trajectory Engine: A Backend for Trajectory Sampling, N.G. Duffield, A. Gerber, M. Grossglauser, IEEE Network Operations and Management Symposium 2002, Florence, Italy, April 15-19, 2002.
• [DLT 01] Charging from sampled network usage, N.G. Duffield, C. Lund, M. Thorup, ACM SIGCOMM Internet Measurement Workshop 2001, San Francisco, CA, November 1-2, 2001
• [DLT 02a] N.G. Duffield, C. Lund, M. Thorup, Learn more, sample less: control of volume and variance in network measurement, http://www.research.att.com/~duffield/pubs/
• [DLT 02b] N.G. Duffield, C. Lund, M. Thorup, Properties and Prediction of Flow Statistics from Sampled Packet Streams, ACM SIGCOMM Internet Measurement Workshop 2002, Marseille, France, November 6-8, 2002
• [ESV 02] Cristian Estan, Stefan Savage, George Varghese, Automated Measurement of High Volume Traffic Clusters, Proceedings of the ACM/USENIX Internet Measurement Workshop (IMW), Marseille, France, November 2002 http://www.cs.ucsd.edu/~cestan/papers/hvclusters-shortabstract.pdf
• [EV 01a] C. Estan and G. Varghese. New Directions in Traffic Measure-ment and Accounting. ACM SIGCOMM Internet MeasurementWorkshop, August 2001
• [EV 01b] C. Estan and G. Varghese, New Directions in Traffic Measurement and Accountingl., ACM SIGCOMM Internet Measurement Workshop 2001, San Francisco, CA, November 1-2, 2001
• [EV 02] Cristian Estan and George Varghese. New directions in traffic measurement and accounting. In Proceedings of the ACM SIGCOMM, August 2002
• [ECF 02] Cristian Estan, George Varghese, Mike Fisk, Counting the number of active flows on a high speed link, May 2002-10-08
• [FD 01] C. Fraleigh, C. Diot, B. Lyles, S. Moon, P. Owezarski, D. Papa-giannaki, F. Tobagi, Design and Deployment of a Passive Moni-toring Infrastructure, Passive and Active Measurement Workshop, Amsterdam, April 2001
• [FM 85] P. Flajolet , G. Martin, Probabilistic counting algorithms for data base applications, Journal of Computer and System Sciences, 31(2) :182-209, Oct. 1985
• [FlowScan 00] D. Plonka, "Flowscan: A network traffic flow reporting and visualization tool," in Proceedings of the USENIX Fourteenth System Administration Conference LISA XIV, New Orleans, LA, December 2000.
• [IDGM 01] G. Iannaccone, C. Diot, I. Graham, N. McKeown. "Monitoring very high speed links". Proceedings of the ACM SIGCOMM Internet Measurement Workshop (IMW 2001). San Francisco. November 2001.
• [LCT 01] W.S. Lai, B. Christian, R.W. Tibbs, S. Van de Berghe, A Framework for Internet Traffic Engineering Measurement, draft-ietf-tewg-measure-02.txt, August 2001
• [Netflow] Cisco's IOS Netflow Feature, http://www.cisco.com/wrap/public/732/netflow.
• [Netflow] Cisco NetFlow. http://www.cisco.com /warp /public /732 /Tech /netflow
• [NeTraMet] http://www2.auckland.ac.nz/net//Accounting/ntm.Release.note.html
• [NETRAMET]Nevil Brownlee: "The Network Traffic Meter - NeTraMet version 4.3, http://www.auckland.ac.nz/net/Accounting/ntm.Release.note.html
• [RTFM-ARC] Nevil Brownlee, Cyndi Mills, Greg Ruth: "Traffic Flow Measurement: Architecture", "Traffic Flow Measurement: Meter MIB", "RTFM: Applicability Statement", RFC 2722/2720/2721, Network Working Group, October 1999.
• [sFlow] Peter Phaal, Sonia Panchen, and Neil McKee. RFC 3176: sFlow, September 2001
• [PTBTSD 02] K. Papagiannaki and N. Taft and S. Bhattacharyya and P. Thiran and K.Salamatian and C. Diot (2002).
• A Pragmatic Definition of Elephants in Internet Backbone Traffic.
• In: ACM SIGCOMM Internet Measurement Workshop. Marseilles, France. Nov 2002.
• [RCB 01] B. Ryu, D. Cheney, H. Braun, Internet Flow Characterization: Adaptive Timeout Strategy and Statistical Modeling, Passive and Active Measurement Workshop, Amsterdam, April 2001
• [ZDPS 01] On the Constancy of Internet Path Properties, Y. Zhang, N.G. Duffield, V. Paxson, S. Shenker, ACM SIGCOMM Internet Measurement Workshop 2001, San Francisco, CA, November 1-2, 2001