MIMIC® Online Documentation

<< Previous Section    Next Section >>

MIMIC NETFLOW Protocol Module Guide

  1. Table of Contents

  2. Overview

    The MIMIC NETFLOW Protocol Module is an optional facility that simulates the de-facto standard Cisco NetFlow service as detailed in this Cisco whitepaper , in RFC 3954 and RFC 5101 over the UDP transport protocol. Currently, the SCTP Transport and DTLS Security are not implemented.

    Since it is completely configurable, MIMIC supports

    • IPFIX Biflows as specified in RFC 5103 via the 29305 Private Enterprise Number and the biflowDirection element.

    • IPFIX Testing as specified in RFC 5471 is mostly supported (currently 25 out of 32 tests). See this doc for details.

    • IPFIX Packet Sampling (PSAMP) as specified in RFC 5473 via the Common Properties Options Template Record and commonPropertiesID scoped element.

    • the information elements in the PSAMP Information Model specified in RFC 5477 numbered 301 through 311 and 313 through 338 are pre-defined in MIMIC.

    • Network Secure Event Logging (NSEL) for Adaptive Security Appliance (ASA)

    • Cisco Application Visibility and Control (AVC) NBAR, its fields, metrics and monitoring

    • Performance Routing (PfRv3) Reporting

    • IPFIX Information Elements for Logging NAT Events RFC 8158

    • Juniper JFlow
    Back to Top

  3. Installation

    NetFlow support is made available in MIMIC as an optional dynamically loadable module. Starting with MIMIC 11.00, you can use the Protocol Wizard to install the NETFLOW module. If you prefer to enable NETFLOW by hand, you need to do the following:

    • Use File->Terminate to stop the any running MIMIC daemon.

    • Copy the NETFLOW shared library (netflow.dll on Windows, netflow.so on Unix) from "bin/dynamic/optional" to "bin/dynamic" in the install directory.

    • Install the license keys as detailed in the instructions e-mailed to you.

    • Restart MIMIC. You should see the following type of message in the MIMICLog that confirms that the NETFLOW module was properly loaded : 
      INFO  - NETFLOW : Loaded protocol from < path-to-DLL >
INFO  - NETFLOW v11.00

    Once NETFLOW is loaded, any agent instance configured to support the NetFlow services will be able to send NetFlow data to a NetFlow collector.

    Back to Top

  4. Using NETFLOW from MIMICView

    If the NETFLOW module is enabled, then Agent->Add, Agent->Configure and Agent->Paste dialogs will display NETFLOW as an additional checkbox in the Advanced pane along with the SNMP protocols. On selecting the checkbox a new NETFLOW pane will appear.

    This NETFLOW configuration pane lets the user configure the parameters for a NETFLOW session:

    • Config file

      This mandatory parameter specifies the NETFLOW configuration file(s), each determining what NetFlow data is generated by the corresponding flow source. Multiple configuration files can be concatenated to produce flows from multiple flow sources at this IP address, eg. simulating multiple flow generators behind a NAT firewall. You will not be able to start the agent unless this parameter is set.

      The configuration file is detailed below. You can either edit configuration files directly, or use the NetFlow Wizard.

    • Collector

      This optional parameter specifies the address of the collector. Both IPv4 and IPv6 addresses are supported. If a IPv6 collector is specified, the agent needs to have at least one IPv6 alias configured. If there is not at least one collector defined, then no flows are exported.

    • Collector Port

      This optional parameter specifies the collector port to use. The default port is 9999.

    • Bundle Flowsets

      This optional boolean parameter lets you bundle template and data flowsets into packets. The default is "0", which means to send template and data flowsets in separate packets.

    Back to Top

  5. Using NETFLOW from MIMICShell

    A few new commands and some enhanced old commands can be used from the MIMICShell to control the NETFLOW functionality. Here is a synopsis:

    • mimic protocol msg NETFLOW get args

      This command lets the user gather the self-defining list of arguments required and their particulars. The parameters are detailed above. A sample exchange for this command would be: 

      mimicsh> mimic protocol msg NETFLOW get args
{{filename} {Config File} {file} {scripts/netflow {{*.cfg {Netflow config files}
{edit yes} {new yes}}} - both} {mandatory} {}}
{{collector} {Collector} {string} {} {mandatory} {}} {{collectorport} {Collector Port}
{string} {} {mandatory} {9999}}
{{bundleflowsets} {Bundle flowsets} {boolean} {0} {optional} {0}}

    • mimic agent get protocol

      This command lets the user look at the protocols currently configured on the agent. A sample exchange for this command would be: 

        mimicsh> mimic agent get protocol
  snmpv1,snmpv2c,NETFLOW

    • mimic agent set protocol

      This command lets the user change the protocol setting for an agent. A sample exchange for this command would be: 

        mimicsh> mimic agent get protocol
  snmpv1
  mimicsh> mimic agent set protocol snmpv1,NETFLOW
  mimicsh> mimic agent get protocol
  snmpv1,NETFLOW

    • mimic agent protocol msg NETFLOW get config

      This command lets the user get the current argument settings. A sample exchange for this command would be: 

        mimicsh> mimic agent protocol msg NETFLOW get config
  {filename=} {collector=} {collectorport=9999} {bundleflowsets=0}

    • mimic agent protocol msg NETFLOW set config [config]

      This command lets the user change the current argument settings of all NETFLOW sessions for an agent. A sample exchange for this command would be: 

        mimicsh> mimic agent protocol msg NETFLOW get config
  {filename=} {collector=} {collectorport=9999} {bundleflowsets=0}

  mimicsh> mimic agent protocol msg NETFLOW set config \{collector=192.9.200.71 192.9.200.72\}

  mimicsh>  mimic agent protocol msg NETFLOW get config
  {filename=} {collector=192.9.200.71 192.9.200.72} {collectorport=9999} {bundleflowsets=0}

  mimicsh> mimic agent protocol msg NETFLOW set config \{filename=file1.cfg file2.cfg\}

  mimicsh>  mimic agent protocol msg NETFLOW get config
  {filename=file1.cfg file2.cfg} {collector=192.9.200.71 192.9.200.72} {collectorport=9999}
  {bundleflowsets=0}

    • mimic agent protocol msg NETFLOW get trace
      mimic agent protocol msg NETFLOW set trace [0 or 1]

      This command lets the user change the NETFLOW tracing configuration for an agent. A sample exchange would be: 

        mimicsh> mimic agent assign 1

  mimicsh> mimic agent protocol msg NETFLOW get trace
  0
  mimicsh>  mimic agent protocol msg NETFLOW set trace 1

  mimicsh> mimic agent protocol msg NETFLOW get trace
  1
      and the log would show: 

      INFO  01/23.11:45:35 - agent 1 trace enabled for NETFLOW
INFO  01/23.11:45:35 - NETFLOW[AGT=1]: sent to [192.9.200.71,9999] - V9 bundled flowsets
INFO  01/23.11:45:35 - NETFLOW[AGT=1]: sent to [192.9.200.71,9999] - V9 bundled flowsets
...

    • mimic protocol msg NETFLOW get stats_hdr
      mimic agent protocol msg NETFLOW get statistics

      Returns NETFLOW statistics information:

      • a list of statistic headers, and
      • current statistics values for the specified server.

      In order, the statistic values are:

      • Total number of NETFLOW packets sent.
      • Total number of NETFLOW packets received.
      • Total number of NETFLOW packets discarded.
      • Total number of NETFLOW template flowsets sent.
      • Total number of NETFLOW data flowsets sent.
      • Total number of NETFLOW data flows sent.

      A sample exchange for these commands would be: 

        mimicsh> mimic protocol msg NETFLOW get stats_hdr
  {{pktSnt} {PktsSent}} {{pktRcvd} {PktsRcvd}} {{pktDisc} {PktsDiscarded}}
  {{tflowsetSnt} {TempFlowsetSent}} {{dflowsetSnt} {DataFlowsetSent}}
  {{dflowSnt} {DataFlowSent}} 

  mimicsh> mimic agent protocol msg NETFLOW get statistics
  190 0 0 18 172 1720

    • mimic agent protocol msg NETFLOW halt
      mimic agent protocol msg NETFLOW set filename ...
      mimic agent protocol msg NETFLOW set collector ...
      mimic agent protocol msg NETFLOW reload
      mimic agent protocol msg NETFLOW resume

      This group of commands lets the user reload the configuration file for an agent without stopping it. This in effect allows dynamic reconfiguration of the flows generated by this exporter. The flow generation needs to first be halted, then the reload command reloads the configuration file, and the resume command continues flow generation.

      The commands mimic agent protocol msg NETFLOW set allow to reconfigure certain attributes at runtime, such as the config file and/or collector(s).

    • mimic agent protocol msg NETFLOW flow list
      mimic agent protocol msg NETFLOW flow change ...

      You can change flow generation parameters at runtime with this group of commands for an agent without stopping it.

      The command mimic agent protocol msg NETFLOW flow list lists the flow generation parameters for a flow source config file, as shown in the filename configurable.

      You can change the following global flow generation parameters:

      • tfs_interval the template flowset interval in msec

      • dfs_interval the data flowset interval in msec

      A sample exchange for these commands would be: 

        mimicsh> mimic agent protocol msg NETFLOW flow list v9_simplest.cfg
  {flow source #0} {version 9} {count 1} {tfs_interval 10000}
  {dfs_interval 1000} {num_flowsets 1}
  {{{uid 0} {id 256} {tfs_length 88} {dfr_length 47} {dfs_length 476}
  {dfs_count 3} {dfs_remaining 0} {dfs_num_rec 10} {dfs_pad 2} {seq 0}
  {scope_count 0} {} {option_count 20}
  {{field #0} {id 21} {length 4} {v_type CONSTANT} {value 3488219376}
  {field #1} {id 22} {length 4} {v_type CONSTANT} {value 3488219284}
  {field #2} {id 1} {length 4} {v_type SEQ} {min 2003} {max 10000}
  ...

  mimicsh> mimic agent protocol msg NETFLOW flow change v9_simplest.cfg dfs_interval 2000

  mimicsh> mimicsh> mimic agent protocol msg NETFLOW flow list v9_simplest.cfg
  {flow source #0} {version 9} {count 4294967238} {tfs_interval 10000}
  {dfs_interval 2000} {num_flowsets 1} ...

      You can change the field parameters with the command
      mimic agent protocol msg NETFLOW flow change flowsrc-config-file flowset-uid field-num attr value
      for the attribute attr with value value for field field-num in flowset flowset-uid.

      For fields with RANGE or SEQ type simulation, the min and max attribute values can be changed. For fields with CONSTANT simulation, the min and max attributes are equivalent to value.

      For example, for the flows above, this command would yield 

        mimicsh> mimic agent protocol msg NETFLOW flow change v9_simplest.cfg 0 0 min 57

  mimicsh> mimic agent protocol msg NETFLOW flow list v9_simplest.cfg
  {flow source #0} {version 9} {count 1} {tfs_interval 10000}
  {dfs_interval 1000} {num_flowsets 1}
  {{{uid 0} {id 256} {tfs_length 88} {dfr_length 47} {dfs_length 476}
  {dfs_count 3} {dfs_remaining 0} {dfs_num_rec 10} {dfs_pad 2} {seq 0}
  {scope_count 0} {} {option_count 20}
  {{field #0} {id 21} {length 4} {v_type CONSTANT} {value 57}
  {field #1} {id 22} {length 4} {v_type CONSTANT} {value 3488219284}
  {field #2} {id 1} {length 4} {v_type SEQ} {min 2003} {max 10000}
  ...

    Back to Top

  6. Recording NetFlow

    7. To create a default simulation, you can record from a NetFlow packet capture (PCAP) with the netflowrec utility. This tool work in either of 2 modes:

    • in file mode, a previously captured PCAP file is read in

    • in collector mode, it captures live NetFlow packets from an Exporter and saves them in a temporary PCAP file

    It looks at NetFlow packets in the PCAP file, and for the first Exporter it finds, creates a simulation that attempts to cause the NetFlow module to generate equivalent flows. For each flow source (combination of UDP port and Source ID / Observation Domain ID) from the Exporter IP address a configuration file will generated for all the flows from that source.

    For better recording, here are the recommendations on how to capture Netflow packets with a packet capture program like Wireshark :

    • in order to capture large, fragmented packets, it is better to filter by exporter or collector IP address than by collector port. If one does the latter, then the filter will discard subsequent fragments in fragmented IP packets, leading to missed Netflow packets in the capture.

    By default, the recorder will attempt to detect client-server bi-directional flows, and will create a configuration with a limited number of flows to/from the clients to servers. Thus, to reproduce flows with more fidelity, it is better to feed a small number of flows at a time. The generated configurations can be loaded to produce the desired mix of flows in the simulation.

    The NetFlow Wizard gives you a user-friendly interface in front of this tool.

    This section documents the command-line options to this utility, either

    • --file pcap-file

      read NetFlow packets from the specified packet capture (PCAP) file.

    or

    • --localport port

      specifying this option causes netflowrec to act as a Collector, reading packets from the specified port.

    • --count count

      optional argument to specify how many packets to collect. If not specified, then the collector will capture packets until interrupted (eg. with CTL-C).

    and the common options

    • --out output-file

      this optional argument specifies the output file name. By default, the output file name will be the same as the PCAP file, but with the .cfg suffix.

    • --exporter host-address

      this optional argument specifies a host-address to filter the captured packets. Only packets from this exporter will be considered.

    • --collector host-address

      this optional argument specifies a host-address to filter the captured packets. Only packets from this collector will be considered.

    • --port port

      this optional argument specifies a port to filter the captured packets. Only packets to the specified collector port will be considered for simulation.

    • --version 5|9|10

      you can filter by NetFlow version number with this argument.

    • --start start
    • --stop stop

      with these 2 options you can specify a range of packets to be recorded.

    • --maxenums number

      this forces a maximum number of enumerations to record, the default being 128.

    The netflowrec tool has hardcoded information elements (fields) from IANA and uses the configuration file scripts/netflow/netflow-fields.cfg to reference vendor proprietary fields, and to override simulation behavior of internal fields.

    For example, a section like 

    field = {
  name = VENDOR_PROP
  vendor_type = 148
  alias =  NF_F_CONN_ID
  type = RANGE
}

    defines an information element number 148 with the name NF_F_CONN_ID and will simulate it as an integer with the range detected in the PCAP.

    The netflow-fields.cfg file can include other configuration files, eg. 

    includes = {
	include = fields/cisco-asa.cfg
}

    will define the Cisco ASA NetFlow Secure Event Logging (NSEL) fields defined by this Cisco page.

    Back to Top

  7. Simulation Configuration

    8. NetFlow data to be generated by the simulated exporter is specified by the configuration file(s) loaded into the agent instance. The NetFlow Wizard gives you a user-friendly interface to edit a NetFlow configuration file.

    Back to Top

    Common

    This section documents the field definition parameters common to all versions of the NetFlow simulation.

    For each field in the template, you can specify how to simulate instances of the field in successive flow records.
    Parameter Description
    Field name The name of the standard field, or a vendor proprietary field.
    length Number of bytes for this field.
    type The type of simulation:
    • CONSTANT - always return the specified value
    • RANGE - return a value in the range between min and max
    • SEQ - sequentially return a value from the specified range between min and max
    • ENUM - return an enumerated value
    • ACTION - return a value as computed by the specified action script
    • RANDOM - return a random value
    select How to select the next value if in a range or enumerated list. If specified as SEQ, then values are traversed sequentially either in the range or enumerated list, rather than randomly, which is the default.
    reverse Simulate bidirectional flow records. If this parameter is specified for any field in the record, then whenever a record is generated containing this field, a second record is automatically generated in the reverse direction. Both fields must be defined in the template.

    If the reverse parameter specifies a field name, then the reverse flow will reverse the values of both fields, maintaining the value of the other fields.

    If the reverse parameter specifies a percentage, then the reverse flow will contain a value with a multiple of the original flow, thus producing asymmetric flows.

    Back to Top

    Version 5

    This section documents the version 5 simulation configuration parameters.

    • Global parameters

      These apply to all the flowsets defined in this configuration file.
      Configurable Description
      Comments User editable comment about the configuration. This value does not impact the simulation and is solely intended for self-documentation.
      version NetFlow version, for version 5 see this Cisco whitepaper
      sysUpTime SysUptime of the first generated flowset. Is updated in real-time after that.
      packet_count
      unix_secs Time in seconds since 0000 UTC 1970, at which the Export Packet leaves the Exporter.
      flow_sequence Starting sequence number, incremented thereafter.
      interval Interval in milliseconds between data flowsets.
      sim_count If non-zero, specifies the number of iterations the simulation in this config file is run, else indefinitely.
      num_rec Number of flow records per flowset.

    Back to Top

    Version 9

    This section documents the version 9 simulation configuration parameters.

    • Global parameters

      These apply to all the flowsets defined in this configuration file.
      Configurable Description
      Comments User editable comment about the configuration. This value does not impact the simulation and is solely intended for self-documentation.
      version NetFlow version, for version 9 see RFC 3954
      sysUpTime SysUptime of the first generated flowset. Is updated in real-time after that.
      UNIXSecs Time in seconds since 0000 UTC 1970, at which the Export Packet leaves the Exporter. A value of 0 sets the current time.
      SequenceNumber Starting sequence number, incremented thereafter.
      SourceID A 32-bit value that identifies the Exporter Observation Domain. One configuration can generate flows from one SourceID.
      tfs_interval Interval in milliseconds between template flowsets.
      dfs_interval Interval in milliseconds between data flowsets.
      sim_count If non-zero, specifies the number of iterations the simulation in this config file is run, else indefinitely.

    • Flowset configurables

      These configurables apply to each flowset.
      Parameter Description
      id Template id for this flowset. Unique for this configuration file.
      dfs_count Number of flowsets for this flowset per iteration. If bundled, they will be in the same packet.
      dfs_num_rec Number of flow records per data flowset.

    • Flowset definitions

      This section defines the flow templates and how to simulate the fields.
      Parameter Description
      id Template id for this flowset. Unique for this configuration file.
      scope_count Number of scopes for this options template.
      field_count Number of fields in the flow.

    • The field definitions are detailed in the common section above.
    Back to Top

    Version 10

    This section documents the version 10 simulation configuration parameters.

    • Global parameters

      These apply to all the flowsets defined in this configuration file.
      Configurable Description
      Comments User editable comment about the configuration. This value does not impact the simulation and is solely intended for self-documentation.
      version NetFlow version, for version 10 see RFC 5101
      ExportTime Time in seconds since 0000 UTC 1970, at which the Export Packet leaves the Exporter. A value of 0 sets the current time.
      SequenceNumber Starting sequence number, incremented thereafter.
      ObservationDomainID A 32-bit value that identifies the Exporter Observation Domain. One configuration can generate flows from one ObservationDomainID.
      tfs_interval Interval in milliseconds between template flowsets.
      dfs_interval Interval in milliseconds between data flowsets.
      sim_count If non-zero, specifies the number of iterations the simulation in this config file is run, else indefinitely.

    • The Flowset configurables and definitions are as in version 9 above, the field definitions are in the common section above.
    Back to Top

  8. Compatibility

    9. Click here for the compatibility document. If you get an error, you need to download the optional update package with the Update Wizard.

    Back to Top
    << Previous Section    Next Section >>