Export of Delay Performance Metrics in IP Flow Information eXport (IPFIX)

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 7 May 2025.¶

Copyright Notice

Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

Network operators usually gather and maintain some forms of statistical delay view of their networks (or segments of their networks). That view is meant to help with understanding where in the network, for which customer traffic or services, how much, and why abnormal delay is being accumulated. To that aim, delay-related data need to be reported from devices covering both data and control planes. In order to understand which customer traffic is affected, delay-related data need to be reported in the context of the customer data-plane context. That enables network operators to quickly identify when the control-plane updates the current path with a different set of intermediate hops (that is, a change of the forwarding path) and interfaces, how the path delay changes for which customer traffic.¶

With On-Path Telemetry, described in the Network Telemetry Framework [RFC9232] and applied in In Situ Operations, Administration, and Maintenance (IOAM) Deployment [RFC9378] and Alternate Marking Deployment Framework [I-D.ietf-ippm-alt-mark-deployment], the path delay between two endpoints is measured by inserting a timestamp in the packet.¶

At least two modes of On-Path Telemetry can be distinguished. Passport mode, where only the last hop in the forwarding path of the On-Path Telemetry domain exposes all the metrics, and postcard mode, where the metrics are also exposed in transit nodes. In both modes the forwarding path exposes performance metrics allowing to determine how much delay has been accumulated on which hop. The proposal in this document makes more sense for the postcard mode.¶

In order to export the delay-related metrics via IPIFX [RFC7011], this document defines four new IPFIX Information Elements (IEs), exposing the On-Path delay on OAM transit and decapsulating nodes, following the postcard mode principles. Since these IPFIX IEs are performance metrics [RFC8911], they must be registered in the "IANA Performance Metric Registry [IANA-PERF-METRIC].¶

Following the guidelines for Registered Performance Metric Requesters and Reviewers [RFC8911], the different characteristics of the performance metrics (Identifier, Name, URI, Status, Requester, Revision, Revision Date, Description, etc.) must be clearly specified in the "IANA Performance Metric Registry [IANA-PERF-METRIC] in order for the measurement results using the Performance Metrics to be comparable even if they are performed using different implementations and in different networks. The first performance metric characteristic is the selection of a meaningful name, following the "MetricType_Method_SubTypeMethod_... Spec_Units_Output" naming convention (See Section 7.1.2 of [RFC8911]).¶

+------------------------------------+-------------------------------+
|      Performance Metric            |  IPFIX Information Element    |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I       |pathDelayMeanDeltaMicroseconds |
|P_RFC[RFC-to-be]_Seconds_Mean (TBD1)|(TBD5)                         |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I       |pathDelayMinDeltaMicroseconds  |
|P_RFC[RFC-to-be]_Seconds_Min (TBD2) |(TBD6)                         |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I       |pathDelayMaxDeltaMicroseconds  |
|P_RFC[RFC-to-be]_Seconds_Max (TBD3) |(TBD7)                         |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I       |pathDelaySumDeltaMicroseconds  |
|P_RFC[RFC-to-be]_Seconds_Sum (TBD4) |(TBD8)                         |
+------------------------------------+-------------------------------+

Table 1: Mapping Between IPFIX IEs and Performance Metrics

Assuming time synchronization on devices, the delay is measured by calculating the difference between the timestamp imposed with On-Path Telemetry in the packet at the OAM encapsulating node and the timestamp exported in the IPFIX flow record from the OAM transit and decapsulating nodes. The lowest, highest, mean, and/or the sum of measured path delay can be exported, thanks to the different IPFIX IE specifications.¶

                       On-Path Telemetry Domain
              .........................................
              .                                       .
              .    D1                                 .
              . x------>                              .
              .                                       .
              .          D2                           .
              . x-------------------->                .
              .                                       .
              .                  D3                   .
              . x-----------------------------------> .
              .                                       .
(H1) ------ (R1) ------- (R2) ------- (R3) -------- (R4) ------ (H2)
Host 1  Encapsulating   Transit      Transit   Decapsulating  Host 2
            Node         Node 1       Node 2        Node
              .                                       .
              .                                       .
              .........................................

In the use case shown in Figure 1 using On-path Telemetry to export the delay metrics, the node R2 exports the delay D1, the node R3 exports the delay D2 and the decapsulating node R4 exports the total delay D3 using IPFIX.¶

The advantage of this solution is that the delay metrics (min, max, and mean) can be computed on the router, and aggregated directly within the Flow Record, saving export bandwidth and computation on the Collector. For the computation of the min, max, and mean delay metric to be computed locally on the router, the exporter Metering Process requires some local caching/processing computation (for each new packets in the flow), specifically the mean value. A less computational heavy solution for the router is the export of the delay sum instead of the delay mean; on the Collector, the delay mean can easily be computed by a single division operation (using the packet count). The alternative, with no delay monitoring on the router, requires the export of every single packet as a separate Flow Record, including the timestamps information, as described in [I-D.ietf-opsawg-ipfix-alt-mark] for Alternate Marking, for the Collector to compute delay metrics (min, max, and mean), before recomputing the aggregated Flow Record.¶

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

This document makes use of the terms defined in [RFC7011] and [RFC9378].¶

The following terms are used as defined in [RFC7011]:¶

• IPFIX¶

• IPFIX Information Elements (IEs)¶

• Flow¶

• Flow Record¶

• Exporter¶

The following terms are used as defined in [RFC8911]:¶

• Performance Metric¶

• Registered Performance Metric¶

• Performance Metrics Registry¶

The following terms are used as defined in Section 5 of [I-D.ietf-opsawg-oam-characterization]:¶

• Encapsulating node¶

• Transit node¶

• Decapsulating node¶

The following terms are used as defined in Section 3.8 of [RFC7799]:¶

• Hybrid Type I Passive¶

3. Performance Metrics

This section defines the new performance metrics following the template defined in Section 11 of [RFC8911].¶

IANA Note (to be removed): RFC 8192 Section 4 was taken a guiding example.¶

 Max = (FiniteDelay[j])
 such that for some index, j, where 1 <= j <= N
 FiniteDelay[j] >= FiniteDelay[n] for all n

4. IPFIX Information Elements

This section specifies the following new IPFIX IEs:¶

pathDelayMeanDeltaMicroseconds

32-bit unsigned integer that identifies the mean path delay of all packets in the Flow, in microseconds, between the OAM encapsulating node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulating node).¶

pathDelayMinDeltaMicroseconds

32-bit unsigned integer that identifies the lowest path delay of all packets in the Flow, in microseconds, between the OAM encapsulating node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulating node).¶

pathDelayMaxDeltaMicroseconds

32-bit unsigned integer that identifies the highest path delay of all packets in the Flow, in microseconds, between the OAM encapsulating node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulating node).¶

pathDelaySumDeltaMicroseconds

64-bit unsigned integer that identifies the sum of the path delay of all packets in the Flow, in microseconds, between the OAM encapsulating node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulating node).¶

5. Use Cases

The measured On-Path delay can be aggregated with Flow Aggregation as defined in [RFC7015] to the following device and control-plane dimensions to determine:¶

• With node id and egressInterface(14), on which node which logical egress interfaces have been contributing to how much delay.¶

• With node id and egressPhysicalInterface(253), on which node which physical egress interfaces have been contributing to how much delay.¶

• With ipNextHopIPv4Address(15) or ipNextHopIPv6Address(62), the forwarding path to which next-hop IP contributed to how much delay.¶

• With mplsTopLabelIPv4Address(47) or destinationIPv6Address and srhActiveSegmentIPv6(495), the forwarding path to which MPLS top label IPv4 address or IPv6 destination address and SRv6 active segment contributed to how much delay.¶

• BGP communities [RFC1997] are often used for setting a path priority or service selection. With bgpDestinationExtendedCommunityList(488) or bgpDestinationCommunityList(485) or bgpDestinationLargeCommunityList(491) which group of prefixes accumulated at which node how much delay.¶

• With destinationIPv4Address(13), destinationTransportPort(11), protocolIdentifier (4) and sourceIPv4Address(8), or equivalent IPFIX IEs for IPv6, the forwarding path delay on each node from each IPv4 source address to a specific application in the network.¶

Let us consider the example depicted in Figure 1 from Section 1 as topology example. Below example table shows the aggregated delay per each node, ingressInterface,(10) egressInterface(14), destinationIPv6Address(28) and srhActiveSegmentIPv6(495).¶

     +-----------+-----------+------+-------------+-------------+------------+
     | ingress   | egress    | Node | destination | srhActive   | Path Delay |
     | Interface | Interface |      | IPv6Address | SegmentIPv6 |            |
     +-----------+-----------+------+-------------+-------------+------------+
     |    271    |    276    |  R1  | 2001:db8::2 | 2001:db8::4 |    0 us    |
     +-----------+-----------+------+-------------+-------------+------------+
     |    301    |    312    |  R2  | 2001:db8::3 | 2001:db8::4 |   22 us    |
     +-----------+-----------+------+-------------+-------------+------------+
     |    22     |     27    |  R3  | 2001:db8::4 | 2001:db8::4 |   42 us    |
     +-----------+-----------+------+-------------+-------------+------------+
     |    852    |    854    |  R4  | 2001:db8::4 | 2001:db8::4 |  122 us    |
     +-----------+-----------+------+-------------+-------------+------------+

           Table 2: Example table of measured delay. Ascending by delay.

6. IANA Considerations

     +-------+--------------------------------+
     |Element|              Name              |
     |   ID  |                                |
     +-------+--------------------------------+
     | TBD5  | pathDelayMeanDeltaMicroseconds |
     |       |                                |
     +-------+--------------------------------+
     | TBD6  | pathDelayMinDeltaMicroseconds  |
     |       |                                |
     +-------+--------------------------------+
     | TBD7  | pathDelayMaxDeltaMicroseconds  |
     |       |                                |
     +-------+--------------------------------+
     | TBD8  | pathDelaySumDeltaMicroseconds  |
     |       |                                |
     +-------+--------------------------------+
  Table 3: New IPFIX IEs in the "IPFIX Information Elements" Registry

7. Operational Considerations

8. Security Considerations

The IPFIX Information Elements introduced in this document do not directly introduce security issues. Rather, they define a set of performance metrics that may, for privacy or business issues, be considered sensitive information.¶

For example, exporting delay metrics may make attacks possible for the receiver of this information; this would otherwise only be possible for direct observers of the reported Flows along the data path.¶

The underlying protocol used to exchange the information described here must therefore apply appropriate procedures to guarantee the integrity and confidentiality of the exported information. These protocols are defined in separate documents, specifically the IPFIX protocol document [RFC7011].¶

9. Implementation Status

Note to the RFC-Editor: Please remove this section before publishing.¶

10. Acknowledgements

The authors would like to thank Al Morton (Rest in Peace Al), Justin Iurman, Giuseppe Fioccola and Yannick Buchs for their review and valuable comments. Special thanks to Paul Aitken (as IPFIX Designated Expert), Greg Mirsky (as IP Performance Metrics Designated Expert), and to Med Boucadair for his very detailed feedback.¶

Appendix A. IPFIX Encoding Examples

This appendix represents two different encodings for the newly introduced IEs. Taking Figure 1 from Section 1 as topology example. Below example Table 4 shows the aggregated delay with ingressInterface, egressInterface, destinationIPv6Address and srhActiveSegmentIPv6.¶

 +------ +------+-----------+-----------+------+---------+---------+---------+---------+
 |ingress|egress|destination|srhActive  |packet|pathDelay|pathDelay|pathDelay|pathDelay|
 |Inter  |Inter |IPv6Address|SegmentIPv6|Delta |MeanDelta|MinDelta |MaxDelta |SumDelta |
 |face   |face  |           |           |Count |Micro..  |Micro..  |Micro..  |Micro..  |
 +-------+------+-----------+-----------+------+---------+---------+---------+---------+
 |  271  |  276 |2001:db8::4|2001:db8::2|  5   |  36 us  |  22 us  |  74 us  | 180 us  |
 +-------+------+-----------+-----------+------+---------+---------+---------+---------+

  Table 4: Aggregated delay with egressInterface and srhActiveSegmentIPv6

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |          SET ID = 2           |       Length = 40             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      Template ID = 256        |      Field Count = 8          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0|     ingressInterface = 10   |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0|     egressInterface = 14    |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| destinationIPv6Address = 28 |      Field Length = 16        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| srhIPv6ActiveSegment = 495  |      Field Length = 16        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| packetDeltaCount = 5        |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| pathDelayMeanDelta.. = TBD5 |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| pathDelayMinDelta.. = TBD6  |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| pathDelayMaxDelta.. = TBD7  |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     Figure 2: Template Record for pathDelayMeanDeltaMicroseconds

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |         SET ID = 256          |           Length = 60         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           ingressInterface =  271                             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           egressInterface =  276                              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           destinationIPv6Address =                            |
      |                          ...                                  |
      |                          ...                                  |
      |                          2001:db8::2                          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           srhIPv6ActiveSegment = ...                          |
      |                          ...                                  |
      |                          ...                                  |
      |                          2001:db8::4                          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           packetDeltaCount = 5                                |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           pathDelayMeanDeltaMicroseconds =  36                |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           pathDelayMinDeltaMicroseconds =  22                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           pathDelayMaxDeltaMicroseconds =  74                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 3: Data Set Encoding for pathDelayMeanDeltaMicroseconds

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          SET ID = 2           |       Length = 40             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Template ID = 257        |      Field Count = 8          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0|     ingressInterface = 10   |      Field Length = 4         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0|     egressInterface = 14    |      Field Length = 4         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| destinationIPv6Address = 28 |      Field Length = 16        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| srhIPv6ActiveSegment = 495  |      Field Length = 16        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| packetDeltaCount = 5        |      Field Length = 4         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| pathDelayMinDelta.. = TBD6  |      Field Length = 4         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| pathDelayMaxDelta.. = TBD7  |      Field Length = 4         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| pathDelaySumDelta.. = TBD8  |      Field Length = 8         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     Figure 4: Template Record for pathDelaySumDeltaMicroseconds

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         SET ID = 257          |           Length = 60         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           ingressInterface =  271                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           egressInterface =  276                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           destinationIPv6Address =                            |
   |                          ...                                  |
   |                          ...                                  |
   |                          2001:db8::2                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           srhIPv6ActiveSegment = ...                          |
   |                          ...                                  |
   |                          ...                                  |
   |                          2001:db8::4                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           packetDeltaCount = 5                                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           pathDelayMinDeltaMicroseconds =  22                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           pathDelayMaxDeltaMicroseconds =  74                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           pathDelaySumDeltaMicroseconds =  180                |
   |                          ...                                  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 5: Data Set Encoding for pathDelaySumDeltaMicroseconds

Authors' Addresses