Internet DRAFT - draft-feher-benchresres
draft-feher-benchresres
Network Working Group Gabor Feher, BUTE
INTERNET-DRAFT Istvan Cselenyi, TRAB
Expiration Date: January 2001 Krisztian Nemeth, BUTE
July 2000
Benchmarking Terminology and Methodology for Routers Supporting
Resource Reservation
<draft-feher-benchresres-00.txt>
1. Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft shadow directories can be accessed at
http://www.ietf.org/shadow.html
This memo provides information for the Internet community. This memo
does not specify an Internet standard of any kind. Distribution of
this memo is unlimited.
2. Table of content
1. Status of this Memo.............................................1
2. Table of content................................................1
3. Abstract........................................................2
4. Introduction....................................................2
5. Existing definitions............................................3
6. Definitions of Terms............................................3
6.1 Resource Reservation Protocol...............................3
6.1.1 Resource Reservation Session...........................4
6.1.2 Multicast Resource Reservation Session.................4
6.1.3 Reservation Capable Router.............................4
6.1.4 Signaling End-point....................................5
6.1.5 Reservation Initiator..................................5
6.1.6 Signaling Path.........................................6
6.1.7 Signaling Message Sequence.............................6
6.1.8 Multicast aware Resource Reservation Protocol..........7
6.1.9 Scalability Limit......................................7
Feher, Cselenyi, Nemeth Expires January 2001 [Page 1]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
6.2 Traffic Types...............................................8
6.2.1 Premium Traffic........................................8
6.2.2 Best-Effort Traffic....................................8
6.3 Router Load Types...........................................8
6.3.1 Signaling Load.........................................8
6.3.2 Signaling Burst........................................9
6.3.3 Reservation Load.......................................9
6.4 Performance Metrics........................................10
6.4.1 Signaling Message Handling Time.......................10
6.4.2 Premium Traffic Extra Delay...........................11
6.4.3 Best-effort Traffic Extra Delay.......................12
6.4.4 Signaling Message Loss................................12
7. Methodology....................................................13
7.1 Evaluating the Results.....................................13
7.2 Test Set up................................................13
7.2.1 One Tester............................................13
7.2.2 Two Testers...........................................14
7.2.3 Test Set up for Unicast Resource Reservation..........14
7.2.4 Test Set up for Multicast Resource Reservation........14
7.2.5 Signaling Message Verification........................15
7.3 Scalability Tests..........................................15
7.3.1 Maximum Signaling Message Burst Size -- Unicast Case..16
7.3.2 Maximum Signaling Message Burst Size -- Multicast Case16
7.3.3 Maximum Signaling Intensity -- Unicast Case...........17
7.3.4 Maximum Signaling Intensity -- Multicast Case.........18
7.3.5 Maximum Number of Reservation Sessions -- Unicast Case18
7.3.6 Maximum Number of Reservation Sessions -- Multicast Case
............................................................19
7.4 Benchmarking Tests.........................................19
7.4.1 Signaling Message Handling Time -- Unicast Case.......20
7.4.2 Signaling Message Handling Time -- Multicast Case.....22
7.4.3 Premium Traffic Extra Delay -- Unicast Case...........22
7.4.4 Premium Traffic Extra Delay -- Multicast Case.........23
7.4.5 Best-effort Traffic Extra Delay.......................24
8. Acknowledgement................................................24
9. References.....................................................24
10. Authors' Addresses:...........................................25
3. Abstract
The purpose of this document is to define terminology and methodology
specific to the performance benchmarking the resource reservation
signaling capability of IP routers.
In addition to defining the performance metrics and the tests, this
document also describes specific formats for reporting the results of
the tests.
4. Introduction
The IntServ over DiffServ framework [3] outlines a heterogeneous
Quality of Service (QoS) architecture for multi domain Internet
services. Signaling based resource reservation (e.g. by RSVP [5]) is
Feher, Cselenyi, Nemeth Expires January 2001 [Page 2]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
an integral part of that model. While this way scalability
constraints are released for most of the core routers, the
performance of border routers that handle signaling is still crucial.
Therefore network operators, who are planning to deploy this model,
shall scrutinize the scalability limitations in reservation capable
routers and the impact of signaling on the routers’ forwarding
performance.
An objective way for quantifying the scalability constraints of QoS
signaling is to perform measurements on routers that are capable of
resource reservation. This document defines a specific set of tests
that vendors or network operators can use to measure and report the
signaling performance characteristics of router devices that support
resource reservation protocols. The results of these tests will
provide comparable data for different products supporting the
decision process of purchase. Moreover, these measurements provide
input characteristics for the dimensioning of a network in which
resources are provisioned dynamically by signaling. Finally, these
test are applicable for characterizing the impact of control plane
signaling on the forwarding performance of routers.
The first part of the document, "Terminology", defines the terms that
are used later on in the document. The second part is the
"Methodology" that defines measurement methods to find the
scalability limits and characterize the signaling performance of the
tested router.
This benchmarking terminology and methodology is based on the
knowledge gained by examination of four very different resource
reservation protocols: RSVP [5], Boomerang [6], YESSIR [7] and ST2+
[8]. Nevertheless, this document aspires to compose terms and methods
that are valid in general and not restricted to these four protocols.
5. Existing definitions
RFC 1242 [1] "Benchmarking Terminology for Network Interconnect
Devices" and RFC 2285 [2] "Benchmarking Terminology for LAN Switching
Devices" contains discussions of a number of terms relevant to the
benchmarking of signaling performance of reservation capable routers
and should be consulted before attempting to make use of this
document.
For the sake of clarity and continuity this document adopts the
template for definitions set out in Section 2 of RFC 1242 [1].
Definitions are indexed and grouped together in sections for ease of
reference.
6. Definitions of Terms
6.1 Resource Reservation Protocol
This group of definitions applies to various resource reservation
protocols implemented in router devices.
Feher, Cselenyi, Nemeth Expires January 2001 [Page 3]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
6.1.1 Resource Reservation Session
Definition:
A resource reservation session (or shortly reservation) expresses
that certain QoS treatment is assigned to the packets of a traffic
flow along the path between two hosts.
Discussion:
The QoS treatment can be specified explicitly by giving the amount
of networking resources (e.g. forwarding capacity or buffer space)
or it can be specified implicitly by referring to a forwarding
behavior (e.g. EF PHB in case of Differentiated Services [9]). The
flow can be specified either by the five-tuple (i.e. as a micro-
flow in [9]) or as traffic aggregate.
See Also:
Signaling Path
6.1.2 Multicast Resource Reservation Session
Definition:
Multicast resource reservation session (or shortly multicast
reservation) denotes that certain QoS treatment is assigned to the
packets of a traffic flow related to a multicast address.
Discussion:
There are resource reservation protocols that allow different
amount of resource dedication in reservation capable routers for
one multicast reservation session. These protocols distinguish
different traffic sources and different traffic destinations and
can have more than one reservation models to express the resource
needs within the reservation. (e.g. RSVP SE/WF/FF [5])
Issues:
- Reservation aggregation
See Also:
Signaling Path
6.1.3 Reservation Capable Router
Definition:
A router is reservation capable if it understands a resource
reservation protocol that signals the set up, tear down or
modification of a reservation in the router.
Discussion:
Resource reservation protocols can be divided into two groups, the
soft state and the hard state protocols. Both kinds of protocol
tear a resource reservation entry upon request but in case of soft
state protocols the router also tears an entry if a refresh
message does not renew the resource reservation entry within a
Feher, Cselenyi, Nemeth Expires January 2001 [Page 4]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
refresh period. Thus in case of soft state protocols the number of
refresh messages may cause considerable signaling load, while on
the other hand it is easier to build a robust protocol that is
soft state.
6.1.4 Signaling End-point
Definition:
A Signaling End-point can be a host or a network node, which is
capable to initiate and/or terminate or proxy signaling sessions.
Discussion:
Usually, signaling end-points have a protocol stack that is
capable to generate and understand signaling messages. However in
some cases this is not necessary. When the messages of the
resource reservation protocol are payloads of other protocols that
the host understands and the resource reservation protocol does
not require any special action from signaling end-points then the
host becomes a signaling end-point without knowing the resource
reservation protocol (e.g. Boomerang [6] encapsulates the
reservation requests to an ICMP Echo message).
Reservation proxies are protocol translators that translate the
signaling messages of one resource reservation protocol into
messages of other resource reservation protocols bridging
different networks with different resource reservation protocols.
Thus the reservation proxy is two signaling end-points in one, as
it is both a signaling terminator and a signaling initiator.
6.1.5 Reservation Initiator
Definition:
The reservation initiator is the signaling end-point, which
initiates the reservation set up.
Discussions:
Resource reservation protocols can be divided into different
groups according to the initiator of the reservation request, what
can be the traffic source(s), the traffic destination(s), or both
of them. Typical example for the later is RSVP, where the data
traffic destination(s) requests the resource reservation. This
scheme is called receiver-oriented where "receiver" refers to the
data traffic receiver.
A receiver-oriented protocol must assure that the data flow(s)
coming from the traffic source(s) to the traffic destination(s)
takes the same path as the reservation request, which travels in
the opposite direction. IP routing does not guarantee the same
path for the different directions between end-points, so the
resource reservation protocol has to take care of it. Therefore
RSVP has a signaling message primitive, called the PATH message,
which establishes the signaling path(s) among traffic sources and
traffic destinations before the reservation can happen and
Feher, Cselenyi, Nemeth Expires January 2001 [Page 5]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
afterwards the signaling messages are forwarded on this path
backward.
See Also:
Signaling end-point
Signaling path
6.1.6 Signaling Path
Definition:
A signaling path is a sequence of network nodes and links on which
signaling messages travel from one signaling end-point to the
other.
Discussion:
In case of reservation session where the data traffic is unicast
and goes point-to-point, the signaling path is mostly the same as
the data path. However in case of multicast reservations there are
more than one signaling path belonging to one reservation session
according to the definition. There is one signaling path between
each traffic source and traffic destination.
Both in the unicast and multicast case, it might happen that the
path of the reservation is not the same as the path of the data
stream, as recent reservation capable routers do not take effort
to guarantee it. Still, the current practice is that the signaling
messages and the data packets are addressed with the same IP
address, trusting that in this case they are forwarded on the same
path.
Issues:
- Multicast signaling path graph
- Route change
6.1.7 Signaling Message Sequence
Definition:
Signaling message sequence is defined as a series of signaling
messages, which are related to the same reservation session and
whose order follows the specification of the reservation protocol.
Discussion:
A typical signaling message sequence is a reservation set up and
reservation tear down pair, where the first signaling message sets
up a reservation in the router and the second one tears that down.
Signaling message sequences are used in measurements where the
changes in the states of the router affect the measurement. Using
signaling message sequences the router states are restored after
the last message in the sequence. (e.g.: RSVP's PATH and PATHTEAR
message pair)
Feher, Cselenyi, Nemeth Expires January 2001 [Page 6]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
6.1.8 Multicast aware Resource Reservation Protocol
Definition:
A resource reservation protocol is multicast aware when it
supports resource reservation for multicast sessions.
Discussion:
There are two types of multicast support, one is the many-to-many
multicast and the other is the one-to-many multicast. The former
allows reservations for many traffic sources in the multicast
group, while the latter allows for only one traffic source in the
group. When there are more than one data traffic sources in a
multicast reservation session then the protocol might aggregate
the reservation towards the data traffic destinations.
Certainly a multicast aware protocol is more complex than a non-
multicast aware. Currently perhaps the most well-known IP resource
reservation protocol, RSVP [5], does support many-to-many
multicast, while another protocol, Boomerang [6], supports one-to-
many multicast.
Issues:
- Multicast traffic
- IP multicast
6.1.9 Scalability Limit
Definition:
Scalability limit is the border between the steady and the
overloaded state of the Device Under Test (DUT).
Discussion:
All existing routers have finite memory buffer range and finite
processing power (CPU). In the steady state of the router the
memory buffers are not fully utilized and the processing power is
enough to cope with the different tasks of the router. However
when the load rises in the router there is a certain point where
no more memory space is available or the processing power is not
enough to satisfy the needs of all the tasks and thus the router
becomes overloaded. The critical load condition, when the router
is still in the steady state but the smallest amount of load
increase would drive it to the overloaded state is the scalability
limit of the router.
Usually the overloaded state of the router can be recognized by
some kinds of data loss. A resource reservation capable router may
drop signaling messages, data packets or reservation entries, as
it does not have enough capacity to process or maintain them.
Smart task scheduling is able to overcome on this for short
periods when the load is above the scalability limit, thus the
router remains in the steady state. Therefore the scalability
limit should be determined under a long lasting constant load.
Feher, Cselenyi, Nemeth Expires January 2001 [Page 7]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
Issues:
- Scalability tests
6.2 Traffic Types
This group of definitions defines traffic types forwarded by resource
reservation capable routers.
6.2.1 Premium Traffic
Definition:
Premium traffic is a traffic type that the router distinguishes
from the best-effort traffic and forwards its packets according to
a QoS agreement.
Discussion:
Each premium traffic flow has a resource reservation entry in the
router set up by signaling messages. The router might define more
than one premium traffic type (e.g. delay sensitive traffic, loss
sensitive traffic) and different premium traffic types have
different QoS treatments.
6.2.2 Best-Effort Traffic
Definition:
Best-effort traffic is the traffic that has no reservation entry
in the router.
Discussion:
Traffic flows that do not require QoS guaranties along their path
are considered as best-effort traffic. Best effort means that the
router takes its best effort to forward every data packets, but it
cannot give any guarantee. This type of traffic is the standard in
today’s Internet.
6.3 Router Load Types
This group of definitions defines different load components, which
are used for testing the resource reservation performance of the
reservation capable router.
6.3.1 Signaling Load
Definition:
The signaling load is characterized by the signaling intensity,
which expresses, how many signaling messages arrive to the
reservation capable router within a time unit.
Discussion:
As the processing of signaling messages consumes router power, the
signaling intensity is correlated with the load. This load is
called signaling load because it is generated by signaling. The
Feher, Cselenyi, Nemeth Expires January 2001 [Page 8]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
higher the signaling intensity is, the more signaling messages hit
the router within a time unit. Therefore the processing capacity
spent on signaling handling increases, resulting higher load in
the router.
Most of the resource reservation protocols have several protocol
primitives realized by different signaling message types. Each of
these message types requires different processing capacity from
the router.
Issues:
- Different signaling message types load the router differently
- Message generation burstiness
Measurement unit:
Number of signaling messages per second [1/s]
See Also:
Signaling burst
6.3.2 Signaling Burst
Definition:
The signaling burst denotes a certain number of signaling
messages, which arrive to the input port(s) of the router
continuously causing persistent load for the signaling message
handler. The single parameter of the signaling burst is the number
of signaling messages in the burst.
Discussion:
Back-to-back signaling messages on one port of the router form a
typical signaling burst. But other cases can be imagined also
where signaling messages arrive on different ports simultaneously
or with an overlap in time (i.e. when the tail of one signaling
message is behind the head of another one arriving on another
port).
Certainly, as in case of signaling load, the signaling message
type is an issue here, too.
Issues:
- Different types of signaling messages
Measurement unit:
The signaling burst size is the number of messages arrived during
the burst
See Also:
Signaling intensity
6.3.3 Reservation Load
Definition:
Feher, Cselenyi, Nemeth Expires January 2001 [Page 9]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
The reservation load is characterized by the number of resource
reservation sessions set up in the router.
Discussion:
In case of micro-flow classification the packet classifier shall
distinguish the flows, which have reservations in the router from
the others, which do not. Therefore the more reservation session
is set up in the router the more complex the traffic
classification is. This is one factor that contributes to the
reservation load. Moreover, most signaling based resource
reservation protocols require that the routers maintain some kind
of state for each flow they keep reservation for. The growth of
this state space is a scalability issue as on one hand it requires
memory capacity and on the other hand searching through a larger
state space may increase the signaling handling time.
Note that some protocols (e.g. RSVP, Boomerang) allow making
reservation for the aggregation of several micro-flows. In this
case the number of reserved sessions can be smaller than the
number of micro-flows.
In case of soft state protocols the refresh messages required to
maintain the reservations cause a considerable signaling load as
well beside the reservation load. Although, some soft state
protocols might be capable to aggregate refresh messages that
decreases significantly the signaling load.
Issues:
- Micro-flow aggregation
- Refresh message aggregation
Measurement unit:
Number of reservation sessions in the router
6.4 Performance Metrics
This group of definitions is the collection of the measurable effects
of the impact of the resource reservation on a router device.
6.4.1 Signaling Message Handling Time
Definition:
The signaling message handling time (or shortly signal handling
time) is the time that a signaling message spends inside the
router before it is forwarded to the next hop.
Discussion:
Usually signaling messages are issued by a signaling end-point and
forwarded via the signaling path by the routers. However, beside
the message forwarding the router interprets the content of the
messages and acts according to it. Thus the message handling time
is longer than forwarding time of data packets of the same size.
Moreover, there are signaling message primitives that are altered
Feher, Cselenyi, Nemeth Expires January 2001 [Page 10]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
during the processing and there might be messages also that are
drained in the router or generated by the router. Thus, the signal
message handling time is the time difference of a signaling
message entering time and the leaving time of the corresponding
processed signaling message. When a message is drained by the
router then the signal handling time is immeasurable therefore it
is not defined.
In case of signaling messages that carry instructions for
multicast flows, the router might issue multiple signaling
messages after processing. In this case, by definition, the signal
handling time is the time interval elapsed between the arrival of
the incoming signaling message to the router and the departure of
the last, related signaling message.
Signal handling time is an important measure as it directly
affects the setup time of a session. On the other hand, it is also
an indication of the signal processing capacity of the measured
router as it is correlated to the number of signaling messages
that can be processed within a time unit.
This metric is also dependent on the type of the signaling
message. For example, it takes generally smaller time to tear down
a session within a router than to set it up.
Issues:
- Refresh message aggregation
- The type of the signaling message
Measurement unit:
seconds
6.4.2 Premium Traffic Extra Delay
Definition:
Premium traffic extra delay is the delay that a data packet of a
reserved flow suffers because of the resource reservation protocol
running on the router.
Discussion:
This metric shows the effect of the classification, policing and
shaping along with the cross-effect of the control plane on the
data plane.
Classification [9] is the mechanism for filtering out premium
traffic. All data packet belonging to premium traffic must be
classified in order to find the right resource bounds for the flow
where the packet belongs. In systems using only one processor for
both the processing of signaling messages and for data forwarding,
this cross influence is expected to be important.
Issues:
- Smart classification, policing, shaping routines
Feher, Cselenyi, Nemeth Expires January 2001 [Page 11]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
- Core and border routers
Measurement unit:
seconds
See Also:
Best-effort traffic extra delay
6.4.3 Best-effort Traffic Extra Delay
Definition:
Best-effort traffic extra delay is the delay that a non-
prioritized data packet transfer because the resource reservation
protocol is running on the router.
Discussion:
It is obvious that classification or policing algorithms do not
address the best-effort traffic. However the cross-effect between
the control and data plane influences the traffic and adds an
extra delay to the forward procedure of each packet.
Measurement unit:
seconds
See Also:
Premium traffic extra delay
6.4.4 Signaling Message Loss
Definition:
Signaling message loss is the ratio of the signaling messages that
has actually left the router and signaling messages that are
expected to leave the router as a response to the corresponding
signaling message entering to the router.
Discussion:
As in case of signaling message handling time there is a problem
here as well with the signaling messages that drained or generated
inside the router. Those signaling messages are immeasurable and
therefore the definition does not apply to them. Signaling loss
considers signaling messages only that leave the router as a
consequence of processing an entering signaling message. Note,
that signaling messages in a multicast reservation session might
trigger several signaling messages.
Losing a signaling message is almost always an indication of
saturation of the router. This measure is therefore suitable for
sounding out the scalability limits of the router.
Issues:
- Multicast signaling
- Refresh message aggregation
Feher, Cselenyi, Nemeth Expires January 2001 [Page 12]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
Measurement unit:
percentage value, but in many cases the existence of signaling
loss is enough
7. Methodology
7.1 Evaluating the Results
RFC2544 [4] describes considerations regarding the implementation and
evaluation of tests, which are certainly valid for this test suite
too. Namely, we shall emphasize that we intended to create a system
from commercially available measurement instruments and devices for
the sake of easy implementation of the described tests. Simple test
scripts and test software for Linux are available from the Boomerang
homepage [10].
Secondly, care should be taken for selecting the proper tests for a
specific router, since not all of the tests apply to all types of
Devices Under Test (DUTs).
Finally, the selection of the relevant measurement data and their
evaluation requires experience and it must be done with an
understanding of generally accepted testing practices regarding
repeatability, variance and statistical significance of small numbers
of trials.
7.2 Test Set up
7.2.1 One Tester
The ideal way to perform the measurements is to connect a tester
device to both the incoming and outgoing network interfaces of the
DUT. The tester sends signaling messages and data traffic on its
outgoing port to one of the incoming ports of the reservation capable
router device, while the outgoing network port of the router, where
the processed signaling messages and the forwarded packets appear, is
connected to an incoming port of the tester device. Thus the tester
device is capable to measure the signaling message handling time, the
various traffic forwarding times and the signaling loss. This
scenario can be seen in Figure 1 [4]. In this case the tester is both
the signaling and data traffic source and the destination device in
one.
+------------+
| |
+------------| tester |<-------------+
| | | |
| +------------+ |
| |
| +------------+ |
| | | |
+----------->| DUT |--------------+
Feher, Cselenyi, Nemeth Expires January 2001 [Page 13]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
| |
+------------+
Figure 1
7.2.2 Two Testers
The measurements can be performed with two tester devices as well,
separating the sender and receiver functionality into two pieces of
equipment. In this case the sender test device is the driver of the
input network interface of the DUT and the second one, the receiver
test device, is connected to the output network interface of the DUT
and collects the messages and traffic packets leaving from the DUT.
This scenario can be seen in Figure 2. Using this test scenario the
synchronization of the clocks in the tester devices must be assured
for certain tests. Nevertheless, the scalability tests do not depend
on time synchronization.
+--------+ +------------+ +----------+
| | | | | |
| sender |-------->| DUT |--------->| receiver |
| | | | | |
+--------+ +------------+ +----------+
Figure 2
The main benefit of using the single tester device scenario described
above is that the time measurement happens within a single device and
does not require any time synchronization. Using one tester device is
recommended during all of the tests. The description of the
benchmarking methodology uses the expressions of sender and receiver
devices but they do not have to be two physically separate
appliances.
When the DUT runs a multicast aware resource reservation protocol
then the test must be performed in the multicast resource reservation
test scenario as well as in the standard unicast scenario.
7.2.3 Test Set up for Unicast Resource Reservation
The test set up in the unicast test scenario is the same as described
generally in the test set up section. During the test the tester
device must use unicast addresses for the data traffic and must send
resource reservation requests for host-to-host (i.e. unicast)
reservation.
7.2.4 Test Set up for Multicast Resource Reservation
The test set up in the multicast test scenario is similar to the one
described in the beginning of this chapter (see sections 7.2.1 and
7.2.2), but all the outgoing ports (where the router emits signaling
messages or data packets as a response to one signaling message or
data packet on the incoming port) must be connected to the tester
device or tester devices. Furthermore the data traffic from the
Feher, Cselenyi, Nemeth Expires January 2001 [Page 14]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
tester must be sent to a multicast address and the tester device must
ask resource reservation requests for multiple hosts.
There are resource reservation protocols, like RSVP, which has more
than one reservation scheme for multicast flows. In this case all
reservation schemes must be tested.
There are many of ways to possibly define a multicast reservation
session. The possibly large number of end-points and the distribution
of traffic sources and traffic destinations within the group results
in a very large number of combinations. The benchmarking procedure
does not require measuring all possible multicast sessions, just one
scenario is required, while others are still recommended. The
proposed multicast scenario consists of a multicast group with 4 end-
points where there are one traffic originator and three traffic
drainers. The traffic originator should be placed to the sender side
of the DUT, while the drainers should be placed to the receiver side.
The benchmarking reports carried on multicast scenarios always have
to include the full description of the scenario itself.
7.2.5 Signaling Message Verification
However the conformance testing of the resource reservation is beyond
the scope of this document, defective signaling message processing
can be expected in an overloaded router. Thus, during the tests, when
signaling messages are processed in the DUT, the receiver device must
verify the messages whether they fully conform to the message format
of the protocol specification and they are the expected signaling
messages at the given situation. When any of the messages do not
conform to the protocol specification the report must indicate
implementation errors. However, further analysis of the conformance
of a malfunctioning protocol implementation is beyond the scope of
this test suit that targets performance analysis.
Verifying data traffic packets are not required, since the signaling
performance benchmarking of reservation capable routers should not
deal with data traffic. For this purpose there are other benchmarking
methodologies that verify data traffic during the measurements, like
the one described in RFC 2544 [4].
7.3 Scalability Tests
Unicast scalability tests are defined to explore the scalability
limits of a reservation capable router when it deals with unicast
resource reservation requests. According to our definition, a router
reaches its scalability limit, when it steps out of the steady state
operation and steps into a phase where it cannot process all of the
signaling messages and begins to drop some of them.
Multicast scalability tests are applied for multicast-aware resource
reservation protocols only. They are similar to unicast scalability
tests, but all the resource reservations should ask resources for
Feher, Cselenyi, Nemeth Expires January 2001 [Page 15]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
multicast reservation sessions and the multicast test scenario must
be used for the measurements.
7.3.1 Maximum Signaling Message Burst Size -- Unicast Case
Objective:
To determine the maximum number of the signaling messages in a burst
that the router is still able to handle without signaling loss.
Procedure:
1. Select one signaling message or a signaling message sequence from
the specification of the resource reservation protocol.
2. A number of signaling messages or signaling message sequences must
be sent back-to-back in order to form a burst. All the signaling
messages in the burst must be valid messages and the sender must
check if the router is able to process them without any error. The
signaling messages must be addressed in a way that they go through
the DUT and the receiver should be able to catch them all.
3. Send the burst towards the DUT and count the signaling messages
received by the receiver.
4. When the number of sent messages equals to the number of received
messages, the number of messages in a burst must be increased and the
test is to be repeated. When the receiver receives less signaling
messages than the sender has sent, the DUT is over its scalability
limit. The scalability limit for the maximum signaling message burst
size is found when the sender is able to send a signaling burst
without signaling loss but one more signaling message in the burst
would result signaling loss.
During the test the DUT must not receive other data packets than the
signaling messages that the sender sends. The test should be repeated
at least 30 times for each signaling message or signaling message
sequence. The DUT should be reset to its initial state before each
step of the test.
Reporting format:
The report should indicate the type of the signaling message or
signaling message sequence and the median of the measured maximum
signaling burst size.
Note:
There are resource reservation protocols, like RSVP, where certain
kind of signaling messages (e.g. RESV) are valid only when some other
signaling messages (e.g. PATH) have already established corresponding
states in the router. In this situation the sender must ensure the
establishment of such states in the router before the test begins.
7.3.2 Maximum Signaling Message Burst Size -- Multicast Case
Objective:
Feher, Cselenyi, Nemeth Expires January 2001 [Page 16]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
To determine the maximum number of the signaling messages in a burst
that the router is still able to handle without signaling loss.
Procedure:
The procedure is the same as it is defined for the unicast case,
except all the signaling messages should refer to multicast sessions.
The test stops when there is a signaling loss on any of the outgoing
interfaces of the DUT.
Reporting format:
The reporting format is the same as in the unicast case.
7.3.3 Maximum Signaling Intensity -- Unicast Case
Objective:
To determine the maximum number of signaling messages that can be
sent to the router within a time unit.
Procedure:
1. Select one signaling message or a signaling message sequence from
the specification of resource reservation protocol.
2. Construct a flow of valid signaling messages or signaling message
sequences where the individual signaling messages are equally spaced
and there are a specified number of messages in one second according
to the signaling intensity. The signaling messages must be addressed
in a way that they must go through the DUT and the receiver should be
able to catch them all.
3. Send the flow towards the DUT for at least one minute. Count the
signaling messages received by the receiver.
4. When the number of sent messages equals to the number of received
messages then the signaling intensity of the flow must be increased
and the test must be repeated. When the receiver receives less
signaling messages than the sender has sent, the DUT is over its
scalability limit. The scalability limit for the maximum signaling
intensity is found, when the signaling flow arrives to the receiver
without any signaling loss but the DUT would lose signaling messages,
if the signaling intensity would be larger.
During the test the DUT must not forward other data packets than the
signaling messages that the sender sends. The test should be repeated
at least 30 times for each signaling message or signaling message
sequence. The DUT should be reset to its initial state before each
step of the test.
Reporting format:
The report should indicate the type of signaling message or signaling
message sequence and the median of the measured maximum signaling
intensity.
Note:
Feher, Cselenyi, Nemeth Expires January 2001 [Page 17]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
There are resource reservation protocols, like RSVP, where certain
kind of signaling messages are valid only when some other signaling
messages have already established corresponding states in the router.
In this situation the sender must ensure the establishment of such
states in the router before the test begins.
7.3.4 Maximum Signaling Intensity -- Multicast Case
Objective:
To determine the maximum number of signaling messages that can be
sent to the router within a time unit.
Procedure:
The procedure is the same as it is defined for the unicast case,
except that each signaling messages should refer to multicast
sessions. The test stops when there is a signaling loss on any of the
outgoing interfaces of the DUT.
Reporting format:
The reporting format is the same as in the unicast case.
7.3.5 Maximum Number of Reservation Sessions -- Unicast Case
Objective:
To determine the maximum number of reservation sessions that can
exist simultaneously in a reservation capable router.
Procedure:
1. Set up a reservation session in the reservation capable router by
sending the appropriate signaling message sequence over the DUT.
2. After a successful reservation setup, wait for a specified amount
of time (T) while still maintaining the established reservations.
Care should be taken of the maintenance of established reservation
sessions. If the DUT uses soft state resource reservation protocol,
then the waiting time, T must be at least as long as the protocol
specification requires for reservation time out. This waiting is
necessary to assure that DUT is able to refresh the reservations. In
case of hard state resource reservation protocols it is not necessary
to wait thus T can be zero. .
3. Repeat the previous steps until the router is not able to set up
any new reservation or it is not able to maintain the existing
reservations. That is, if you have set up N reservations and the DUT
starts dropping reservations during the waiting time or after it,
then the maximum number of reserved sessions is passed over and the
actual number of maximum reservation sessions is N-1.
The test should be repeated at least 5 times and the DUT should be
reset between the tests cycles.
Reporting format:
Feher, Cselenyi, Nemeth Expires January 2001 [Page 18]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
The report must indicate the median of the measured maximum number of
reservation sessions. When the number of reserved sessions grow over
a number that is surely enough in the given technology conditions,
then the test can be canceled and the report can state that the
resource reservation protocol implementation performs the maximum
number of reservation sessions over that limit. (e.g. "Over 10.000
sessions")
Checking the reservation sessions in a reservation capable router
might be difficult if the router does not support any interface to
monitor its interior states. Lack of such support the signaling-end
point might try to send overrated data traffic across all the
reservation sessions and dropping the right amount of data traffic
means that the reservation sessions are alive. Of course other smart
methods can be used also.
7.3.6 Maximum Number of Reservation Sessions -- Multicast Case
Objective:
To determine the maximum number of reservation sessions that can
exist simultaneously in a reservation capable router.
Procedure:
The procedure is the same as it is defined for the unicast case,
except all the signaling messages should refer to multicast
reservation sessions.
Reporting format:
The reporting format is the same as in the unicast case.
7.4 Benchmarking Tests
Benchmarking tests are defined to measure the QoS signaling related
performance metrics on the router device.
During the tests the DUT must stay in steady state. This means that
the router must not drop any signaling message or data packet, i.e.
it should operate below its scalability limits. In case of signaling
message or data traffic loss, the test must be stopped, and the
parameters of the test must be adjusted to prevent the DUT to leave
its steady state operating range.
During all of the benchmarking tests the sender tester loads the DUT
by sending a specified amount of signaling and data traffic to the
receiver device across the DUT. Moreover, the signaling end-points
must also assure that the DUT maintains a certain number of
reservation sessions. All of the performance metrics are measured
under different load conditions, where the load is a combination of:
a. Signaling load
b. Premium traffic load
c. Best-effort traffic load
d. Reservation load
Feher, Cselenyi, Nemeth Expires January 2001 [Page 19]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
Signaling load must be generated by sending signaling messages
periodically to the DUT. This way the signaling load is measured by
the signaling intensity and expressed with the same signaling
measurement unit as it is in case of signaling intensity. The
signaling messages sent by the sender device should be equally
distributed on the time scale. Most of the resource reservation
protocols have more than one signaling message type and the
benchmarking is complete when the test has been carried over for each
signaling message type.
Premium traffic load must be generated by sending a data flow from
the sender to the receiver across the DUT where the DUT must have a
valid reservation for that flow. The traffic must consist of equally
spaced and equally sized data packets. The premium traffic must be
reported by its traffic parameters: data packet size in octets and
the calculated bandwidth of the stream in kbps unit. The data packet
size should include both the payload and the header of the IP packet.
Signaling messages maintaining the flow do not belong to the data
traffic flow and must be ignored when calculating the data traffic
parameters.
The best-effort traffic load must be generated by sending a data flow
from the sender tester to the receiver tester across the DUT where
the DUT must not have any reservation for that flow. The traffic must
consist of equally spaced and equally sized data packets. The best-
effort traffic must be reported by traffic parameters as it is
described in case of the premium traffic.
The reservation load must be generated by reserving resources for
traffic flows via signaling in the DUT. During the test the sender
device must maintain the reservation sessions with signaling messages
if the resource reservation protocol requires it. The reservation
sessions should not need to be loaded with data traffic. The
reservations must differ at least in one of the IP addresses or port
number of the endpoints in the reservation entry.
The four load types have influence on each other from their nature,
but during the tests these cross-effects must be minimized. The
signaling load must establish as few temporary resource reservations
in the DUT as possible. When a new resource reservation is set up in
the DUT the signaling end-points must arrange to restore the number
of reservations in the router as soon as possible. Signaling messages
are datagrams in the real word, but during the measurements they are
not treated as premium or best-effort traffic.
7.4.1 Signaling Message Handling Time -- Unicast Case
Objective:
To determine how fast the resource reservation protocol responds to
one type of signaling message or signaling message sequence in
various load conditions.
Feher, Cselenyi, Nemeth Expires January 2001 [Page 20]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
Procedure:
1. Select one signaling message or a signaling message sequence from
the resource reservation protocol specification.
2. Load the DUT with a certain kind of load except signaling load.
The DUT should preserve its steady state operation during the load.
3. Form a signaling message flow from the selected signaling message
type or signaling message sequence. The signaling messages should be
distributed equally in time, this way the signaling flow has a
constant signaling intensity.
4. Send the signaling flow to the receiver across the DUT and measure
the time intervals that signaling messages spend inside the router as
the difference of absolute time (e.g. time stamp) when a signaling
message enters to the DUT and the time when the corresponding
signaling message leaves it. The signaling flow must be up for at
least one minute and the measurement should begin 30 seconds after
the first signaling message is processed in the DUT and should last
at least 30 seconds. The result of the measurement is the average of
the individually measured signaling message handling times grouped by
signaling message types.
5. Repeat the test 30 times for each desired signaling intensity
value for the signaling flow. Always reset the DUT between two test
cycles.
6. Alter the signaling intensity and the load conditions and repeat
the whole test. The signaling message handling time must be measured
in function of many different load conditions. Measuring all possible
load condition however is almost impossible so it is not required,
instead the parameters of the load generation is given in a range and
a step.
The data traffic parameters have to be selected from common traffic
parameters. It is recommended to choose a packet size of: 54, 64,
128, 256, 1024, 1518, 2048 and 4472. The same values used in RFC 2544
that gives methodology for benchmarking network interconnect devices.
The packet rate is recommended to be 1, 10, 100 and 1000 kbit/s. At
least 2 different data traffic parameter set should be chosen both
for the premium traffic and the best-effort traffic. It is
recommended to use UDP packets constructing data traffic but any
other transfer protocol can be used.
The number of reserved sessions in the DUT should be in the range of
1 and the maximum number of reservation sessions for the tested
resource reservation protocol implementation. At least 3 different
values must be chosen.
The signaling intensity of the signaling message flow should be in
the range of 1 and the maximum signaling intensity for the tested
resource reservation protocol implementation. At least 3 different
signaling message intensity values must be chosen.
Feher, Cselenyi, Nemeth Expires January 2001 [Page 21]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
Additionally, the signaling message handling time must be measured in
unloaded conditions as well. In this case there is no data traffic on
the DUT, no reserved resources and the signaling intensity should
take the 1 signaling message per second value.
Reporting format:
The results should be reported in a table. The columns of the table
are the values of the four load components and the fifth column is
the measured signaling message handling time in seconds. The table
entries for the traffic parameters must include the packet size, the
packet rate and the transfer protocol type.
There should be one table for each signaling message type or
signaling message sequence.
Note: The number of the tests is the product of the number of
different load parameters. The minimum is 37 which includes: 2
parameter sets for the premium traffic, 2 parameter sets for best-
effort traffic, 3 different reservation session number and 3
different signaling intensity values for the signaling sequence plus
one measurement where the signaling load is the only load on the DUT.
7.4.2 Signaling Message Handling Time -- Multicast Case
Objective:
To determine how fast the resource reservation protocol responds to
one type of signaling messages or signaling message sequence in
various load conditions.
Procedure:
The procedure is similar to the unicast case but the signaling
messages should refer to multicast sessions, the reserved sessions in
the DUT must be entries for multicast reservation sessions and the
premium traffic must be addressed to a multicast group.
Reporting format:
The reporting format is the same as in the unicast case.
7.4.3 Premium Traffic Extra Delay -- Unicast Case
Objective:
To determine how the resource reservation protocol affects the
premium traffic forwarding performance of the router when it handles
resource reservations for unicast flows.
Procedure:
1. Load the DUT with a certain kind of load except premium traffic
load. The DUT should preserve its steady state operation during the
load.
2. Construct a premium traffic flow from data packets of the same
size. The flow should have a constant bitrate.
Feher, Cselenyi, Nemeth Expires January 2001 [Page 22]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
3. Send the premium flow to the receiver across the DUT with
different traffic parameters and measure the time intervals that data
packets spend inside the router as they are being forwarded. The
signaling must set up a reservation session for this flow in the DUT
before the test starts and it should not be torn down during the
test. The traffic flow must be active for at least one minute. The
measurement should begin 30 seconds after the first data packet
belonging to the measured flow is forwarded in the DUT, and should
last at least 30 seconds. It is not necessary to measure all premium
packets, but at least 20 samples are required in a second and the
samples should be distributed equally in time scale. The result of
the measurement is the average of the individually measured packet
forwarding time values.
4. Repeat the test 30 times for each desired parameter set of the
premium traffic. Always reset the DUT between two test cycles.
5. Alter the parameters of the premium traffic and the load
conditions and repeat the whole test. The premium traffic extra delay
must be measured in function of many different load conditions.
Choose the load parameters according to the test description of the
signaling message handling time, but here at least 3 traffic
parameter sets are required to be tested in case of the premium flow.
Additionally, the premium traffic extra delay must be measured in
unloaded conditions as well. In this case there is neither best-
effort traffic and nor signaling load on the DUT and there is only
one resource reservation entry, which one reserves the resources for
the premium traffic flow used in the test.
Reporting format:
The results should be reported in a table. The columns of the table
are the values of the four load components, the fifth column is the
premium traffic delay in seconds and the sixth column is the
difference of the premium traffic delay with the given load
conditions and the premium traffic delay measured in the unloaded
router. The table entries for the traffic parameters must include the
packet size, the packet rate and the transfer protocol type. The
column of the signaling load must contain the signaling message type
or the signaling message sequence along with the signaling intensity.
Note: The number of the tests is at least as large as in case of the
signaling message handling time test. Using the same parameters as in
case of the signaling message handling time test is highly
recommended.
7.4.4 Premium Traffic Extra Delay -- Multicast Case
Objective:
Feher, Cselenyi, Nemeth Expires January 2001 [Page 23]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
To determine how the resource reservation protocol affects the
premium traffic forwarding performance of the router when it handles
resource reservations for multicast flows.
Procedure:
The procedure is similar to the unicast case but the signaling
messages should refer to multicast sessions, the reservation sessions
in the DUT must be multicast reservations sessions and the premium
traffic must be addressed to a multicast group.
Reporting format:
The reporting format is the same as in the unicast case.
7.4.5 Best-effort Traffic Extra Delay
Objective:
To determine how the resource reservation protocol affects the best-
effort traffic forwarding performance of the router when it handles
resource reservations for unicast flows.
Procedure:
The procedure is almost the same as in case of the premium traffic
extra delay test, but the best-effort traffic must be measured
instead of the premium traffic and it should be measured for at least
3 different traffic parameter sets. Of course the best-effort traffic
does not require any reservation session entry.
Reporting format:
The reporting format is the same as in case of the premium traffic
delay test, except that the fifth column is for the best-effort
traffic instead of the premium traffic.
Note:
Measuring the best-effort traffic delay for multicast flows is not
necessary as the vast amount of the best-effort traffic is unicast
and considered to remain unicast in the future as multicast flows
usually requires QoS services.
8. Acknowledgement
We would like to thank the following individuals for their help in
designing and implementing the benchmarking framework presented in
this document: Joakim Bergkvist and Norbert Vegh from Telia Research
AB, Sweden, Gabor Kovacs, Anrdas Korn, Balazs Szabo from High Speed
Netwroks Laboratory of BUTE.
9. References
[1] S. Bradner, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, July 1991
[2] R. Mandeville, "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, February 1998
Feher, Cselenyi, Nemeth Expires January 2001 [Page 24]
INTERNET-DRAFT <draft-feher-benchresres-00.txt> July 2000
[3] Y. Bernet, et. al., "A Framework For Integrated Services
Operation Over Diffserv Networks", Internet Draft, May 2000,
<draft-ietf-issll-diffserv-rsvp-05.txt>
[4] S. Bradner, J. McQuaid, "Benchmarking Methodology for Network
Interconnect Devices", RFC 2544, March 1999
[5] B. Braden, Ed., et. al., "Resource Reservation Protocol (RSVP) -
Version 1 Functional Specification", RFC 2205, September 1997.
[6] J. Bergkvist, I. Cselenyi, "Boomerang Protocol Specification",
Internet Draft, June 1999, <draft-bergkvist-boomerang-spec-
00.txt>
[7] P. Pan, H. Schulzrinne, "YESSIR: A Simple Reservation Mechanism
for the Internet", Computer Communication Review, on-line
version, volume 29, number 2, April 1999
[8] L. Delgrossi, L. Berger, "Internet Stream Protocol Version 2
(ST2) Protocol Specification - Version ST2+", RFC 1819, August
1995
[9] D. Black, S. Blake, M. Carlson, E. Davies, Z. Wang and W. Weiss,
"An Architecture for Differentiated Services", RFC 2475,
December 1998
[10] Boomerang Team, "Boomerang homepage - Benchmarking Tools",
http://boomerang.ttt.bme.hu
10. Authors' Addresses:
Gabor Feher
Budapest University of Technology and Economics (BUTE)
Department of Telecommunications and Telematics
Pazmany Peter Setany 1/D, H-1117, Budapest,
Phone: +36 1 463-3110
Email: feher@ttt-atm.ttt.bme.hu
Istvan Cselenyi
Telia Research AB
Vitsandsgatan 9B
SE 12386, Farsta
SWEDEN,
Phone: +46 8 713-8173
Email: istvan.i.cselenyi@telia.se
Krisztian Nemeth
Budapest University of Technology and Economics (BUTE)
Department of Telecommunications and Telematics
Pazmany Peter Setany 1/D, H-1117, Budapest,
Phone: +36 1 463-2225
Email: nemeth_k@ttt-atm.ttt.bme.hu
Feher, Cselenyi, Nemeth Expires January 2001 [Page 25]
