-----Original Message-----
From: Poretsky, Scott
Sent: Saturday, October 21, 2006 7:43 AM
To: 'Tom Alexander'; shankar.rao@xxxxxxxxx
Cc: 'Al Morton'
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts
Thanks Tom! Responses for the comments on the Terminology are below.
Scott
####################
------------------------------------------------------------------------
1. Introduction
Routers in an operational network are simultaneously configured with
multiple protocols and security policies while forwarding traffic and
being managed. To accurately benchmark a router for deployment it is
necessary to test that router in operational conditions by
simultaneously configuring and scaling network protocols and security
policies, forwarding traffic, and managing the device. It is helpful
to accelerate these network operational conditions so that the
router under test can be benchmarked with faster test duration.
Testing a router in accelerated network conditions is known as
Accelerated Stress Benchmarking.
[Tom A: The above paragraph is essential to understanding the rationale
for the rest of the draft, but is somewhat confusing. Suggest rewriting
above paragraph as:
"Routers in an operational network are configured with multiple
protocols
and security policies while simultaneously forwarding traffic and
being managed. To accurately benchmark a router prior to deployment, it
is necessary to test that router under operational conditions by
configuring and scaling network protocols and security policies, and
simultaneously forwarding traffic as well as managing the device.
Scott> Added the 3 words changes above
[Tom A:
configuration and management represent relatively infrequent activities
during actual operation
Scott> This statement is incorrect.
[Tom A:
(compared to traffic forwarding), it is useful
to artificially overstress (accelerate) these network operational
conditions so that the router under test can be benchmarked with lower
test durations.
Scott> Partially agree. Now using the sentence "It is useful to
accelerate these network operational conditions so that the router under
test can be benchmarked with a shorter test duration."
------------------------------------------------------------------------
This document provides the Terminology for performing Stress
Benchmarking of networking devices. The three phases of the Stress
Test: Startup, Instability and Recovery are defined along with the
benchmark and configuration terms associated with the each phase.
Benchmarks for stress testing are defined using the Aggregate
Forwarding Rate and control plane Session Count during each phase
of the test. For each plane, the Configuration Set, Startup
Conditions, and Instability Conditions are defined. Also defined are
the Benchmark Planes fundamental to stress testing configuration,
setup and measurement. These are the Control Plane, Data Plane,
Management Plane and Security Plane Multiple benchmarks are made
for each Benchmark Plane during each Phase. Benchmarks can be
compared across multiple planes for the same DUT or at the same
plane for 2 or more DUTS. These benchmarks White Box benchmarks
are provided in Appendix 1 for additional DUT behavior
measurements. The terminology is to be used with the companion
methodology document [4]. The sequence of phases, actions, and
benchmarks are shown in Table 1.
[Tom A: Suggest rewording above sentence beginning with "These
benchmarks
White Box benchmarks ..." as follows:
"Benchmarks of internal DUT characteristics such as memory and CPU
utilization (also known as White Box benchmarks) are described in
Appendix 1, to allow additional characterization of DUT behavior."]
Scott> Agree
------------------------------------------------------------------------
Table 1. Phase Sequence and Benchmarks
III. Recovery Phase II. Instability Phase I. Startup Phase
<-----------------<---<-------------------<----<--------------<
Remove Instability Achieve Configuration Apply Startup
Conditions Set Conditions
Benchmark: Benchmark: Benchmark:
Recovered Aggregate Unstable Aggregate Stable Aggregate
Forwarding Rate Forwarding Rate Forwarding Rate
Degraded Aggregate
Forwarding Rate
Average Degraded
Forwarding Rate
Recovered Latency Unstable Latency Startup Latency
Recovered Uncontrolled Recovered Uncontrolled Stable Session Count
Sessions Lost Sessions Lost
Recovery Time
[Tom A: Is there some reason why the above table reads from right to
left? It would be more logical for it to read from left to right.]
Scott> Yes. The BMWG has discussed this on previous occasions. Test
Equipment displays time on a scale that scrolls right to left. We
want to match that.
------------------------------------------------------------------------
3. Term definitions
3.1 General Terms
3.1.1 Benchmark Planes
Definition:
The features, conditions, and behavior for the Accelerated Stress
Benchmarking.
Discussion:
There are four Benchmark Planes: Control Plane, Data Plane,
Management Plane, and Security Plane as shown in Figure 1. The
Benchmark Planes define the Configuration, Startup Conditions,
Instability Conditions, and Failure Conditions used for the test.
[Tom A: The last sentence does not make sense. Benchmark planes cannot
define the conditions; instead, conditions are defined for each plane.
Suggest rewording as:
"Configuration, Startup Conditions, Instability Conditions, and
Failure Conditions used for each test are defined for each of these
four Benchmark Planes."]
Scott> Agree
------------------------------------------------------------------------
3.1.2 Configuration Sets
Definition:
The features and scaling limits used during the Accelerated
Stress
Benchmarking.
Discussion:
There are four Configuration Sets: Control Plane Configuration
Set,
Data Plane Configuration Set, Management Plane Configuration
Set,
and Security Plane Configuration Set.
[Tom A: The above discussion does not help me understand what a
Configuration Set is. Please consider adding text to clarify. A
couple of examples of Configuration Sets would be MOST helpful.]
Scott> Examples are in the methodology. Al requested to have one
mandatory minimum configuration set. I will add that here to also
satisfy your request for an example.
------------------------------------------------------------------------
3.1.6 Controlled Session Loss
Definition:
Control Plane sessions that are intentionally brought
down during the Stress test.
Discussion:
The test equipment is able to control protocol
session state with the DUT.
[Tom A: The above discussion specifies some capability of the
test equipment, and does not clarify what Controlled Session
Loss is. Suggest rewording as follows:
"Controlled Session Loss is performed during the test in order
to stress the DUT by forcing it to tear down Control Plane
sessions while handling traffic. It is assumed that the test
equipment is able to control protocol session state with the
DUT and is therefore able to introduce Controlled Session Loss."]
Scott> Agree
------------------------------------------------------------------------
3.2.3 Management Plane
Definition:
The Management features and tools used for the
Accelerated Stress Benchmarking.
Poretsky and Rao [Page 8]
INTERNET-DRAFT Terminology for Accelerated June 2006
Stress Benchmarking
Discussion:
A key component of the Accelerated Stress Benchmarking is the
Management Plane to assess manageability of the router
under stress. The Management Plane defines the Configuration,
Startup Conditions, and Instability Conditions of the
management protocols and features. The Management Plane
includes SNMP, Logging/Debug, Statistics Collection, and
management configuration sessions such as telnet, SSH, and
serial console. SNMP Gets SHOULD be performed continuously.
Management configuration sessions should be open
simultaneously and be repeatedly open and closed. Open
management sessions should have valid and invalid
configuration and show commands entered.
[Tom A: It is difficult, in the above discussion, to determine where
terminology ends and methodology takes up. It appears that the
discussion is trying to specify methodology together with terminology.
Suggest moving the last 2 sentences (beginning with "SNMP Gets ..."
into the methodology.]
Scott> Agree
------------------------------------------------------------------------
3.2.4 Security Plane
Definition:
The Security features used during the Accelerated Stress
Benchmarking.
Discussion:
The Security Plane defines the Configuration, Startup
Conditions, and Instability Conditions of the security
features and protocols. The Security Plane includes the
ACLs, Firewall, Secure Protocols, and User Login. Tunnels
for those such as IPsec should be established and flapped.
Policies for Firewalls and ACLs should be repeatedly added
and removed from the configuration via telnet, SSH, or
serial management sessions.
[Tom A: Same issue as for Management Plane: the above discussion
mixes terminology and methodology. Suggest moving the last 2
sentences (starting with "Tunnels for ...") to the methodology.
Scott> Agree.
------------------------------------------------------------------------
3.3.5.1 Management Plane Configuration Set
Definition:
The router management features enabled for the
Accelerated Stress Benchmark.
Discussion:
A key component of the Accelerated Stress Benchmark is the
Management Configuration Set to assess manageability of the
router under stress. The Management Configuration Set defines
the management configuration of the DUT. Features that are
part of the Management Configuration Set include access, SNMP,
Logging/Debug, and Statistics Collection, and services such as
FTP, as shown in Figure 3. These features should be enabled
throughout the Stress test. SNMP Gets should be made
continuously with multiple FTP and Telnet sessions operating
simultaneously. FTP sessions should be opened and closed at
varying intervals and get and put files while open. Telnet
sessions should be opened and closed at varying intervals and
enter valid and invalid show and configuration commands while
open.
[Tom A: Same issue as before - methodology is mixed in with the
terminology. Suggest moving the last 4 sentences (starting with
"These features should be enabled ...") to the methodology.]
Scott> Agree.
------------------------------------------------------------------------
Appendix 1. White Box Benchmarking Terminology
Minimum Available Memory
Definition:
Minimum DUT Available Memory during the duration of the
Accelerated Stress Benchmark.
Discussion:
It is necessary to monitor DUT memory to measure this
benchmark.
[Tom A: It is nice that one has to monitor DUT memory to measure
this benchmark, but what is the significance of this benchmark?
Please explain. Even a sentence like "This benchmark enables the
assessment of reserve capacity in the DUT, as well as capacity
degradation over repeated trials" would be fine.]
Scott> Used the sentences, "This benchmark enables the assessment of
resources in the DUT. It is necessary to monitor DUT memory to measure
this benchmark."
Please note that when benchmarking "Minimum Available Memory" it is
necessary
to monitor DUT memory .
------------------------------------------------------------------------
Maximum CPU Utilization
Definition:
Maximum DUT CPU utilization during the duration of the
Accelerated Stress Benchmark.
Discussion:
It is necessary to monitor DUT CPU Utilization to measure
this benchmark.
[Tom A: Same comment as for previous White Box benchmark.]
Scott> Used the sentences, " This benchmark enables the assessment of
resources in the DUT. It is necessary to monitor DUT CPU Utilization
to measure this benchmark." Please note that when benchmarking "Maximum
CPU Utilization" it is necessary to monitor the DUT CPU.
-----Original Message-----
From: Tom Alexander [
mailto:tom@xxxxxxxxxxxx]
Sent: Saturday, October 21, 2006 3:47 AM
To: Poretsky, Scott; shankar.rao@xxxxxxxxx
Cc: 'Al Morton'
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts
Scott, Shankar,
I read the accelerated benchmarking drafts and have some comments, in
the
enclosed files. As I'm a relative newbie in BMWG I didn't feel
comfortable
sending them to the whole list (when you're about to put your foot in
your
mouth, keeping the audience small is a good idea!) so I'm sending them
directly to you instead.
Note that I've excerpted pieces of the drafts and added my comments
below,
using the general syntax "[Tom A. - ..... ]".
Best regards,
- Tom