To whom it might concern,
We have observed bandwidth problems of TCP when running benchmarks which
gradually increase its payload and having (most of its) data flowing
unidirectional. The degradation is observed between two dual XEON/E7500
machines, using Gbe and a cross-over cable. The Gbe is eth1 and has an
MTU of 9000. eth0 is a FE with an MTU of 1500. The machines is running
linux 2.4.20, and the NIC in question is an Intel Corp. 82544GC Gigabit
Ethernet Controller (rev 02).
During an attempt to change TCP parameter settings, and also
systematically changing socket rx/tx buffer sizes, I discovered that once
in a while, the benchmark ran well. The OK run was not deterministic, and
had nothing to do with change of the parameter settings. Hence, I let the
machine work during the week-end, and took tcpdumps which I saved for the
successful run. An analysis of the "OK" run vs. the
"BAD" run, discovers a couple of interesting things. The
problem is that the advertised window of the receiver does not increase.
The second problem, which also affects the "OK" run, is that
the ratio of packets sent (from src to dst) and the number of packets
received (#advertisements) is 1 for the "BAD" scenario
(actually this is probably a consequence of the window being only ~2xMTU
size). Surprisingly, it is as large as 0.5 for the "OK"
scenario. IMHO, this violates RFC813, which states:
- There are two reasons for prompt
acknowledgement. One is to
- prevent retransmission. We will discuss later how to
determine whether
- unnecessary retransmission is
occurring. The other
reason one
- acknowledges promptly is to permit further data to
be sent. However,
- the previous section makes quite clear that it is not
always desirable
- to send a little bit of data, even though the receiver may have room
for
- it. Therefore, one can
state a general rule that under normal
- operation, the receiver of data need not,
and for efficiency reasons
- should not, acknowledge the data unless either the
acknowledgement is
- intended to produce an increased useable window
, is
necessary in order
- to prevent retransmission or is
being sent as part of a reverse
- direction segment being sent for some other reason. We will
consider an
- algorithm to achieve these goals.
The two tcpdumps has been analyzed by a simple awk scripts, which
gives information for every 1/10 of a second of the runtime:
time: elapsed time relative to the first packet
MB/s: sum of TCP payload (based on TCP sequence numbers) *1e-6 / delta
time
avg_len: average packet length (TCP payload) sent
nsent: no of packets sent from src to dst
avg_win: average window size advertised by the src
adv/sent: ratio between #advertisements (sum of prompt acks and
piggybacked acks) and #packets sent
The extract of the analysis is enclosed as "ok.txt" and
"bad.txt". Also, the information is presented as graphs in
"bw_winsiz.png" and "bw_adv-sent.png". In the graphs,
the "OK" data uses the bottom x-axis, the "BAD" data
uses the upper x-axis (the "BAD" run-time is longer due to the
lower bandwidth). Bandwidth uses the left y-axes, whereas the average
advertised window size and the ratio of the #advertisements and #packets
sent uses the rightmost x-axis.
I do hope this information is useful 4u and that the problem can be
fixed. Since I do not read the mailing lists, I would appreciate a note
back by email of someone triggers on this.
Cheers, Håkon
--
Håkon Bugge; VP Product Development; Scali AS;
mailto:hob@xxxxxxxx;
http://www.scali.com;
fax: +47 22 62 89 51;
Voice: +47 22 62 89 50; Cellular (Europe+US): +47 924 84 514;
Visiting Addr: Olaf Helsets vei 6, Bogerud, N-0621 Oslo, Norway;
Mail Addr: Scali AS, Postboks 150, Oppsal, N-0619 Oslo,
Norway;
bw_adv-sent.png
Description: PNG image
bad.txt
Description: Text document
bw_winsiz.png
Description: PNG image
ok.txt
Description: Text document
|
