[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Fwd: [Bug 62590] performance drop after moving from apache 2.2 to apache 2.4


Having a brief look at ERR_clear_error in openssl it looks like that various locking operations are performed to get the
error state. I assume that this causes a lot of contention in the load case.
The question is whether there is another solution to BZ62590 but to call ERR_clear_error before every read and write. So
far, I have no idea for an efficient check of not processed errors in the stack and only clear if they are present /
consider them correctly for further error processing.

Regards

Rüdiger


On 08/29/2018 09:16 AM, Ruediger Pluem wrote:
> I guess it would be helpful to know which openssl version was used for the test. There was a valid use case for BZ62590
> and the question is why ERR_clear_error is that expensive. But the expensiveness might depend on the used openssl version.
> 
> Regards
> 
> Rüdiger
> 
> 
> On 08/28/2018 04:54 PM, William A Rowe Jr wrote:
>> As we unwind various regressions and breakage, one non-lethal but somewhat horrid report stands out. Eric correctly tied
>> it to the patch applied for https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 in the 2.4.24 timeframe.
>>
>> Server Software:        Apache/2.2.34
>> SSL/TLS Protocol:       TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
>>
>> vs
>>
>> Server Software:        Apache/2.4.34SSL/TLS Protocol:       TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
>>
>> Measures out with
>>
>> Time taken for tests:   192.131 seconds
>> Total transferred:      731130414 bytes
>> HTML transferred:       88000000 bytes
>> Requests per second:    10409.59 [#/sec] (mean)
>> Time per request:       5.764 [ms] (mean)
>> Time per request:       0.096 [ms] (mean, across all concurrent requests)
>> Transfer rate:          3716.20 [Kbytes/sec] received
>>
>> vs
>>
>> Time taken for tests:   571.058 seconds
>> Total transferred:      689130083 bytes
>> HTML transferred:       90000000 bytes
>> Requests per second:    3502.27 [#/sec] (mean)
>> Time per request:       17.132 [ms] (mean)
>> Time per request:       0.286 [ms] (mean, across all concurrent requests)
>> Transfer rate:          1178.48 [Kbytes/sec] received
>>
>>
>>
>>
>> Connection Times (ms)
>>               min  mean[+/-sd] median   max
>> Connect:        0    0   0.4      0      87
>> Processing:     0    6   1.2      6      71
>> Waiting:        0    6   1.2      5      70
>> Total:          0    6   1.3      6      98
>>
>> Percentage of the requests served within a certain time (ms)
>>   50%      6
>>   66%      6
>>   75%      6
>>   80%      6
>>   90%      7
>>   95%      8
>>   98%      9
>>   99%     10
>>  100%     98 (longest request)
>>  
>> ----
>>
>> I did then the same for Apache/2.4.34 (with apr-1.6.3 and apr-util-1.6.1), with the following changes in the httpd.conf (including ssl-support):
>> StartServers         1
>> ServerLimit          1
>> ThreadLimit          2500
>> ThreadsPerChild      2500
>> ThreadStackSize      1048576 
>> MinSpareThreads      1
>> MaxSpareThreads      500
>> MaxRequestWorkers      2500
>> MaxConnectionsPerChild  0
>>
>>  
>> and here the output of ApacheBench:
>>
>> ab -k -n 2000000 -c 60 'https://adnvl005:44300/'
>> This is ApacheBench, Version 2.3 <$Revision: 655654 $>
>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
>> Licensed to The Apache Software Foundation, http://www.apache.org/
>>
>> Benchmarking adnvl005 (be patient)
>> Completed 200000 requests
>> Completed 400000 requests
>> Completed 600000 requests
>> Completed 800000 requests
>> Completed 1000000 requests
>> Completed 1200000 requests
>> Completed 1400000 requests
>> Completed 1600000 requests
>> Completed 1800000 requests
>> Completed 2000000 requests
>> Finished 2000000 requests
>>
>>
>> Connection Times (ms)
>>               min  mean[+/-sd] median   max
>> Connect:        0    0   2.1      0     208
>> Processing:     0   17  20.3     10     285
>> Waiting:        0   17  20.3     10     285
>> Total:          0   17  20.4     10     285
>>
>> Percentage of the requests served within a certain time (ms)
>>   50%     10
>>   66%     16
>>   75%     23
>>   80%     28
>>   90%     44
>>   95%     59
>>   98%     79
>>   99%     94
>>  100%    285 (longest request)
>>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: ** <bugzilla@xxxxxxxxxx <mailto:bugzilla@xxxxxxxxxx>>
>> Date: Mon, Aug 27, 2018 at 9:11 AM
>> Subject: [Bug 62590] performance drop after moving from apache 2.2 to apache 2.4
>> To: bugs@xxxxxxxxxxxxxxxx <mailto:bugs@xxxxxxxxxxxxxxxx>
>>
>>
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 <https://bz.apache.org/bugzilla/show_bug.cgi?id=62590>
>>
>> --- Comment #1 from paolo <paolo@xxxxxxxxxx <mailto:paolo@xxxxxxxxxx>> ---
>> After some debug-session I found out that  the problem are the ERR_clear_error
>> calls in ssl_filter_write and ssl_io_input_read. If I remove those calls the
>> performance is the same like with httpd/2.2.
>>
>> Are those calls really needed in the ssl_io_input_read/ssl_filter_write
>> function?
>> Isn't it enough to have it only in the ssl_io_filter_handshake function.
>>
>> Or what about to call this function only if an error occurred:
>>
>>         else /* (rc < 0) */ {
>>             int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc);
>>             conn_rec *c = (conn_rec*)SSL_get_app_data(inctx->filter_ctx->pssl);
>>
>>  +           ERR_clear_error();
>>
>>             if (ssl_err == SSL_ERROR_WANT_READ) {
>>
>>
>> Many thanks for any answer.
>>
>> -- 
>> You are receiving this mail because:
>> You are the assignee for the bug.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: bugs-unsubscribe@xxxxxxxxxxxxxxxx <mailto:bugs-unsubscribe@xxxxxxxxxxxxxxxx>
>> For additional commands, e-mail: bugs-help@xxxxxxxxxxxxxxxx <mailto:bugs-help@xxxxxxxxxxxxxxxx>
>>
>>
>