Apacheserver-status w sending reply
OS: CentOS 6.7; Apache/2.2.15; nginx/1.13.5
Symptoms: The number of httpd processes increases; at about 2250, the server starts returning 502 errors.
Check:
Tool: Apache Server Status Vizualization
SS values are very high in the W Sending Reply status.
Determine what the processes are doing (strace):
# strace -p 18404
Process 18404 attached
wait4(18772, ^CProcess 18404 detached
# strace -p 18772
Process 18772 attached
restart_syscall(<... resuming interrupted call ...>) = 0
poll([{fd=29, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
poll([{fd=29, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 0 (Timeout)
poll([{fd=29, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
poll([{fd=29, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 0 (Timeout)
poll([{fd=29, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
poll([{fd=29, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000^CProcess 18772 detached
# lsof -p 18772 -ad 29
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
httpd.itk 18772 domain_user 29u IPv4 14414303 0t0 TCP domain.com:54646->static.9.20.40.188.clients.your-server.de:http (ESTABLISHED)
Next, identify which PHP scripts on the server request data from static.9.20.40.188.clients.your-server.de.
Search using any available method.
Found it:
/**.
* Function TWS_INC_SALESDOUBLER_order_success
**/
function TWS_INC_SELLACTION_order_success($mOrderId, $mAmount)
{
if (isset($_COOKIE[SELLACTION_COOKIE_NAME]))
{
$mOrderId = 99999 + $mOrderId;
$url = "http://sellaction.net/reg.php?id=".$_COOKIE[SELLACTION_COOKIE_NAME].'-'.SELLACTION_TARIFF_ID.'_'.$mAmount.'&order_id='.$mOrderId;
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => $url
));
$resp = curl_exec($curl);
curl_close($curl);
}
}
This is the culprit. Remove or disable it.