connectionIdlePollingInterval delays requests for the specified time

[martin-tarjanyi]

Describe the bug

When client is setup with connectionIdlePollingInterval, it causes delay for the response to become available.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected behavior

No additional delay regardless of the value of connectionIdlePollingInterval

Current behavior

There is an additional delay for each request which corresponds with the connectionIdlePollingInterval setting.

Steps to Reproduce

package com.example.playground

import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.headObject
import aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine
import kotlin.time.Duration.Companion.milliseconds
import kotlin.time.measureTime

suspend fun main() {
    val bucket = "my-bucket"
    val key = "my-key"

    S3Client {
        region = "us-west-2"
        httpClient(OkHttpEngine) {
            connectionIdlePollingInterval = 500.milliseconds
        }
    }.use { client ->
        repeat(5) {
            measureTime {
                client.headObject {
                    this.bucket = bucket
                    this.key = key
                }
            }.also { println("Request duration: $it") }
        }
    }
}

Example output:

Request duration: 3.028859417s
Request duration: 725.529083ms
Request duration: 751.692167ms
Request duration: 722.700250ms
Request duration: 728.030250ms

Example output without connectionIdlePollingInterval:

Request duration: 2.925923958s
Request duration: 232.163250ms
Request duration: 231.357042ms
Request duration: 225.145542ms
Request duration: 236.862875ms

We can discard the first request as warmup.

Possible Solution

No response

Context

I'm trying to apply the connectionIdlePollingInterval setting to prevent intermittent connection errors on production. However, it causes noticeable delay.

In addition I think it would be nice to make the hardcoded Dispatchers.IO configurable in ConnectionMonitoringEventListener as applications often use this dispatcher for other purposes which could cause some conflicts. Alternatively, it could be configured using limitedParallelism which would create a dedicated pool for the connection monitor. Note: this probably wouldn't solve the issue in itself, this is just an additional observation that I made while debugging the issue.

AWS SDK for Kotlin version

1.5.62

Platform (JVM/JS/Native)

Java 17/21

Operating system and version

macOs 14.8.2

[ianbotsf]

Hello @martin-tarjanyi, you are correct that the connection idle polling interval can cause slowdowns in request throughput.

Specifically, the connection idle poller uses the JVM sockets API to attempt reading from the connection to determine whether it's been closed remotely. Operations on that API are blocking and so the polling interval is used as a timeout for the speculative reads. This means that OkHttp cannot use the connection until the socket read has either failed (indicating remote closure) or timed out (indicating the connection is still valid).

My suggestion is to use a lower polling interval such as 50ms, to disable connection polling altogether, or to switch to using a different HTTP engine like CRT.

On the note about hardcoded Dispatchers.IO, we plan to address that as part of smithy-lang/smithy-kotlin#1448.

Does that address your concerns?