Skip to content

Fix up S3 repo logging docs #4301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DaveCTurner wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Fix up S3 repo logging docs #4301

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
90bef4e
Fix up S3 repo logging docs
DaveCTurner Dec 11, 2025
6bf376f
Nested tabs?
DaveCTurner Dec 11, 2025
54d3aba
WARNING instead of nested tabs
DaveCTurner Dec 11, 2025
64f5ddd
Fix
DaveCTurner Dec 11, 2025
0fbaaf7
9.1 docs should be match the 9.2+ ones
DaveCTurner Dec 11, 2025
d295b0f
Merge branch 'main' into 2025/12/11/repo-s3-logging
theletterf Dec 12, 2025
5326ef6
Merge branch 'main' into 2025/12/11/repo-s3-logging
DaveCTurner Dec 16, 2025
d77854f
Update deploy-manage/tools/snapshot-and-restore/s3-repository.md
DaveCTurner Dec 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 76 additions & 4 deletions deploy-manage/tools/snapshot-and-restore/s3-repository.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -405,11 +405,12 @@ $$$using-minio-with-elasticsearch$$$
The performance, reliability, and durability of a MinIO-backed repository depend on the properties of the underlying infrastructure and on the details of your MinIO configuration. You must design your storage infrastructure and configure MinIO in a way that ensures your MinIO-backed repository has performance, reliability, and durability characteristics which match AWS S3 in order for it to be fully S3-compatible. If you need assistance with your MinIO configuration, please contact the MinIO support team.
::::

Most storage systems can be configured to log the details of their interaction with {{es}}. If you are investigating a suspected incompatibility with AWS S3, it is usually simplest to collect these logs and provide them to the supplier of your storage system for further analysis. If the incompatibility is not clear from the logs emitted by the storage system, configure {{es}} to log every request it makes to the S3 API by [setting the logging level](/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md) of the `com.amazonaws.request` logger to `DEBUG`.
Most storage systems can be configured to log the details of their interaction with {{es}}. If you are investigating a suspected incompatibility with AWS S3, it is usually simplest to collect these logs from your storage system and provide them to the supplier of your storage system for further analysis. Contact the supplier of your storage system for advice on how to configure it to log requests sufficiently verbosely for this troubleshooting.

To prevent leaking sensitive information such as credentials and keys in logs, {{es}} rejects configuring this logger at high verbosity unless [insecure network trace logging](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#http-rest-request-tracer) is enabled. To do so, you must explicitly enable it on each node by setting the system property `es.insecure_network_trace_enabled` to `true`.
::::::{applies-switch}

Once enabled, you can configure the `com.amazonaws.request` logger:
:::::{applies-item} { "stack": "ga 9.0" }
If the incompatibility is not clear from the logs emitted by the storage system, configure {{es}} to log every request it makes to the S3 API by [setting the logging level](/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md) of the `com.amazonaws.request` logger to `DEBUG`:

```console
PUT /_cluster/settings
Expand All @@ -420,7 +421,78 @@ PUT /_cluster/settings
}
```

Collect the {{es}} logs covering the time period of the failed analysis from all nodes in your cluster and share them with the supplier of your storage system along with the analysis response so they can use them to determine the problem. See the [AWS Java SDK](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html) documentation for further information, including details about other loggers that can be used to obtain even more verbose logs. When you have finished collecting the logs needed by your supplier, set the logger settings back to `null` to return to the default logging configuration and disable insecure network trace logging again. See [Logger](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-logger) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) for more information.
To prevent leaking sensitive information such as credentials and keys in logs, {{es}} rejects configuring this logger at high verbosity unless [insecure network trace logging](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#http-rest-request-tracer) is activated. To do so, you must explicitly configure it on each node by setting the system property `es.insecure_network_trace_enabled` to `true`.

Collect the {{es}} logs covering the time period of the failed analysis from all nodes in your cluster and share them with the supplier of your storage system along with the analysis response so they can use them to determine the problem. Refer to [Logging AWS SDK for Java Calls](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html) for further information, including details about other loggers that can be used to obtain even more verbose logs.
:::::

:::::{applies-item} { "stack": "ga 9.1" }
Copy link
Collaborator

@shainaraskas shainaraskas Dec 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest using just one tab for 9.1+, and simplify the note:

:::{warning}
In the following patch versions of {{es}}, it is not possible to obtain more detailed logs from the AWS Java SDK. Use the logs from the storage system itself, or upgrade to a newer version of {{es}}.

* 9.1.0 to 9.1.9
* 9.2.0 to 9.2.3
:::

officially (and this was true prior to this docs system as well), we don't really document patch-level changes like this, because features shouldn't really be introduced in patches per semver. If you think this note is critical and would avoid significant support debt, we can definitely still include it.

Copy link
Contributor Author

@DaveCTurner DaveCTurner Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could if you're sure but my concern is that we'd then have this warning in place for the rest of the 9.x series and as time passes it will become relevant to fewer and fewer users. Users are already pretty bad at ignoring even relevant admonitions so I don't think it's good psychology to have irrelevant ones too.

This isn't a new feature, it's a bug fix. I know we generally don't call out bugs but I think it'll cause frustration to try these instructions in a version with the bug and see no results nor any error and it won't be a great user experience to contact support and be pointed to the known-and-fixed bug report hidden in the release notes.

Copy link
Collaborator

@shainaraskas shainaraskas Dec 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can tag the admonition with 9.1 and 9.2 to draw the relevant users' eyes as well. that's best done with the ranges thing we are juuuust waiting to be merged, but this syntax below should work right now

I believe people will likely just try do do this without seeing the admonition anyway, so the admonition is more there to catch them when it errors out and they read a little more deeply.

:::{warning}
:applies_to: {"stack": "ga 9.1, ga 9.2"}
In the following patch versions of {{es}}, it is not possible to obtain more detailed logs from the AWS Java SDK. Use the logs from the storage system itself, or upgrade to a newer version of {{es}}.

* 9.1.0 to 9.1.9
* 9.2.0 to 9.2.3
:::

Copy link
Contributor Author

@DaveCTurner DaveCTurner Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when it errors out

Sadly it doesn't error out, it just silently does nothing at all in the buggy versions so you can't tell whether you've done the config right or not.

But ok I'll trim this down to a 9.1-only tab with the warning and we can refine it soon. I guess eventually we could reduce it from a warning to a notice once these buggy versions are far enough behind us.


::::{warning}
In versions of {{es}} 9.1 prior to 9.1.9 it is not possible to obtain more detailed logs from the AWS Java SDK. Use the logs from the storage system itself, or upgrade to a newer version of {{es}}.
::::

If the incompatibility is not clear from the logs emitted by the storage system, configure {{es}} to log every request it makes to the S3 API by [setting the logging level](/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md) of the `software.amazon.awssdk.request` logger to `DEBUG`:

```console
PUT /_cluster/settings
{
"persistent": {
"logger.software.amazon.awssdk.request": "DEBUG"
}
}
```
Comment on lines +437 to +444
Copy link
Collaborator

@shainaraskas shainaraskas Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as an even more lightweight alternative, consider just creating tabs for the code snippet, and skipping the logger name in the preamble


To prevent leaking sensitive information such as credentials and keys in logs, {{es}} rejects configuring this logger at high verbosity unless [insecure network trace logging](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#http-rest-request-tracer) is enabled. To do so, you must explicitly enable it on each node by setting the system property `es.insecure_network_trace_enabled` to `true`.

Collect the {{es}} logs covering the time period of the failed analysis from all nodes in your cluster and share them with the supplier of your storage system along with the analysis response so they can use them to determine the problem. Refer to [Logging with the SDK for Java 2.x](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/logging-slf4j.html) for further information, including details about other loggers that can be used to obtain even more verbose logs. When configuring other loggers, note that {{es}} configures the AWS Java SDK to use the `ApacheHttpClient` synchronous HTTP client.

:::::

:::::{applies-item} { "stack": "ga 9.2" }

::::{warning}
In versions of {{es}} 9.2 prior to 9.2.3 it is not possible to obtain more detailed logs from the AWS Java SDK. Use the logs from the storage system itself, or upgrade to a newer version of {{es}}.
::::

If the incompatibility is not clear from the logs emitted by the storage system, configure {{es}} to log every request it makes to the S3 API by [setting the logging level](/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md) of the `software.amazon.awssdk.request` logger to `DEBUG`:

```console
PUT /_cluster/settings
{
"persistent": {
"logger.software.amazon.awssdk.request": "DEBUG"
}
}
```

To prevent leaking sensitive information such as credentials and keys in logs, {{es}} rejects configuring this logger at high verbosity unless [insecure network trace logging](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#http-rest-request-tracer) is enabled. To do so, you must explicitly enable it on each node by setting the system property `es.insecure_network_trace_enabled` to `true`.

Collect the {{es}} logs covering the time period of the failed analysis from all nodes in your cluster and share them with the supplier of your storage system along with the analysis response so they can use them to determine the problem. Refer to [Logging with the SDK for Java 2.x](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/logging-slf4j.html) for further information, including details about other loggers that can be used to obtain even more verbose logs, noting that {{es}} configures the AWS Java SDK to use the `ApacheHttpClient` synchronous HTTP client.

:::::

:::::{applies-item} { "stack": "ga 9.3" }

If the incompatibility is not clear from the logs emitted by the storage system, configure {{es}} to log every request it makes to the S3 API by [setting the logging level](/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md) of the `software.amazon.awssdk.request` logger to `DEBUG`:

```console
PUT /_cluster/settings
{
"persistent": {
"logger.software.amazon.awssdk.request": "DEBUG"
}
}
```

To prevent leaking sensitive information such as credentials and keys in logs, {{es}} rejects configuring this logger at high verbosity unless [insecure network trace logging](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#http-rest-request-tracer) is enabled. To do so, you must explicitly enable it on each node by setting the system property `es.insecure_network_trace_enabled` to `true`.

Collect the {{es}} logs covering the time period of the failed analysis from all nodes in your cluster and share them with the supplier of your storage system along with the analysis response so they can use them to determine the problem. Refer to [Logging with the SDK for Java 2.x](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/logging-slf4j.html) for further information, including details about other loggers that can be used to obtain even more verbose logs, noting that {{es}} configures the AWS Java SDK to use the `ApacheHttpClient` synchronous HTTP client.
:::::

::::::

When you have finished collecting the logs needed by your supplier, set the logger settings back to `null` to return to the default logging configuration and deactivate insecure network trace logging again. Refer to [Logger](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-logger) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) for more information.


## Linearizable register implementation [repository-s3-linearizable-registers]
Expand Down
Loading