Allow to pass botocore session to REST Catalog (S3 Tables) like Glue Catalog does

Feature Request / Improvement

Background
Glue catalogs allows to setup access credentials when creating boto3.Session. Unfortunately, REST Catalog is able to use default session only in class SigV4Adapter(HTTPAdapter):.
Ideally exactly the same mechanism could be used here (OFC with constants renamed)

session = boto3.Session(
                profile_name=properties.get(GLUE_PROFILE_NAME),
                region_name=get_first_property_value(properties, GLUE_REGION, AWS_REGION),
                botocore_session=properties.get(BOTOCORE_SESSION),
                aws_access_key_id=get_first_property_value(properties, GLUE_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
                aws_secret_access_key=get_first_property_value(properties, GLUE_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
                aws_session_token=get_first_property_value(properties, GLUE_SESSION_TOKEN, AWS_SESSION_TOKEN),
            )

Implementation
I'm happy to implement the change. I have it working locally. I can also remove some of the code duplication (FileIO also creates boto session - it looks as above)

Pls tell me if you are interested in PR

@rudolfix

@rudolfix what's the implication of how it currently works? because I am seeing some weird iam errors when writing data in sometimes w/lambda, not sure if related.

@rudolfix what's the implication of how it currently works? because I am seeing some weird iam errors when writing data in sometimes w/lambda, not sure if related.

I can't easily configure S3 Table access credentials - I have to provide default AWS profile in .aws or AWS env variables. If that is not done then I get this:

  File "/home/rudolfix/src/dlt-plus/playground/demo_aws_catalogs/error_catalog_credentials.py", line 24, in <module>
    raise e
  File "/home/rudolfix/src/dlt-plus/playground/demo_aws_catalogs/error_catalog_credentials.py", line 22, in <module>
    rest_catalog = RestCatalog(name="s3tables", uri=uri, **properties)
  File "/home/rudolfix/src/iceberg-python/pyiceberg/catalog/rest/__init__.py", line 265, in __init__
    self._fetch_config()
  File "/home/rudolfix/src/iceberg-python/pyiceberg/catalog/rest/__init__.py", line 382, in _fetch_config
    response = session.get(self.url(Endpoints.get_config, prefixed=False), params=params)
  File "/home/rudolfix/src/dlt-plus/.venv/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/home/rudolfix/src/dlt-plus/.venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/rudolfix/src/dlt-plus/.venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/rudolfix/src/dlt-plus/.venv/lib/python3.10/site-packages/requests/adapters.py", line 641, in send
    self.add_headers(
  File "/home/rudolfix/src/iceberg-python/pyiceberg/catalog/rest/__init__.py", line 484, in add_headers
    credentials = boto_session.get_credentials().get_frozen_credentials()
AttributeError: 'NoneType' object has no attribute 'get_frozen_credentials'

we use pyiceberg from our library (dlt) where many configurations should coexist as the same time and such "hardcoded" session makes that harder

Hello!
We're encountering the same issue using PyIceberg to access AWS S3 tables with the AWS S3 Table REST Catalog. Our use case involves multiple clients accessing different S3 tables concurrently, so we need PyIceberg to support dynamically specifying credentials in code.

@rudolfix

Hi @rudolfix , just wonder do you still have plans to implement this change?

looks like it is implemented here: #2071 and this exactly what I planned to do!

Robustify boto3 session handling (DynamoDB, RestCatalog) (apache#2071)

Navigation Menu

Allow to pass botocore session to REST Catalog (S3 Tables) like Glue Catalog does #2008

Feature Request / Improvement

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow to pass botocore session to REST Catalog (S3 Tables) like Glue Catalog does #2008

Description

Feature Request / Improvement

Activity

potatochipcoconut commented on May 17, 2025

rudolfix commented on May 18, 2025

YangKian commented on Jun 5, 2025

YangKian commented on Jun 6, 2025

rudolfix commented on Jun 9, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions