Payload limit configuration and validation #1288

jmaeagle99 · 2026-01-20T22:02:27Z

What was changed

Update the SDK to check the size of payload collections and issue warnings and errors when the size exceeds limits:

Exceeding the error threshold will raise a PayloadsSizeError. When running within a worker, this exception is caught, logged as a warning, and the corresponding workflow task or activity is failed.
Exceeding the warning threshold will log a warning.

This is done by:

Create a PayloadLimitsConfig class for configuring the warning and error limits. The error limit are not defined by default whereas the warning limit is set to 512 KiB.
Add a payload_limits property to DataConverter of type PayloadLimitsConfig.
Add non-public helper methods to DataConverter for encoding and decoding payloads in all forms. The encoding methods call _validate_payload_limits before returning.
The rest of the SDK is updated to use the non-public helper methods on DataConveter instead of using payload_codec directly.

Examples

Log output when an activity attempts to return a result that exceeds the error limit:

13:26:25 [ WARNING] [TMPRL1103] Attempted to upload payloads with size that exceeded the error limit. Size: 12325 bytes, Limit: 5120 bytes ({'activity_id': '1', 'activity_type': 'large_payload_activity', 'attempt': 3, 'namespace': 'default', 'task_queue': 'b6a43612-b0dc-4e74-9fc8-f112bb64f264', 'workflow_id': 'workflow-e645f290-513d-44a1-8a7e-6ef8d72d63e4', 'workflow_run_id': '019bdd4d-3268-7395-bb2e-451b692ec217', 'workflow_type': 'LargePayloadWorkflow'}) (_activity.py:387)

Log output when a workflow attempts to provide an activity input that exceeds the error limit:

16:20:22 [ WARNING] [TMPRL1103] Attempted to upload payloads with size that exceeded the error limit. Size: 12346 bytes, Limit: 5120 bytes (_workflow.py:371)

Note that the above example is missing the extra context that the activity result failure example has. This is due to the available logging infra where these errors are raised and can be fixed separately with some log refactoring (see deferred items).

Log output when a workflow attempts to provide activity input that is above the warning threshold but below the error limit:

16:24:46 [ WARNING] [TMPRL1103] Attempted to upload payloads with size that exceeded the error limit. Size: 4154 bytes, Limit: 1024 bytes (converter.py:1291)

Same note about the missing extra context.

Deferred

Work that has been deferred to later PRs (unless requested to pull back in):

Use a new WorkflowTaskFailedCause to indicate the specific cause of the failure scenario. Pending Add a new workflow failure cause for oversized payloads api#697, integration into sdk-core, and sdk-core into sdk-python.
Use a context-aware logger into _validate_payload_limits to get rich information about the execution context when issuing a warning.
Use a context-aware logger in _WorkflowWorker::_handle_activation to get rich information about the execution context when issuing a warning when exceeding the payload error limit.
~~Get payload error limits from server to be used as defaults for workers.~~

Why?

Users need to know when payload sizes are approaching or have exceeded size limits. This will help prevent workflow outages and inform users to adjust their workflows to make use of alternate storage methods or to break down their payloads more granularly.

Checklist

Closes Warn if the SDK tried to send a payload above a specific size #1284
Closes SDK should fail workflow task if payloads size it known to be too large #1285
How was this tested: Unit tests
Any docs updates needed? Yes

temporalio/converter.py

temporalio/exceptions.py

temporalio/worker/_activity.py

temporalio/converter.py

temporalio/worker/_activity.py

temporalio/exceptions.py

temporalio/converter.py

temporalio/exceptions.py

temporalio/converter.py

tests/worker/test_workflow.py

temporalio/converter.py

…rors

cretz

Looks great, this is much cleaner IMO than before, though I still think server limits should be wholly unrelated to SDK ones

cretz · 2026-01-28T19:59:28Z

temporalio/worker/_worker.py

+        namespace_info = await self._bridge_worker.validate()
+        payload_error_limits = (
+            _PayloadErrorLimits(
+                memo_upload_error_limit=namespace_info.Limits.memo_size_limit_error,
+                payload_upload_error_limit=namespace_info.Limits.blob_size_limit_error,
+            )
+            if namespace_info.HasField("limits")
+            else None
+        )


Just want to make the general comment that I still think it's confusing for users to understand that we have different error limits/approaches for clients and workers for the same payloads. We're ok w/ server validated sometimes and SDK validated other times in an inconsistent manner.

I still think it would have been worth exploring the alternative of divorcing SDK-side limits from server ones, which only has negative effect for self-hosted users, but we are already breaking compatibility for even cloud users with proxies.

cretz · 2026-01-28T20:50:12Z

temporalio/converter.py

+    memo_upload_warning_limit: int = 2 * 1024
+    """The limit (in bytes) at which a memo size warning is logged."""
+
+    payload_upload_error_disabled: bool = False


Hrmm, reviewing how this works, I was going to first suggest that be called worker_payload_upload_error_disabled (because it has nothing to do with general payload upload), but now I'm wondering if this specific error option should be a worker option, not a data converter one. And if set, just ignore the namespace info that comes back.

I also think you only need one option for disabling the error check, not sure it should be split between memos and not.

I also am a bit concerned at the term "upload" as I think that will have a different meaning with external payload stuff, maybe "create" or "encode"?

cretz · 2026-01-28T20:51:22Z

temporalio/converter.py

+    """The limit (in bytes) at which a memo size warning is logged."""
+
+    payload_upload_error_disabled: bool = False
+    """Field indiciating that the payload size checks should be disabled in the SDK.


Suggested change

"""Field indiciating that the payload size checks should be disabled in the SDK.

"""Field indicating that the payload size checks should be disabled in the SDK.

cretz · 2026-01-28T20:55:45Z

temporalio/converter.py

+            "[TMPRL1103] Attempted to upload payloads with size that exceeded the warning limit.",
+        )
+
+    def _validate_limits(


Not sure this has value as an instance method

cretz · 2026-01-28T20:59:45Z

temporalio/converter.py

+        warning_limit: int,
+        warning_message: str,
+    ):
+        total_size = sum(payload.ByteSize() for payload in payloads)


Technically if the server was comparing sizes of the Payloads message wrapper it is 2-3 bytes larger, but that is probably negligible, though still technically backwards incompatible if this sum is like 3.99999 MB heh.

cretz · 2026-01-28T21:02:10Z

temporalio/converter.py

+
+        if warning_limit > 0 and total_size > warning_limit:
+            # TODO: Use a context aware logger to log extra information about workflow/activity/etc
+            warnings.warn(


I think we should consider a custom Warning category class here

cretz · 2026-01-28T21:09:07Z

temporalio/converter.py

+        total_size = sum(payload.ByteSize() for payload in payloads)
+
+        if error_limit > 0 and total_size > error_limit:
+            raise temporalio.exceptions.PayloadSizeError(


I think we should prepend a 💥 emoji to this PR title and clarify in description it is technically a backwards incompatible change to eagerly error client side on payload failures (even though they can turn it off). It's always good to be safer when pointing out these incompatibilities in our release notes.

Where an error occurs can have significant effect on existing logic, e.g. finally clauses that may never have triggered before (though we recognize the unrecoverability of server-side validation of this before).

Also it should be noted that some users have proxies (which will proxy the namespace capabilities) which already do external storage or encryption today, and they are going to start seeing errors they didn't otherwise which can cause havoc on their workflows that were working just fine today. We need to really get out the message that those users should be disabling this error handling.

There is of course the alternative which is to stop using server information that only helps with half the payloads anyways.

jmaeagle99 marked this pull request as ready for review January 21, 2026 05:04

jmaeagle99 requested a review from a team as a code owner January 21, 2026 05:04

cretz reviewed Jan 21, 2026

View reviewed changes

temporalio/converter.py Outdated Show resolved Hide resolved

temporalio/converter.py Outdated Show resolved Hide resolved

temporalio/worker/_activity.py Outdated Show resolved Hide resolved

temporalio/exceptions.py Outdated Show resolved Hide resolved

jmaeagle99 marked this pull request as draft January 22, 2026 04:54

drewhoskins-temporal reviewed Jan 24, 2026

View reviewed changes

Payload limit configuration and validation

04a1c6d

jmaeagle99 force-pushed the payload-limits branch from bd14d85 to 04a1c6d Compare January 27, 2026 18:54

jmaeagle99 added 11 commits January 27, 2026 21:49

Change error limits to disabling fields

9ca018d

Use error limits from server

fb7f857

Remove tests that do not test real code paths

98b1eb2

Don't pass error limits to workflow replayer

861269b

Fix formatting

526f595

Use consistent message and use marker values for improved searchability

dc38edd

Handle failure encoding payload limit errors for activities

a9aa5ce

Nest workflow payload error handle in case it causes payload limit er…

7f621ef

…rors

Remove remaining investigative comments

58081db

Better messages and use TMPRL rule code

22de18b

Merge branch 'main' into payload-limits

dd7bcc2

jmaeagle99 marked this pull request as ready for review January 28, 2026 19:17

More test comments

76ff2a9

cretz reviewed Jan 28, 2026

View reviewed changes

	"""Field indiciating that the payload size checks should be disabled in the SDK.
	"""Field indicating that the payload size checks should be disabled in the SDK.

Payload limit configuration and validation #1288

Are you sure you want to change the base?

Payload limit configuration and validation #1288

Conversation

jmaeagle99 commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Examples

Deferred

Why?

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cretz left a comment

Choose a reason for hiding this comment

Uh oh!

cretz Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cretz Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cretz Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cretz Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cretz Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cretz Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cretz Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jmaeagle99 commented Jan 20, 2026 •

edited

Loading

cretz Jan 28, 2026 •

edited

Loading