Skip to content

feat: make AWS::LanguageExtensions local processing opt-in via --language-extensions flag#9033

Merged
bnusunny merged 33 commits into
developfrom
le-disable-and-backcompat-tests
May 21, 2026
Merged

feat: make AWS::LanguageExtensions local processing opt-in via --language-extensions flag#9033
bnusunny merged 33 commits into
developfrom
le-disable-and-backcompat-tests

Conversation

@bnusunny

@bnusunny bnusunny commented May 20, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Make local AWS::LanguageExtensions processing opt-in (off by default), restoring pre-1.160.0 behavior for templates that declare the transform but don't need local expansion
  • Add --language-extensions / --no-language-extensions CLI flag to all 8 affected commands: build, package, deploy, sync, validate, local invoke, local start-api, local start-lambda
  • Support SAM_CLI_ENABLE_LANGUAGE_EXTENSIONS=1 env var as fallback (flag wins)
  • Support samconfig.toml persistence via language_extensions = true

Motivation

SAM CLI 1.160.0 enabled local processing of AWS::LanguageExtensions implicitly when the transform was declared. This caused regressions (#9004, #9005, #9027, #9029) for templates that declared the transform for non-Fn::ForEach reasons or as a forward-compat placeholder. Making it opt-in lets customers adopt at their own pace.

Changes

  1. expand_language_extensions() now requires enabled: bool as a keyword-only arg — unwired callers fail with TypeError
  2. resolve_language_extensions_enabled(flag_value: Optional[bool]) -> bool resolver (flag → env var → False)
  3. @language_extensions_option Click decorator shared across all commands
  4. SamLocalStackProvider.get_stacks(..., language_extensions_enabled=False) kwarg threaded through recursion
  5. Each command context (Build, Package, Deploy, Sync, Validate, InvokeContext) wired end-to-end
  6. Template class in artifact_exporter wired for nested-stack recursion
  7. Inspection-path callers (sam list, companion stack, image-repo validation) explicitly pass False
  8. Telemetry regression tests lock in: event fires only when opt-in AND template uses LE
  9. Integration tests updated to pass --language-extensions for ForEach templates
  10. Documentation updated with opt-in instructions

Test plan

  • 9237 unit tests pass (pytest tests/unit/ -q -W ignore::pytest.PytestUnknownMarkWarning)
  • --language-extensions parameter registered on all 8 commands (verified programmatically)
  • make format clean
  • make lint shows zero errors
  • Integration test test_build_without_flag_does_not_expand_foreach verifies opt-out behavior
  • Existing ForEach integration tests pass with --language-extensions flag

bnusunny added 19 commits May 19, 2026 20:43
Add language_extensions_enabled kwarg to SamLocalStackProvider.get_stacks
with default=False to match pre-1.160.0 behavior. Forward to
expand_language_extensions and recursive get_stacks calls for nested
stacks.
…alidator

Wire the --language-extensions flag from validate command through to
SamTemplateValidator and expand_language_extensions. Validator takes a
resolved bool (not Optional) since resolution happens at command boundary.

Add language_extensions parameter to __init__, forward enabled kwarg to
expand_language_extensions call, and update all test call sites.
@bnusunny bnusunny requested a review from a team as a code owner May 20, 2026 03:54
@github-actions github-actions Bot added area/package sam package command area/deploy sam deploy command area/validate sam validate command area/build sam build command area/local/invoke sam local invoke command area/sync sam sync command area/schema JSON schema file pr/internal labels May 20, 2026

@aws-sam-cli-bot aws-sam-cli-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: 9ffa5ed..0d96fac
Files: 60
Comments: 1

Comment thread tests/integration/buildcmd/test_build_cmd_language_extensions.py

@aws-sam-cli-bot aws-sam-cli-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: 9ffa5ed..ed0d4f9
Files: 66
Comments: 1


Comments on lines outside the diff:

[samcli/lib/sync/infra_sync_executor.py:609] [BUG] With the new opt-in semantics, _detect_foreach_code_changes can crash on templates that declare AWS::LanguageExtensions but where the user did not pass --language-extensions. The early-return guard only short-circuits when the template has no LE transform at all:

if not check_using_language_extension(current_template):
   return
...
current_expanded = expand_language_extensions(
   current_template,
   parameter_values=current_params,
   enabled=self._sync_context.language_extensions_enabled,
).expanded_template

When the transform is declared but enabled=False, expand_language_extensions now returns a passthrough (per the new if not enabled branch in sam_integration.py), so current_expanded still contains the unexpanded Fn::ForEach::* keys whose values are lists, not dicts. The subsequent loop then does:

for resource_id, resource_dict in current_resources.items():
   if resource_id in original_regular_keys:  # Fn::ForEach key is excluded from this set
       continue
   resource_type = resource_dict.get("Type")  # AttributeError: 'list' object has no attribute 'get'

This is the exact regression scenario the PR aims to support (template declares the transform, user opts out), and the call site in _auto_skip_infra_sync has no surrounding try/except, so the AttributeError propagates and breaks sam sync's auto-skip-infra path.

Add an early return for the disabled case, e.g.:

if not self._sync_context.language_extensions_enabled:
   return
if not check_using_language_extension(current_template):
   return

Or guard the loop body to skip is_foreach_key(resource_id) entries when expansion was a passthrough.

@bnusunny

Copy link
Copy Markdown
Contributor Author

Re: PackageContext not threaded in deploy command (review 2026-05-20T17:51)

Fixed in ed0d4f9 — language_extensions=language_extensions is now passed to the PackageContext(...) call inside do_cli, matching the sync command's wiring.

@bnusunny

Copy link
Copy Markdown
Contributor Author

Re: _detect_foreach_code_changes crash when LE disabled (review 2026-05-20T18:04)

Fixed in 9f5934b — added early return when language_extensions_enabled is False, merged with the existing check_using_language_extension guard into a single condition.

valerena
valerena previously approved these changes May 20, 2026

@valerena valerena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple of questions, but it looks good.

Comment thread samcli/lib/cfn_language_extensions/sam_integration.py
Comment thread samcli/lib/cli_validation/image_repository_validation.py
Comment thread tests/unit/commands/deploy/test_command.py

@aws-sam-cli-bot aws-sam-cli-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: 9ffa5ed..d0f7f41
Files: 66
Comments: 1

Comment thread samcli/lib/cli_validation/image_repository_validation.py
Comment thread samcli/commands/build/build_context.py
Comment thread samcli/commands/_utils/options.py
Comment thread samcli/commands/deploy/deploy_context.py
Comment thread samcli/commands/local/cli_common/invoke_context.py
Comment thread samcli/lib/cfn_language_extensions/sam_integration.py

@aws-sam-cli-bot aws-sam-cli-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: 78c43a8..8ceafc3
Files: 66
Comments: 1

Comment thread samcli/lib/bootstrap/companion_stack/companion_stack_manager.py
@bnusunny bnusunny enabled auto-merge May 20, 2026 23:17
@bnusunny bnusunny added this pull request to the merge queue May 20, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 20, 2026
@bnusunny bnusunny enabled auto-merge May 21, 2026 00:49
@bnusunny bnusunny added this pull request to the merge queue May 21, 2026
Merged via the queue into develop with commit fe16ab0 May 21, 2026
53 checks passed
bnusunny added a commit that referenced this pull request May 21, 2026
After #9033, language extensions are opt-in. The two ForEach zipfile
tests added by this PR omitted the flag, causing SAM transform plugins
to choke on unexpanded Fn::ForEach blocks.
bnusunny added a commit that referenced this pull request May 21, 2026
PR #9033 made AWS::LanguageExtensions local processing opt-in. Two API
changes broke three tests on this branch after the merge from develop:

- expand_language_extensions() now requires a keyword-only `enabled` arg
- PackageContext reads self._language_extensions_enabled, set in __init__

Pass enabled=True to expand_language_extensions in the two artifact-exporter
tests, and set _language_extensions_enabled on the PackageContext stub that
bypasses __init__ via __new__. All three tests exercise templates with
Transform: AWS::LanguageExtensions, so enabling LE is the intended behavior.
hoangsetup pushed a commit to hoangsetup/aws-sam-cli that referenced this pull request May 23, 2026
… build merge (aws#9029) (aws#9031)

* fix(cfn-lang-ext): preserve inline-source intrinsics through LE-aware build merge (aws#9029)

Under `Transform: AWS::LanguageExtensions`, `sam build` was overwriting
`Code: {ZipFile: !Sub ...}` with the LE-resolved value, baking default
pseudo-parameter substitutions (`us-east-1`, `123456789012`) into the
built template. The same merge path covered three sites; gate all three
on a single artifacts-lookup primitive matching the non-LE
`ApplicationBuilder.update_template` early-skip:

- Root-level merge (`_update_original_template_paths`): skip resources
  whose `get_full_path(stack_path, key)` is absent from `artifacts`.
- ForEach static branch (`_merge_static_artifact_path`): same check per
  expanded key.
- ForEach dynamic branch (`_collect_dynamic_mapping_entries` + nested):
  same check; reject with `UserException` naming the non-buildable
  property when no iteration produced an artifact, since Mappings hold
  only static strings and the loop variable cannot survive deploy.

Adds case A/B/C integration tests + testdata templates and unit tests
covering the new skip and refusal paths.

* refactor(cfn-lang-ext): address PR aws#9031 review feedback

- Hoist inline `get_full_path` imports to the top of build_context.py.
- Allow inline `Code: {ZipFile: !Sub ...}` inside Fn::ForEach bodies that
  reference the loop variable: skip the merge instead of raising, since
  CFN's LanguageExtensions transform expands the body at deploy time and
  substitutes the loop variable per iteration. Drop the unused
  `_raise_dynamic_no_artifacts_unsupported` helper and flip the related
  integration test to assert pass-through preservation.
- Stop coercing `Optional[Dict[str, str]]` artifacts to `{}` in functions
  that only forward the value. Where it's used directly, fold the None
  guard into the existing membership check (`not artifacts or ...`).
  `_merge_static_artifact_path` becomes Optional to match its callers.

* test(cfn-lang-ext): flip dynamic-no-artifact unit test to pass-through

Companion to the prior commit: the dynamic-branch path no longer raises
when an inline-source Fn::ForEach iteration has no build artifact. Update
the unit test to assert that no Mapping is generated and the original
Code (with Fn::Sub intact) is left untouched, so CFN's LanguageExtensions
transform can substitute the loop variable per-iteration at deploy time.

* refactor(build_context): hoist all inline imports, drop redundant `or {}`

Hoist every inline import in build_context.py to the module-level import
block (`itertools`, the cfn_language_extensions models/sam_integration
symbols, and all language_extensions_packaging helpers). None of these
introduce a circular dependency — language_extensions_packaging and
cfn_language_extensions only import from samcli.lib.* and samcli.commands.
validate, never from samcli.commands.build.build_context.

Drop the redundant `stack_output_template_path_by_stack_path or {}` at
the `_update_original_template_paths` call site: the receiver declares
the parameter Optional and already coerces to {} internally.

* test(integ): opt-in to --language-extensions for foreach zipfile tests

After aws#9033, language extensions are opt-in. The two ForEach zipfile
tests added by this PR omitted the flag, causing SAM transform plugins
to choke on unexpanded Fn::ForEach blocks.
Ali-Razmjoo pushed a commit to fork-the-planet/aws-sam-cli that referenced this pull request May 29, 2026
…tension functions (aws#9030)

* refactor(package): replace METADATA_EXPORT_LIST and GLOBAL_EXPORT_DICT with typed registries

Migrate two flat package exporter registries to typed dataclass form:

- METADATA_EXPORT_LIST (list of Resource subclasses) -> METADATA_EXPORTS
  (List[MetadataExportSpec]). Each spec captures both the metadata_type
  ('AWS::ServerlessRepo::Application') and the per-property exporter
  classes that handle LicenseUrl/ReadmeUrl. Template._export_metadata
  now dispatches via a metadata_type lookup instead of a per-class
  RESOURCE_TYPE filter.

- GLOBAL_EXPORT_DICT (dict keyed by Fn::Transform) -> GLOBAL_TRANSFORM_EXPORTS
  (List[GlobalTransformExportSpec]). Each spec carries a discriminator
  callable (e.g. _is_aws_include for matching AWS::Include) plus the
  handler. _export_global_artifacts now dispatches through the
  discriminator so future Fn::Transform variants can register without
  touching the walker.

The typed shape is the contract a follow-up commit will read from to
process AWS::Include before language-extension expansion.

* feat(package): merge SAR Metadata exports back into LE child templates

Extend merge_language_extensions_s3_uris with a registry-driven Metadata
pass that copies rewritten property values (LicenseUrl, ReadmeUrl) from
the exported template back into the original (Fn::ForEach-preserving)
template after LE expansion.

Without this pass, when a child stack uses Transform: AWS::LanguageExtensions
and declares AWS::ServerlessRepo::Application metadata, sam package
silently dropped the License/Readme S3 URLs from the merged output —
they were uploaded but never wired into the template the user deploys.

Implementation iterates METADATA_EXPORTS (the registry added in the prior
commit) so future metadata types pick up merge support without touching
the merge walker.

* fix(package): process AWS::Include before language-extension expansion (aws#9027)

Closes aws#9027.

Symptom: when a template uses both Transform: AWS::LanguageExtensions and
contains Fn::Transform: AWS::Include buried inside an LE function (e.g.
Fn::ToJsonString), sam package fails to rewrite the include's Location to
an S3 URL. CloudFormation then rejects the deploy with 'The location
parameter is not a valid S3 uri.'

Root cause: PackageContext._export ran expand_language_extensions BEFORE
the artifact exporter walked Fn::Transform nodes. LE functions like
Fn::ToJsonString json.dumps() their argument, collapsing the structural
Fn::Transform subtree into a JSON-string literal. By the time the
exporter ran, the include was no longer a structural dict node and was
invisible to the global-transform walker.

Fix: process AWS::Include (and any other GLOBAL_TRANSFORM_EXPORTS
handler) on the original template BEFORE LE expansion runs, mirroring
CloudFormation's own server-side transform ordering — CFN resolves
inline Fn::Transform macros before evaluating AWS::LanguageExtensions.

Implementation:

- Extract Template._export_global_artifacts to a module-level
  _export_global_artifacts_pass(template_dict, uploader, template_dir).
  The instance method becomes a one-line delegate so existing callers
  keep working.
- Call _export_global_artifacts_pass on the original template before
  expand_language_extensions in PackageContext._export (root flow) and
  in CloudFormationStackResource.do_export (nested-stack child flow).

Dynamic-Location AWS::Include inside Fn::ForEach (e.g.
Location: ./swagger-${Name}.yaml) is not supported by sam package: a
local file path with literal ${...} placeholders does not exist on disk,
so is_local_file fails and the existing InvalidLocalPathError fires —
which is the right user-facing failure. CloudFormation does not
substitute loop variables into Fn::Transform paths server-side either,
so the limitation matches CFN's actual capability.

* test(package): integration coverage for AWS::Include + SAR Metadata in LE templates

Three end-to-end tests that exercise the package_context._export-equivalent
flow on language-extension templates:

- test_le_template_with_top_level_aws_include_merges_location verifies
  AWS::Include in Outputs gets its Location rewritten to s3:// after
  the pre-LE pass.
- test_le_template_with_serverless_repo_metadata_merges_license_url
  verifies SAR LicenseUrl/ReadmeUrl in Metadata get merged back.
- TestPackageContextIssue9027 reproduces the user template from
  aws#9027 (Fn::ToJsonString over Fn::Transform: AWS::Include
  buried inside AWS::SSM::Parameter) and asserts the buried Location is
  rewritten to s3://. Locks down the regression.

* docs(cfn-lang-ext): document AWS::Include processing order vs language extensions

Add a section explaining that sam package processes Fn::Transform: AWS::Include
before language-extension expansion, mirroring CloudFormation's server-side
transform ordering. This means AWS::Include Location rewrites work correctly
even when the include lives buried inside language-extension functions like
Fn::ToJsonString or Fn::ForEach bodies.

* test(package): align LE tests with opt-in API from aws#9033

PR aws#9033 made AWS::LanguageExtensions local processing opt-in. Two API
changes broke three tests on this branch after the merge from develop:

- expand_language_extensions() now requires a keyword-only `enabled` arg
- PackageContext reads self._language_extensions_enabled, set in __init__

Pass enabled=True to expand_language_extensions in the two artifact-exporter
tests, and set _language_extensions_enabled on the PackageContext stub that
bypasses __init__ via __new__. All three tests exercise templates with
Transform: AWS::LanguageExtensions, so enabling LE is the intended behavior.

* test(package): hoist inline imports in TestPackageContextBuriedAWSInclude

Move os, Destination/Uploaders, and yaml_parse to module-level imports;
drop the redundant inline tempfile/PackageContext (already at module level).

Addresses inline-imports review comment on aws#9030.

* refactor(package): hoist InvalidSamDocumentException and expand_language_extensions imports

Move the two inline imports inside _export() to module scope. No
behavior change; this is preparation for the upcoming _export() split
which references these from a new branch method.

* refactor(package): add _export_without_language_extensions (off-path branch)

New private method that mirrors pre-1.160.0 sam-cli behavior: a tight
Template.export() pipeline with no LE machinery. Unused by _export()
until the dispatcher is cut over in a follow-up commit.

Also adds two structural-gate smoke tests that stay red until the
dispatcher is wired up — they assert that the off path never invokes
expand_language_extensions or the pre-LE _export_global_artifacts_pass.

* refactor(package): add _export_with_language_extensions (on-path branch)

New private method containing the existing aws#9027 ordering: pre-LE
include pass, LE expansion, Template.export(), merge, Mappings.
Unused by _export() until the dispatcher is cut over in the next commit.

* refactor(package): split _export() into LE / non-LE branches gated on flag

_export() becomes a thin dispatcher: read the template, branch on
self._language_extensions_enabled, dump. The two branch methods added
in the prior two commits now own the actual work.

Treats --language-extensions as a hard correctness boundary: when off,
no LE machinery is invoked at all (no expand_language_extensions, no
pre-LE _export_global_artifacts_pass, no merge, no Mappings).
Template.export() still handles AWS::Include for non-LE templates via
its own internal _export_global_artifacts pass — that path has worked
since long before the aws#9027 fix and is unchanged by this refactor.

Public surface (PackageContext._export(template_path, use_json) -> str)
is preserved; all existing _export tests pass unchanged.

* test(package): repoint expand_language_extensions patches to package_context use site

The hoist in 756c502 moved `expand_language_extensions` to a
module-level import in `package_context.py`. The use-site binding is
now captured at module load, so existing tests that patched the source
module `samcli.lib.cfn_language_extensions.sam_integration.expand_language_extensions`
no longer intercept calls from `_export()`.

Repoint two patches at the use-site
`samcli.commands.package.package_context.expand_language_extensions`:

- `test_phase_separation.py::test_package_context_export_calls_expand_language_extensions`
  was failing intermittently in CI depending on test-collection order.
- `test_package_context.py::test_off_path_does_not_invoke_expand_language_extensions`
  is retargeted for consistency now that the inline-import shadow is
  gone (post-Task-4 cutover); the dead `had_language_extensions=False`
  scaffolding can also go since the off path no longer calls expand.

Other source-module patches in `test_artifact_exporter.py` are correct
as-is — those tests cover `Template.export()` paths whose use site is
the artifact_exporter module.

* refactor(package): hoist do_export inline imports and repoint test patches

Hoist expand_language_extensions, IntrinsicsSymbolTable,
generate_and_apply_artifact_mappings, and merge_language_extensions_s3_uris
to module-level imports in artifact_exporter.py — preparation for the
LE / non-LE structural-gate split of CloudFormationStackResource.do_export.

Module-level binding requires existing tests that patched
samcli.lib.cfn_language_extensions.sam_integration.expand_language_extensions
(the source module) to repoint at the use-site
samcli.lib.package.artifact_exporter.expand_language_extensions, since
the use-site name is captured at module load and source-module patches
no longer intercept once the inline import is gone. Same lesson as
PR aws#9030 commit 45dea4b for package_context.

* refactor(package): add _do_export_without_language_extensions branch method

LE-off branch for CloudFormationStackResource: path-based Template
construction with no pre-LE pass, no parameter_values, no deepcopy.
Method is unused until the dispatcher rewrite — wiring lands in the
follow-up commit. Adds TestDoExportLanguageExtensionsStructuralGate
with two red structural-gate assertions that the dispatcher rewrite
will turn green.

* refactor(package): add _do_export_with_language_extensions branch method

Lifts the existing LE-expansion body from CloudFormationStackResource.do_export
into a dedicated method (resource_id, template_path, parent_dir,
abs_template_path, resource_dict) -> Dict. No behavior change yet — the
dispatcher in the follow-up commit will route LE-on calls into it.

Threads resource_dict so the branch can read nested-stack Parameters
without the dispatcher passing them separately. Returns the exported
template dict so the dispatcher owns the final yaml_dump + upload tail.

* refactor(package): split do_export into LE / non-LE branches gated on flag

CloudFormationStackResource.do_export is now a thin dispatcher that:
  1) does the early-exit guards (None / S3 URL / non-local file)
  2) routes to _do_export_with_language_extensions or
     _do_export_without_language_extensions based on
     self.language_extensions_enabled
  3) owns the final yaml_dump + upload + property mutation tail.

The off path is now structurally LE-free: no expand_language_extensions
call, no _export_global_artifacts_pass, no IntrinsicsSymbolTable
pseudo-param plumbing, no parent_parameter_values, no copy.deepcopy.
Template.export() runs its own internal _export_global_artifacts so
AWS::Include still resolves on the off path.

The structural-gate tests added in the previous commit are now green.
Existing LE tests that rely on language_extensions_enabled = True now
set the flag explicitly (where they previously depended on the LE
machinery running unconditionally as a passthrough).

* chore(package): apply make format to do_export refactor

Black collapsed multi-line calls and signatures whose single-line
form fits under the 120-char limit. No behavior change.

* refactor(package): add _build_child_parameter_values helper

Adds a CFN-parity helper that builds the parameter_values dict for a
nested child stack: pseudo-params (with parent pseudo-name overrides
honored) plus parent-rebound values via the nested-stack Parameters
property. Non-pseudo parent names are not copied; child Defaults are
read by the LE expander itself via parsed_template.parameters.

Helper is unused production code in this commit. Wired into
CloudFormationStackResource._do_export_with_language_extensions in
the next commit.

* fix(package): tighten parent_parameter_values scoping in LE path

CloudFormationStackResource._do_export_with_language_extensions used to
bulk-copy the parent stack's full parameter_values into the child, so a
parent's `Foo=parent_foo` could silently shadow an unrelated child
Parameter named Foo. This diverged from CloudFormation's nested-stack
contract (only parent-rebound names reach the child) and from the
non-LE path (which threads no parameters at all).

Replace the merge block with _build_child_parameter_values, which
returns pseudo-params (with parent pseudo-name overrides honored) plus
parent-rebound values from the nested-stack Parameters property. Child
template Defaults still resolve via the LE expander's parsed_template
fallback (resolvers/fn_ref.py:_resolve_parameter).

Adds end-to-end coverage:
- non-pseudo parent param does not leak into child's parameter_values
- child Default still resolves via resolver fallback after the change

* refactor(package): trim Task 2 redundant comments and verbose test docstrings

Code-review polish for the LE parent-param scoping fix:
- drop call-site comment block at _do_export_with_language_extensions
  that duplicated _build_child_parameter_values's docstring
- shorten the two new test docstrings in
  TestCloudFormationStackResourceChildExpansion to match neighbor
  style (no internal module paths)

* refactor(tests): hoist inline imports added in LE parent-param fix

Move imports introduced by the previous two commits from inside test
bodies up to the module-level import block:
- _build_child_parameter_values, IntrinsicsSymbolTable
- yaml_dump (samcli.yamlhelper), shutil, inspect (stdlib)

None of these symbols are @patch targets, so hoisting does not affect
any mock binding (use-site patches in artifact_exporter remain correct).

* refactor(tests): hoist remaining inline imports and extract LE fixtures

Address review feedback on PR aws#9030: inline imports should live at
module scope, and on-the-fly file writes for LE child templates should
use checked-in fixtures instead of tempfile.mkdtemp() + yaml_dump().

Hoist inline imports (LanguageExtensionResult, _resolve_nested_stack_parameters,
yaml_dump, shutil, the packageable_resources block) to the module-level
import block. None are @patch targets, so use-site patches remain correct.

Add tests/unit/lib/package/test_data/ following the precedent in
tests/unit/lib/intrinsic_resolver/test_data, with TEST_DATA_PATH
constant defined as Path(__file__).resolve().parent / "test_data".

Convert the LE parent-param test methods, the buried-AWS::Include test,
the expansion-error-handling pair, and the structural-gate pair to read
their child templates (and the export-events.json include target) from
the fixture tree. The on-the-fly tempdir + yaml_dump scaffolding is
removed entirely along with the corresponding _write_child /
_write_minimal_child helpers.

No production code change; all 110 unit tests in test_artifact_exporter
still pass.

* refactor(tests): extract TestPackageContextBuriedAWSInclude inline fixture

Replace the on-the-fly tempfile + open()+write() scaffolding in
TestPackageContextBuriedAWSInclude.setUp with a checked-in fixture under
tests/unit/commands/package/test_data/buried_aws_include/. Mirrors the
TEST_DATA_PATH pattern already in tests/unit/lib/package/test_data and
tests/unit/lib/intrinsic_resolver/test_data.

setUp now resolves template_path from TEST_DATA_PATH; the export-events
include target lives next to template.yaml so the relative Location in
the template still resolves at sam-package time.

No production code change; all 146 tests in test_package_context still
pass.
singledigit pushed a commit to singledigit/aws-sam-cli that referenced this pull request Jun 6, 2026
…tension functions (aws#9030)

* refactor(package): replace METADATA_EXPORT_LIST and GLOBAL_EXPORT_DICT with typed registries

Migrate two flat package exporter registries to typed dataclass form:

- METADATA_EXPORT_LIST (list of Resource subclasses) -> METADATA_EXPORTS
  (List[MetadataExportSpec]). Each spec captures both the metadata_type
  ('AWS::ServerlessRepo::Application') and the per-property exporter
  classes that handle LicenseUrl/ReadmeUrl. Template._export_metadata
  now dispatches via a metadata_type lookup instead of a per-class
  RESOURCE_TYPE filter.

- GLOBAL_EXPORT_DICT (dict keyed by Fn::Transform) -> GLOBAL_TRANSFORM_EXPORTS
  (List[GlobalTransformExportSpec]). Each spec carries a discriminator
  callable (e.g. _is_aws_include for matching AWS::Include) plus the
  handler. _export_global_artifacts now dispatches through the
  discriminator so future Fn::Transform variants can register without
  touching the walker.

The typed shape is the contract a follow-up commit will read from to
process AWS::Include before language-extension expansion.

* feat(package): merge SAR Metadata exports back into LE child templates

Extend merge_language_extensions_s3_uris with a registry-driven Metadata
pass that copies rewritten property values (LicenseUrl, ReadmeUrl) from
the exported template back into the original (Fn::ForEach-preserving)
template after LE expansion.

Without this pass, when a child stack uses Transform: AWS::LanguageExtensions
and declares AWS::ServerlessRepo::Application metadata, sam package
silently dropped the License/Readme S3 URLs from the merged output —
they were uploaded but never wired into the template the user deploys.

Implementation iterates METADATA_EXPORTS (the registry added in the prior
commit) so future metadata types pick up merge support without touching
the merge walker.

* fix(package): process AWS::Include before language-extension expansion (aws#9027)

Closes aws#9027.

Symptom: when a template uses both Transform: AWS::LanguageExtensions and
contains Fn::Transform: AWS::Include buried inside an LE function (e.g.
Fn::ToJsonString), sam package fails to rewrite the include's Location to
an S3 URL. CloudFormation then rejects the deploy with 'The location
parameter is not a valid S3 uri.'

Root cause: PackageContext._export ran expand_language_extensions BEFORE
the artifact exporter walked Fn::Transform nodes. LE functions like
Fn::ToJsonString json.dumps() their argument, collapsing the structural
Fn::Transform subtree into a JSON-string literal. By the time the
exporter ran, the include was no longer a structural dict node and was
invisible to the global-transform walker.

Fix: process AWS::Include (and any other GLOBAL_TRANSFORM_EXPORTS
handler) on the original template BEFORE LE expansion runs, mirroring
CloudFormation's own server-side transform ordering — CFN resolves
inline Fn::Transform macros before evaluating AWS::LanguageExtensions.

Implementation:

- Extract Template._export_global_artifacts to a module-level
  _export_global_artifacts_pass(template_dict, uploader, template_dir).
  The instance method becomes a one-line delegate so existing callers
  keep working.
- Call _export_global_artifacts_pass on the original template before
  expand_language_extensions in PackageContext._export (root flow) and
  in CloudFormationStackResource.do_export (nested-stack child flow).

Dynamic-Location AWS::Include inside Fn::ForEach (e.g.
Location: ./swagger-${Name}.yaml) is not supported by sam package: a
local file path with literal ${...} placeholders does not exist on disk,
so is_local_file fails and the existing InvalidLocalPathError fires —
which is the right user-facing failure. CloudFormation does not
substitute loop variables into Fn::Transform paths server-side either,
so the limitation matches CFN's actual capability.

* test(package): integration coverage for AWS::Include + SAR Metadata in LE templates

Three end-to-end tests that exercise the package_context._export-equivalent
flow on language-extension templates:

- test_le_template_with_top_level_aws_include_merges_location verifies
  AWS::Include in Outputs gets its Location rewritten to s3:// after
  the pre-LE pass.
- test_le_template_with_serverless_repo_metadata_merges_license_url
  verifies SAR LicenseUrl/ReadmeUrl in Metadata get merged back.
- TestPackageContextIssue9027 reproduces the user template from
  aws#9027 (Fn::ToJsonString over Fn::Transform: AWS::Include
  buried inside AWS::SSM::Parameter) and asserts the buried Location is
  rewritten to s3://. Locks down the regression.

* docs(cfn-lang-ext): document AWS::Include processing order vs language extensions

Add a section explaining that sam package processes Fn::Transform: AWS::Include
before language-extension expansion, mirroring CloudFormation's server-side
transform ordering. This means AWS::Include Location rewrites work correctly
even when the include lives buried inside language-extension functions like
Fn::ToJsonString or Fn::ForEach bodies.

* test(package): align LE tests with opt-in API from aws#9033

PR aws#9033 made AWS::LanguageExtensions local processing opt-in. Two API
changes broke three tests on this branch after the merge from develop:

- expand_language_extensions() now requires a keyword-only `enabled` arg
- PackageContext reads self._language_extensions_enabled, set in __init__

Pass enabled=True to expand_language_extensions in the two artifact-exporter
tests, and set _language_extensions_enabled on the PackageContext stub that
bypasses __init__ via __new__. All three tests exercise templates with
Transform: AWS::LanguageExtensions, so enabling LE is the intended behavior.

* test(package): hoist inline imports in TestPackageContextBuriedAWSInclude

Move os, Destination/Uploaders, and yaml_parse to module-level imports;
drop the redundant inline tempfile/PackageContext (already at module level).

Addresses inline-imports review comment on aws#9030.

* refactor(package): hoist InvalidSamDocumentException and expand_language_extensions imports

Move the two inline imports inside _export() to module scope. No
behavior change; this is preparation for the upcoming _export() split
which references these from a new branch method.

* refactor(package): add _export_without_language_extensions (off-path branch)

New private method that mirrors pre-1.160.0 sam-cli behavior: a tight
Template.export() pipeline with no LE machinery. Unused by _export()
until the dispatcher is cut over in a follow-up commit.

Also adds two structural-gate smoke tests that stay red until the
dispatcher is wired up — they assert that the off path never invokes
expand_language_extensions or the pre-LE _export_global_artifacts_pass.

* refactor(package): add _export_with_language_extensions (on-path branch)

New private method containing the existing aws#9027 ordering: pre-LE
include pass, LE expansion, Template.export(), merge, Mappings.
Unused by _export() until the dispatcher is cut over in the next commit.

* refactor(package): split _export() into LE / non-LE branches gated on flag

_export() becomes a thin dispatcher: read the template, branch on
self._language_extensions_enabled, dump. The two branch methods added
in the prior two commits now own the actual work.

Treats --language-extensions as a hard correctness boundary: when off,
no LE machinery is invoked at all (no expand_language_extensions, no
pre-LE _export_global_artifacts_pass, no merge, no Mappings).
Template.export() still handles AWS::Include for non-LE templates via
its own internal _export_global_artifacts pass — that path has worked
since long before the aws#9027 fix and is unchanged by this refactor.

Public surface (PackageContext._export(template_path, use_json) -> str)
is preserved; all existing _export tests pass unchanged.

* test(package): repoint expand_language_extensions patches to package_context use site

The hoist in 756c502 moved `expand_language_extensions` to a
module-level import in `package_context.py`. The use-site binding is
now captured at module load, so existing tests that patched the source
module `samcli.lib.cfn_language_extensions.sam_integration.expand_language_extensions`
no longer intercept calls from `_export()`.

Repoint two patches at the use-site
`samcli.commands.package.package_context.expand_language_extensions`:

- `test_phase_separation.py::test_package_context_export_calls_expand_language_extensions`
  was failing intermittently in CI depending on test-collection order.
- `test_package_context.py::test_off_path_does_not_invoke_expand_language_extensions`
  is retargeted for consistency now that the inline-import shadow is
  gone (post-Task-4 cutover); the dead `had_language_extensions=False`
  scaffolding can also go since the off path no longer calls expand.

Other source-module patches in `test_artifact_exporter.py` are correct
as-is — those tests cover `Template.export()` paths whose use site is
the artifact_exporter module.

* refactor(package): hoist do_export inline imports and repoint test patches

Hoist expand_language_extensions, IntrinsicsSymbolTable,
generate_and_apply_artifact_mappings, and merge_language_extensions_s3_uris
to module-level imports in artifact_exporter.py — preparation for the
LE / non-LE structural-gate split of CloudFormationStackResource.do_export.

Module-level binding requires existing tests that patched
samcli.lib.cfn_language_extensions.sam_integration.expand_language_extensions
(the source module) to repoint at the use-site
samcli.lib.package.artifact_exporter.expand_language_extensions, since
the use-site name is captured at module load and source-module patches
no longer intercept once the inline import is gone. Same lesson as
PR aws#9030 commit 45dea4b for package_context.

* refactor(package): add _do_export_without_language_extensions branch method

LE-off branch for CloudFormationStackResource: path-based Template
construction with no pre-LE pass, no parameter_values, no deepcopy.
Method is unused until the dispatcher rewrite — wiring lands in the
follow-up commit. Adds TestDoExportLanguageExtensionsStructuralGate
with two red structural-gate assertions that the dispatcher rewrite
will turn green.

* refactor(package): add _do_export_with_language_extensions branch method

Lifts the existing LE-expansion body from CloudFormationStackResource.do_export
into a dedicated method (resource_id, template_path, parent_dir,
abs_template_path, resource_dict) -> Dict. No behavior change yet — the
dispatcher in the follow-up commit will route LE-on calls into it.

Threads resource_dict so the branch can read nested-stack Parameters
without the dispatcher passing them separately. Returns the exported
template dict so the dispatcher owns the final yaml_dump + upload tail.

* refactor(package): split do_export into LE / non-LE branches gated on flag

CloudFormationStackResource.do_export is now a thin dispatcher that:
  1) does the early-exit guards (None / S3 URL / non-local file)
  2) routes to _do_export_with_language_extensions or
     _do_export_without_language_extensions based on
     self.language_extensions_enabled
  3) owns the final yaml_dump + upload + property mutation tail.

The off path is now structurally LE-free: no expand_language_extensions
call, no _export_global_artifacts_pass, no IntrinsicsSymbolTable
pseudo-param plumbing, no parent_parameter_values, no copy.deepcopy.
Template.export() runs its own internal _export_global_artifacts so
AWS::Include still resolves on the off path.

The structural-gate tests added in the previous commit are now green.
Existing LE tests that rely on language_extensions_enabled = True now
set the flag explicitly (where they previously depended on the LE
machinery running unconditionally as a passthrough).

* chore(package): apply make format to do_export refactor

Black collapsed multi-line calls and signatures whose single-line
form fits under the 120-char limit. No behavior change.

* refactor(package): add _build_child_parameter_values helper

Adds a CFN-parity helper that builds the parameter_values dict for a
nested child stack: pseudo-params (with parent pseudo-name overrides
honored) plus parent-rebound values via the nested-stack Parameters
property. Non-pseudo parent names are not copied; child Defaults are
read by the LE expander itself via parsed_template.parameters.

Helper is unused production code in this commit. Wired into
CloudFormationStackResource._do_export_with_language_extensions in
the next commit.

* fix(package): tighten parent_parameter_values scoping in LE path

CloudFormationStackResource._do_export_with_language_extensions used to
bulk-copy the parent stack's full parameter_values into the child, so a
parent's `Foo=parent_foo` could silently shadow an unrelated child
Parameter named Foo. This diverged from CloudFormation's nested-stack
contract (only parent-rebound names reach the child) and from the
non-LE path (which threads no parameters at all).

Replace the merge block with _build_child_parameter_values, which
returns pseudo-params (with parent pseudo-name overrides honored) plus
parent-rebound values from the nested-stack Parameters property. Child
template Defaults still resolve via the LE expander's parsed_template
fallback (resolvers/fn_ref.py:_resolve_parameter).

Adds end-to-end coverage:
- non-pseudo parent param does not leak into child's parameter_values
- child Default still resolves via resolver fallback after the change

* refactor(package): trim Task 2 redundant comments and verbose test docstrings

Code-review polish for the LE parent-param scoping fix:
- drop call-site comment block at _do_export_with_language_extensions
  that duplicated _build_child_parameter_values's docstring
- shorten the two new test docstrings in
  TestCloudFormationStackResourceChildExpansion to match neighbor
  style (no internal module paths)

* refactor(tests): hoist inline imports added in LE parent-param fix

Move imports introduced by the previous two commits from inside test
bodies up to the module-level import block:
- _build_child_parameter_values, IntrinsicsSymbolTable
- yaml_dump (samcli.yamlhelper), shutil, inspect (stdlib)

None of these symbols are @patch targets, so hoisting does not affect
any mock binding (use-site patches in artifact_exporter remain correct).

* refactor(tests): hoist remaining inline imports and extract LE fixtures

Address review feedback on PR aws#9030: inline imports should live at
module scope, and on-the-fly file writes for LE child templates should
use checked-in fixtures instead of tempfile.mkdtemp() + yaml_dump().

Hoist inline imports (LanguageExtensionResult, _resolve_nested_stack_parameters,
yaml_dump, shutil, the packageable_resources block) to the module-level
import block. None are @patch targets, so use-site patches remain correct.

Add tests/unit/lib/package/test_data/ following the precedent in
tests/unit/lib/intrinsic_resolver/test_data, with TEST_DATA_PATH
constant defined as Path(__file__).resolve().parent / "test_data".

Convert the LE parent-param test methods, the buried-AWS::Include test,
the expansion-error-handling pair, and the structural-gate pair to read
their child templates (and the export-events.json include target) from
the fixture tree. The on-the-fly tempdir + yaml_dump scaffolding is
removed entirely along with the corresponding _write_child /
_write_minimal_child helpers.

No production code change; all 110 unit tests in test_artifact_exporter
still pass.

* refactor(tests): extract TestPackageContextBuriedAWSInclude inline fixture

Replace the on-the-fly tempfile + open()+write() scaffolding in
TestPackageContextBuriedAWSInclude.setUp with a checked-in fixture under
tests/unit/commands/package/test_data/buried_aws_include/. Mirrors the
TEST_DATA_PATH pattern already in tests/unit/lib/package/test_data and
tests/unit/lib/intrinsic_resolver/test_data.

setUp now resolves template_path from TEST_DATA_PATH; the export-events
include target lives next to template.yaml so the relative Location in
the template still resolves at sam-package time.

No production code change; all 146 tests in test_package_context still
pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/build sam build command area/deploy sam deploy command area/local/invoke sam local invoke command area/package sam package command area/schema JSON schema file area/sync sam sync command area/validate sam validate command pr/internal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants