Cost Optimization on Amazon SageMaker

This is the fifth and final post in our Amazon SageMaker series, building on everything from notebooks and endpoints through training, deployment, and Pipelines.

The two most common, most expensive SageMaker mistakes are rarely a single bad decision — they're a forgotten always-on resource and an oversized instance choice, both accruing cost quietly, for weeks or months, before anyone happens to check a bill closely enough to notice. This closing post is a practical rundown of where SageMaker cost actually accumulates, and the concrete, specific levers for controlling each source.

The endpoint you forgot about

We flagged this directly in the deployment post earlier in this series: a real-time endpoint stays running, and billing, continuously until explicitly deleted. This is, in practice, the single most common source of real, avoidable SageMaker spend.

import boto3
 
client = boto3.client("sagemaker")
endpoints = client.list_endpoints()
 
for ep in endpoints["Endpoints"]:
    print(ep["EndpointName"], ep["CreationTime"], ep["EndpointStatus"])

Running this kind of audit on a schedule — not just once, reactively, after noticing an unexpectedly large bill — is the actual fix, the same "check this deliberately and regularly, not just when something already looks wrong" instinct behind rebuilding Docker images on a schedule and re-evaluating model choice periodically, both covered earlier in this blog.

Matching instance type to actual workload

estimator = Estimator(
    image_uri=xgb_image,
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",  # is this actually the right size?
)

The exact same instinct behind setting accurate resource requests in Kubernetes, covered earlier in this blog, applies directly here: an oversized training or inference instance wastes money on unused capacity; an undersized one risks failure or unacceptably slow performance. SageMaker's instance-type recommendation tooling, and simply testing a workload against 2-3 candidate instance types before committing to one for ongoing use, both meaningfully beat defaulting to whichever instance type a tutorial happened to use as its example.

Spot instances for training: substantial savings, with a real trade-off

estimator = Estimator(
    image_uri=xgb_image,
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    use_spot_instances=True,
    max_wait=3600,
    max_run=1800,
    checkpoint_s3_uri=f"s3://{bucket}/checkpoints/",
)

Spot instances offer meaningfully discounted compute — often a substantial reduction versus on-demand pricing — in exchange for AWS being able to reclaim the instance with short notice if capacity is needed elsewhere. For training jobs specifically (as opposed to always-on inference endpoints, where an unexpected interruption is a much more serious problem), this is often a genuinely favorable trade: checkpoint_s3_uri lets an interrupted training job resume from its last checkpoint rather than restarting entirely from scratch, meaningfully limiting the actual cost of an interruption when one occurs.

Automatic scaling for real-time endpoints

client = boto3.client("application-autoscaling")
 
client.register_scalable_target(
    ServiceNamespace="sagemaker",
    ResourceId=f"endpoint/churn-prediction-endpoint/variant/AllTraffic",
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    MinCapacity=1,
    MaxCapacity=4,
)
 
client.put_scaling_policy(
    PolicyName="cpu-scaling",
    ServiceNamespace="sagemaker",
    ResourceId=f"endpoint/churn-prediction-endpoint/variant/AllTraffic",
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 70.0,
        "PredefinedMetricSpecification": {"PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"},
    },
)

This is directly analogous to a Kubernetes Horizontal Pod Autoscaler, covered earlier in this blog — an endpoint scales its instance count based on actual traffic, rather than being permanently provisioned for peak load it experiences only occasionally, or permanently under-provisioned for load spikes it can't actually handle. For traffic with genuine daily or seasonal variation, this is frequently a larger cost saving than instance-type selection alone.

Multi-variant endpoints: consolidating capacity across model versions

Beyond scaling one model's capacity, an endpoint can host multiple model variants (from the canary-deployment pattern covered in the previous post) with weighted traffic splits, sharing the same underlying capacity-management infrastructure rather than each version needing its own separately-scaled endpoint:

predictor.update_endpoint_weights_and_capacities(
    desired_weights_and_capacities=[
        {"VariantName": "current", "DesiredWeight": 70},
        {"VariantName": "canary", "DesiredWeight": 30},
    ]
)

Shifting traffic weights between variants incrementally — rather than provisioning two entirely separate endpoints for a canary rollout — keeps the cost of testing a new model version proportional to the traffic actually sent to it, rather than doubling the fixed infrastructure cost for the duration of the test.

Right-sizing through SageMaker Inference Recommender

Rather than guessing at instance type through trial and error, SageMaker's Inference Recommender runs a model against several candidate instance types automatically and reports actual latency and cost trade-offs:

client = boto3.client("sagemaker")
 
client.create_inference_recommendations_job(
    JobName="churn-model-recommender",
    JobType="Default",
    RoleArn=role,
    InputConfig={
        "ModelPackageVersionArn": model_package_arn,
        "JobDurationInSeconds": 3600,
    },
)

response = client.describe_inference_recommendations_job(JobName="churn-model-recommender")
for rec in response["InferenceRecommendations"]:
    print(rec["EndpointConfiguration"]["InstanceType"], rec["Metrics"]["CostPerInference"], rec["Metrics"]["MaxInvocations"])

This turns "which instance type should this model actually run on" from a guess into a measured comparison — the same evaluation discipline covered for comparing foundation models on Bedrock in the previous series, applied here to instance-type selection for a self-trained model, rather than relying on whichever instance type a tutorial or a previous, possibly outdated decision happened to use.

Choosing the right deployment shape, revisited

The previous post's framework — real-time for continuous, latency-sensitive traffic; serverless for intermittent, latency-tolerant traffic; asynchronous for genuinely long-running requests; multi-model for many similar, individually lower-traffic models — is itself the single highest-leverage cost decision covered in this entire series. An always-on real-time endpoint provisioned for a model called a handful of times a day is a structural cost mismatch no amount of instance-type tuning fully corrects; the right fix is choosing serverless (or a different shape entirely) for that traffic pattern in the first place, not right-sizing the wrong deployment shape.

Tagging resources for accurate cost attribution

A SageMaker bill with no consistent tagging strategy tells you the total spend, but not which team, project, or model is actually responsible for it — the same cost-visibility gap covered for untagged cloud infrastructure generally, applied here to ML-specific resources:

estimator = Estimator(
    image_uri=xgb_image,
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    tags=[
        {"Key": "Project", "Value": "churn-prediction"},
        {"Key": "Team", "Value": "data-science"},
        {"Key": "Environment", "Value": "production"},
    ],
)

Tagging every training job, endpoint, and pipeline consistently — the same tagging discipline covered for Terraform resources earlier in this blog — is what makes a Cost Explorer breakdown by project or team actually possible, rather than a single undifferentiated SageMaker line item on the monthly bill that nobody can attribute to a specific initiative without manually cross-referencing resource names.

Using SageMaker Savings Plans for predictable long-term workloads

Beyond spot instances for training and provisioned throughput for endpoints, AWS SageMaker Savings Plans offer a further discount in exchange for a one- or three-year compute commitment, applicable flexibly across instance types and SageMaker features:

aws savingsplans create-savings-plan \
  --savings-plan-offering-id "abc-123" \
  --commitment "10.0"

This is worth evaluating specifically once an organization's SageMaker usage has been running long enough to establish a genuinely stable baseline — committing to a savings plan before usage patterns are actually understood risks committing to more (or less) capacity than turns out to be needed, the same "measure before committing" caution covered for Bedrock's provisioned throughput in the previous series.

Reviewing endpoint utilization, not just existence

Beyond checking whether an endpoint still exists at all, it's worth checking whether an existing endpoint's provisioned capacity actually matches its real traffic — a two-instance endpoint receiving traffic that a single instance could easily handle is a quieter, less obvious version of the same waste as a fully forgotten one:

cw = boto3.client("cloudwatch")
response = cw.get_metric_statistics(
    Namespace="AWS/SageMaker",
    MetricName="Invocations",
    Dimensions=[{"Name": "EndpointName", "Value": "churn-prediction-endpoint"}],
    StartTime=start, EndTime=end, Period=3600, Statistics=["Sum"],
)

Reviewing actual invocation volume against provisioned instance count on the same schedule as the endpoint-existence audit catches over-provisioning that a simple "is it still running" check alone would miss entirely.

What to actually remember from this series

• A forgotten, always-on endpoint is the single most common source of avoidable SageMaker cost — audit running endpoints on a schedule, not just reactively after noticing a large bill.
• Match instance type to actual measured workload needs, the same instinct as setting accurate Kubernetes resource requests — Inference Recommender turns that into a measured comparison rather than a guess.
• Spot instances offer substantial training-cost savings, with checkpointing as the concrete mitigation for the interruption risk that trade-off carries.
• Autoscaling and weighted multi-variant traffic shifting both keep capacity proportional to actual demand, rather than fixed at worst-case or duplicated across every model version under test.
• Choosing the right deployment shape for actual traffic patterns is a bigger cost lever than instance-level tuning within the wrong shape.

That closes out our Amazon SageMaker series — from notebooks and endpoints through training, deployment, Pipelines, and now cost optimization — and with it, our full 50-post series spanning Terraform, Ansible, Docker, Kubernetes, GitHub Actions, OpenAI, LangChain, LangGraph, Bedrock, and now SageMaker. If your team is building real production infrastructure or AI systems on any of these tools, this is exactly the kind of hands-on work we build our training programs around.