Deploying Models with SageMaker Real-Time and Serverless Endpoints

This is the third post in our Amazon SageMaker series, building on notebooks and training jobs and built-in algorithms.

A trained model sitting in S3 isn't useful to anything until it's deployed somewhere that can actually serve predictions. SageMaker offers several distinct deployment shapes, each suited to a genuinely different traffic pattern — choosing the wrong one for a given workload means either paying for idle capacity or accepting latency a use case can't actually tolerate.

Real-time endpoints: always on, lowest latency

predictor = estimator.deploy(
    initial_instance_count=2,
    instance_type="ml.m5.large",
    endpoint_name="churn-prediction-endpoint",
)
 
result = predictor.predict(new_customer_data)

A real-time endpoint provisions instances that stay running continuously, ready to respond with the lowest possible latency at any moment. This is the right shape for genuinely latency-sensitive, continuous, unpredictable traffic — a live production application where a user is actively waiting on a response. It's also, per the next post in this series, the shape most likely to accrue real, ongoing cost for capacity that sits idle outside of actual traffic — appropriate specifically when that always-on cost is justified by genuinely continuous demand.

Serverless inference: scale-to-zero, cold-start trade-off

from sagemaker.serverless import ServerlessInferenceConfig
 
serverless_config = ServerlessInferenceConfig(
    memory_size_in_mb=2048,
    max_concurrency=5,
)
 
predictor = estimator.deploy(
    serverless_inference_config=serverless_config,
    endpoint_name="churn-prediction-serverless",
)

A serverless endpoint scales down to zero provisioned capacity during idle periods, and back up automatically when a request arrives — billed only for actual inference compute time, genuinely analogous to the disposable, pay-for-what-you-use model covered for training jobs, Docker containers, and GitHub Actions runners throughout this blog. The real trade-off: a request arriving after idle time incurs a cold start — the time to provision capacity before that first request can actually be served — meaningfully higher latency than an already-warm real-time endpoint. This is the right choice for genuinely intermittent, unpredictable traffic (a handful of calls per hour, batch-adjacent use cases) where that occasional cold-start latency is an acceptable trade for not paying for continuously idle capacity.

Asynchronous inference: for requests that genuinely take a while

from sagemaker.async_inference import AsyncInferenceConfig
 
async_config = AsyncInferenceConfig(
    output_path=f"s3://{bucket}/async-predictions",
    max_concurrent_invocations_per_instance=4,
)
 
predictor = estimator.deploy(
    async_inference_config=async_config,
    instance_type="ml.m5.xlarge",
    initial_instance_count=1,
)
 
response = predictor.predict_async(input_path=f"s3://{bucket}/large-batch-input.csv")

For requests that genuinely take longer than a typical synchronous HTTP call should reasonably block for — processing a large input payload, a computationally heavy model — asynchronous inference queues the request and writes the result to S3 once complete, rather than holding a connection open and waiting. This is a genuinely different pattern than either real-time or serverless: neither of those is well-suited to a request that might legitimately take minutes to process.

Async inference also supports scaling all the way down to zero instances during idle periods, combining the "no cost when idle" benefit of serverless with support for genuinely long-running requests serverless's concurrency model doesn't accommodate well — worth considering specifically when a workload has both properties (infrequent, and individually slow) at once.

Multi-model endpoints: consolidating cost across many models

from sagemaker.multidatamodel import MultiDataModel
 
mdm = MultiDataModel(
    name="multi-churn-models",
    model_data_prefix=f"s3://{bucket}/multi-models/",
    role=role,
)
 
predictor = mdm.deploy(initial_instance_count=1, instance_type="ml.m5.xlarge")
result = predictor.predict(data, target_model="customer-segment-a.tar.gz")

An organization serving many similar models — one per customer segment, one per region — behind separate real-time endpoints pays for each endpoint's infrastructure independently, even though most of them see comparatively low individual traffic. A multi-model endpoint hosts many models behind one endpoint's infrastructure, loading a specific model's artifact into memory on demand based on the target_model parameter — meaningfully reducing infrastructure cost for exactly this "many similar, individually lower-traffic models" pattern, at the cost of a load-time delay the first time a specific model is requested after being evicted from memory.

Testing a new model version safely: shadow and canary deployments

Deploying a new model version directly to a production endpoint risks a regression reaching all traffic immediately, the same risk covered for a bad rollout in our Kubernetes and GitHub Actions posts. SageMaker supports two mitigation patterns directly:

from sagemaker.model import Model
 
predictor.update_endpoint(
    production_variants=[
        {"VariantName": "current", "ModelName": "churn-model-v1", "InitialVariantWeight": 90, "InstanceType": "ml.m5.large", "InitialInstanceCount": 1},
        {"VariantName": "canary", "ModelName": "churn-model-v2", "InitialVariantWeight": 10, "InstanceType": "ml.m5.large", "InitialInstanceCount": 1},
    ]
)

A canary deployment like this one routes a small percentage of real traffic (10% here) to a new model version, comparing its actual production behavior against the existing version before shifting more traffic to it — the same gradual-rollout principle behind a Kubernetes Deployment's rolling update, or a GitHub Actions deploy gated on staging succeeding first, applied here to model versions specifically. A shadow deployment goes a step further, sending a full copy of live traffic to a new model version without its predictions actually being returned to users at all — purely for comparing behavior against the current version under genuine production load, with zero risk to real users if the new version behaves unexpectedly.

Attaching a custom inference container

The examples so far deploy a model trained with a SageMaker built-in algorithm, using its matching pre-built serving container automatically. A model with genuinely custom preprocessing or postprocessing logic — normalizing input features a specific way, applying a business rule to a raw prediction before returning it — needs its own inference code, supplied via a custom entry point script rather than relying purely on the default container behavior:

from sagemaker.pytorch import PyTorchModel
 
model = PyTorchModel(
    model_data=f"s3://{bucket}/model-output/model.tar.gz",
    role=role,
    entry_point="inference.py",
    framework_version="2.1",
)
 
predictor = model.deploy(initial_instance_count=1, instance_type="ml.m5.large")

# inference.py
def model_fn(model_dir):
    return load_model(model_dir)
 
def predict_fn(input_data, model):
    raw_prediction = model.predict(input_data)
    return apply_business_rules(raw_prediction)  # custom postprocessing

model_fn and predict_fn are the specific hooks SageMaker's framework containers expect — the same "the platform defines the interface, you supply the implementation" pattern as a Kubernetes readiness probe or a Bedrock Agent's Lambda handler, covered elsewhere in this blog, applied here to how a served model actually processes a request.

Invoking an endpoint directly, without the SDK wrapper

predictor.predict() wraps a lower-level API call worth knowing directly, since it's what a non-Python client (a Lambda function written in another language, an application backend) actually calls:

runtime = boto3.client("sagemaker-runtime")
response = runtime.invoke_endpoint(
    EndpointName="churn-prediction-endpoint",
    ContentType="text/csv",
    Body="5.1,3.5,1.4,0.2",
)
print(response["Body"].read())

This is the same underlying boto3 client used for every other AWS service call throughout this blog — a real-time endpoint, once deployed, is reachable from any application with appropriate IAM permissions, not only from the SageMaker Python SDK used to create it.

Serving multiple model versions behind one variant during evaluation

Before committing to a canary split with real traffic, predictor.predict() against a specific TargetVariant lets you send a test request to a chosen version explicitly, useful for smoke-testing a new variant before it receives any real production traffic at all:

result = predictor.predict(new_customer_data, target_variant="canary")

Choosing between them

• Real-time: continuous, latency-sensitive, unpredictable traffic — a live production application.
• Serverless: intermittent, low-volume traffic where occasional cold-start latency is an acceptable trade for not paying for idle capacity.
• Asynchronous: requests that genuinely take longer than a synchronous call should reasonably block for — and can also scale to zero when idle.
• Multi-model: many similar models, each individually lower-traffic, where consolidating infrastructure meaningfully reduces total cost.

What to actually remember from this post

• Real-time endpoints stay running continuously — lowest latency, at the cost of paying for capacity even during idle periods.
• Serverless endpoints scale to zero, trading occasional cold-start latency for genuinely pay-per-use cost — the right fit for intermittent traffic specifically.
• Asynchronous inference is for requests too long for a synchronous call — queued, with results delivered to S3 rather than blocking a connection, and can also scale to zero when idle.
• Multi-model endpoints consolidate infrastructure cost across many similar, individually lower-traffic models behind one endpoint.
• Canary and shadow deployments let you test a new model version against real traffic safely before shifting all traffic to it — the same gradual-rollout discipline covered elsewhere in this blog for Kubernetes and CI/CD.

Next in the series: SageMaker Pipelines: Automating the ML Lifecycle, where training and deployment stop being manual notebook steps and become a repeatable, automated pipeline.