Choosing the Right Foundation Model on Amazon Bedrock

This is the fifth and final post in our Amazon Bedrock series, building on getting started, Knowledge Bases, Agents, and Guardrails.

Bedrock's actual value proposition — one consistent API and IAM model across multiple providers' foundation models, covered in the first post in this series — only pays off if you're genuinely choosing between models deliberately, rather than picking one and never revisiting the decision. We flagged the same instinct for routing tasks to a smaller, cheaper model in our OpenAI series; Bedrock's multi-provider access makes that same discipline available across an even broader set of options.

The trade-offs that actually matter

• Capability vs. cost vs. latency. A larger, more capable model (Claude 3.5 Sonnet, for instance) genuinely outperforms a smaller one (Claude 3 Haiku, or Amazon's Titan Lite) on complex reasoning, nuanced writing, and multi-step tasks — and costs meaningfully more per token, with correspondingly higher latency. A classification task, a simple extraction task, or the internal summarization pattern covered in our OpenAI context-management post rarely needs the larger model's additional capability at all.
• Context window size, if an application genuinely needs to process long documents in a single call, rather than relying on chunked retrieval (covered throughout our RAG posts) to surface only the relevant portion.
• Fine-tuning support varies by model and provider — relevant specifically if the style-and-format-consistency fine-tuning case covered in our OpenAI series applies to your use case, on a model actually available through Bedrock.

Comparing models directly, on your own actual data

import boto3
import json
 
client = boto3.client("bedrock-runtime", region_name="eu-west-2")
 
models_to_compare = [
    "anthropic.claude-3-haiku-20240307-v1:0",
    "anthropic.claude-3-5-sonnet-20241022-v2:0",
    "meta.llama3-1-70b-instruct-v1:0",
]
 
test_prompt = "Summarize the key benefits of Kubernetes autoscaling in 2 sentences."
 
for model_id in models_to_compare:
    if model_id.startswith("anthropic"):
        body = {
            "anthropic_version": "bedrock-2023-05-31",
            "max_tokens": 200,
            "messages": [{"role": "user", "content": test_prompt}],
        }
    else:
        body = {"prompt": test_prompt, "max_gen_len": 200}
 
    response = client.invoke_model(modelId=model_id, body=json.dumps(body))
    result = json.loads(response["body"].read())
    print(f"--- {model_id} ---")
    print(result.get("content", result.get("generation")))

Because Bedrock exposes multiple providers through the same client and the same billing account, direct comparison on your own real prompts — not a generic public benchmark that may not reflect your actual task at all — is genuinely straightforward to set up. This is the practical payoff of Bedrock's consolidation: evaluating three different providers' models costs nothing beyond the API calls themselves, with no separate account, contract, or billing relationship required for any of them.

Building this into a repeatable evaluation, not a one-off test

A single test prompt is a reasonable starting point; a real decision deserves the same evaluation-dataset discipline covered for LangChain applications in an earlier series — a set of real examples from your actual use case, with a consistent scoring method applied across every candidate model:

test_cases = [
    {"prompt": "Summarize the key benefits of Kubernetes autoscaling in 2 sentences.", "expected_topics": ["HPA", "cost", "traffic"]},
    {"prompt": "Classify this support ticket urgency: 'my production database is down'", "expected": "high"},
]
 
def score_model(model_id, test_cases):
    results = []
    for case in test_cases:
        response = invoke(model_id, case["prompt"])
        results.append(evaluate_against_expected(response, case))
    return sum(results) / len(results)
 
for model_id in models_to_compare:
    print(model_id, score_model(model_id, test_cases))

Running the same evaluation harness against every candidate model, on the same set of real test cases, turns "which model feels better" into an actual measured comparison — genuinely important once a model choice is meant to be revisited periodically rather than decided once and left alone, since a repeatable harness makes re-evaluating against a newly released model a matter of re-running the same script, not redesigning the comparison from scratch each time.

A framework for the actual decision

• Start with the smallest, cheapest model that could plausibly handle the task — Claude 3 Haiku or a similarly-sized model — and evaluate its output on real examples from your own use case, not a hypothetical one.
• Escalate to a larger model specifically when evaluation reveals a genuine capability gap — noticeably worse reasoning, missed nuance, or an accuracy shortfall on your own real test cases — rather than defaulting to the largest available model preemptively, "just in case."
• Re-evaluate periodically, not just once. Model pricing and capability both shift as providers release new versions — a choice that was correct six months ago is worth revisiting, the same "rebuild and rescan on a schedule" instinct we recommended for Docker base images earlier in this blog, applied here to model selection instead.

Provisioned throughput: a distinct decision from model choice

aws bedrock create-provisioned-model-throughput \
  --model-id "anthropic.claude-3-5-sonnet-20241022-v2:0" \
  --provisioned-model-name "novuspark-production-throughput" \
  --commitment-duration "OneMonth" \
  --model-units 2

On-demand Bedrock pricing (pay per token, no capacity reservation) is the right default for most applications, including genuinely variable production traffic. Provisioned throughput — committing to reserved capacity for a specific model, at a different pricing structure — becomes worth evaluating specifically once an application has consistently high, predictable volume, where guaranteed throughput and a different cost structure at that specific volume level actually pay off. This is a distinct decision from which model to use at all, worth evaluating separately, and only once real production usage patterns are actually established and predictable.

Multi-model routing: using the framework in production, not just at decision time

The "start small, escalate on evaluated need" framework applies just as well as a live routing strategy, not only a one-time selection:

def route_request(task_type: str, prompt: str):
    if task_type in ("classification", "extraction", "summarization"):
        model_id = "anthropic.claude-3-haiku-20240307-v1:0"
    elif task_type == "complex_reasoning":
        model_id = "anthropic.claude-3-5-sonnet-20241022-v2:0"
    else:
        model_id = "anthropic.claude-3-haiku-20240307-v1:0"  # default to cheaper
 
    return invoke_model(model_id, prompt)

A production application routing different call sites to different models based on the actual task — rather than every call in the system defaulting to the same, largest model regardless of what it's actually being asked to do — is the direct operational expression of the model-selection discipline covered throughout this post, applied continuously rather than decided once at launch and never revisited per call site.

Latency as its own dimension, separate from throughput

Two models with similar published throughput numbers can still feel meaningfully different in a real, latency-sensitive application — time-to-first-token matters more than total tokens-per-second for anything a user is actively watching stream in:

import time
 
start = time.time()
response = client.invoke_model_with_response_stream(modelId=model_id, body=json.dumps(body))
first_chunk_time = None
for event in response["body"]:
    if first_chunk_time is None:
        first_chunk_time = time.time() - start
    # ... process chunk
print(f"Time to first token: {first_chunk_time:.2f}s")

Measuring time-to-first-token directly, alongside total generation time, is worth doing explicitly for any user-facing application — a model that's slightly slower overall but starts streaming a response noticeably faster often feels more responsive than one that's faster end-to-end but has a longer initial delay, the same perceived-versus-actual latency distinction covered for streaming responses in our OpenAI series.

Regional availability as a real constraint, not an afterthought

Not every model is available in every AWS region, and a model comparison that ignores this can end up choosing a model that isn't actually deployable where an application needs to run:

aws bedrock list-foundation-models --region eu-west-2 --query "modelSummaries[].modelId"

Checking regional availability early — particularly for an application with genuine data-residency requirements that constrain which region it can run in at all — avoids discovering, after a model comparison has already concluded, that the winning model simply isn't available in the region the rest of the application is committed to.

What to actually remember from this series

• Model choice is a real, ongoing decision — not a one-time pick, made once and never revisited, no differently than routing tasks to cheaper models is an active discipline in our OpenAI series.
• Bedrock's consolidated billing and IAM model make direct, low-friction comparison across providers genuinely practical — a real advantage over managing separate accounts and credentials per provider.
• Build a repeatable evaluation harness against real test cases, not a one-off manual comparison — it's what makes re-evaluating against new model releases a re-run, not a redesign.
• Start small, escalate based on evaluated capability gaps on your own real data — not a default assumption that the largest available model is always the safer choice.
• Route different call sites to different models based on actual task requirements, in production, not just at initial launch decision time.
• Provisioned throughput is a separate decision from model selection, worth evaluating only once real, predictable production volume actually justifies it.

That closes out our Amazon Bedrock series — from getting started through Knowledge Bases, Agents, Guardrails, and now model selection. If your team is building production AI applications on AWS, this is exactly the kind of hands-on work we build our AI & Generative AI training around.