cloudcostdown.euAWS AI/ML Cost Optimization · Startup Guide
AWS AI/ML Cost Optimization: Reduce SageMaker & Bedrock Costs 40–70%
AI/ML is the fastest-growing AWS cost for tech startups. SageMaker Spot training, Bedrock model selection, multi-model endpoints, and idle resource cleanup can cut AI infrastructure costs 40–70% without changing your outputs.
Why AI/ML AWS Costs Surprise Teams
The model selection problem
Most teams pick a model that works and never revisit the selection. Claude Sonnet costs 12× more than Claude Haiku. GPT-4 costs 100× more than Llama 3 for the same token count. For classification, extraction, and summarization tasks, cheaper models often achieve equivalent quality.
Audit every model in production. For each: run the task on a cheaper model with your real data and measure quality. In most cases, you’ll find 2–3 tasks where a cheaper model works just as well.
The idle resource problem
SageMaker endpoints, notebook instances, and Studio kernels charge by the hour - whether or not they’re processing requests. A data science team’s exploratory notebooks left running over a weekend can generate $500–2,000 in charges before anyone notices.
Enable auto-shutdown policies for all SageMaker resources and audit running instances weekly.
5 AI/ML Cost Optimizations for AWS
Apply these in order - highest ROI first.
Use SageMaker Spot training for model training jobs
SageMaker Managed Spot training uses EC2 Spot instances for training jobs, delivering up to 90% savings compared to on-demand. Unlike regular Spot instances, SageMaker handles checkpointing and automatic job restart on interruption - you just enable it and set a max wait time.
How to implement
- In your SageMaker Estimator: set use_spot_instances=True and max_wait=7200 (seconds)
- Configure checkpoint_s3_uri to save checkpoints to S3 so training resumes after interruption
- Set max_run to your estimated training time - the job runs until completion or max_wait
- Use training_max_wait_time in the TrainingJobDefinition for SageMaker Pipelines
- Monitor spot interruptions in SageMaker console → Training jobs → Events tab
Note: Spot interruptions during SageMaker training automatically resume from the last checkpoint. For jobs under 4 hours without GPU scarcity, interruption rates are typically under 10%.
Choose the right Bedrock pricing tier
Amazon Bedrock offers three pricing modes: on-demand (pay per token, highest flexibility), batch inference (25% discount for async workloads), and Provisioned Throughput (committed throughput for consistent latency). Most startups use on-demand when batch or provisioned would be significantly cheaper.
How to implement
- Audit your Bedrock usage type: real-time interactive (on-demand), bulk processing (batch), or constant high-volume (provisioned)
- For document processing, content generation, and batch classification: use Bedrock Batch Inference (25% discount)
- For consistent high-volume inference: evaluate Provisioned Throughput - commit to model units for predictable throughput and cost
- Use the cheapest model that meets your quality bar: Claude Haiku vs. Sonnet vs. Opus - cost differences are 10–100×
- Enable Bedrock Model Evaluation to compare quality vs. cost across model tiers
Note: The biggest AI/ML cost lever is model selection. Claude Haiku costs $0.00025 per 1K input tokens; Claude Sonnet costs $0.003 - 12× more expensive. Many tasks (classification, extraction, summarization) achieve the same quality on Haiku.
Consolidate SageMaker endpoints with multi-model endpoints
SageMaker real-time endpoints charge for the underlying instance regardless of traffic. A startup with 10 models each on separate ml.m5.large endpoints pays 10× the cost of hosting all 10 on a single multi-model endpoint. Multi-model endpoints load models from S3 on demand and keep frequently accessed models in memory.
How to implement
- Package models in the SageMaker multi-model format (tar.gz in S3)
- Create a Multi-Model Endpoint with MULTI_MODEL mode: create-endpoint --endpoint-name mmep --endpoint-config-name config
- Invoke specific models: pass TargetModel=model_name.tar.gz in the InvokeEndpoint call
- Configure auto scaling on the endpoint to scale instances based on InvocationsPerInstance metric
- Monitor per-model latency in CloudWatch - add instances if model loading causes latency spikes
Note: Multi-model endpoints work best for models with similar memory footprints and when not all models receive traffic simultaneously. Large foundation models (LLMs) are not compatible - use for smaller specialist models.
Delete idle SageMaker Studio resources
SageMaker Studio domains create persistent EBS volumes and may have idle notebook kernels, running apps, and space instances that charge by the hour even when no work is happening. A team of 5 data scientists with idle ml.t3.medium instances left running overnight each week accumulates thousands in monthly waste.
How to implement
- SageMaker Console → Studio → Domains → Users → Running instances
- Identify notebook instances that have been idle for hours/days (no kernel activity)
- Shut down idle kernels and apps: JupyterLab → Running Terminals and Kernels → Shut down all
- Enable auto-shutdown: set StudioWebPortal shutdown timeout in domain settings
- Use SageMaker lifecycle configurations to auto-stop idle kernels after 30 minutes
Note: This is especially common in data science teams where notebooks are started for exploration and left running overnight or over weekends. A single ml.p3.2xlarge instance costs $3.06/hour - $22/night, $66/weekend.
Use SageMaker Savings Plans for predictable inference
SageMaker Savings Plans provide a discount of up to 64% on SageMaker ML instance usage (training, hosting, processing jobs, Studio) in exchange for a 1-year commitment. If you have stable SageMaker usage - production endpoints that run 24/7 - this is a straightforward discount.
How to implement
- AWS Cost Management → Savings Plans → Discover Savings Plans → SageMaker Savings Plans
- Review recommendations based on your 90-day SageMaker usage
- Purchase a 1-year no-upfront plan for your baseline ML instance commitment
- Plans apply automatically to all SageMaker ML instance usage in the account
- Review coverage monthly in Savings Plans utilization report
Note: SageMaker Savings Plans are separate from Compute Savings Plans and only cover SageMaker ML instances. If you use SageMaker Serverless Inference, that usage is not covered - only on-demand ML instances.
Frequently Asked Questions
What is the biggest AI/ML cost on AWS for startups?
For startups building with LLMs: usually Bedrock inference costs. For startups training custom models: SageMaker training compute. The key is matching the model tier to the task - using a large, expensive model for tasks a smaller model handles equally well is the most common form of AI/ML waste.
Is AWS Bedrock cheaper than OpenAI?
It depends on the model. Bedrock hosts Anthropic Claude, Meta Llama, Amazon Titan, and other models. Claude Haiku on Bedrock ($0.00025/1K tokens) is significantly cheaper than GPT-4 ($0.03/1K tokens). For many tasks, Haiku or Llama 3 on Bedrock is the most cost-effective option.
How do SageMaker Spot instances handle training interruptions?
SageMaker automatically restarts interrupted Spot training jobs from the last checkpoint. You configure a checkpoint S3 path, and SageMaker saves checkpoints there during training. When the job resumes, it loads the last checkpoint and continues from that point - no manual intervention needed.
What is a SageMaker multi-model endpoint?
A single SageMaker real-time endpoint that hosts multiple models, loading them from S3 on demand. You invoke a specific model by passing its S3 key as TargetModel. Models are cached in memory on the instance - frequently accessed models stay loaded, rarely accessed ones are evloaded to free memory.
Does AWS offer discounts for AI/ML workloads?
Yes: SageMaker Managed Spot Training (60–90% off), SageMaker Savings Plans (up to 64% off ML instances), Bedrock Batch Inference (25% off), and Provisioned Throughput (negotiated rates). For large enterprise commitments, AWS Enterprise Discount Program (EDP) applies across all services including AI/ML.