Cloud Computing
AWS Core Services: IAM, VPC, EC2, S3, RDS, Lambda and the Glue Between Them
Beginner11 min readUpdated
AWS has hundreds of services, but a dozen of them carry almost every workload. This guide explains what each core service is for, how they fit together in a typical three-tier deployment, and the configuration decisions that matter, with IAM policy, CLI and CloudFormation examples you can adapt. You will be able to read an AWS architecture diagram and know what each box does and why it is there.
What you will learn
- Describe the role of IAM users, roles and policies and write a least-privilege policy document
- Lay out a VPC with public and private subnets, route tables, security groups and gateways
- Choose between EC2, containers on ECS/EKS and Lambda for a given workload
- Pick the right storage and database service — EBS, S3, RDS, Aurora or DynamoDB — for a given access pattern
- Connect components with SQS, SNS and EventBridge and observe them with CloudWatch and CloudTrail
Amazon Web Services publishes well over two hundred services, yet a typical production system uses perhaps fifteen, and the same fifteen appear again and again. This guide is a tour of those: what each is for, the decisions you make when you configure it, and how it plugs into its neighbours. It assumes you have read the cloud fundamentals guide and know what a region, an availability zone and the shared-responsibility model are. Service names are the current official ones; where AWS renamed something recently, the old name is mentioned once.
Accounts, organisations and IAM
Everything in AWS lives inside an account, which is both a billing boundary and a hard security boundary. Organisations rarely run on a single account: AWS Organizations lets you create many accounts under one management account, group them into organisational units and apply service control policies (SCPs) that cap what any identity inside those accounts may do — for example, forbidding the creation of resources outside approved regions. Separate accounts for production, non-production, security tooling and log archiving is the standard baseline.
Inside an account, AWS Identity and Access Management (IAM) decides who can call which API on which resource. The vocabulary:
- Principals are identities: IAM users (long-lived, with a password or access keys), IAM roles (assumed temporarily, no permanent credentials) and federated identities from an external provider through IAM Identity Center or SAML/OIDC.
- Policies are JSON documents that allow or deny actions on resources, optionally under conditions. Identity-based policies attach to principals; resource-based policies attach to the resource itself (an S3 bucket, a queue, a KMS key).
- Roles are the workhorse. A role has a trust policy saying who may assume it and permission policies saying what the assumer may do. EC2 instances, Lambda functions, ECS tasks and Kubernetes pods all obtain their permissions by being associated with a role; the platform injects short-lived credentials that rotate automatically.
The evaluation logic is simple to state and worth memorising: everything is denied by default; an explicit Allow in an applicable policy grants access; an explicit Deny anywhere overrides every Allow; SCPs and permission boundaries set an outer ceiling that identity policies cannot exceed.
A least-privilege policy for a service that reads objects from one bucket prefix and writes to one queue looks like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReadInvoices",
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": "arn:aws:s3:::acme-billing-prod/invoices/*"
},
{
"Sid": "ListInvoicePrefixOnly",
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": "arn:aws:s3:::acme-billing-prod",
"Condition": { "StringLike": { "s3:prefix": ["invoices/*"] } }
},
{
"Sid": "EnqueueWork",
"Effect": "Allow",
"Action": ["sqs:SendMessage"],
"Resource": "arn:aws:sqs:eu-west-1:123456789012:invoice-processing"
}
]
}
Every resource is identified by an Amazon Resource Name (ARN) with the pattern arn:partition:service:region:account-id:resource. Global services such as S3 and IAM leave the region field empty. Wildcards in the Resource field are where most over-permissive policies come from; "Resource": "*" combined with "Action": "s3:*" gives the principal every bucket in the account.
Networking: the VPC
An Amazon Virtual Private Cloud (VPC) is a logically isolated network in one region with an IPv4 CIDR range you choose (and optionally IPv6). Inside it you create subnets, each bound to one availability zone. A subnet is public if its route table sends 0.0.0.0/0 to an internet gateway; otherwise it is private. That distinction is entirely a routing decision — there is no other flag.
A conventional layout for a three-tier application in three zones:
VPC 10.20.0.0/16 (eu-west-1)
├── public-a 10.20.0.0/24 ALB nodes, NAT gateway → route 0.0.0.0/0 → internet gateway
├── public-b 10.20.1.0/24 ALB nodes, NAT gateway
├── public-c 10.20.2.0/24 ALB nodes, NAT gateway
├── app-a 10.20.10.0/24 EC2 / ECS tasks → route 0.0.0.0/0 → NAT gateway (same AZ)
├── app-b 10.20.11.0/24
├── app-c 10.20.12.0/24
├── data-a 10.20.20.0/24 RDS primary → no default route at all
├── data-b 10.20.21.0/24 RDS standby
└── data-c 10.20.22.0/24
Application instances sit in private subnets and reach the internet for outbound calls through a NAT gateway that lives in a public subnet. Deploy one NAT gateway per zone; a single shared one is a cross-zone dependency and a cross-zone data-transfer charge. Database subnets have no route to the internet at all.
Traffic control has two layers. Security groups are stateful firewalls attached to network interfaces, and their best feature is that a rule can reference another security group: "allow 5432 from the security group of the application tier" survives instances being replaced, whereas IP-based rules do not. Network ACLs are stateless, subnet-level lists; leave them at their permissive default unless you have a specific blocking requirement.
Two more pieces appear in almost every real VPC. VPC endpoints let private subnets reach AWS services such as S3, DynamoDB, Secrets Manager and ECR without traversing the internet or a NAT gateway. VPC peering or, at scale, AWS Transit Gateway connects VPCs to each other and to on-premises networks over AWS Site-to-Site VPN or AWS Direct Connect.
Compute: EC2, containers and Lambda
Amazon Elastic Compute Cloud (EC2) provides virtual machines. The choices that matter when launching one are the Amazon Machine Image (AMI), which is the boot disk template; the instance type, whose family letter encodes the purpose (general purpose, compute-, memory- or storage-optimised, accelerated) and whose size sets vCPU and memory; the subnet and security groups; the IAM instance profile carrying its role; and user data, a script run at first boot. Instances have a lifecycle (pending, running, stopping, stopped, terminated) and, importantly, the instance store disk on some types vanishes on stop, whereas EBS volumes persist.
Purchasing options change the bill without changing the machine: On-Demand for flexibility, Savings Plans or Reserved Instances for commitment discounts on steady workloads, and Spot Instances — spare capacity that AWS can reclaim with a two-minute warning — for interruptible batch and stateless workloads.
An Auto Scaling group keeps a desired number of instances running from a launch template, replaces unhealthy ones, spreads them across the subnets you list, and adjusts the count according to scaling policies (target tracking on CPU or request count is the usual starting point). In front of it, Elastic Load Balancing offers an Application Load Balancer for HTTP/HTTPS with path- and host-based routing, a Network Load Balancer for TCP/UDP at very high throughput with static IPs, and a Gateway Load Balancer for virtual appliances.
# Launch one instance in a private subnet with a role and a startup script
aws ec2 run-instances \
--image-id ami-0abcdef1234567890 \
--instance-type t3.small \
--subnet-id subnet-0a1b2c3d \
--security-group-ids sg-0f1e2d3c \
--iam-instance-profile Name=order-service-profile \
--user-data file://bootstrap.sh \
--tag-specifications 'ResourceType=instance,Tags=[{Key=service,Value=orders},{Key=env,Value=prod}]'
Containers have two orchestrators. Amazon Elastic Container Service (ECS) is AWS's own scheduler, simpler to operate, with AWS Fargate as a serverless launch type that removes the need to manage hosts at all. Amazon Elastic Kubernetes Service (EKS) runs upstream Kubernetes for teams that want portability or already have Kubernetes skills; see the Kubernetes fundamentals guide for the orchestration model itself. Both pull images from Amazon Elastic Container Registry (ECR).
AWS Lambda runs a function in response to an event — an HTTP request through API Gateway or a function URL, a new S3 object, a queue message, a schedule — with no instance to manage. You choose a runtime (Java, Python, Node.js, .NET, Go and custom runtimes via containers), a memory size that also scales the CPU share, and a timeout with a hard upper limit measured in minutes. Lambda suits event-driven, bursty or glue workloads; long-running processes, workloads needing persistent connections, and anything sensitive to cold-start latency are better on containers.
| Need | Reach for |
|---|---|
| Full control of the OS, licences bound to cores, legacy software | EC2 |
| Long-running services packaged as containers, predictable load | ECS on Fargate or EKS |
| Event-driven, bursty, short tasks; minimal operations | Lambda |
| Batch jobs tolerant of interruption | EC2 Spot via Auto Scaling or AWS Batch |
Storage: EBS, EFS and S3
Amazon Elastic Block Store (EBS) provides block volumes that attach to one EC2 instance in the same availability zone — the disk of a virtual machine. Volume types trade cost for throughput and IOPS; general-purpose SSD (gp3) is the default choice, provisioned-IOPS SSD (io2) exists for demanding databases. Snapshots are incremental copies stored in S3 and are how you back up and clone volumes. Because a volume is zone-bound, an instance in another zone cannot mount it; that constraint alone rules EBS out as shared storage.
Amazon Elastic File System (EFS) is a managed NFS file system mountable from many instances across zones at once, for workloads that genuinely need a shared POSIX file system such as content management systems or lift-and-shift applications that write to a shared directory.
Amazon Simple Storage Service (S3) is object storage: buckets hold objects addressed by key, accessed over HTTPS, with no file system semantics and effectively unlimited capacity. It is the default home for backups, logs, static websites, data-lake files, media and anything else that is written once and read many times. Features worth knowing on day one:
- Storage classes (Standard, Intelligent-Tiering, Standard-IA, One Zone-IA, Glacier Instant/Flexible/Deep Archive) trade retrieval speed and per-request cost against storage cost; lifecycle rules move objects between classes and expire them.
- Versioning keeps every overwrite and delete, which is your protection against accidental deletion and ransomware-style overwrites.
- Block Public Access at the account level should be on unless you are deliberately serving a public site.
- Server-side encryption is on by default; you can choose an AWS-managed key or your own key in AWS Key Management Service (KMS).
- Pre-signed URLs grant time-limited access to a single object without making the bucket public.
aws s3api put-bucket-versioning --bucket acme-billing-prod \
--versioning-configuration Status=Enabled
aws s3 cp ./reports/2026-09.pdf s3://acme-billing-prod/reports/2026-09.pdf \
--storage-class STANDARD_IA
Databases: RDS, Aurora and DynamoDB
Amazon Relational Database Service (RDS) runs managed instances of PostgreSQL, MySQL, MariaDB, Oracle Database, Microsoft SQL Server and Db2. AWS handles provisioning, patching in a maintenance window, automated backups with point-in-time recovery, and — with Multi-AZ enabled — a synchronously replicated standby in another zone with automatic failover. Read replicas are asynchronous copies for scaling reads and are not a substitute for Multi-AZ. You still own the schema, the queries, the indexes and the parameter group.
Amazon Aurora is AWS's own relational engine, wire-compatible with PostgreSQL and MySQL, with storage replicated six ways across three zones and failover measured in seconds. Aurora Serverless variants scale capacity with load. Choose Aurora when you want RDS ergonomics with higher availability and read scaling and are not tied to another engine.
Amazon DynamoDB is a fully managed key-value and document database with consistent single-digit-millisecond latency at any scale. Tables have a partition key (and optionally a sort key), and you query by those keys or through secondary indexes you design in advance. Capacity is on-demand or provisioned; there is no server, no patching and no connection pool to size. The cost of that simplicity is that access patterns must be known and modelled up front.
Amazon ElastiCache (Redis or Valkey and Memcached) covers caching and session stores; Amazon OpenSearch Service covers full-text search and log analytics.
Messaging and integration
Decoupling components with queues and events is how AWS architectures absorb bursts and isolate failures.
Amazon Simple Queue Service (SQS) is a message queue. Standard queues offer at-least-once delivery with best-effort ordering and near-unlimited throughput; FIFO queues guarantee order and exactly-once processing within a message group at lower throughput. A consumer receives a message, processes it, and deletes it; if it fails to delete within the visibility timeout the message reappears, and after a configured number of attempts it lands in a dead-letter queue.
Amazon Simple Notification Service (SNS) is publish/subscribe: a topic fans a message out to many subscribers — SQS queues, Lambda functions, HTTPS endpoints, e-mail, SMS. The SNS-to-SQS fan-out is the standard pattern when several services need the same event.
Amazon EventBridge is an event bus with content-based rules that route events from AWS services, your applications and SaaS partners to targets, plus a scheduler. It replaces the older CloudWatch Events. For orchestrating multi-step workflows with retries and branching, AWS Step Functions provides a state machine.
Edge, DNS and observability
Amazon Route 53 is DNS with health checks and routing policies (weighted, latency, failover, geolocation) that turn DNS into a traffic-steering tool. Amazon CloudFront is the content delivery network; it caches at edge locations, terminates TLS with certificates from AWS Certificate Manager, and fronts S3 buckets, load balancers or any HTTP origin. AWS WAF attaches to CloudFront or an ALB to filter requests.
Amazon CloudWatch collects metrics, logs and alarms. Every service emits metrics automatically; your applications push logs through the CloudWatch agent or the Lambda runtime; alarms fire on thresholds and can trigger Auto Scaling or SNS notifications. AWS CloudTrail records every API call in the account — who did what, from where, when — and is your audit trail; enable it in every region and deliver it to a locked-down bucket in the log-archive account. AWS X-Ray adds distributed tracing.
Configuration and secrets have homes too: AWS Systems Manager Parameter Store for configuration values and AWS Secrets Manager for credentials with automatic rotation. Reading them at start-up via the instance role is the cloud-native replacement for property files with passwords inside — a practice the twelve-factor guide develops further.
Putting it together in CloudFormation
AWS CloudFormation describes resources in YAML or JSON templates and creates, updates or deletes them as a stack. It is the native infrastructure-as-code option; Terraform and the AWS CDK are common alternatives, and the infrastructure-as-code guide compares them. A minimal stack with a queue, its dead-letter queue and a role a service could assume:
AWSTemplateFormatVersion: "2010-09-09"
Description: Invoice processing queue and worker role
Resources:
InvoiceDlq:
Type: AWS::SQS::Queue
Properties:
MessageRetentionPeriod: 1209600 # 14 days, the maximum
InvoiceQueue:
Type: AWS::SQS::Queue
Properties:
VisibilityTimeout: 120 # longer than the worst-case processing time
RedrivePolicy:
deadLetterTargetArn: !GetAtt InvoiceDlq.Arn
maxReceiveCount: 5
WorkerRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal: { Service: ecs-tasks.amazonaws.com } # only ECS tasks may assume it
Action: sts:AssumeRole
Policies:
- PolicyName: consume-invoices
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: [sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes]
Resource: !GetAtt InvoiceQueue.Arn
Outputs:
QueueUrl:
Value: !Ref InvoiceQueue
Common mistakes
Access keys in code or CI variables. Use roles on compute and OIDC federation from CI systems such as GitHub Actions; if a key must exist, scope it narrowly and rotate it.
One NAT gateway, one zone. When that zone fails, every private subnet loses internet access. Deploy per zone and route each private subnet to its local gateway.
Read replicas as high availability. A replica is asynchronous and promotion is manual unless you build it. Multi-AZ is the availability feature; replicas are for read scaling.
Public S3 buckets by accident. Keep account-level Block Public Access on and serve public content through CloudFront with an origin access control instead.
Ignoring CloudTrail and cost allocation tags. Without CloudTrail you cannot answer "who deleted that", and without a consistent service/env/owner tagging scheme you cannot answer "who is paying for this". Both are cheap to set up on day one and painful to retrofit.
Key takeaways
- Accounts are the security and billing boundary; IAM roles with short-lived credentials are how workloads get permissions, and policies deny by default.
- A VPC is your private network: public subnets route to an internet gateway, private ones through per-zone NAT gateways or VPC endpoints, and security groups reference each other rather than IPs.
- EC2 gives full control, ECS/EKS runs containers, Lambda runs event-driven functions; pick by control needed and duty cycle, not by fashion.
- EBS is a zone-bound disk, EFS a shared file system, S3 the durable object store with versioning, lifecycle rules and Block Public Access.
- RDS and Aurora cover relational needs with Multi-AZ failover; DynamoDB covers key-based access at any scale when patterns are known in advance.
- SQS, SNS and EventBridge decouple components; CloudWatch, CloudTrail and X-Ray tell you what is happening; CloudFormation makes all of it reproducible.
Sources and further reading
Frequently asked questions
- Should my application use an IAM user with access keys or an IAM role?
- A role, almost always. Code running on EC2, ECS, EKS or Lambda can be given a role, and the platform delivers short-lived credentials to it automatically through the instance metadata service or the container credential endpoint. Long-lived access keys belong to IAM users and end up in repositories, laptops and CI logs. Reserve users for the rare human who cannot federate through an identity provider, and rotate their keys.
- What is the difference between a security group and a network ACL?
- A security group is a stateful firewall attached to a network interface: if you allow inbound port 443, the reply traffic is allowed automatically, and rules can reference other security groups instead of IP ranges. A network ACL is a stateless allow/deny list attached to a subnet, evaluated in rule order, where you must open both directions. Security groups do most of the work; network ACLs are a coarse second layer for blocking known-bad ranges.
- When should I choose DynamoDB over RDS?
- Choose DynamoDB when you know the access patterns up front, they are key-based lookups or queries within a partition, you need single-digit-millisecond latency at any scale, and you do not need ad-hoc joins or transactions across many tables. Choose RDS or Aurora when the data model is relational, queries are varied, or you rely on SQL features, ORMs and reporting tools. Modelling a relational schema in DynamoDB and then discovering you need a join is the classic mistake.
- Do I need a NAT gateway?
- Only if instances in private subnets must reach the internet — to download packages, call third-party APIs or reach AWS services that you have not exposed through VPC endpoints. A NAT gateway is billed per hour and per gigabyte processed, so for traffic that goes only to S3, DynamoDB or other AWS services, a gateway or interface VPC endpoint is both cheaper and more secure.