title: Cartographer - Multi-Cloud Infrastructure Visualization slug: cartographer-cloud-visualization description: AWS-first infrastructure graph with pattern language and automated diagram mining status: Product published: published-wip category: Cloud Infrastructure technologies: - Python - FastAPI - Neo4j - React - D3.js - AWS CDK github: https://github.com/macleodlabs/cartographer date: 2025-01-15 featured: false hero: false
Cartographer - Multi-Cloud Infrastructure Visualization
Composable multi-cloud infrastructure graph ingestion and visualization with AWS Pattern Language (LoD 0-4).
The AWS Pattern Language
Traditional cloud diagrams fail at scale. Cartographer introduces Levels of Detail (LoD) for consistent, scalable architecture visualization:
LoD 0 (Executive) → Stakeholders, platforms, inter-cloud
LoD 1 (System) → Region + VPC, 6-12 services, no AZ
LoD 2 (Network) → VPC → AZ → Subnets, IGW/NAT
LoD 3 (Service) → Concrete services, sync/async flows
LoD 4 (Resource) → Instances, configs, scaling
Architecture
graph TB
subgraph "Data Ingestion"
CF[CloudFormation<br/>Templates]
GH[GitHub<br/>IaC Repos]
AWS[AWS APIs<br/>Cartography]
end
subgraph "Graph Storage"
NEO[Neo4j<br/>Graph Database]
RULES[Pattern Rules<br/>YAML]
end
subgraph "Processing Layer"
PARSER[CF Parser]
MINER[Diagram Miner]
RESOLVER[Relationship<br/>Resolver]
end
subgraph "Visualization"
UI[React UI<br/>Pattern Viewer]
D3[D3.js<br/>Renderer]
LAYOUT[Layout Engine<br/>Bottom-Up]
end
CF --> PARSER
GH --> PARSER
AWS --> NEO
PARSER --> RESOLVER
RESOLVER --> NEO
NEO --> UI
RULES --> LAYOUT
LAYOUT --> D3
D3 --> UI
Bottom-Up Layout Algorithm
Hierarchical space-filling layout:
class BottomUpLayoutEngine:
"""
Calculate node sizes from leaves upward,
then position from root downward.
"""
def calculate_node_sizes(self, graph: NetworkGraph) -> Dict[str, Size]:
"""
Phase 1: Bottom-up size calculation
"""
sizes = {}
# Process leaf nodes first (resources)
for node in graph.leaves():
sizes[node.id] = self.base_size(node.type)
# Process parents bottom-up
for level in reversed(range(graph.depth())):
for node in graph.nodes_at_level(level):
children = node.children()
child_sizes = [sizes[c.id] for c in children]
# Container must fit all children + padding
layout = self.pack_children(child_sizes, node.layout_type)
sizes[node.id] = Size(
width=layout.width + 2 * node.padding,
height=layout.height + 2 * node.padding
)
return sizes
def adjust_peer_dimensions(self, graph: NetworkGraph, sizes: Dict):
"""
Phase 2: Align peer components (VPCs, AZs)
for visual consistency
"""
adjusted = sizes.copy()
# Group nodes by parent and level
for parent in graph.parents():
peers = parent.children()
# Match heights, preserve content-based widths
max_height = max(sizes[p.id].height for p in peers)
for peer in peers:
adjusted[peer.id].height = max_height
return adjusted
def recalculate_cascading(self, graph: NetworkGraph,
original: Dict, adjusted: Dict):
"""
Phase 3: Cascade size changes to children
when parents were resized
"""
final = adjusted.copy()
changed = self.find_changed_nodes(original, adjusted)
for node in changed:
if node.type == "vpc":
# AZs expand to fill VPC height
for az in node.children():
available_height = final[node.id].height - 2 * node.padding
final[az.id].height = available_height
# Recursively update subnets
self.expand_children(az, final)
return final
def position_nodes(self, graph: NetworkGraph, sizes: Dict):
"""
Phase 4: Position nodes recursively from root
"""
positions = {}
def position_subtree(node, parent_bounds):
size = sizes[node.id]
# Center within parent bounds
x = parent_bounds.x + (parent_bounds.width - size.width) / 2
y = parent_bounds.y + (parent_bounds.height - size.height) / 2
positions[node.id] = Point(x, y)
# Position children based on node type
if node.type == "region":
self.position_vpcs_horizontally(node, positions, sizes)
elif node.type == "vpc":
self.position_azs_horizontally(node, positions, sizes)
elif node.type == "az":
self.position_subnets_by_scope(node, positions, sizes)
elif node.type == "subnet":
self.position_resources_grid(node, positions, sizes)
# Start from root
root = graph.root()
root_bounds = Bounds(0, 0, viewport.width, viewport.height)
position_subtree(root, root_bounds)
return positions
Pattern Language Implementation
LoD Rules Engine:
class PatternLanguage:
"""
AWS architecture pattern language with LoD rules
"""
def __init__(self):
self.rules = self.load_rules("schema/awsdl_rules.yaml")
def apply_lod_filter(self, graph: NetworkGraph, lod: int) -> NetworkGraph:
"""
Filter graph based on Level of Detail
"""
lod_config = self.rules["lod"][str(lod)]
filtered = NetworkGraph()
# Add nodes based on LoD configuration
for node in graph.nodes():
if node.kind in lod_config["show"]:
filtered.add_node(node)
# Collapse services if needed
if "collapse" in lod_config:
for pattern, label in lod_config["collapse"].items():
matching = filtered.find_pattern(pattern)
filtered.replace_with_group(matching, label)
# Add edges with style based on LoD
edge_styles = lod_config.get("edgeStyles", {})
for edge in graph.edges():
if edge.src in filtered and edge.dst in filtered:
edge.style = edge_styles.get(edge.type, "solid")
filtered.add_edge(edge)
return filtered
# Example LoD configuration
AWSDL_RULES = {
"lod": {
"0": { # Executive/Context
"show": ["region"],
"hide": ["vpc", "az", "subnet", "resource"],
"collapse": {
"microservices": "Service Group",
"event_mesh": "Event Mesh"
}
},
"1": { # System/Architecture
"show": ["region", "vpc"],
"hide": ["az", "subnet"],
"maxServiceIcons": 20
},
"2": { # Network/VPC
"show": ["region", "vpc", "az", "subnet"],
"hide": ["resource"],
"layout": {
"azColumns": True,
"subnetRows": ["public", "private", "db"]
}
},
"3": { # Service/Dataflow
"show": ["region", "vpc", "az", "subnet", "resource"],
"edgeStyles": {
"sync": "solid",
"async": "dashed",
"control": "dotted"
}
}
}
}
CloudFormation Import
Automatic template parsing:
class CloudFormationImporter:
"""
Import CloudFormation templates into Neo4j graph
"""
def import_template(self, template_path: str) -> ImportResult:
"""
Parse CF template and create graph nodes/relationships
"""
with open(template_path) as f:
cf = yaml.safe_load(f)
resources = cf.get("Resources", {})
stack_name = self.extract_stack_name(template_path)
# Create stack node
stack = self.create_node("Stack", name=stack_name)
# Create resource nodes
for logical_id, resource in resources.items():
resource_type = resource["Type"]
properties = resource.get("Properties", {})
node = self.create_node(
self.map_cf_type(resource_type),
logical_id=logical_id,
properties=properties
)
# Link to stack
self.create_relationship(stack, "CONTAINS", node)
# Resolve references
for key, value in properties.items():
if isinstance(value, dict) and "Ref" in value:
ref_target = value["Ref"]
if ref_target in resources:
target = self.find_node(logical_id=ref_target)
self.create_relationship(
node, "DEPENDS_ON", target
)
return ImportResult(
stack_id=stack.id,
resources_imported=len(resources)
)
GitHub Repository Scraping
Automatic IaC discovery:
def import_github_repo(repo_url: str, region: str = "us-east-1"):
"""
Scan GitHub repo for CloudFormation templates
and import into graph
"""
# Clone repo (or use GitHub API)
templates = find_cloudformation_templates(repo_url)
imported = []
for template in templates:
# Create synthetic stack name from file path
stack_name = f"github/{repo_url.split('/')[-1]}/{template.name}"
# Parse and import
result = cf_importer.import_template(
template.path,
stack_name=stack_name,
region=region,
source="github"
)
imported.append(result)
return {
"repository": repo_url,
"templates_found": len(templates),
"stacks_created": len(imported)
}
Diagram Mining System
Learn patterns from AWS reference diagrams:
┌──────────────────────────────────────────────┐
│ Diagram Mining Pipeline │
├──────────────────────────────────────────────┤
│ │
│ 1. Fetch AWS Icon Pack │
│ └─> Official architecture icons │
│ │
│ 2. Scrape AWS Reference Diagrams │
│ ├─> PPTX presentations │
│ ├─> PDF whitepapers │
│ └─> PNG diagrams │
│ │
│ 3. Parse Layout Features │
│ ├─> Container hierarchy │
│ ├─> Service placement │
│ ├─> Edge routing │
│ └─> Color schemes │
│ │
│ 4. Mine LoD Rules │
│ ├─> Which services at which LoD? │
│ ├─> Container visibility rules │
│ └─> Edge style patterns │
│ │
│ 5. Export YAML Rules │
│ └─> schema/awsdl_rules.yaml │
│ │
└──────────────────────────────────────────────┘
Neo4j Data Model
Graph schema for AWS infrastructure:
// Nodes
(:Region {name, id})
(:VPC {cidr, id})
(:AvailabilityZone {name, id})
(:Subnet {cidr, scope, id}) // scope: public|private|database
(:Resource {type, name, config})
(:Stack {name, template_url})
// Relationships
(Region)-[:CONTAINS]->(VPC)
(VPC)-[:CONTAINS]->(AvailabilityZone)
(AvailabilityZone)-[:CONTAINS]->(Subnet)
(Subnet)-[:HOSTS]->(Resource)
(Resource)-[:CONNECTS_TO {type}]->(Resource) // type: sync|async
(Stack)-[:DEFINES]->(Resource)
// Example query: Find all resources in VPC
MATCH (vpc:VPC {id: $vpc_id})-[:CONTAINS*]->(subnet:Subnet)
-[:HOSTS]->(resource:Resource)
RETURN resource.type, resource.name, subnet.scope
Interactive UI Features
React-based pattern viewer:
- LoD Selector - Switch between detail levels on the fly
- Drill-Down - Click to expand collapsed groups
- Scope Coloring - Public (green), Private (orange), DB (blue)
- Pan & Zoom - Smooth D3 interactions
- CloudFormation Import - Upload templates directly
- GitHub Integration - Import IaC repos by URL
Performance Metrics
Operation | Time | Details
---------------------------|-----------|---------------------------
Parse CF template | 50-200ms | 50-500 resources
Import to Neo4j | 100-500ms | With relationship resolution
Render LoD 2 diagram | 80-150ms | Single VPC, 3 AZs
Render LoD 3 diagram | 200-400ms | 20-50 services
Layout calculation | 30-100ms | Bottom-up algorithm
GitHub repo scan | 2-5s | 10-50 templates
Technical Stack
- Backend: FastAPI + Python 3.11
- Database: Neo4j 5.x (graph storage)
- Frontend: React 18 + D3.js
- Layout: Custom bottom-up algorithm
- Import: CloudFormation + GitHub API
- Export: SVG, PNG, Neo4j backup
Quick Start
cd cartographer
# Start Neo4j and API
docker compose up -d
# Import CloudFormation
curl -X POST http://localhost:8001/ingest/cloudformation \
-d '{"region": "us-east-1"}'
# Import GitHub repo
curl -X POST http://localhost:8001/ingest/github \
-d '{"github_url": "https://github.com/aws-samples/ecs-refarch"}'
# View diagram
open http://localhost:8080?lod=2
Use Cases
1. Architecture Documentation
Automatically generate LoD-appropriate diagrams for different audiences.
2. Infrastructure Discovery
Visualize existing AWS environments from CloudFormation/Terraform.
3. Pattern Mining
Learn common AWS patterns from reference architectures.
4. Compliance Visualization
Show network segmentation and security boundaries.
Cloud infrastructure visualization from MacLeod Labs