Cartographer - Multi-Cloud Infrastructure Visualization

Composable multi-cloud infrastructure graph ingestion and visualization with AWS Pattern Language (LoD 0-4).

The AWS Pattern Language

Traditional cloud diagrams fail at scale. Cartographer introduces Levels of Detail (LoD) for consistent, scalable architecture visualization:

LoD 0 (Executive)    → Stakeholders, platforms, inter-cloud
LoD 1 (System)       → Region + VPC, 6-12 services, no AZ
LoD 2 (Network)      → VPC → AZ → Subnets, IGW/NAT
LoD 3 (Service)      → Concrete services, sync/async flows
LoD 4 (Resource)     → Instances, configs, scaling

Architecture

graph TB
    subgraph "Data Ingestion"
        CF[CloudFormation
Templates]
        GH[GitHub
IaC Repos]
        AWS[AWS APIs
Cartography]
    end
    
    subgraph "Graph Storage"
        NEO[Neo4j
Graph Database]
        RULES[Pattern Rules
YAML]
    end
    
    subgraph "Processing Layer"
        PARSER[CF Parser]
        MINER[Diagram Miner]
        RESOLVER[Relationship
Resolver]
    end
    
    subgraph "Visualization"
        UI[React UI
Pattern Viewer]
        D3[D3.js
Renderer]
        LAYOUT[Layout Engine
Bottom-Up]
    end
    
    CF --> PARSER
    GH --> PARSER
    AWS --> NEO
    PARSER --> RESOLVER
    RESOLVER --> NEO
    NEO --> UI
    RULES --> LAYOUT
    LAYOUT --> D3
    D3 --> UI

Bottom-Up Layout Algorithm

Hierarchical space-filling layout:

class BottomUpLayoutEngine:
    """
    Calculate node sizes from leaves upward,
    then position from root downward.
    """
    
    def calculate_node_sizes(self, graph: NetworkGraph) -> Dict[str, Size]:
        """
        Phase 1: Bottom-up size calculation
        """
        sizes = {}
        
        # Process leaf nodes first (resources)
        for node in graph.leaves():
            sizes[node.id] = self.base_size(node.type)
        
        # Process parents bottom-up
        for level in reversed(range(graph.depth())):
            for node in graph.nodes_at_level(level):
                children = node.children()
                child_sizes = [sizes[c.id] for c in children]
                
                # Container must fit all children + padding
                layout = self.pack_children(child_sizes, node.layout_type)
                sizes[node.id] = Size(
                    width=layout.width + 2 * node.padding,
                    height=layout.height + 2 * node.padding
                )
        
        return sizes
    
    def adjust_peer_dimensions(self, graph: NetworkGraph, sizes: Dict):
        """
        Phase 2: Align peer components (VPCs, AZs)
        for visual consistency
        """
        adjusted = sizes.copy()
        
        # Group nodes by parent and level
        for parent in graph.parents():
            peers = parent.children()
            
            # Match heights, preserve content-based widths
            max_height = max(sizes[p.id].height for p in peers)
            for peer in peers:
                adjusted[peer.id].height = max_height
        
        return adjusted
    
    def recalculate_cascading(self, graph: NetworkGraph, 
                              original: Dict, adjusted: Dict):
        """
        Phase 3: Cascade size changes to children
        when parents were resized
        """
        final = adjusted.copy()
        changed = self.find_changed_nodes(original, adjusted)
        
        for node in changed:
            if node.type == "vpc":
                # AZs expand to fill VPC height
                for az in node.children():
                    available_height = final[node.id].height - 2 * node.padding
                    final[az.id].height = available_height
                    
                    # Recursively update subnets
                    self.expand_children(az, final)
        
        return final
    
    def position_nodes(self, graph: NetworkGraph, sizes: Dict):
        """
        Phase 4: Position nodes recursively from root
        """
        positions = {}
        
        def position_subtree(node, parent_bounds):
            size = sizes[node.id]
            
            # Center within parent bounds
            x = parent_bounds.x + (parent_bounds.width - size.width) / 2
            y = parent_bounds.y + (parent_bounds.height - size.height) / 2
            
            positions[node.id] = Point(x, y)
            
            # Position children based on node type
            if node.type == "region":
                self.position_vpcs_horizontally(node, positions, sizes)
            elif node.type == "vpc":
                self.position_azs_horizontally(node, positions, sizes)
            elif node.type == "az":
                self.position_subnets_by_scope(node, positions, sizes)
            elif node.type == "subnet":
                self.position_resources_grid(node, positions, sizes)
        
        # Start from root
        root = graph.root()
        root_bounds = Bounds(0, 0, viewport.width, viewport.height)
        position_subtree(root, root_bounds)
        
        return positions

Pattern Language Implementation

LoD Rules Engine:

class PatternLanguage:
    """
    AWS architecture pattern language with LoD rules
    """
    
    def __init__(self):
        self.rules = self.load_rules("schema/awsdl_rules.yaml")
    
    def apply_lod_filter(self, graph: NetworkGraph, lod: int) -> NetworkGraph:
        """
        Filter graph based on Level of Detail
        """
        lod_config = self.rules["lod"][str(lod)]
        
        filtered = NetworkGraph()
        
        # Add nodes based on LoD configuration
        for node in graph.nodes():
            if node.kind in lod_config["show"]:
                filtered.add_node(node)
                
        # Collapse services if needed
        if "collapse" in lod_config:
            for pattern, label in lod_config["collapse"].items():
                matching = filtered.find_pattern(pattern)
                filtered.replace_with_group(matching, label)
        
        # Add edges with style based on LoD
        edge_styles = lod_config.get("edgeStyles", {})
        for edge in graph.edges():
            if edge.src in filtered and edge.dst in filtered:
                edge.style = edge_styles.get(edge.type, "solid")
                filtered.add_edge(edge)
        
        return filtered

# Example LoD configuration
AWSDL_RULES = {
    "lod": {
        "0": {  # Executive/Context
            "show": ["region"],
            "hide": ["vpc", "az", "subnet", "resource"],
            "collapse": {
                "microservices": "Service Group",
                "event_mesh": "Event Mesh"
            }
        },
        "1": {  # System/Architecture  
            "show": ["region", "vpc"],
            "hide": ["az", "subnet"],
            "maxServiceIcons": 20
        },
        "2": {  # Network/VPC
            "show": ["region", "vpc", "az", "subnet"],
            "hide": ["resource"],
            "layout": {
                "azColumns": True,
                "subnetRows": ["public", "private", "db"]
            }
        },
        "3": {  # Service/Dataflow
            "show": ["region", "vpc", "az", "subnet", "resource"],
            "edgeStyles": {
                "sync": "solid",
                "async": "dashed",
                "control": "dotted"
            }
        }
    }
}

CloudFormation Import

Automatic template parsing:

class CloudFormationImporter:
    """
    Import CloudFormation templates into Neo4j graph
    """
    
    def import_template(self, template_path: str) -> ImportResult:
        """
        Parse CF template and create graph nodes/relationships
        """
        with open(template_path) as f:
            cf = yaml.safe_load(f)
        
        resources = cf.get("Resources", {})
        stack_name = self.extract_stack_name(template_path)
        
        # Create stack node
        stack = self.create_node("Stack", name=stack_name)
        
        # Create resource nodes
        for logical_id, resource in resources.items():
            resource_type = resource["Type"]
            properties = resource.get("Properties", {})
            
            node = self.create_node(
                self.map_cf_type(resource_type),
                logical_id=logical_id,
                properties=properties
            )
            
            # Link to stack
            self.create_relationship(stack, "CONTAINS", node)
            
            # Resolve references
            for key, value in properties.items():
                if isinstance(value, dict) and "Ref" in value:
                    ref_target = value["Ref"]
                    if ref_target in resources:
                        target = self.find_node(logical_id=ref_target)
                        self.create_relationship(
                            node, "DEPENDS_ON", target
                        )
        
        return ImportResult(
            stack_id=stack.id,
            resources_imported=len(resources)
        )

GitHub Repository Scraping

Automatic IaC discovery:

def import_github_repo(repo_url: str, region: str = "us-east-1"):
    """
    Scan GitHub repo for CloudFormation templates
    and import into graph
    """
    # Clone repo (or use GitHub API)
    templates = find_cloudformation_templates(repo_url)
    
    imported = []
    for template in templates:
        # Create synthetic stack name from file path
        stack_name = f"github/{repo_url.split('/')[-1]}/{template.name}"
        
        # Parse and import
        result = cf_importer.import_template(
            template.path,
            stack_name=stack_name,
            region=region,
            source="github"
        )
        
        imported.append(result)
    
    return {
        "repository": repo_url,
        "templates_found": len(templates),
        "stacks_created": len(imported)
    }

Diagram Mining System

Learn patterns from AWS reference diagrams:

┌──────────────────────────────────────────────┐
│ Diagram Mining Pipeline                      │
├──────────────────────────────────────────────┤
│                                              │
│ 1. Fetch AWS Icon Pack                      │
│    └─> Official architecture icons          │
│                                              │
│ 2. Scrape AWS Reference Diagrams            │
│    ├─> PPTX presentations                   │
│    ├─> PDF whitepapers                      │
│    └─> PNG diagrams                         │
│                                              │
│ 3. Parse Layout Features                    │
│    ├─> Container hierarchy                  │
│    ├─> Service placement                    │
│    ├─> Edge routing                         │
│    └─> Color schemes                        │
│                                              │
│ 4. Mine LoD Rules                           │
│    ├─> Which services at which LoD?        │
│    ├─> Container visibility rules           │
│    └─> Edge style patterns                  │
│                                              │
│ 5. Export YAML Rules                        │
│    └─> schema/awsdl_rules.yaml             │
│                                              │
└──────────────────────────────────────────────┘

Neo4j Data Model

Graph schema for AWS infrastructure:

// Nodes
(:Region {name, id})
(:VPC {cidr, id})
(:AvailabilityZone {name, id})
(:Subnet {cidr, scope, id})  // scope: public|private|database
(:Resource {type, name, config})
(:Stack {name, template_url})

// Relationships
(Region)-[:CONTAINS]->(VPC)
(VPC)-[:CONTAINS]->(AvailabilityZone)
(AvailabilityZone)-[:CONTAINS]->(Subnet)
(Subnet)-[:HOSTS]->(Resource)
(Resource)-[:CONNECTS_TO {type}]->(Resource)  // type: sync|async
(Stack)-[:DEFINES]->(Resource)

// Example query: Find all resources in VPC
MATCH (vpc:VPC {id: $vpc_id})-[:CONTAINS*]->(subnet:Subnet)
      -[:HOSTS]->(resource:Resource)
RETURN resource.type, resource.name, subnet.scope

Interactive UI Features

React-based pattern viewer:

• LoD Selector - Switch between detail levels on the fly
• Drill-Down - Click to expand collapsed groups
• Scope Coloring - Public (green), Private (orange), DB (blue)
• Pan & Zoom - Smooth D3 interactions
• CloudFormation Import - Upload templates directly
• GitHub Integration - Import IaC repos by URL

Performance Metrics

Operation                  | Time      | Details
---------------------------|-----------|---------------------------
Parse CF template          | 50-200ms  | 50-500 resources
Import to Neo4j            | 100-500ms | With relationship resolution
Render LoD 2 diagram       | 80-150ms  | Single VPC, 3 AZs
Render LoD 3 diagram       | 200-400ms | 20-50 services
Layout calculation         | 30-100ms  | Bottom-up algorithm
GitHub repo scan           | 2-5s      | 10-50 templates

Technical Stack

• Backend: FastAPI + Python 3.11
• Database: Neo4j 5.x (graph storage)
• Frontend: React 18 + D3.js
• Layout: Custom bottom-up algorithm
• Import: CloudFormation + GitHub API
• Export: SVG, PNG, Neo4j backup

Quick Start

cd cartographer

# Start Neo4j and API
docker compose up -d

# Import CloudFormation
curl -X POST http://localhost:8001/ingest/cloudformation \
  -d '{"region": "us-east-1"}'

# Import GitHub repo
curl -X POST http://localhost:8001/ingest/github \
  -d '{"github_url": "https://github.com/aws-samples/ecs-refarch"}'

# View diagram
open http://localhost:8080?lod=2

Use Cases

1. Architecture Documentation

Automatically generate LoD-appropriate diagrams for different audiences.

2. Infrastructure Discovery

Visualize existing AWS environments from CloudFormation/Terraform.

3. Pattern Mining

Learn common AWS patterns from reference architectures.

4. Compliance Visualization

Show network segmentation and security boundaries.

Cloud infrastructure visualization from MacLeod Labs