Isaac Sim AI Perception and Computer Vision: AI-Enhanced Sensing

Purpose

This chapter explores Isaac Sim's AI perception capabilities and how to leverage them for computer vision applications. You'll learn how to generate synthetic training data, implement perception pipelines, and apply domain randomization techniques to create robust AI models for robotics.

Learning Outcomes

• Understand AI perception capabilities in Isaac Sim
• Generate synthetic training data using Isaac Sim
• Implement computer vision pipelines with AI integration
• Apply domain randomization for robust AI models

AI Perception in Isaac Sim

Synthetic Data Generation

Isaac Sim excels at generating synthetic data for AI training:

import omni.replicator.core as rep

# Configure replicator for synthetic data generation
rep.settings.carb_settings("/Omniverse/Replicator/RTSubframes", 8)

# Define a camera to generate data
camera = rep.create.camera(position=(0, 0, 2), rotation=(60, 0, 0))

# Create light sources for realistic illumination
lights = rep.create.light(
    position=lambda: (rep.distribution.uniform(-300, 300, 3),),
    rotation=lambda: (rep.distribution.uniform(-30, 30, 3),),
    light_type="distant",
    intensity=rep.distribution.normal(3000, 1000)
)

# Generate various annotation types
with rep.trigger.on_frame(num_frames=100):
    with camera:
        rep.modify.pose(
            position=rep.distribution.uniform((-1, -1, 1), (1, 1, 3)),
            rotation=rep.distribution.uniform((-15, -15, -180), (15, 15, 180))
        )

    # Generate RGB data
    rgb = rep.WriterRegistry.get("RgbSchemaProvider")->write_attribute("data", "output/rgb_{frame}.png")

    # Generate semantic segmentation
    seg = rep.WriterRegistry.get("SemanticSchemaProvider")->write_attribute("data", "output/seg_{frame}.png")

    # Generate instance segmentation
    inst = rep.WriterRegistry.get("InstanceSchemaProvider")->write_attribute("data", "output/inst_{frame}.png")

    # Generate depth data
    depth = rep.WriterRegistry.get("DepthSchemaProvider")->write_attribute("data", "output/depth_{frame}.exr")

    # Generate optical flow
    flow = rep.WriterRegistry.get("FlowSchemaProvider")->write_attribute("data", "output/flow_{frame}.exr")

Annotation Types

Isaac Sim provides various annotation types for AI training:

• RGB: Standard color images
• Semantic Segmentation: Per-pixel class labels
• Instance Segmentation: Per-pixel instance IDs
• Depth: Distance from camera
• Normal Maps: Surface normals
• Bounding Boxes: 2D and 3D bounding boxes
• Optical Flow: Motion vectors
• Material IDs: Per-pixel material information

Domain Randomization

Randomization Techniques

Domain randomization makes AI models robust to real-world variations:

# Randomize lighting conditions
with rep.randomizer.register("randomize_lighting"):
    lights = rep.get.light()

    with lights.group:
        rep.modify.visibility(rep.distribution.choice([True, False], weights=[0.7, 0.3]))
        rep.light.intensity(rep.distribution.normal(3000, 1000))
        rep.light.color(rep.distribution.uniform((0.5, 0.5, 0.5), (1.0, 1.0, 1.0)))

# Randomize object appearances
with rep.randomizer.register("randomize_appearances"):
    props = rep.get.prim_at_path("/World/Props")

    with props:
        rep.material.roughness(rep.distribution.uniform(0.1, 0.9))
        rep.material.metallic(rep.distribution.uniform(0.0, 0.5))
        rep.material.diffuse_reflection_color(rep.distribution.uniform((0.2, 0.2, 0.2), (0.8, 0.8, 0.8)))

# Randomize textures
with rep.randomizer.register("randomize_textures"):
    surfaces = rep.get.prim_at_path("/World/Surfaces")

    with surfaces:
        rep.material.albedo_texture(
            rep.distribution.choice(
                ["texture1.png", "texture2.png", "texture3.png"],
                weights=[0.3, 0.4, 0.3]
            )
        )

Environment Randomization

Randomize the entire environment for comprehensive training:

# Randomize backgrounds
def randomize_backgrounds():
    # Create random backgrounds from a list of textures
    background_textures = [
        "textures/outdoor_1.jpg",
        "textures/indoor_1.jpg",
        "textures/warehouse_1.jpg"
    ]

    # Randomly select and apply background
    bg_texture = rep.distribution.choice(background_textures)
    rep.background.set_texture(bg_texture)

# Randomize object positions
def randomize_object_positions():
    objects = rep.get.prim_at_path("/World/Objects")

    with objects:
        rep.modify.pose(
            position=rep.distribution.uniform((-2, -2, 0), (2, 2, 1)),
            rotation=rep.distribution.uniform((0, 0, 0), (0, 0, 360))
        )

# Randomize camera poses
def randomize_camera_poses():
    cameras = rep.get.prim_at_path("/World/Cameras")

    with cameras:
        rep.modify.pose(
            position=rep.distribution.uniform((-1, -1, 1), (1, 1, 3)),
            rotation=rep.distribution.uniform((-30, -30, -180), (30, 30, 180))
        )

Computer Vision Pipelines

Object Detection Pipeline

Implement an object detection pipeline in Isaac Sim:

import torch
import torchvision.transforms as transforms
from PIL import Image
import numpy as np
import cv2

class IsaacSimObjectDetectionPipeline:
    def __init__(self, model_path):
        # Load pre-trained model
        self.model = torch.load(model_path)
        self.model.eval()

        # Define transforms
        self.transform = transforms.Compose([
            transforms.Resize((416, 416)),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406],
                              std=[0.229, 0.224, 0.225])
        ])

        # Class names for detection
        self.class_names = [
            "person", "bicycle", "car", "motorcycle", "airplane",
            "bus", "train", "truck", "boat", "traffic light"
        ]

    def preprocess_image(self, raw_image):
        """Preprocess image from Isaac Sim for detection"""
        # Convert Isaac Sim image format to PIL
        pil_image = Image.fromarray(raw_image)

        # Apply transformations
        tensor = self.transform(pil_image)
        tensor = tensor.unsqueeze(0)  # Add batch dimension

        return tensor

    def detect_objects(self, image_tensor):
        """Run object detection on image tensor"""
        with torch.no_grad():
            predictions = self.model(image_tensor)

        # Process predictions
        detections = self.post_process_predictions(predictions)
        return detections

    def post_process_predictions(self, predictions):
        """Post-process model predictions"""
        # Implementation depends on specific model architecture
        # This would typically involve:
        # - Applying non-maximum suppression
        # - Converting normalized coordinates to pixel coordinates
        # - Filtering detections by confidence threshold
        pass

    def visualize_detections(self, image, detections):
        """Draw bounding boxes on image"""
        img_with_boxes = image.copy()

        for detection in detections:
            x1, y1, x2, y2 = detection['bbox']
            class_idx = detection['class']
            confidence = detection['confidence']

            # Draw bounding box
            cv2.rectangle(img_with_boxes, (x1, y1), (x2, y2), (0, 255, 0), 2)

            # Add label
            label = f"{self.class_names[class_idx]}: {confidence:.2f}"
            cv2.putText(img_with_boxes, label, (x1, y1-10),
                       cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)

        return img_with_boxes

Semantic Segmentation Pipeline

Implement semantic segmentation in Isaac Sim:

class IsaacSimSemanticSegmentationPipeline:
    def __init__(self, model_path):
        # Load segmentation model
        self.model = torch.load(model_path)
        self.model.eval()

        # Define color mapping for classes
        self.color_map = {
            0: (0, 0, 0),      # Background
            1: (128, 0, 0),    # Person
            2: (0, 128, 0),    # Car
            3: (128, 128, 0),  # Road
            4: (0, 0, 128),    # Building
            # Add more classes as needed
        }

    def segment_image(self, image):
        """Perform semantic segmentation on image"""
        # Preprocess image
        input_tensor = self.preprocess_for_segmentation(image)

        # Run inference
        with torch.no_grad():
            output = self.model(input_tensor)

        # Convert to segmentation mask
        segmentation_mask = torch.argmax(output, dim=1).squeeze().cpu().numpy()

        return segmentation_mask

    def create_segmentation_overlay(self, image, mask):
        """Create colored overlay for segmentation mask"""
        overlay = np.zeros_like(image)

        for class_id, color in self.color_map.items():
            mask_indices = (mask == class_id)
            overlay[mask_indices] = color

        # Blend original image with overlay
        blended = cv2.addWeighted(image, 0.7, overlay, 0.3, 0)

        return blended, overlay

Depth Estimation Pipeline

Implement depth estimation using Isaac Sim's ground truth depth:

class IsaacSimDepthEstimationPipeline:
    def __init__(self, model_path=None):
        if model_path:
            # Load depth estimation model
            self.model = torch.load(model_path)
            self.model.eval()
        else:
            # Use Isaac Sim's ground truth depth
            self.use_ground_truth = True

    def estimate_depth(self, rgb_image, ground_truth_depth=None):
        """Estimate depth from RGB image"""
        if self.use_ground_truth and ground_truth_depth is not None:
            # Use ground truth depth from Isaac Sim
            return ground_truth_depth

        # Preprocess RGB image
        input_tensor = self.preprocess_rgb(rgb_image)

        # Run depth estimation model
        with torch.no_grad():
            predicted_depth = self.model(input_tensor)

        return predicted_depth.squeeze().cpu().numpy()

    def calculate_depth_metrics(self, predicted_depth, ground_truth_depth):
        """Calculate depth estimation metrics"""
        # Calculate Mean Absolute Error
        mae = np.mean(np.abs(predicted_depth - ground_truth_depth))

        # Calculate Root Mean Square Error
        rmse = np.sqrt(np.mean((predicted_depth - ground_truth_depth) ** 2))

        # Calculate relative error
        relative_error = np.mean(
            np.abs(predicted_depth - ground_truth_depth) / (ground_truth_depth + 1e-6)
        )

        return {
            'MAE': mae,
            'RMSE': rmse,
            'Relative Error': relative_error
        }

    def visualize_depth(self, depth_map):
        """Visualize depth map with color mapping"""
        # Normalize depth to 0-255 range
        normalized_depth = (depth_map - depth_map.min()) / (depth_map.max() - depth_map.min())
        normalized_depth = (normalized_depth * 255).astype(np.uint8)

        # Apply colormap
        colored_depth = cv2.applyColorMap(normalized_depth, cv2.COLORMAP_JET)

        return colored_depth

Training AI Models with Isaac Sim

Data Collection Pipeline

Create a pipeline for collecting training data:

import os
import json
from datetime import datetime

class IsaacSimTrainingDataCollector:
    def __init__(self, output_dir):
        self.output_dir = output_dir
        self.session_id = datetime.now().strftime("%Y%m%d_%H%M%S")

        # Create output directories
        self.rgb_dir = os.path.join(output_dir, "rgb")
        self.seg_dir = os.path.join(output_dir, "segmentation")
        self.depth_dir = os.path.join(output_dir, "depth")
        self.annotations_dir = os.path.join(output_dir, "annotations")

        for directory in [self.rgb_dir, self.seg_dir, self.depth_dir, self.annotations_dir]:
            os.makedirs(directory, exist_ok=True)

    def save_training_sample(self, sample_data):
        """Save a training sample with all annotations"""
        # Generate unique filename
        sample_id = f"{self.session_id}_{sample_data['frame_num']:06d}"

        # Save RGB image
        rgb_path = os.path.join(self.rgb_dir, f"{sample_id}.jpg")
        cv2.imwrite(rgb_path, sample_data['rgb'])

        # Save segmentation
        seg_path = os.path.join(self.seg_dir, f"{sample_id}.png")
        cv2.imwrite(seg_path, sample_data['segmentation'])

        # Save depth
        depth_path = os.path.join(self.depth_dir, f"{sample_id}.exr")
        cv2.imwrite(depth_path, sample_data['depth'])

        # Save annotations
        annotation_data = {
            'sample_id': sample_id,
            'timestamp': sample_data['timestamp'],
            'camera_pose': sample_data['camera_pose'],
            'objects': sample_data['objects'],
            'lighting_conditions': sample_data['lighting_conditions']
        }

        annotation_path = os.path.join(self.annotations_dir, f"{sample_id}.json")
        with open(annotation_path, 'w') as f:
            json.dump(annotation_data, f, indent=2)

    def collect_training_session(self, num_samples, randomizers):
        """Collect a complete training session"""
        # Apply randomizers
        for randomizer in randomizers:
            randomizer.apply()

        # Collect samples
        for i in range(num_samples):
            # Get data from Isaac Sim
            sample_data = self.get_isaac_sim_data()

            # Save sample
            self.save_training_sample(sample_data)

            # Log progress
            if i % 100 == 0:
                print(f"Collected {i}/{num_samples} samples")

    def get_isaac_sim_data(self):
        """Get synchronized data from Isaac Sim sensors"""
        # Implementation to get synchronized RGB, segmentation, depth, etc.
        # This would interface with Isaac Sim sensors
        pass

Model Training Integration

Integrate Isaac Sim with model training workflows:

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, Dataset
import torchvision.models as models

class IsaacSimDataset(Dataset):
    def __init__(self, data_dir, transform=None):
        self.data_dir = data_dir
        self.transform = transform

        # Load file lists
        self.rgb_files = self._get_file_list(os.path.join(data_dir, "rgb"))
        self.seg_files = self._get_file_list(os.path.join(data_dir, "segmentation"))

    def _get_file_list(self, dir_path):
        """Get list of files in directory"""
        return [f for f in os.listdir(dir_path) if f.endswith(('.jpg', '.png'))]

    def __len__(self):
        return len(self.rgb_files)

    def __getitem__(self, idx):
        # Load RGB image
        rgb_path = os.path.join(self.data_dir, "rgb", self.rgb_files[idx])
        rgb_img = Image.open(rgb_path).convert('RGB')

        # Load segmentation mask
        seg_path = os.path.join(self.data_dir, "segmentation", self.seg_files[idx])
        seg_img = Image.open(seg_path).convert('L')

        if self.transform:
            rgb_img = self.transform(rgb_img)
            seg_img = self.transform(seg_img)

        return rgb_img, seg_img

def train_model_with_isaac_data():
    """Train model using Isaac Sim generated data"""
    # Define transforms
    transform = transforms.Compose([
        transforms.Resize((224, 224)),
        transforms.ToTensor(),
    ])

    # Create dataset
    dataset = IsaacSimDataset("./isaac_sim_data", transform=transform)
    dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

    # Create model
    model = models.segmentation.deeplabv3_resnet50(pretrained=False, num_classes=21)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)

    # Training loop
    for epoch in range(10):
        for batch_idx, (data, target) in enumerate(dataloader):
            optimizer.zero_grad()
            output = model(data)['out']
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()

            if batch_idx % 100 == 0:
                print(f'Epoch: {epoch}, Batch: {batch_idx}, Loss: {loss.item():.6f}')

AI Perception Applications

Object Recognition

Use Isaac Sim for object recognition tasks:

class IsaacSimObjectRecognition:
    def __init__(self, model_path):
        # Load object recognition model
        self.feature_extractor = self.load_feature_extractor(model_path)
        self.classifier = self.load_classifier(model_path)

        # Object database
        self.object_database = {}
        self.load_object_database()

    def extract_features(self, image):
        """Extract features from image for recognition"""
        # Extract deep features using CNN
        features = self.feature_extractor(image)
        return features

    def recognize_object(self, image):
        """Recognize objects in image"""
        # Extract features
        features = self.extract_features(image)

        # Compare with known objects
        similarities = self.compare_with_database(features)

        # Return best matches
        best_matches = self.get_best_matches(similarities)

        return best_matches

    def add_to_database(self, object_name, features):
        """Add object to recognition database"""
        if object_name not in self.object_database:
            self.object_database[object_name] = []

        self.object_database[object_name].append(features)

Pose Estimation

Implement 6D pose estimation using Isaac Sim:

class IsaacSimPoseEstimation:
    def __init__(self, model_path):
        self.model = torch.load(model_path)
        self.model.eval()

        # Camera intrinsic parameters
        self.fx, self.fy = 554.25, 554.25
        self.cx, self.cy = 320, 240

    def estimate_pose(self, image, object_template):
        """Estimate 6D pose of object in image"""
        # Preprocess image and template
        processed_image = self.preprocess(image)
        processed_template = self.preprocess(object_template)

        # Run pose estimation
        with torch.no_grad():
            rotation_matrix, translation_vector = self.model(
                processed_image,
                processed_template
            )

        # Convert to 6D pose
        pose_6d = self.rotation_translation_to_6d(
            rotation_matrix,
            translation_vector
        )

        return pose_6d

    def calculate_reprojection_error(self, pose_6d, points_3d, points_2d):
        """Calculate reprojection error for pose validation"""
        # Project 3D points to 2D using estimated pose
        projected_points = self.project_3d_to_2d(
            points_3d,
            pose_6d['rotation'],
            pose_6d['translation']
        )

        # Calculate error
        error = np.mean(np.linalg.norm(projected_points - points_2d, axis=1))

        return error

    def project_3d_to_2d(self, points_3d, rotation, translation):
        """Project 3D points to 2D image coordinates"""
        # Apply rotation and translation
        transformed_points = np.dot(rotation, points_3d.T).T + translation

        # Apply camera projection
        projected_x = (transformed_points[:, 0] * self.fx) / transformed_points[:, 2] + self.cx
        projected_y = (transformed_points[:, 1] * self.fy) / transformed_points[:, 2] + self.cy

        return np.stack([projected_x, projected_y], axis=1)

Performance Optimization

GPU Acceleration

Leverage GPU acceleration for AI perception:

import torch.cuda as cuda

class IsaacSimAIPipeline:
    def __init__(self, gpu_id=0):
        # Check GPU availability
        if cuda.is_available():
            self.device = torch.device(f'cuda:{gpu_id}')
            print(f"Using GPU: {cuda.get_device_name(gpu_id)}")
        else:
            self.device = torch.device('cpu')
            print("Using CPU")

        # Move models to device
        self.model = self.model.to(self.device)

    def run_inference_batch(self, image_batch):
        """Run inference on batch of images"""
        # Move batch to GPU
        batch_tensor = image_batch.to(self.device)

        # Run inference
        with torch.no_grad():
            results = self.model(batch_tensor)

        # Move results back to CPU if needed
        if self.device != torch.device('cpu'):
            results = results.cpu()

        return results

    def optimize_memory_usage(self):
        """Optimize memory usage for large models"""
        # Enable mixed precision training
        torch.backends.cudnn.benchmark = True

        # Clear GPU cache periodically
        if hasattr(torch.cuda, 'empty_cache'):
            torch.cuda.empty_cache()

Hands-on Exercise

Create a complete AI perception pipeline:

1. Set up Isaac Sim with domain randomization
2. Create a scene with multiple objects
3. Implement a synthetic data generation pipeline
4. Train a simple neural network on the generated data
5. Test the trained model in Isaac Sim for real-time inference
6. Evaluate the model's performance and robustness

Summary

Isaac Sim's AI perception capabilities provide powerful tools for generating synthetic training data, implementing computer vision pipelines, and applying domain randomization techniques. The combination of high-fidelity simulation with AI capabilities enables the development of robust perception systems that can be effectively transferred to real-world applications. Understanding these capabilities is crucial for leveraging Isaac Sim for AI-driven robotics development.

Further Reading

• Isaac Sim Replicator Documentation
• AI Perception in Isaac Sim
• Domain Randomization Tutorial

Practice Questions

1. What is domain randomization and why is it important for AI models?
2. What annotation types does Isaac Sim provide for AI training?
3. How does synthetic data generation benefit AI perception systems?

Check Your Understanding

Multiple Choice Questions:

1. What is the primary benefit of synthetic data generation in Isaac Sim? A) It's cheaper than real data B) It provides unlimited, diverse training data with perfect annotations C) It eliminates the need for real sensors D) It runs faster than real robots

2. Which of the following is NOT an annotation type provided by Isaac Sim? A) Semantic segmentation B) Instance segmentation C) Perfect sensor readings D) Optical flow

3. What is domain randomization used for? A) Reducing simulation costs B) Making AI models more robust to domain shift C) Improving graphics quality D) Increasing simulation speed