Realitykit

by dpearson26998d90fd121a26No license1.1K starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated 2 months ago

Build iOS augmented reality and 3D experiences with RealityKit and ARKit. Use when adding RealityView content, loading entities or USDZ models, anchoring objects to planes or world positions, distinguishing entity hit tests from ARKit real-world raycasts, handling AR camera availability, world tracking, scene updates, or RealityKit entity gestures and interactions.

Instructions onlySoftware Development

AI-generated overview

Guides building iOS augmented reality and 3D experiences with RealityKit and ARKit, covering RealityView, entities, anchoring, raycasting and gestures.

What it does: This skill provides reference guidance for building iOS AR and 3D experiences using RealityKit for rendering and ARKit for world tracking. It covers RealityView setup, loading and creating entities, anchoring and placement, raycasting distinctions, gesture interaction, scene understanding, common mistakes and a review checklist. It produces explanatory instructions and code examples rather than files or scripts.
When to use it: Use it when adding RealityView content, loading entities or USDZ models, anchoring objects to planes or world positions, or handling AR camera availability and world tracking. It also fits distinguishing entity hit tests from ARKit real-world raycasts and implementing RealityKit entity gestures.
Requirements: No scripts are shipped; it is instructions only. It targets Swift 6.3 and iOS 26+, and assumes familiarity with SwiftUI, RealityKit and ARKit. AR use requires camera permission (NSCameraUsageDescription) and device support checks.

RealityKit

Build AR experiences on iOS using RealityKit for rendering and ARKit for world tracking. Covers RealityView, entity management, raycasting, scene understanding, and gesture-based interactions. Targets Swift 6.3 / iOS 26+.

Setup
RealityView Basics
Loading and Creating Entities
Anchoring and Placement
Raycasting
Gestures and Interaction
Scene Understanding
Common Mistakes
Review Checklist
References

Setup

Project Configuration

Add NSCameraUsageDescription to Info.plist
On iOS, RealityViewCameraContent displays an AR camera view by default (iOS 18+, macOS 15+); use .virtual camera mode for explicit non-AR fallback
No entitlement is required for basic AR. If AR is core to the app, add the arkit required-device capability; otherwise gate AR UI with isSupported.

Device Requirements

Check the exact AR configuration's isSupported value before presenting AR UI.

swift

import ARKit
guard ARWorldTrackingConfiguration.isSupported else {    showUnsupportedDeviceMessage()    return}

import ARKit
guard ARWorldTrackingConfiguration.isSupported else {    showUnsupportedDeviceMessage()    return}

Key Types

Type	Platform	Role
`RealityView`	iOS 18+, visionOS 1+	SwiftUI view that hosts RealityKit content
`RealityViewCameraContent`	iOS 18+, macOS 15+	Content displayed through an AR camera view on iOS, non-AR on macOS
`Entity`	All	Base class for all scene objects
`ModelEntity`	All	Entity with a visible 3D model
`AnchorEntity`	All	Tethers entities to a real-world anchor

RealityView Basics

RealityView is the SwiftUI entry point for RealityKit. RealityViewCameraContent is the iOS/macOS content type. On iOS, it uses an AR camera view by default and can use content.camera = .virtual for non-AR mode when requested or when AR/camera access is unavailable.

swift

import ARKitimport SwiftUIimport RealityKit
struct ARExperienceView: View {    var body: some View {        RealityView { (content: RealityViewCameraContent) in            if !ARWorldTrackingConfiguration.isSupported {                content.camera = .virtual            }
            let sphere = ModelEntity(                mesh: .generateSphere(radius: 0.05),                materials: [SimpleMaterial(                    color: .blue,                    isMetallic: true                )]            )            sphere.position = [0, 0, -0.5]  // 50cm in front of camera            content.add(sphere)        }    }}

import ARKitimport SwiftUIimport RealityKit
struct ARExperienceView: View {    var body: some View {        RealityView { (content: RealityViewCameraContent) in            if !ARWorldTrackingConfiguration.isSupported {                content.camera = .virtual            }
            let sphere = ModelEntity(                mesh: .generateSphere(radius: 0.05),                materials: [SimpleMaterial(                    color: .blue,                    isMetallic: true                )]            )            sphere.position = [0, 0, -0.5]  // 50cm in front of camera            content.add(sphere)        }    }}

Make and Update Pattern

Use the update closure to respond to SwiftUI state changes:

swift

struct PlacementView: View {    @State private var modelColor: UIColor = .red
    var body: some View {        RealityView { content in            let box = ModelEntity(                mesh: .generateBox(size: 0.1),                materials: [SimpleMaterial(                    color: .red,                    isMetallic: false                )]            )            box.name = "colorBox"            box.position = [0, 0, -0.5]            content.add(box)        } update: { content in            if let box = content.entities.first(                where: { $0.name == "colorBox" }            ) as? ModelEntity {                box.model?.materials = [SimpleMaterial(                    color: modelColor,                    isMetallic: false                )]            }        }
        Button("Change Color") {            modelColor = modelColor == .red ? .green : .red        }    }}

struct PlacementView: View {    @State private var modelColor: UIColor = .red
    var body: some View {        RealityView { content in            let box = ModelEntity(                mesh: .generateBox(size: 0.1),                materials: [SimpleMaterial(                    color: .red,                    isMetallic: false                )]            )            box.name = "colorBox"            box.position = [0, 0, -0.5]            content.add(box)        } update: { content in            if let box = content.entities.first(                where: { $0.name == "colorBox" }            ) as? ModelEntity {                box.model?.materials = [SimpleMaterial(                    color: modelColor,                    isMetallic: false                )]            }        }
        Button("Change Color") {            modelColor = modelColor == .red ? .green : .red        }    }}

Loading and Creating Entities

Loading from USDZ Files

Load 3D models asynchronously to avoid blocking the main thread:

swift

RealityView { content in    if let robot = try? await ModelEntity(named: "robot") {        robot.position = [0, -0.2, -0.8]        robot.scale = [0.01, 0.01, 0.01]        content.add(robot)    }}

RealityView { content in    if let robot = try? await ModelEntity(named: "robot") {        robot.position = [0, -0.2, -0.8]        robot.scale = [0.01, 0.01, 0.01]        content.add(robot)    }}

Adding Components

Entities use an ECS (Entity Component System) architecture. Add components to give entities behavior:

swift

let box = ModelEntity(    mesh: .generateBox(size: 0.1),    materials: [SimpleMaterial(color: .red, isMetallic: false)])
// Make it respond to physicsbox.components.set(PhysicsBodyComponent(    massProperties: .default,    material: .default,    mode: .dynamic))
// Add collision shape for interactionbox.components.set(CollisionComponent(    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]))
// Enable input targeting for gesturesbox.components.set(InputTargetComponent())

let box = ModelEntity(    mesh: .generateBox(size: 0.1),    materials: [SimpleMaterial(color: .red, isMetallic: false)])
// Make it respond to physicsbox.components.set(PhysicsBodyComponent(    massProperties: .default,    material: .default,    mode: .dynamic))
// Add collision shape for interactionbox.components.set(CollisionComponent(    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]))
// Enable input targeting for gesturesbox.components.set(InputTargetComponent())

Anchoring and Placement

AnchorEntity

Use AnchorEntity to anchor content to detected surfaces or world positions:

swift

RealityView { content in    // Anchor to a horizontal surface    let floorAnchor = AnchorEntity(.plane(        .horizontal,        classification: .floor,        minimumBounds: [0.2, 0.2]    ))
    let model = ModelEntity(        mesh: .generateBox(size: 0.1),        materials: [SimpleMaterial(color: .orange, isMetallic: false)]    )    floorAnchor.addChild(model)    content.add(floorAnchor)}

RealityView { content in    // Anchor to a horizontal surface    let floorAnchor = AnchorEntity(.plane(        .horizontal,        classification: .floor,        minimumBounds: [0.2, 0.2]    ))
    let model = ModelEntity(        mesh: .generateBox(size: 0.1),        materials: [SimpleMaterial(color: .orange, isMetallic: false)]    )    floorAnchor.addChild(model)    content.add(floorAnchor)}

Anchor Targets

Target	Description
`.plane(.horizontal, ...)`	Horizontal surfaces (floors, tables)
`.plane(.vertical, ...)`	Vertical surfaces (walls)
`.plane(.any, ...)`	Any detected plane
`.world(transform:)`	Fixed world-space position

Raycasting

Keep RealityKit scene queries separate from ARKit real-world raycasts:

RealityViewCameraContent.ray(through:in:to:) returns a camera ray in RealityKit coordinate spaces. It projects a screen point into the virtual scene; it is not proof of a detected physical surface.
RealityViewCameraContent.hitTest(point:in:query:mask:) hits virtual entities made hittable by CollisionComponent shapes. Use those shapes for entity picking and targeted gestures, not ARKit plane detection.
Use AnchorEntity(.plane(...)) for simple placement on detected planes.
Use ARKit ARRaycastQuery plus ARSession.raycast(_:) when the task needs a one-shot intersection with real-world surfaces, then anchor with AnchorEntity(raycastResult:).

swift

let results = session.raycast(query)if let result = results.first {    let anchor = AnchorEntity(raycastResult: result)    anchor.addChild(model)    content.add(anchor)}

let results = session.raycast(query)if let result = results.first {    let anchor = AnchorEntity(raycastResult: result)    anchor.addChild(model)    content.add(anchor)}

Do not treat entity hit tests as substitutes for ARKit surface raycasts.

Gestures and Interaction

For gesture-based entity interaction, add CollisionComponent for the hittable shape and InputTargetComponent for input targeting.

Drag Gesture on Entities

swift

struct DraggableARView: View {    var body: some View {        RealityView { content in            let box = ModelEntity(                mesh: .generateBox(size: 0.1),                materials: [SimpleMaterial(color: .blue, isMetallic: true)]            )            box.position = [0, 0, -0.5]            box.components.set(CollisionComponent(                shapes: [.generateBox(size: [0.1, 0.1, 0.1])]            ))            box.components.set(InputTargetComponent())            box.name = "draggable"            content.add(box)        }        .gesture(            DragGesture()                .targetedToAnyEntity()                .onChanged { value in                    let entity = value.entity                    guard let parent = entity.parent else { return }                    entity.position = value.convert(                        value.location3D,                        from: .local,                        to: parent                    )                }        )    }}

struct DraggableARView: View {    var body: some View {        RealityView { content in            let box = ModelEntity(                mesh: .generateBox(size: 0.1),                materials: [SimpleMaterial(color: .blue, isMetallic: true)]            )            box.position = [0, 0, -0.5]            box.components.set(CollisionComponent(                shapes: [.generateBox(size: [0.1, 0.1, 0.1])]            ))            box.components.set(InputTargetComponent())            box.name = "draggable"            content.add(box)        }        .gesture(            DragGesture()                .targetedToAnyEntity()                .onChanged { value in                    let entity = value.entity                    guard let parent = entity.parent else { return }                    entity.position = value.convert(                        value.location3D,                        from: .local,                        to: parent                    )                }        )    }}

Scene Understanding

Per-Frame Updates

Subscribe to SceneEvents.Update for continuous scene work rather than driving RealityKit with a SwiftUI timer. Keep the subscription alive and use event.deltaTime; see Entity Animations [blocked].

Platform Boundaries

On visionOS, ARKit provides a different API surface with ARKitSession, WorldTrackingProvider, and PlaneDetectionProvider. These visionOS-specific types are not available on iOS. On iOS, RealityKit handles world tracking automatically through RealityViewCameraContent.

For iOS architecture or migration notes, gate AR with ARWorldTrackingConfiguration.isSupported, host content with RealityViewCameraContent, and build scenes from Entity/ModelEntity placed with AnchorEntity.

Handoffs: CollisionComponent + InputTargetComponent handle RealityKit interaction; AccessibilityComponent handles entity accessibility metadata; detailed SwiftUI gestures and VoiceOver/Switch Control policy belong to siblings.

Treat existing SCNView/SCNNode work as either a separate SceneKit path or an explicit migration to RealityKit, not a mixed scene graph.

Common Mistakes

DON'T: Skip AR capability checks

Use the configuration-specific support check from Setup before presenting AR. Show non-AR content or an explicit unavailable state when it fails.

DON'T: Load heavy models synchronously

Loading large USDZ files on the main thread causes frame drops and hangs. The make closure of RealityView is async -- use it.

swift

// WRONG -- synchronous load blocks the main threadRealityView { content in    let model = try! Entity.load(named: "large-scene")    content.add(model)}
// CORRECT -- async loadRealityView { content in    if let model = try? await ModelEntity(named: "large-scene") {        content.add(model)    }}

// WRONG -- synchronous load blocks the main threadRealityView { content in    let model = try! Entity.load(named: "large-scene")    content.add(model)}
// CORRECT -- async loadRealityView { content in    if let model = try? await ModelEntity(named: "large-scene") {        content.add(model)    }}

DON'T: Forget collision and input target components for interactive entities

Interactive entities need both components shown in Gestures and Interaction; without them, taps and drags pass through.

DON'T: Create new entities in the update closure

The update closure runs on every SwiftUI state change. Creating entities there duplicates content on each render pass.

swift

// WRONG -- duplicates entities on every state changeRealityView { content in    // empty} update: { content in    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))    content.add(sphere)  // Added again on every update}
// CORRECT -- create in make, modify in updateRealityView { content in    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))    sphere.name = "mySphere"    content.add(sphere)} update: { content in    if let sphere = content.entities.first(        where: { $0.name == "mySphere" }    ) as? ModelEntity {        // Modify existing entity        sphere.position.y = newYPosition    }}

// WRONG -- duplicates entities on every state changeRealityView { content in    // empty} update: { content in    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))    content.add(sphere)  // Added again on every update}
// CORRECT -- create in make, modify in updateRealityView { content in    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))    sphere.name = "mySphere"    content.add(sphere)} update: { content in    if let sphere = content.entities.first(        where: { $0.name == "mySphere" }    ) as? ModelEntity {        // Modify existing entity        sphere.position.y = newYPosition    }}

DON'T: Ignore camera permission

RealityKit on iOS needs camera access. If the user denies permission, the view shows a black screen with no explanation.

swift

// WRONG -- no permission handlingRealityView { content in    // Black screen if camera denied}
// CORRECT -- check and request permissionstruct ARContainerView: View {    @State private var cameraAuthorized = false
    var body: some View {        Group {            if cameraAuthorized {                RealityView { content in                    // AR content                }            } else {                ContentUnavailableView(                    "Camera Access Required",                    systemImage: "camera.fill",                    description: Text("Enable camera in Settings to use AR.")                )            }        }        .task {            let status = AVCaptureDevice.authorizationStatus(for: .video)            if status == .authorized {                cameraAuthorized = true            } else if status == .notDetermined {                cameraAuthorized = await AVCaptureDevice                    .requestAccess(for: .video)            }        }    }}

// WRONG -- no permission handlingRealityView { content in    // Black screen if camera denied}
// CORRECT -- check and request permissionstruct ARContainerView: View {    @State private var cameraAuthorized = false
    var body: some View {        Group {            if cameraAuthorized {                RealityView { content in                    // AR content                }            } else {                ContentUnavailableView(                    "Camera Access Required",                    systemImage: "camera.fill",                    description: Text("Enable camera in Settings to use AR.")                )            }        }        .task {            let status = AVCaptureDevice.authorizationStatus(for: .video)            if status == .authorized {                cameraAuthorized = true            } else if status == .notDetermined {                cameraAuthorized = await AVCaptureDevice                    .requestAccess(for: .video)            }        }    }}

Review Checklist

References

Read references/realitykit-patterns.md [blocked] for physics, animations, lighting, ECS, accessibility, and performance patterns.
RealityKit framework
RealityView
RealityViewCameraContent
RealityViewCamera
Entity
ModelEntity
AnchorEntity
ARKit framework
ARKit in iOS
Verifying Device Support and User Permission
ARWorldTrackingConfiguration
ARRaycastQuery
ARSession.raycast(_:)
Loading entities from a file

Source and attribution

Source:dpearson2699/swift-ios-skillsinskills/realitykitat commit8d90fd1

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

More from dpearson2699/swift-ios-skills

Widgetkit

dpearson2699

Guides implementing, reviewing, and improving WidgetKit widgets and controls for iOS, iPadOS, watchOS, and CarPlay.

Software Development1.1Kupdated 2 months ago

Weatherkit

dpearson2699

Guides iOS developers in fetching WeatherKit forecasts, alerts, and attribution using WeatherService.

Software Development1.1Kupdated 2 months ago

Vision Framework

dpearson2699

Implement computer vision features including text recognition (OCR), face detection, barcode scanning, image segmentation, object tracking, and document scanning in iOS apps. Covers both the modern Swift-native Vision API (iOS 18+) and legacy VNRequest patterns, VisionKit DataScannerViewController for live camera scanning, and CoreMLRequest/VNCoreMLRequest for custom model inference. Use when adding OCR, barcode scanning, face detection, or custom Core ML model inference with Vision.

Awaiting classification1.1Kupdated 2 months ago

Tipkit

dpearson2699

Implement and review Apple TipKit feature-discovery UI for iOS 17+ apps. Use when adding or auditing in-app tips, contextual help, coach marks, Tip, TipView, popoverTip, rules, events, actions, display frequency, testing overrides, reusable tip identifiers, or iOS 18+ TipGroup and CloudKit tip sync; avoid for generic SwiftUI navigation or layout outside tip presentation.

Awaiting classification1.1Kupdated 2 months ago

Tabletopkit

dpearson2699

Guides building multiplayer spatial board games on visionOS with Apple's TabletopKit and RealityKit.

Software Development1.1Kupdated 2 months ago

Swiftui Webkit

dpearson2699

Guides embedding and controlling web content in SwiftUI apps with WebKit for SwiftUI on iOS 26 and later.

Software Development1.1Kupdated 2 months ago

Realitykit

RealityKit

Contents

Setup

Project Configuration

Device Requirements

Key Types

RealityView Basics

Make and Update Pattern

Loading and Creating Entities

Loading from USDZ Files

Adding Components

Anchoring and Placement

AnchorEntity

Anchor Targets

Raycasting

Gestures and Interaction

Drag Gesture on Entities

Scene Understanding

Per-Frame Updates

Platform Boundaries

Common Mistakes

DON'T: Skip AR capability checks

DON'T: Load heavy models synchronously

DON'T: Forget collision and input target components for interactive entities

DON'T: Create new entities in the update closure

DON'T: Ignore camera permission

Review Checklist

References

Source and attribution

More from dpearson2699/swift-ios-skills

Widgetkit

Weatherkit

Vision Framework

Tipkit

Tabletopkit

Swiftui Webkit

More in Software Development

Playground

M5 Onboard

Cardputer Buddy

Web Design Guidelines

Vercel React Native Skills

Vercel React Best Practices