{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Visualizing predictions with vision3d\n\nThis example demonstrates overlaying detector predictions on ground truth\nwith :func:`vision3d.viz.log_sample` and :func:`vision3d.viz.log_boxes_3d`.\nGround truth and predictions are logged to separate [Rerun](https://rerun.io/) entities so they can be toggled independently: both\nkeep per-class colors, ground truth is drawn as translucent colored boxes\nand predictions as a wireframe with their confidence score in the label.\n\nThe visualization is dataset-agnostic, so any vision3d dataset works here;\nwe use :class:`~vision3d.datasets.NuScenes3D` to source real frames.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load a frame\nPrediction visualization only needs a\n:class:`~vision3d.datasets.SampleInputs` /\n:class:`~vision3d.datasets.SampleTargets` pair to overlay on. See the\n`dataset examples <sphx_glr_auto_examples_datasets>` for the full\nloading and batching pipeline.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from pathlib import Path\n\nfrom vision3d.datasets import NuScenes3D\n\nNUSCENES_ROOT = Path(\"~/.cache/vision3d/nuscenes-mini\").expanduser()\n\ndataset = NuScenes3D(NUSCENES_ROOT, version=\"v1.0-mini\", split=\"train\", download=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Synthesize predictions\nA real detector would produce these, but to keep the example\nself-contained we synthesize a :class:`~vision3d.metrics.Prediction3D`\nfrom each frame's targets: drop a few objects, jitter the boxes, and\nassign random confidence scores.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import torch\n\nfrom vision3d.datasets import SampleTargets\nfrom vision3d.metrics import Prediction3D\nfrom vision3d.tensors import BoundingBoxes3D\n\n\ndef fake_predictions(\n    targets: SampleTargets, *, generator: torch.Generator\n) -> Prediction3D:\n    \"\"\"Perturb ground-truth boxes into plausible predictions.\n\n    Returns:\n        A :class:`~vision3d.metrics.Prediction3D` derived from ``targets``.\n    \"\"\"\n    boxes = targets[\"boxes\"]\n    labels = targets[\"labels\"]\n    n = boxes.shape[0]\n\n    # Keep ~80% of the objects as detections.\n    keep = torch.rand(n, generator=generator) < 0.8\n    raw = boxes.as_subclass(torch.Tensor)[keep].clone()\n    labels = labels[keep]\n\n    # Jitter centers (+/- 0.5 m) and sizes (+/- 10%).\n    raw[:, :3] += torch.empty_like(raw[:, :3]).uniform_(-0.5, 0.5, generator=generator)\n    raw[:, 3:6] *= 1 + torch.empty_like(raw[:, 3:6]).uniform_(\n        -0.1, 0.1, generator=generator\n    )\n\n    scores = torch.empty(len(raw)).uniform_(0.3, 0.99, generator=generator)\n    return Prediction3D(\n        boxes=BoundingBoxes3D(raw, format=boxes.format),\n        scores=scores,\n        labels=labels,\n    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Overlay predictions with :func:`vision3d.viz.log_sample`\nPassing ``predictions`` overlays detections alongside the ground truth.\n``label_to_id`` keeps per-class colors consistent across frames, and\n``score_threshold`` drops low-confidence detections before logging.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import rerun as rr\nimport rerun.blueprint as rrb\n\nfrom vision3d.viz import fusion_layout, log_sample\n\nrr.init(\"vision3d_predictions\", spawn=True)\nrr.send_blueprint(\n    rrb.Blueprint(\n        fusion_layout(NuScenes3D.camera_names, NuScenes3D.camera_grid),\n        rrb.TimePanel(state=\"collapsed\"),\n    )\n)\nrr.log(\"world\", rr.ViewCoordinates.RIGHT_HAND_Z_UP, static=True)\n\ngenerator = torch.Generator().manual_seed(0)\nfor frame_idx in range(10):\n    f_inputs, f_targets = dataset[frame_idx]\n    f_preds = fake_predictions(f_targets, generator=generator)\n    rr.set_time(\"frame\", sequence=frame_idx)\n    log_sample(\n        f_inputs,\n        f_targets,\n        predictions=f_preds,\n        label_to_id=dataset.class_to_idx,\n        score_threshold=0.4,\n        jpeg_quality=75,\n    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Logging boxes directly with :func:`vision3d.viz.log_boxes_3d`\n:func:`~vision3d.viz.log_sample` is a convenience wrapper; for finer\ncontrol you can drive :func:`vision3d.viz.log_boxes_3d` yourself. It\naccepts ``scores``, a ``score_threshold``, and a ``fill_mode`` so you can\nstyle ground truth and predictions however you like. Here we log the same\npredictions to a standalone entity as a dense wireframe with a stricter\nthreshold.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from vision3d.viz import log_boxes_3d\n\nf_inputs, f_targets = dataset[0]\nf_preds = fake_predictions(f_targets, generator=generator)\n\nrr.set_time(\"frame\", sequence=0)\nlog_boxes_3d(\n    \"world/pred_strict/boxes\",\n    f_preds[\"boxes\"],\n    class_ids=f_preds[\"labels\"].tolist(),\n    label_to_id=dataset.class_to_idx,\n    scores=f_preds[\"scores\"],\n    score_threshold=0.7,\n    fill_mode=\"densewireframe\",\n    show_labels=True,\n)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.14.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}