# Visual autonomous driving
Source: https://kaustubhkr.github.io/projects/visual-driving.html

← Experiments & archive (https://kaustubhkr.github.io/archive.html) 08 / Robot learning 2024–2026

# Learning to drive from vision

Built expert-data collection and visual imitation learning, then reused the learned features in continuous-control training.
System Visual autonomous driving
My contribution Data pipeline, model integration & training system
Access Public implementation

Engineering & evidence CARLA environments, image and vehicle-state fusion, actor–critic training and checkpoint recovery.
Public simulation work; physical deployment is documented separately in the golf-cart project.
Physical golf-cart deployment ↗ (https://kaustubhkr.github.io/projects/golf-cart.html)

Technical deep dive Architecture, implementation & tradeoffs
System architecture CARLA / representation transfer

### Demonstrations become control features.

- 01

#### Observe

RGB + additional channels Five-channel input representation paired with expert controls.

- 02

#### Imitate

visual features → control heads Transfer RGB features and train steering, throttle and brake predictions.

- 03

#### Reuse

checkpoint feature extractor Carry the learned visual representation into an interaction-based agent.

- 04

#### Improve

visual features + vehicle state Condition SAC on perception features and low-dimensional vehicle state.

↖ A checkpoint is also a reusable representation. The next experiment asks what interaction can add to features first learned from expert driving.
Learning stages Supervised imitation → representation reuse → actor–critic
Control Steering, throttle and brake predictions
Temporal scope The current recurrent path receives a single timestep

01 / Implementation

## Observation to control

- Adapted the visual model to five input channels while retaining pretrained RGB weights for the first three.

- Used separate steering, throttle and brake heads for the imitation path.

- Built the CARLA environment interface and the observation/action plumbing for interactive training.

02 / Implementation

## Representation reuse

- Loaded the imitation checkpoint into a custom feature extractor and removed the final control-regression heads.

- Combined the extracted visual representation with state inputs through a learned projection.

- Integrated the extractor with Stable-Baselines3 SAC and separate policy/critic networks.

03 / Implementation

## The experiment infrastructure

- Implemented checkpoint discovery and resume logic, evaluation and monitoring utilities.

- Added multi-environment training paths and tools for recovering interrupted experiments.

- Kept collected demonstrations, imitation models and interactive-learning experiments in the same development trail.

04 / Implementation

## Private control extensions

- Extended the autonomous-driving work with MPCC comparison, modified PPO/SAC and a multi-head attention encoder.

- Used the same methods in the physical golf-cart work, including an RL adapter and industrial-control integration. These extensions are maintained separately from the public CARLA repository.

Engineering note

## Transfer the representation; inspect the control problem again.

A representation learned from expert examples supplies a useful starting point, but interactive training changes the observations the agent encounters. The pipeline makes the feature-transfer boundary explicit so the demonstration model, state inputs and actor–critic objective can be inspected separately.

Project context CARLA simulation work with separate private MPCC, modified PPO/SAC and attention-encoder extensions. The related physical golf-cart case study covers the vehicle deployment.
Source on GitHub ↗ (https://github.com/kaustubhkr/Self_Driving_Car)

Keep exploring

## A memory system I can run.

↗ (https://kaustubhkr.github.io/projects/personal-os.html) Personal OS

END OF SECTION 13: projects/visual-driving.html