NavLab: the autonomous vehicle of the 80s

(1)

How artificial intelligence could impact the car industry

Stéphane Canu

asi.insa-rouen.fr/enseignants/~scanu scanu@insa-rouen.fr

« APPC, ASI »

(2)

Road map

1 A very brief history of autonomous vehicles

2 Autonomous vehicles today

3 How has this happened? (Deep Learninng)

4 How Artificial Intelligence will change the Automotive Industry

5 Conclusion

(3)

Artificial intelligence and autonomous vehicles

Artificial intelligence is about doing things better than human

→ It can do a lot of things better than a human driver

(4)

NavLab: the autonomous vehicle of the 80s

1 M $ , 10 km/h

http://www.rediscoverthe80s.com/2016/11/navlab- the- selfdriving- car- of- the- 80s.html/

(5)

DARPA Grand Challenge 2004

for American autonomous vehicles only 1 million $

140 miles (225km) from Barstow, California to Primm, Neveda

(6)

DARPA Grand Challenge 2005

2 million $ – 132 miles (213 km) in the desert, Primm, Neveda.

Stanford vs Carnegie Mellon

212.7 km at an average speed of 30.7 km/h Key issue

"The specific transfer function emulates human driving characteristics, and is learned from data gathered through human driving."

(7)

CMU autonomous vehicles

(8)

Road map

5 Conclusion

(9)

(10)

Level 2/3 Autonomous vehicles for sale

100 000 $

Tesla Model X vs. Audi A8

Motivations Today:

I driver comfort (10 000e→0 because of openpilot) Tomorrow

I save lives (safety)

I environmental issues

(11)

Comparaison des systèmes d’aide à la conduite (see

openpilot)

(12)

Level 4 experences in Rouen, Phoenix, 13 cities in China...

Waymo’s cars (Google) hit the 10 million-mile milestone on public roads New uses

public transportation (last kilometer) isolated people

autonomous ride services (taxi) . . .

pole- moveo.org/actualites/rouen- normandy- autonomous- lab- la- metropole- rouen- normandie- teste- un- service- de- mobilite- autonome/

(13)

Autonomous vehicle performance ranking

In total, 48 companies reported driving 2 million miles in autonomous mode on California roads and highways 2018

(14)

Advanced Driver Assistance Systems ADAS Ratings

Consumer Reports’ for major Advanced Driver Assistance Systems (nov 2020)

(15)

Safety Ratings

Safety Assist, which evaluated driver-assistance and crash-avoidance technologies.

(16)

Accidents: 8 lethal since 2015 (+7 processing)

https://www.republicanmatters.com/police- release- footage- of- fatal- uber- selfdriving- car- accident/

What autonomy level?

→ Levels for AI ?

https://en.wikipedia.org/wiki/List_of_self- driving_car_fatalities

(17)

The 3 IA Levels

What do you see ? (is it a cat?) specific intelligence

Why is it a cat? general intelligence

Is she going up or down? super intelligence

(18)

Specific, General and Super Intelligence

What do you see ? (is it a cat?) specific intelligence

Why is it a cat? general intelligence

Is she going up or down? super intelligence

(19)

When do we see super intelligence?

(20)

Autonomy and Artificial Intelligence Levels

Level 3 autonomy specific intelligence

Level 4 autonomy general intelligence

Level 5 autonomy super intelligence

(21)

Road map

(22)

Reconnaissance d’objets : caltech 101 database (2004)

101 classes,

30 images par catégorie ...and the winner is NOT a deep network

I pas assez d’images

in What is the Best Multi-Stage Architecture for Object Recognition? Jarrett et al, 2009

(23)

The image net database (Deng et al., 2012)

ImageNet = 15 million labeled high-resolution images of 22,000 categories.

Large-Scale Visual Recognition Challenge (a subset of ImageNet) 1000 categories.

1.2 million training images, 50,000 validation images,

(24)

A new fashion in image processing

shallow approaches deep learning

Y. LeCun StatLearn tutorial

(25)

Deep architecture and the image net (15%)

The Alex Net architecture [Krizhevsky, Sutskever, Hinton, 2012]

Convolution + Recitification (ReLU) + Normalization + Pooling 60 million parameters

using 2 GPU – 6 days regularization

(26)

ImageNet results

2012 Alex Net 2013 ZFNet 2014 VGG

2015 GoogLeNet / Inception 2016 Residual Network 2018 NAS Network

2020 EfficientNet (Transformers)

(27)

Reconnaissance d’objets par apprentissage

2020: GPT-3 contains 175 billion parameters Pour résoudre des problèmes spécifiques :

traitement d’image

reconnaissance de la parole, traduction automatique le texte : répondre à des questions, assistants, (GPT-3) planifier : Alpha GO

(28)

Big Models

GPT-3 requiert au moins 175 Go de mémoire vive

(29)

Detection, tracking and recognition of traffic signs (2011-13)

Recognition German Traffic Sign Recognition Benchmark (GTSRB) data set, containing 51839 labelled images of real-world traffic signs.

Detection The German Traffic Sign Detection Benchmark is a single-image detection

assessment 900 images (600 for training and 300 for test) and the winner is

(30)

Object recognition for autonomous vehicles

https://www.nvidia.com/en- au/selfdriving- cars/drive- px/

https://pjreddie.com/darknet/yolo/

Two components are needed detection

recognition (characterization)

(31)

Perception is scene understanding

Scene understanding is

Multi-task learning

(32)

Components of self driving cars software (1)

(33)

Components of self driving cars software

AI inside: uses data

(Deep) learning based programming

(34)

Decision making is an imitation model

Imitation model is

Multi-Task and multi objective learning

Mayank Bansal , ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst, ICML 2019

(35)

Machine learning issues in self driving

multi task multi objective

architecture design issues

the never ending learning (long tails events) under budget

(36)

HydraNet: Tesla Autopilot Neural network architecture

in 2020 : 48 networks, 1,000 outputs, 70,000 GPU h to train Multi task learning

Sharing feature over space, cameras and time

(37)

HydraNet: Tesla Autopilot Neural network architecture

Multi task learning

Sharing feature over space, cameras and time

(38)

Tesla Neural Network

backbone ResNet50 (2020), RegNet (2021) (a Facebook 2020 paper) feature fusion : EfficientDet (from a Google 2019 paper)

decision module (multi task fine tuning)

I head

I trunk

I terminal

(39)

Tesla car crash

1916 Tesla’s car crash was due to a confusion between white side of trailer with brightly lit sky

(40)

Managing the confidence

(41)

Data: the long tail of situations

Rare driving situations

Drago Anguelov (Waymo) - MIT Self-Driving Cars lectures https://medium.com/waymo/how- do- we- prepare- for- rare- and- odd- situations- on- the- road- by- following- these- 3- steps- 1abd73d93e15

(42)

Tesla improving: iterative process

data

multi task learning fleet learing

testing = shadow mode for more training data

(43)

Tesla’s point of view on data

Gathering process

I 221 triggering situations manual labelling (1000 people)

I 2d -> 3d

I reconstruction labelling auto labelling

I use specificly trained networks

I human to clean simulation

I rare event

I sensor robustness

I adversarial exemples

(44)

Tesla’s release and validation

(45)

Waymo’s point of view

uses autoML (NAS cells)

Drago Anguelov (Waymo) - MIT Self-Driving Cars lectures

(46)

Waymo’s AutoML

Proxy end-to-end search: Explore thousands of architecture on a scaled-down proxy task, apply the 100 best ones to the original task,

validate and deploy the best of the best architectures on the car.

Drago Anguelov (Waymo) - MIT Self-Driving Cars lectures https://medium.com/waymo/automl- automating- the- design- of- machine- learning- models- for- autonomous- driving- 141a5583ec2a

(47)

Waymo’s AutoML (mobil eye is missing)

1) The first graph shows about 4,000 architectures discovered with a random search on a simple set of architectures. Each point is an architecture that was trained and evaluated. The solid line

marks the best architectures at different inference time constraints. The red dot shows the latency and performance of the net built with transfer learning. In this random search, the nets were not as good as the one from transfer learning. 2) In the second graph, the yellow and blue points show the results of two other search algorithms. The yellow one was a random search on a refined set of architectures. The blue one used reinforcement learning as in [1] and explored

(48)

Tesla vs. Waymo (vs. Openpilot, MobilEye, Appolo. . . )

Tesla Waymo

Autonomy Diver assist Self driving

objective Hands on the wheel no driver

Sensors Camera Lidar + Camera

Autopilot on highway Taxi in a known area

Data from the Fleet Simulation

3 millions miles per day 16 billions simulated miles AI multi task deep learning

semi supervised

hybrid deep learning reinforcement & Auto ML Running

by now 1 000 000 100

Project

2014/2016 2010

(49)

Open Pilot: 1000 $

(50)

Openpilot : l’étiquetage des données par crowd sourcing

(51)

Openpilot : surveillance du conducteur

EfficentNet inside

https://github.com/commaai/openpilot

(52)

Road map

5 Conclusion

(53)

Programmation par l’exemple : le pari de Tesla & Waymo

Tesla is collecting “just over 3 million miles [of data] per day.”

Waymo train the car with 6 million miles driven on public roads and 5 billion driven in simulation

Learn agent for driving situation simulations

(54)

Massive open data sets

and simulators (Carla, google & microsoft)

(55)

Les 5 composants d’un système de conduite autonome

https://devblogs.nvidia.com/parallelforall/deep- learning- selfdriving- cars https://medium.com/udacity/how- self- driving- cars- work- f77c49dca47e

(56)

(57)

Les données de ApolloScape

(58)

Des compétitions académiques

(59)

Des plateformes de développement ouvertes

Open source code, written by hundreds of students from across the globe!

Autoware (japon), NVIDIA’s DriveWorks and others

https://www.udacity.com/selfdriving- car

(60)

Data for self driving cars: CNN

Tesla Project Dojo

(61)

L’IA dans l’automobile au delà du véhicule autonome

Assistance à la conduite (AI copilote)

Maintenance prédictive

Production : usine 2.0

(62)

Pour quoi faire ?

Accepter les voitures autonomes Une histoire de confiance

Nous voulons comprendre les enjeux conduite du changement

(63)

Pour quoi faire ?

self driving trucks and tractors

(64)

Road map (done)

5 Conclusion

(65)

Future of AI in Automotive Industry

The value of your data (IA fuel) big data

Robustness (degraded conditions) deep learning theory Level 4 Autonomous driving common sense (cf Y. LeCun) unsupervised learning Predictive Maintenance data + prior knowledge

Acceptability (safety) Ethic

Interpretable AI

(66)

Questions?

http://asi.insa-rouen.fr/enseignants/~scanu/