Haste makes waste

Nano01(自動運転)-U04-Lesson19-Project:Behavioral cloning(准备篇)

Posted on By lijun

3. Project Resources

这个项目通过workspace完成,需要的文件包括:

  • drive.py: python脚本,当神经网络训练好了之后进行汽车自动驾驶
  • video.py: 当汽车自动驾驶时用于生成video
  • sample driving data: /opt/carnd_p3/data/中,如果只使用自己的训练数据时,注意将其保存到其他目录
  • a simulator

4. Running the Simulator

介绍如何使用simulator收集数据:

  1. 可以使用键盘key,也可以使用鼠标来控制方向。
  2. 使用R录像。

When you first run the simulator, you’ll see a configuration screen asking what size and graphical quality you would like. We suggest running at the smallest size and the fastest graphical quality.

5. Data Collection Tactics(/’tæktɪks/ 策略)

In order to start collecting training data, you’ll need to do the following:

  • Enter Training Mode in the simulator.
  • Start driving the car to get a feel for the controls.
  • When you are ready, hit the record button in the top right to start recording.
  • Continue driving for a few laps or till you feel like you have enough data.
  • Hit the record button in the top right again to stop recording.

6. Data Collection Strategies

It’s time to think about collecting data that will ensure a successful model. There are a few general concepts to think about that we will later discuss in more detail:

  • the car should stay in the center of the road as much as possible
  • if the car veers off (突然转向) to the side, it should recover back to center
  • driving counter-clockwise (逆时针方向的) can help the model generalize ( /’dʒɛnrəlaɪz/ 一般化)
  • flipping(v. 轻弹) the images is a quick way to augment(vt. 增加;/ɔɡ’mɛnt/ ) the data
  • collecting data from the second track can also help generalize the model
  • we want to avoid overfitting or underfitting when training the model
  • knowing when to stop collecting more data

7. Data Visualization

如果上面的simulator一切顺利的话:

  1. IMG文件夹中,记录了汽车运行的每一帧
  2. driving_log.csv,每一行代表汽车运行中的关键数据,比如角度,刹车,速度等。

image

前三行表示各个摄像头拍下的图片,后面分别是转向角,油门,刹车,速度,转向角的区间是-1到1,油门是0到1。 在这里,我们把这些图像作为特征集,转向角作为标签集,然后用这些数据来训练网络,来预测转向角。

通过使用3张不同角度的图片,能有效的泛化模型。

8. Training Your Network

本节介绍了怎么去创建和训练网络,通过Keras训练一个网络完成下面的事情:

  1. 将之前拍摄的照片作为神经网络的输入
  2. 输出是这个车的转向角

首先通过scp将本地的图片copy到AWS的EC2上,然后在上面提取数据:

scp ~/Desktop/data.zip carnd@35.160.30.126

视频中的代码如下:

import csv
import cv2
import numpy as np

lines = []
with open("../data/driving_log.csv") as csvfile:
	reader = csv.reader(csvfile)
	for line in reader:
		lines.append(line)

images = []
measurements = []

# 读取图片和第四行的转向角数据
for line in lines:
	source_path = line[0]
	filename = source_path.split('/')[-1]
	current_path = "../data/IMG" + filename

	image = cv2.imread(current_path)
	images.append(image)

	measurement = float(line[3])
	measurements.append(measurement)

# 上面读取的图片作为输入特征数据,转向角作为标签数据
X_train = np.array(images)
y_train = np.array(measurements)
from keras.models import Sequential
from keras.layers import Flatten,Dense

# 建立keras神经网络
model = Sequential()
model.add(Flatten(input_shape=(160,320,3)))
model.add(Dense(1))

# 训练神经网络
model.compile(loss="mse",optimizer="adam")
model.fit(X_train,y_train,validation_split=0.2,shuffle=True,nb_epoch=7)

# 保存好训练的神经网络
model.save("model.h5")
exit()

9. Running Your Network

现在将训练好的神经网络作为drive.py的输入,运行drive.py后就能为simulator提供转向数据,simulator启动无人驾驶模式后,就能运行起来。

如果这个模型在训练数据和验证数据中的表现都不错(最后的误差函数值小),那说明是数据收集的有问题,收集的驾驶行为数据应该是车始终在中心校位置。

9.1 验证神经网络

在进入上面的simulator之前,需要保证输出的神经网络是好的,一般将数据分为80%的训练数据,20%的验证数据,另外记得要将数据顺序打乱。

如果训练数据和验证数据中,该神经网络的表现都不好,即误差函数的值都很高,说明是低度拟合了(underfitting),需要如下处理:

  1. 增加epoch次数
  2. 追加更多的卷积层

如果训练数据中表现不错,但验证数据中表现不好,说明是过拟合了,需要如下处理:

  1. 使用dropout或池化层
  2. 使用更少的卷积层或全连接层
  3. 收集更多的驾驶数据

理想情况是,在训练数据和验证数据中,神经网络表现都好,这样能保证驾驶时预测到正确的角度。

10. Data Preprocessing

  1. 将数据数据正规化,保证其在 -0.5 到 0.5 之间
  2. 减少epoch为2
# 建立keras神经网络
model = Sequential()
model.add(Lambda(lambda x:(x / 255.0) - 0.5,input_shape=(160,320,3)))

model.add(Flatten(input_shape=(160,320,3)))
model.add(Dense(1))

Kera中,lambda layer用于生成一个任意函数,在每个图片数据通过该layer时进行处理。

上面例子中的lambda layer将所有的input数据都进行了正规化处理。

11. More Networks

添加更多的卷积层:

from keras.models import Sequential
from keras.layers import Flatten,Dense
from keras.layers.Convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

# 建立keras神经网络
model = Sequential()
model.add(Lambda(lambda x:(x / 255.0) - 0.5,input_shape=(160,320,3)))
model.add(Convolution2D(6,5,5,activation="relu"))
model.add(MaxPooling2D())
model.add(Convolution2D(6,5,5,activation="relu"))
model.add(MaxPooling2D())
model.add(Flatten())
model.add(Dense(120))
model.add(Dense(84))
model.add(Dense(1))

# 训练神经网络
model.compile(loss="mse",optimizer="adam")
model.fit(X_train,y_train,validation_split=0.2,shuffle=True,nb_epoch=7)

# 保存好训练的神经网络
model.save("model.h5")
exit()

12. Data Augmentation

通过将数据翻转,生成对应的反方向图片和负的角度,以达到数据增强的目的:

augmented_images,augmented_measurements = [],[]

for image,measurement in zip(images,measurements):
	augmented_images.append(image)
	augmented_measurements.append(measurement)

	augmented_images.append(cv2.flip(iamge,1))
	augmented_measurements.append(measurements*(-1))
# 上面增强后的图片作为输入特征数据,转向角作为标签数据
X_train = np.array(images)
y_train = np.array(measurements)

Flipping Images And Steering Measurements:

A effective technique for helping with the left turn bias involves flipping images and taking the opposite sign of the steering measurement. For example:

import numpy as np
image_flipped = np.fliplr(image)
measurement_flipped = -measurement

翻转后效果如下:

image

13. using Multiple Cameras

读取更多的图片数据(之前只取了一个摄像头的数据)

# 读取图片和第四行的转向角数据
for line in lines:
	for i in range(3)
		source_path = line[i]
		filename = source_path.split('/')[-1]
		current_path = "../data/IMG" + filename

		image = cv2.imread(current_path)
		images.append(image)

		measurement = float(line[3])
		measurements.append(measurement)
  1. If you train the model to associate a given image from the center camera with a left turn, then you could also train the model to associate the corresponding image from the left camera with a somewhat softer left turn.
  2. And you could train the model to associate the corresponding image from the right camera with an even harder left turn.

Explanation of How Multiple Cameras Work:

image

  1. From the right camera’s perspective, the steering angle would be larger than the angle from the center camera.
  2. From the perspective of the left camera, the steering angle would be less than the steering angle from the center camera.

Here is some example code to give an idea of how all three images can be used:

with open(csv_file, 'r') as f:
        reader = csv.reader(f)
        for row in reader:
            steering_center = float(row[3])

            # create adjusted steering measurements for the side camera images
            correction = 0.2 # this is a parameter to tune
            steering_left = steering_center + correction
            steering_right = steering_center - correction

            # read in images from center, left and right cameras
            path = "..." # fill in the path to your training IMG directory
            img_center = process_image(np.asarray(Image.open(path + row[0])))
            img_left = process_image(np.asarray(Image.open(path + row[1])))
            img_right = process_image(np.asarray(Image.open(path + row[2])))

            # add images and angles to data set
            car_images.extend(img_center, img_left, img_right)
            steering_angles.extend(steering_center, steering_left, steering_right)

14. Cropping Images in Keras

因为有一部分数据,比如图片顶端的天空和树木,图片底端的汽车引擎盖,都是不需要的数据。

添加model.add(Cropping2D(cropping=((70,25),(0,0))))

from keras.models import Sequential
from keras.layers import Flatten,Dense,Lambda,Cropping2D
from keras.layers.Convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

# 建立keras神经网络
model = Sequential()
model.add(Lambda(lambda x:(x / 255.0) - 0.5,input_shape=(160,320,3)))
model.add(Cropping2D(cropping=((70,25),(0,0))))
model.add(Convolution2D(6,5,5,activation="relu"))
model.add(MaxPooling2D())
model.add(Convolution2D(6,5,5,activation="relu"))
model.add(MaxPooling2D())
model.add(Flatten())
model.add(Dense(120))
model.add(Dense(84))
model.add(Dense(1))

# 训练神经网络
model.compile(loss="mse",optimizer="adam")
model.fit(X_train,y_train,validation_split=0.2,shuffle=True,nb_epoch=7)

# 保存好训练的神经网络
model.save("model.h5")
exit()

Cropping2D Layer:

Keras provides the Cropping2D layer for image cropping within the model.

The Cropping2D layer might be useful for choosing an area of interest that excludes the sky and/or the hood of the car.

如下图是使用Cropping2D layer处理后的对比图:

image

下面的代码,剪切掉了如下的图形:

  1. 图片顶端往下的50像素
  2. 图片底端网上的20像素
  3. 图片左端开始的0像素(不剪切)
  4. 图片右端开始的0像素(不剪切)
from keras.models import Sequential, Model
from keras.layers import Cropping2D
import cv2

# set up cropping2D layer
model = Sequential()
model.add(Cropping2D(cropping=((50,20), (0,0)), input_shape=(3,160,320)))
...

15. Even More Powerful Network

该网络参考End-to-End Deep Learning for Self-Driving Cars

这里使用的网络结构如下:

image

  1. input后接着1个正规化层,这个已经有了
  2. 5个卷积层,每个卷积层的维度不同
  3. 4个全连接层
from keras.models import Sequential
from keras.layers import Flatten,Dense,Lambda,Cropping
from keras.layers.Convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

# 建立keras神经网络
model = Sequential()
model.add(Lambda(lambda x:(x / 255.0) - 0.5,input_shape=(160,320,3)))
model.add(Cropping2D(cropping=((70,25),(0,0))))

model.add(Convolution2D(24,5,5,subsample=(2,2),activation="relu"))
model.add(Convolution2D(36,5,5,subsample=(2,2),activation="relu"))
model.add(Convolution2D(48,5,5,subsample=(2,2),activation="relu"))
model.add(Convolution2D(64,3,3,activation="relu"))
model.add(Convolution2D(64,3,3,activation="relu"))
model.add(Flatten())
model.add(Dense(100))
model.add(Dense(50))
model.add(Dense(10))
model.add(Dense(1))

# 训练神经网络
model.compile(loss="mse",optimizer="adam")
model.fit(X_train,y_train,validation_split=0.2,shuffle=True,nb_epoch=7)

# 保存好训练的神经网络
model.save("model.h5")
exit()

16. More Data Collection

优化模型最好的方式就是尽可能收集更多的数据,如果只收集在道路中心线上正常行驶的数据,可能不够训练出合适的驾驶模型。

因为如果训练数据总是正常驾驶数据,模型就无法学习到车偏离后该如何处理(特定图片下转向角多大)。

So you need to teach the car what to do when it’s off on the side of the road.

  1. One approach might be to constantly wander off to the side of the road and then steer back to the middle.
  2. A better approach is to only record data when the car is driving from the side of the road back toward the center line.

下面还有几种方式能增加驾驶数据:

Driving Counter-Clockwise: 逆时针驾驶

Track one有一些偏左的偏差,如果只是在Track one上进行顺时针的驾驶,那数据会有偏左的偏差,消除这种偏差的方法就是沿着该track的反向,即逆时针驾驶。这样相当于又多了一个track的数据,能让模型更加泛化。

Using Both Tracks:

多用一个Track去生成驾驶数据,这样最终的模型会更加泛化。

Collecting Enough Data:

那么怎么才算收集了足够的数据呢,因为模型会输出一个连续的数值,这个数值就是均方差(误差函数的值)。

  1. If the mean squared error is high on both a training and validation set, the model is underfitting(低拟合)。
  2. If the mean squared error is low on a training set but high on a validation set, the model is overfitting(过拟合)。Collecting more data can help improve a model when the model is overfitting.

如果模型的均方差不错,但就是用该模型驾驶时,驾驶行为不好呢,说明之前用的驾驶数据有些地方有问题.

请确保收集的驾驶数据中:

  1. 有沿着中心线的2-3圈数据
  2. 有一圈从道路边缘回复到中心线的数据
  3. 有一圈在弯道平滑驾驶的数据

17. Visualizing Loss

Keras中,model.fit()model.fit_generator()中有一个变量verbose,用于设定是否输出损失值(loss metrics),可以设定为1和2.

设定为1的时候:

  1. 训练模型的时候,在终端输出一个progressbar
  2. 训练模型的时候,将损失值输出到训练集
  3. 训练和验证集的每个epoch中,输出其损失值

设定为2的时候,只输出上面的3,即每个epoch的损失值,如下是一个例子:

from keras.models import Model
import matplotlib.pyplot as plt

history_object = model.fit_generator(train_generator, samples_per_epoch =
    len(train_samples), validation_data = 
    validation_generator,
    nb_val_samples = len(validation_samples), 
    nb_epoch=5, verbose=1)

### print the keys contained in the history object
print(history_object.history.keys())

### plot the training and validation loss for each epoch
plt.plot(history_object.history['loss'])
plt.plot(history_object.history['val_loss'])
plt.title('model mean squared error loss')
plt.ylabel('mean squared error loss')
plt.xlabel('epoch')
plt.legend(['training set', 'validation set'], loc='upper right')
plt.show()

image

18. Generators

参考python中的生成器

这里通过simulator生成的图片大小是 160×320×3,存储1万张要消耗1.5GB内存,另外在数据处理的时候要变成float型,又要增大到4倍。

通过生成器可以解决这个问题,能实现只有需要用的时候才去计算,避免不必要内存浪费。

下面是一个示例程序:

def fibonacci():
    numbers_list = []
    while 1:
        if(len(numbers_list) < 2):
            numbers_list.append(1)
        else:
            numbers_list.append(numbers_list[-1] + numbers_list[-2])
        yield 1 # change this line so it yields its list instead of 1

our_generator = fibonacci()
my_output = []

for i in range(10):
    my_output = (next(our_generator))
    
print(my_output)

输出为1,可以看出上面只调用了一次。

下面的代码是如何将generator应用到本项目中:

import os
import csv

samples = []
with open('./driving_log.csv') as csvfile:
    reader = csv.reader(csvfile)
    for line in reader:
        samples.append(line)

from sklearn.model_selection import train_test_split
train_samples, validation_samples = train_test_split(samples, test_size=0.2)

import cv2
import numpy as np
import sklearn

def generator(samples, batch_size=32):
    num_samples = len(samples)
    while 1: # Loop forever so the generator never terminates
        shuffle(samples)
        for offset in range(0, num_samples, batch_size):
            batch_samples = samples[offset:offset+batch_size]

            images = []
            angles = []
            for batch_sample in batch_samples:
                name = './IMG/'+batch_sample[0].split('/')[-1]
                center_image = cv2.imread(name)
                center_angle = float(batch_sample[3])
                images.append(center_image)
                angles.append(center_angle)

            # trim image to only see section with road
            X_train = np.array(images)
            y_train = np.array(angles)
            yield sklearn.utils.shuffle(X_train, y_train)

# compile and train the model using the generator function
train_generator = generator(train_samples, batch_size=32)
validation_generator = generator(validation_samples, batch_size=32)

ch, row, col = 3, 80, 320  # Trimmed image format

model = Sequential()
# Preprocess incoming data, centered around zero with small standard deviation 
model.add(Lambda(lambda x: x/127.5 - 1.,
        input_shape=(ch, row, col),
        output_shape=(ch, row, col)))
model.add(... finish defining the rest of your model architecture here ...)

model.compile(loss='mse', optimizer='adam')
model.fit_generator(train_generator, samples_per_epoch= /
            len(train_samples), validation_data=validation_generator, /
            nb_val_samples=len(validation_samples), nb_epoch=3)

"""
If the above code throw exceptions, try 
model.fit_generator(train_generator, steps_per_epoch= len(train_samples),
validation_data=validation_generator, validation_steps=len(validation_samples), epochs=5, verbose = 1)
"""

19. Recording Video in Autonomous Mode

在这里说明了如何生成video:Behavioral Cloning Project

drive.py:在驾驶模式下生成一帧帧的图片

python drive.py model.h5 run1

run1是生成了图片的目录,可以查看下:

ls run1

[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_424.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_451.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_477.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_528.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_573.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_618.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_697.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_723.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_749.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_817.jpg
...

上面的图片是为了下一步生成video使用的:

Using video.py:将图片整合成video

python video.py run1

最终整合成run1目录下的mp4文件,另外也可以指定video的FPS,默认是60.

python video.py run1 --fps 48

20. Project Workspace Instructions

Accessing and using the workspace:

  • Click YES to run in GPU-enabled mode only if you:
    1. Want to collect training data.
    2. Want to test your solution.
    3. Want to train your neural network using GPU (network training will be challenging without using a GPU).
  • Click NO to run in GPU-enabled mode only if you:
    1. Want to code your solution on the text editor.

Simulator:

To run the simulator click on the button Simulator on the bottom right, then go to the newly opened tab in your browser and double-click on the simulator icon in the desktop. This will open the simulator’s configuration window, finally, run and test your code or gather data.

image

Note: If you get an error when clicking on the Simulator button, please check that you have enabled the GPU.

Project submission when using the workspace:

Make sure that your directory /home/workspace/ doesn’t include your training images since the Reviews system is limited to 10,000 files and 500MB. If you have more than that you will get the following error too many files when trying to submit. To fix this error just move your images to directory below~/opt/.

Things to keep in mind:

  1. Try to not save data directly within /home/workspace/ because this directly only have the storage capacity of 3 GB, and storing a large amount of data within this directory, can freeze your workspace. If you need to save a large amount of data use /opt/ but all the data saved there will be lost once you leave the classroom.
  2. If you want to avoid collecting data every time you refresh workspace, the data can be stored in your local environment (Dropbox, GitHub, Google Drive, etc.) and can be pulled over to the workspace, using the command 'wget' command. Common Issues

Common Issues

Further Help:

  1. fit_generator
  2. a helpful guide
  3. You can use our sample data for track 1
  4. Keep in mind that training images are loaded in BGR colorspace using cv2 while drive.py load images in RGB to predict the steering angles.

Using GitHub and Creating Effective READMEs If you are unfamiliar with GitHub , Udacity has a brief GitHub tutorial to get you started. Udacity also provides a more detailed free course on git and GitHub. To learn about REAMDE files and Markdown, Udacity provides a free course on READMEs, as well. GitHub also provides a tutorial about creating Markdown files.

21. 整合后代码:

import csv
import cv2
import numpy as np
import datetime
import random
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split
lines = []

#load csv file
with open('data/driving_log.csv') as csvfile:
    reader = csv.reader(csvfile)
    header = next(reader)
    for line in reader:
        lines.append(line)

#crop resize and change color space of image
def crop_and_resize_change_color_space(image):
    image = cv2.cvtColor(cv2.resize(image[80:140,:], (32,32)), cv2.COLOR_BGR2RGB)
    return image

#generator to yeild processed images for training as well as validation data set
def generator_images(data, batchSize = 32):
    while True:
        data = shuffle(data)
        for i in range(0, len(data), int(batchSize/4)):
            X_batch = []
            y_batch = []
            details = data[i: i+int(batchSize/4)]
            for line in details:
                image = crop_and_resize_change_color_space(cv2.imread('data/IMG/'+ line[0].split('/')[-1]))
                steering_angle = float(line[3])
                #appending original image
                X_batch.append(image)
                y_batch.append(steering_angle)
                #appending flipped image
                X_batch.append(np.fliplr(image))
                y_batch.append(-steering_angle)
                # appending left camera image and steering angle with offset
                X_batch.append(crop_and_resize_change_color_space(cv2.imread('./data/IMG/'+ line[1].split('/')[-1])))
                y_batch.append(steering_angle+0.4)
                # appending right camera image and steering angle with offset
                X_batch.append(crop_and_resize_change_color_space(cv2.imread('./data/IMG/'+ line[2].split('/')[-1])))
                y_batch.append(steering_angle-0.3)
            # converting to numpy array
            X_batch = np.array(X_batch)
            y_batch = np.array(y_batch)
            yield shuffle(X_batch, y_batch)

#Diving data among training and validation set
training_data, validatio_data = train_test_split(lines, test_size = 0.2)

from keras.models import Sequential
from keras.layers import Convolution2D, Dropout, MaxPooling2D, Flatten, Activation, Dense, Cropping2D, Lambda

#creating model to be trained
model = Sequential()
model.add(Lambda(lambda x: x /255.0 - 0.5, input_shape=(32,32,3) ))
model.add(Convolution2D(15, 3, 3, subsample=(2, 2), activation = 'relu'))
model.add(Dropout(0.4))
model.add(MaxPooling2D((2,2)))
model.add(Flatten())
model.add(Dense(1))

#compiling and running the model
model.compile('adam', 'mse')
model.fit_generator(generator_images(training_data), samples_per_epoch = len(training_data)*4, nb_epoch = 2, validation_data=generator_images(validatio_data), nb_val_samples=len(validatio_data))

#saving the model
model.save('model.h5')