Coursera

Week 2 Assignment: Zombie Detection

Welcome to this week’s programming assignment! You will use the Object Detection API and retrain RetinaNet to spot Zombies using just 5 training images. You will setup the model to restore pretrained weights and fine tune the classification layers.

Important: This colab notebook has read-only access so you won’t be able to save your changes. If you want to save your work periodically, please click File -> Save a Copy in Drive to create a copy in your account, then work from there.

Exercises

• Exercise 1 - Import Object Detection API packages
• Exercise 2 - Visualize the training images
• Exercise 3 - Define the category index dictionary
• Exercise 4 - Download checkpoints
• Exercise 5.1 - Locate and read from the configuration file
• Exercise 5.2 - Modify the model configuration
• Exercise 5.3 - Modify model_config
• Exercise 5.4 - Build the custom model
• Exercise 6.1 - Define Checkpoints for the box predictor
• Exercise 6.2 - Define the temporary model checkpoint
• Exercise 6.3 - Restore the checkpoint
• Exercise 7 - Run a dummy image to generate the model variables
• Exercise 8 - Set training hyperparameters
• Exercise 9 - Select the prediction layer variables
• Exercise 10 - Define the training step
• Exercise 11 - Preprocess, predict, and post process an image

Installation

You’ll start by installing the Tensorflow 2 Object Detection API.

# uncomment the next line if you want to delete an existing models directory
!rm -rf ./models/

# clone the Tensorflow Model Garden
!git clone --depth 1 https://github.com/tensorflow/models/

Cloning into 'models'...
remote: Enumerating objects: 3993, done.
remote: Counting objects: 100% (3993/3993), done.
remote: Compressing objects: 100% (3105/3105), done.
remote: Total 3993 (delta 1153), reused 1970 (delta 831), pack-reused 0
Receiving objects: 100% (3993/3993), 49.77 MiB | 33.33 MiB/s, done.
Resolving deltas: 100% (1153/1153), done.

# Compile the Object Detection API protocol buffers and install the necessary packages
!cd models/research/ && protoc object_detection/protos/*.proto --python_out=. && cp object_detection/packages/tf2/setup.py . && python -m pip install .

Processing /content/models/research
  Preparing metadata (setup.py) ... [?25l[?25hdone
Collecting avro-python3 (from object-detection==0.1)
  Downloading avro-python3-1.10.2.tar.gz (38 kB)
  Preparing metadata (setup.py) ... [?25l[?25hdone
Collecting apache-beam (from object-detection==0.1)
  Downloading apache_beam-2.50.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.7/14.7 MB 40.2 MB/s eta 0:00:00
[?25hRequirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (9.4.0)
Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (4.9.3)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (3.7.1)
Requirement already satisfied: Cython in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (3.0.2)
Requirement already satisfied: contextlib2 in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (21.6.0)
Requirement already satisfied: tf-slim in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (1.1.0)
Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (1.16.0)
Requirement already satisfied: pycocotools in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (2.0.7)
Collecting lvis (from object-detection==0.1)
  Downloading lvis-0.5.3-py3-none-any.whl (14 kB)
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (1.11.3)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (1.5.3)
Collecting tf-models-official>=2.5.1 (from object-detection==0.1)
  Downloading tf_models_official-2.13.2-py2.py3-none-any.whl (2.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 44.6 MB/s eta 0:00:00
[?25hCollecting tensorflow_io (from object-detection==0.1)
  Downloading tensorflow_io-0.34.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (28.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 28.8/28.8 MB 17.3 MB/s eta 0:00:00
[?25hRequirement already satisfied: keras in /usr/local/lib/python3.10/dist-packages (from object-detection==0.1) (2.13.1)
Collecting pyparsing==2.4.7 (from object-detection==0.1)
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.8/67.8 kB 8.5 MB/s eta 0:00:00
[?25hCollecting sacrebleu<=2.2.0 (from object-detection==0.1)
  Downloading sacrebleu-2.2.0-py3-none-any.whl (116 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.6/116.6 kB 15.5 MB/s eta 0:00:00
[?25hCollecting portalocker (from sacrebleu<=2.2.0->object-detection==0.1)
  Downloading portalocker-2.8.2-py3-none-any.whl (17 kB)
Requirement already satisfied: regex in /usr/local/lib/python3.10/dist-packages (from sacrebleu<=2.2.0->object-detection==0.1) (2023.6.3)
Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.10/dist-packages (from sacrebleu<=2.2.0->object-detection==0.1) (0.9.0)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from sacrebleu<=2.2.0->object-detection==0.1) (1.23.5)
Collecting colorama (from sacrebleu<=2.2.0->object-detection==0.1)
  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Requirement already satisfied: gin-config in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (0.5.0)
Requirement already satisfied: google-api-python-client>=1.6.7 in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (2.84.0)
Collecting immutabledict (from tf-models-official>=2.5.1->object-detection==0.1)
  Downloading immutabledict-3.0.0-py3-none-any.whl (4.0 kB)
Requirement already satisfied: kaggle>=1.3.9 in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (1.5.16)
Requirement already satisfied: oauth2client in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (4.1.3)
Requirement already satisfied: opencv-python-headless in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (4.8.1.78)
Requirement already satisfied: psutil>=5.4.3 in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (5.9.5)
Requirement already satisfied: py-cpuinfo>=3.3.0 in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (9.0.0)
Requirement already satisfied: pyyaml>=6.0.0 in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (6.0.1)
Collecting sentencepiece (from tf-models-official>=2.5.1->object-detection==0.1)
  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 71.4 MB/s eta 0:00:00
[?25hCollecting seqeval (from tf-models-official>=2.5.1->object-detection==0.1)
  Downloading seqeval-1.2.2.tar.gz (43 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.6/43.6 kB 5.9 MB/s eta 0:00:00
[?25h  Preparing metadata (setup.py) ... [?25l[?25hdone
Requirement already satisfied: tensorflow-datasets in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (4.9.3)
Requirement already satisfied: tensorflow-hub>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (0.14.0)
Collecting tensorflow-model-optimization>=0.4.1 (from tf-models-official>=2.5.1->object-detection==0.1)
  Downloading tensorflow_model_optimization-0.7.5-py2.py3-none-any.whl (241 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 241.2/241.2 kB 15.2 MB/s eta 0:00:00
[?25hCollecting tensorflow-text~=2.13.0 (from tf-models-official>=2.5.1->object-detection==0.1)
  Downloading tensorflow_text-2.13.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.5/6.5 MB 38.5 MB/s eta 0:00:00
[?25hRequirement already satisfied: tensorflow~=2.13.0 in /usr/local/lib/python3.10/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (2.13.0)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->object-detection==0.1) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->object-detection==0.1) (2023.3.post1)
Requirement already satisfied: absl-py>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from tf-slim->object-detection==0.1) (1.4.0)
Collecting crcmod<2.0,>=1.7 (from apache-beam->object-detection==0.1)
  Downloading crcmod-1.7.tar.gz (89 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.7/89.7 kB 12.4 MB/s eta 0:00:00
[?25h  Preparing metadata (setup.py) ... [?25l[?25hdone
Collecting orjson<4.0 (from apache-beam->object-detection==0.1)
  Downloading orjson-3.9.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.7/138.7 kB 19.5 MB/s eta 0:00:00
[?25hCollecting dill<0.3.2,>=0.3.1.1 (from apache-beam->object-detection==0.1)
  Downloading dill-0.3.1.1.tar.gz (151 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 152.0/152.0 kB 21.3 MB/s eta 0:00:00
[?25h  Preparing metadata (setup.py) ... [?25l[?25hdone
Requirement already satisfied: cloudpickle~=2.2.1 in /usr/local/lib/python3.10/dist-packages (from apache-beam->object-detection==0.1) (2.2.1)
Collecting fastavro<2,>=0.23.6 (from apache-beam->object-detection==0.1)
  Downloading fastavro-1.8.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 80.2 MB/s eta 0:00:00
[?25hCollecting fasteners<1.0,>=0.3 (from apache-beam->object-detection==0.1)
  Downloading fasteners-0.19-py3-none-any.whl (18 kB)
Requirement already satisfied: grpcio!=1.48.0,<2,>=1.33.1 in /usr/local/lib/python3.10/dist-packages (from apache-beam->object-detection==0.1) (1.58.0)
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam->object-detection==0.1)
  Downloading hdfs-2.7.2.tar.gz (43 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.4/43.4 kB 6.2 MB/s eta 0:00:00
[?25h  Preparing metadata (setup.py) ... [?25l[?25hdone
Requirement already satisfied: httplib2<0.23.0,>=0.8 in /usr/local/lib/python3.10/dist-packages (from apache-beam->object-detection==0.1) (0.22.0)
Collecting objsize<0.7.0,>=0.6.1 (from apache-beam->object-detection==0.1)
  Downloading objsize-0.6.1-py3-none-any.whl (9.3 kB)
Collecting pymongo<5.0.0,>=3.8.0 (from apache-beam->object-detection==0.1)
  Downloading pymongo-4.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (671 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 671.3/671.3 kB 65.9 MB/s eta 0:00:00
[?25hRequirement already satisfied: proto-plus<2,>=1.7.1 in /usr/local/lib/python3.10/dist-packages (from apache-beam->object-detection==0.1) (1.22.3)
Requirement already satisfied: protobuf<4.24.0,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from apache-beam->object-detection==0.1) (3.20.3)
Requirement already satisfied: pydot<2,>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam->object-detection==0.1) (1.4.2)
Requirement already satisfied: requests<3.0.0,>=2.24.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam->object-detection==0.1) (2.31.0)
Requirement already satisfied: typing-extensions>=3.7.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam->object-detection==0.1) (4.5.0)
Collecting zstandard<1,>=0.18.0 (from apache-beam->object-detection==0.1)
  Downloading zstandard-0.21.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 107.1 MB/s eta 0:00:00
[?25hRequirement already satisfied: pyarrow<12.0.0,>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam->object-detection==0.1) (9.0.0)
Requirement already satisfied: cycler>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from lvis->object-detection==0.1) (0.12.0)
Requirement already satisfied: kiwisolver>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from lvis->object-detection==0.1) (1.4.5)
Requirement already satisfied: opencv-python>=4.1.0.25 in /usr/local/lib/python3.10/dist-packages (from lvis->object-detection==0.1) (4.8.0.76)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->object-detection==0.1) (1.1.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->object-detection==0.1) (4.43.0)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->object-detection==0.1) (23.1)
Requirement already satisfied: tensorflow-io-gcs-filesystem==0.34.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow_io->object-detection==0.1) (0.34.0)
Requirement already satisfied: google-auth<3.0.0dev,>=1.19.0 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (2.17.3)
Requirement already satisfied: google-auth-httplib2>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (0.1.1)
Requirement already satisfied: google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (2.11.1)
Requirement already satisfied: uritemplate<5,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (4.1.1)
Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam->object-detection==0.1)
  Downloading docopt-0.6.2.tar.gz (25 kB)
  Preparing metadata (setup.py) ... [?25l[?25hdone
Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (2023.7.22)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (4.66.1)
Requirement already satisfied: python-slugify in /usr/local/lib/python3.10/dist-packages (from kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (8.0.1)
Requirement already satisfied: urllib3 in /usr/local/lib/python3.10/dist-packages (from kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (2.0.5)
Requirement already satisfied: bleach in /usr/local/lib/python3.10/dist-packages (from kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (6.0.0)
Collecting dnspython<3.0.0,>=1.16.0 (from pymongo<5.0.0,>=3.8.0->apache-beam->object-detection==0.1)
  Downloading dnspython-2.4.2-py3-none-any.whl (300 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 300.4/300.4 kB 39.5 MB/s eta 0:00:00
[?25hRequirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.24.0->apache-beam->object-detection==0.1) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.24.0->apache-beam->object-detection==0.1) (3.4)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (1.6.3)
Requirement already satisfied: flatbuffers>=23.1.21 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (23.5.26)
Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (0.4.0)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (0.2.0)
Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (3.9.0)
Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (16.0.6)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (3.3.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (67.7.2)
Requirement already satisfied: tensorboard<2.14,>=2.13 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (2.13.0)
Requirement already satisfied: tensorflow-estimator<2.14,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (2.13.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (2.3.0)
Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (1.15.0)
Requirement already satisfied: dm-tree~=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow-model-optimization>=0.4.1->tf-models-official>=2.5.1->object-detection==0.1) (0.1.8)
Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.10/dist-packages (from oauth2client->tf-models-official>=2.5.1->object-detection==0.1) (0.5.0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python3.10/dist-packages (from oauth2client->tf-models-official>=2.5.1->object-detection==0.1) (0.3.0)
Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from oauth2client->tf-models-official>=2.5.1->object-detection==0.1) (4.9)
Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.10/dist-packages (from seqeval->tf-models-official>=2.5.1->object-detection==0.1) (1.2.2)
Requirement already satisfied: array-record in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (0.4.1)
Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (8.1.7)
Requirement already satisfied: etils[enp,epath,etree]>=0.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (1.5.0)
Requirement already satisfied: promise in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (2.3)
Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (1.14.0)
Requirement already satisfied: toml in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (0.10.2)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (0.41.2)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from etils[enp,epath,etree]>=0.9.0->tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (2023.6.0)
Requirement already satisfied: importlib_resources in /usr/local/lib/python3.10/dist-packages (from etils[enp,epath,etree]>=0.9.0->tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (6.1.0)
Requirement already satisfied: zipp in /usr/local/lib/python3.10/dist-packages (from etils[enp,epath,etree]>=0.9.0->tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (3.17.0)
Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (1.60.0)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.0dev,>=1.19.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (5.3.1)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.21.3->seqeval->tf-models-official>=2.5.1->object-detection==0.1) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.21.3->seqeval->tf-models-official>=2.5.1->object-detection==0.1) (3.2.0)
Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.14,>=2.13->tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (1.0.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.14,>=2.13->tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (3.4.4)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.14,>=2.13->tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (0.7.1)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.14,>=2.13->tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (2.3.7)
Requirement already satisfied: webencodings in /usr/local/lib/python3.10/dist-packages (from bleach->kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (0.5.1)
Requirement already satisfied: text-unidecode>=1.3 in /usr/local/lib/python3.10/dist-packages (from python-slugify->kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (1.3)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard<2.14,>=2.13->tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (1.3.1)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.14,>=2.13->tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (2.1.3)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard<2.14,>=2.13->tensorflow~=2.13.0->tf-models-official>=2.5.1->object-detection==0.1) (3.2.2)
Building wheels for collected packages: object-detection, avro-python3, crcmod, dill, hdfs, seqeval, docopt
  Building wheel for object-detection (setup.py) ... [?25l[?25hdone
  Created wheel for object-detection: filename=object_detection-0.1-py3-none-any.whl size=1697354 sha256=3e20ec9a8d5a05582305cd3b3b2bdd32f0cafb4cbc333fdb560bc2a9fd68b5d9
  Stored in directory: /tmp/pip-ephem-wheel-cache-szp5u16o/wheels/53/dd/70/2de274d6c443c69d367bd6a5606f95e5a6df61aacf1435ec0d
  Building wheel for avro-python3 (setup.py) ... [?25l[?25hdone
  Created wheel for avro-python3: filename=avro_python3-1.10.2-py3-none-any.whl size=43991 sha256=cc3d8db5f8e7974e7ed8fb7f0218153049c2d97480bbfa6a1da9697ffe4bee19
  Stored in directory: /root/.cache/pip/wheels/bc/85/62/6cdd81c56f923946b401cecff38055b94c9b766927f7d8ca82
  Building wheel for crcmod (setup.py) ... [?25l[?25hdone
  Created wheel for crcmod: filename=crcmod-1.7-cp310-cp310-linux_x86_64.whl size=31404 sha256=32bc59778c186d966911e6637d804ba231677b924e6666ae6012f8f08f15aa53
  Stored in directory: /root/.cache/pip/wheels/85/4c/07/72215c529bd59d67e3dac29711d7aba1b692f543c808ba9e86
  Building wheel for dill (setup.py) ... [?25l[?25hdone
  Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl size=78542 sha256=1657b7bb1940ad687f3715c304038cc2cfcd54af4edc13fbd11f7fe6e1ada5ad
  Stored in directory: /root/.cache/pip/wheels/ea/e2/86/64980d90e297e7bf2ce588c2b96e818f5399c515c4bb8a7e4f
  Building wheel for hdfs (setup.py) ... [?25l[?25hdone
  Created wheel for hdfs: filename=hdfs-2.7.2-py3-none-any.whl size=34168 sha256=a758f9b10ae255ded45d6e5349a4910cf27603bf108797e00cce21ef990aeb0a
  Stored in directory: /root/.cache/pip/wheels/ab/39/8e/e1905de9af8ae74911cd3e53e721995cd230816f63776e5825
  Building wheel for seqeval (setup.py) ... [?25l[?25hdone
  Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16162 sha256=8a245c56cb4b6cc464f98a10c3d5b5c03e1f4929ae28171a9ab0b81df52078ce
  Stored in directory: /root/.cache/pip/wheels/1a/67/4a/ad4082dd7dfc30f2abfe4d80a2ed5926a506eb8a972b4767fa
  Building wheel for docopt (setup.py) ... [?25l[?25hdone
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13705 sha256=fe38e0f34bb3c6840ad9e32a668506c4c768f5699fc1561a5ae2cf438474578d
  Stored in directory: /root/.cache/pip/wheels/fc/ab/d4/5da2067ac95b36618c629a5f93f809425700506f72c9732fac
Successfully built object-detection avro-python3 crcmod dill hdfs seqeval docopt
Installing collected packages: sentencepiece, docopt, crcmod, zstandard, tensorflow-model-optimization, tensorflow_io, pyparsing, portalocker, orjson, objsize, immutabledict, fasteners, fastavro, dnspython, dill, colorama, avro-python3, sacrebleu, pymongo, hdfs, seqeval, lvis, apache-beam, tensorflow-text, tf-models-official, object-detection
  Attempting uninstall: pyparsing
    Found existing installation: pyparsing 3.1.1
    Uninstalling pyparsing-3.1.1:
      Successfully uninstalled pyparsing-3.1.1
Successfully installed apache-beam-2.50.0 avro-python3-1.10.2 colorama-0.4.6 crcmod-1.7 dill-0.3.1.1 dnspython-2.4.2 docopt-0.6.2 fastavro-1.8.4 fasteners-0.19 hdfs-2.7.2 immutabledict-3.0.0 lvis-0.5.3 object-detection-0.1 objsize-0.6.1 orjson-3.9.7 portalocker-2.8.2 pymongo-4.5.0 pyparsing-2.4.7 sacrebleu-2.2.0 sentencepiece-0.1.99 seqeval-1.2.2 tensorflow-model-optimization-0.7.5 tensorflow-text-2.13.0 tensorflow_io-0.34.0 tf-models-official-2.13.2 zstandard-0.21.0

Imports

Let’s now import the packages you will use in this assignment.

import matplotlib
import matplotlib.pyplot as plt

import os
import random
import zipfile
import io
import scipy.misc
import numpy as np

import glob
import imageio
from six import BytesIO
from PIL import Image, ImageDraw, ImageFont
from IPython.display import display, Javascript
from IPython.display import Image as IPyImage

try:
  # %tensorflow_version only exists in Colab.
  %tensorflow_version 2.x
except Exception:
  pass

import tensorflow as tf
tf.get_logger().setLevel('ERROR')

Colab only includes TensorFlow 2.x; %tensorflow_version has no effect.

Exercise 1: Import Object Detection API packages

Import the necessary modules from the object_detection package.

• From the utils package:
  • label_map_util
  • config_util: You’ll use this to read model configurations from a .config file and then modify that configuration
  • visualization_utils: please give this the alias viz_utils, as this is what will be used in some visualization code that is given to you later.
  • colab_utils
• From the builders package:
  • model_builder: This builds your model according to the model configuration that you’ll specify.

### START CODE HERE (Replace Instances of `None` with your code) ###
# import the label map utility module
from object_detection.utils import label_map_util

# import module for reading and updating configuration files.
from object_detection.utils import config_util

# import module for visualization. use the alias `viz_utils`
from object_detection.utils import visualization_utils as viz_utils

# import module for building the detection model
from object_detection.builders import model_builder
### END CODE HERE ###

# import module for utilities in Colab
from object_detection.utils import colab_utils

Utilities

You’ll define a couple of utility functions for loading images and plotting detections. This code is provided for you.

def load_image_into_numpy_array(path):
    """Load an image from file into a numpy array.

    Puts image into numpy array to feed into tensorflow graph.
    Note that by convention we put it into a numpy array with shape
    (height, width, channels), where channels=3 for RGB.

    Args:
    path: a file path.

    Returns:
    uint8 numpy array with shape (img_height, img_width, 3)
    """

    img_data = tf.io.gfile.GFile(path, 'rb').read()
    image = Image.open(BytesIO(img_data))
    (im_width, im_height) = image.size

    return np.array(image.getdata()).reshape(
        (im_height, im_width, 3)).astype(np.uint8)


def plot_detections(image_np,
                    boxes,
                    classes,
                    scores,
                    category_index,
                    figsize=(12, 16),
                    image_name=None):
    """Wrapper function to visualize detections.

    Args:
    image_np: uint8 numpy array with shape (img_height, img_width, 3)
    boxes: a numpy array of shape [N, 4]
    classes: a numpy array of shape [N]. Note that class indices are 1-based,
          and match the keys in the label map.
    scores: a numpy array of shape [N] or None.  If scores=None, then
          this function assumes that the boxes to be plotted are groundtruth
          boxes and plot all boxes as black with no classes or scores.
    category_index: a dict containing category dictionaries (each holding
          category index `id` and category name `name`) keyed by category indices.
    figsize: size for the figure.
    image_name: a name for the image file.
    """

    image_np_with_annotations = image_np.copy()

    viz_utils.visualize_boxes_and_labels_on_image_array(
        image_np_with_annotations,
        boxes,
        classes,
        scores,
        category_index,
        use_normalized_coordinates=True,
        min_score_thresh=0.8)

    if image_name:
        plt.imsave(image_name, image_np_with_annotations)

    else:
        plt.imshow(image_np_with_annotations)

Download the Zombie data

Now you will get 5 images of zombies that you will use for training.

• The zombies are hosted in a Google bucket.
• You can download and unzip the images into a local training/ directory by running the cell below.

# uncomment the next 2 lines if you want to delete an existing zip and training directory
!rm training-zombie.zip
!rm -rf ./training

# download the images
!wget --no-check-certificate \
    https://storage.googleapis.com/tensorflow-3-public/datasets/training-zombie.zip \
    -O ./training-zombie.zip

# unzip to a local directory
local_zip = './training-zombie.zip'
zip_ref = zipfile.ZipFile(local_zip, 'r')
zip_ref.extractall('./training')
zip_ref.close()

rm: cannot remove 'training-zombie.zip': No such file or directory
--2023-10-04 10:51:05--  https://storage.googleapis.com/tensorflow-3-public/datasets/training-zombie.zip
Resolving storage.googleapis.com (storage.googleapis.com)... 108.177.98.207, 74.125.197.207, 74.125.135.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|108.177.98.207|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1915446 (1.8M) [application/zip]
Saving to: ‘./training-zombie.zip’

./training-zombie.z 100%[===================>]   1.83M  --.-KB/s    in 0.01s   

2023-10-04 10:51:05 (143 MB/s) - ‘./training-zombie.zip’ saved [1915446/1915446]

Exercise 2: Visualize the training images

Next, you’ll want to inspect the images that you just downloaded.

• Please replace instances of None below to load and visualize the 5 training images.
• You can inspect the training directory (using the Files button on the left side of this Colab) to see the filenames of the zombie images. The paths for the images will look like this:

./training/training-zombie1.jpg
./training/training-zombie2.jpg
./training/training-zombie3.jpg
./training/training-zombie4.jpg
./training/training-zombie5.jpg

• To set file paths, you’ll use os.path.join. As an example, if you wanted to create the path ‘./parent_folder/file_name1.txt’, you could write:

os.path.join('parent_folder', 'file_name' + str(1) + '.txt')

• You should see the 5 training images after running this cell. If not, please inspect your code, particularly the image_path.

%matplotlib inline

### START CODE HERE (Replace Instances of `None` with your code) ###

# assign the name (string) of the directory containing the training images
train_image_dir = './training'

# declare an empty list
train_images_np = []

# run a for loop for each image
for i in range(1, 6):

    # define the path (string) for each image
    image_path = os.path.join(train_image_dir, f"training-zombie{i}.jpg")
    print(image_path)

    # load images into numpy arrays and append to a list
    train_images_np.append(load_image_into_numpy_array(image_path))
### END CODE HERE ###

# configure plot settings via rcParams
plt.rcParams['axes.grid'] = False
plt.rcParams['xtick.labelsize'] = False
plt.rcParams['ytick.labelsize'] = False
plt.rcParams['xtick.top'] = False
plt.rcParams['xtick.bottom'] = False
plt.rcParams['ytick.left'] = False
plt.rcParams['ytick.right'] = False
plt.rcParams['figure.figsize'] = [14, 7]

# plot images
for idx, train_image_np in enumerate(train_images_np):
    plt.subplot(1, 5, idx+1)
    plt.imshow(train_image_np)

plt.show()

./training/training-zombie1.jpg
./training/training-zombie2.jpg
./training/training-zombie3.jpg
./training/training-zombie4.jpg
./training/training-zombie5.jpg

Prepare data for training (Optional)

In this section, you will create your ground truth boxes. You can either draw your own boxes or use a prepopulated list of coordinates that we have provided below.

# Define the list of ground truth boxes
gt_boxes = []

Option 1: draw your own ground truth boxes

If you want to draw your own, please run the next cell and the following test code. If not, then skip these optional cells.

• Draw a box around the zombie in each image.
• Click the next image button to go to the next image
• Click submit when it says “All images completed!!”.

• Make sure to not make the bounding box too big.
  • If the box is too big, the model might learn the features of the background (e.g. door, road, etc) in determining if there is a zombie or not.
• Include the entire zombie inside the box.
• As an example, scroll to the beginning of this notebook to look at the bounding box around the zombie.

# Option 1: draw your own ground truth boxes

# annotate the training images
# colab_utils.annotate(train_images_np, box_storage_pointer=gt_boxes)

# Option 1: draw your own ground truth boxes
# TEST CODE:
# try:
#   assert(len(gt_boxes) == 5), "Warning: gt_boxes is empty. Did you click `submit`?"

# except AssertionError as e:
#   print(e)

# # checks if there are boxes for all 5 images
# for gt_box in gt_boxes:
#     try:
#       assert(gt_box is not None), "There are less than 5 sets of box coordinates. " \
#                                   "Please re-run the cell above to draw the boxes again.\n" \
#                                   "Alternatively, you can run the next cell to load pre-determined " \
#                                   "ground truth boxes."

#     except AssertionError as e:
#         print(e)
#         break


# ref_gt_boxes = [
#         np.array([[0.27333333, 0.41500586, 0.74333333, 0.57678781]]),
#         np.array([[0.29833333, 0.45955451, 0.75666667, 0.61078546]]),
#         np.array([[0.40833333, 0.18288394, 0.945, 0.34818288]]),
#         np.array([[0.16166667, 0.61899179, 0.8, 0.91910903]]),
#         np.array([[0.28833333, 0.12543962, 0.835, 0.35052755]]),
#       ]

# for gt_box, ref_gt_box in zip(gt_boxes, ref_gt_boxes):
#     try:
#       assert(np.allclose(gt_box, ref_gt_box, atol=0.04)), "One of the boxes is too big or too small. " \
#                                                           "Please re-draw and make the box tighter around the zombie."

#     except AssertionError as e:
#       print(e)
#       break

Option 2: use the given ground truth boxes

You can also use this list if you opt not to draw the boxes yourself.

# Option 2: use given ground truth boxes
# set this to `True` if you want to override the boxes you drew
override = False

# bounding boxes for each of the 5 zombies found in each image.
# you can use these instead of drawing the boxes yourself.
ref_gt_boxes = [
        np.array([[0.27333333, 0.41500586, 0.74333333, 0.57678781]]),
        np.array([[0.29833333, 0.45955451, 0.75666667, 0.61078546]]),
        np.array([[0.40833333, 0.18288394, 0.945, 0.34818288]]),
        np.array([[0.16166667, 0.61899179, 0.8, 0.91910903]]),
        np.array([[0.28833333, 0.12543962, 0.835, 0.35052755]]),
      ]

# if gt_boxes is empty, use the reference
if not gt_boxes or override is True:
  gt_boxes = ref_gt_boxes

# if gt_boxes does not contain 5 box coordinates, use the reference
for gt_box in gt_boxes:
    try:
      assert(gt_box is not None)

    except:
      gt_boxes = ref_gt_boxes

      break

View your ground truth box coordinates

Whether you chose to draw your own or use the given boxes, please check your list of ground truth box coordinates.

# print the coordinates of your ground truth boxes
for gt_box in gt_boxes:
  print(gt_box)

[[0.27333333 0.41500586 0.74333333 0.57678781]]
[[0.29833333 0.45955451 0.75666667 0.61078546]]
[[0.40833333 0.18288394 0.945      0.34818288]]
[[0.16166667 0.61899179 0.8        0.91910903]]
[[0.28833333 0.12543962 0.835      0.35052755]]

Below, we add the class annotations. For simplicity, we assume just a single class, though it should be straightforward to extend this to handle multiple classes. We will also convert everything to the format that the training loop expects (e.g., conversion to tensors, one-hot representations, etc.).

Exercise 3: Define the category index dictionary

You’ll need to tell the model which integer class ID to assign to the ‘zombie’ category, and what ‘name’ to associate with that integer id.

• zombie_class_id: By convention, class ID integers start numbering from 1,2,3, onward.

  • If there is ever a ‘background’ class, it could be assigned the integer 0, but in this case, you’re just predicting the one zombie class.
  • Since you are just predicting one class (zombie), please assign 1 to the zombie class ID.

• category_index: Please define the category_index dictionary, which will have the same structure as this:

{human_class_id :
  {'id'  : human_class_id,
   'name': 'human_so_far'}
}

• Define category_index similar to the example dictionary above, except for zombies.

• This will be used by the succeeding functions to know the class id and name of zombie images.

• num_classes: Since you are predicting one class, please assign 1 to the number of classes that the model will predict.

  • This will be used during data preprocessing and again when you configure the model.

### START CODE HERE (Replace instances of `None` with your code ###

# Assign the zombie class ID
zombie_class_id = 1

# define a dictionary describing the zombie class
category_index = {
    zombie_class_id: {
        'id': zombie_class_id,
        "name": "zombie"
    }
}

# Specify the number of classes that the model will predict
num_classes = 1
### END CODE HERE ###

# TEST CODE:

print(category_index[zombie_class_id])

{'id': 1, 'name': 'zombie'}

Expected Output:

>{'id': 1, 'name': 'zombie'}

Data preprocessing

You will now do some data preprocessing so it is formatted properly before it is fed to the model:

• Convert the class labels to one-hot representations
• convert everything (i.e. train images, gt boxes and class labels) to tensors.

This code is provided for you.

# The `label_id_offset` here shifts all classes by a certain number of indices;
# we do this here so that the model receives one-hot labels where non-background
# classes start counting at the zeroth index.  This is ordinarily just handled
# automatically in our training binaries, but we need to reproduce it here.

label_id_offset = 1
train_image_tensors = []

# lists containing the one-hot encoded classes and ground truth boxes
gt_classes_one_hot_tensors = []
gt_box_tensors = []

for (train_image_np, gt_box_np) in zip(train_images_np, gt_boxes):

    # convert training image to tensor, add batch dimension, and add to list
    train_image_tensors.append(tf.expand_dims(tf.convert_to_tensor(
        train_image_np, dtype=tf.float32), axis=0))

    # convert numpy array to tensor, then add to list
    gt_box_tensors.append(tf.convert_to_tensor(gt_box_np, dtype=tf.float32))

    # apply offset to to have zero-indexed ground truth classes
    zero_indexed_groundtruth_classes = tf.convert_to_tensor(
        np.ones(shape=[gt_box_np.shape[0]], dtype=np.int32) - label_id_offset)

    # do one-hot encoding to ground truth classes
    gt_classes_one_hot_tensors.append(tf.one_hot(
        zero_indexed_groundtruth_classes, num_classes))

print('Done prepping data.')

Done prepping data.

Visualize the zombies with their ground truth bounding boxes

You should see the 5 training images with the bounding boxes after running the cell below. If not, please re-run the annotation tool again or use the prepopulated gt_boxes array given.

# give boxes a score of 100%
dummy_scores = np.array([1.0], dtype=np.float32)

# define the figure size
plt.figure(figsize=(30, 15))

# use the `plot_detections()` utility function to draw the ground truth boxes
for idx in range(5):
    plt.subplot(2, 4, idx+1)
    plot_detections(
      train_images_np[idx],
      gt_boxes[idx],
      np.ones(shape=[gt_boxes[idx].shape[0]], dtype=np.int32),
      dummy_scores, category_index)

plt.show()

Download the checkpoint containing the pre-trained weights

Next, you will download RetinaNet and copy it inside the object detection directory.

When working with models that are at the frontiers of research, the models and checkpoints may not yet be organized in a central location like the TensorFlow Garden (https://github.com/tensorflow/models).

• You’ll often read a blog post from the researchers, who will usually provide information on:
  • how to use the model
  • where to download the models and pre-trained checkpoints.

It’s good practice to do some of this “detective work”, so that you’ll feel more comfortable when exploring new models yourself! So please try the following steps:

• Go to the TensorFlow Blog, where researchers announce new findings.
• In the search box at the top of the page, search for “retinanet”.
• In the search results, click on the blog post titled “TensorFlow 2 meets the Object Detection API” (it may be the first search result).
• Skim through this blog and look for links to either the checkpoints or to Colabs that will show you how to use the checkpoints.
• Try to fill out the following code cell below, which does the following:
  • Download the compressed SSD Resnet 50 version 1, 640 x 640 checkpoint.
  • Untar (decompress) the tar file
  • Move the decompressed checkpoint to models/research/object_detection/test_data/

If you want some help getting started, please click on the “Initial Hints” cell to get some hints.

Initial Hints

General Hints to get started

• The link to the blog is TensorFlow 2 meets the Object Detection API
• In the blog, you'll find the text "COCO pre-trained weights, which links to a list of checkpoints in GitHub titled TensorFlow 2 Detection Model Zoo.
• If you read each checkpoint name, you'll find the one for SSD Resnet 50 version 1, 640 by 640. If you hover your mouse over
• If you right-click on the desired checkpoint link, you can save the link address, and use it in the code cell below to get the checkpoint.
• For more hints, please click on the cell "More Hints"

More Hints

More Hints

• To see how to download the checkpoint, look in the blog for links to Colab tutorials.
• For example, the blog links to a Colab titled Intro to Object Detection Colab
• In the Colab, you'll see the section titled "Build a detection model and load pre-trained model weights", which is followed by a code cell showing how to download, decompress, and relocate a checkpoint. Use similar syntax, except use the URL to the ssd resnet50 version 1 640x640 checkpoint instead.
• If you're feeling stuck, please click on the cell "Even More Hints".

Even More Hints

Even More Hints

• The blog post also links to a notebook titled Eager Few Shot Object Detection Colab
• In this notebook, look for the section titled "Create model and restore weights for all but last layer". The code cell below it shows how to download the exact checkpoint that you're interested in.
• You can also review the lecture videos for this week, which show the same code.

Exercise 4: Download checkpoints

• Download the compressed SSD Resnet 50 version 1, 640 x 640 checkpoint.
• Untar (decompress) the tar file
• Move the decompressed checkpoint to models/research/object_detection/test_data/

### START CODE HERE ###
# Download the SSD Resnet 50 version 1, 640x640 checkpoint
!wget http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.tar.gz

# untar (decompress) the tar file
!tar -xf ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.tar.gz

# copy the checkpoint to the test_data folder models/research/object_detection/test_data/
!mv ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/checkpoint models/research/object_detection/test_data/

### END CODE HERE

--2023-10-04 10:51:20--  http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.tar.gz
Resolving download.tensorflow.org (download.tensorflow.org)... 74.125.199.207, 74.125.20.207, 108.177.98.207, ...
Connecting to download.tensorflow.org (download.tensorflow.org)|74.125.199.207|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 244817203 (233M) [application/x-tar]
Saving to: ‘ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.tar.gz’

ssd_resnet50_v1_fpn 100%[===================>] 233.48M   147MB/s    in 1.6s    

2023-10-04 10:51:22 (147 MB/s) - ‘ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.tar.gz’ saved [244817203/244817203]

Configure the model

Here, you will configure the model for this use case.

Exercise 5.1: Locate and read from the configuration file

pipeline_config

• In the Colab, on the left side table of contents, click on the folder icon to display the file browser for the current workspace.
• Navigate to models/research/object_detection/configs/tf2. The folder has multiple .config files.
• Look for the file corresponding to ssd resnet 50 version 1 640x640.
• You can double-click the config file to view its contents. This may help you as you complete the next few code cells to configure your model.
• Set the pipeline_config to a string that contains the full path to the resnet config file, in other words: models/research/.../... .config

configs

If you look at the module config_util that you imported, it contains the following function:

def get_configs_from_pipeline_file(pipeline_config_path, config_override=None):

• Please use this function to load the configuration from your pipeline_config.
  • configs will now contain a dictionary.

tf.keras.backend.clear_session()


### START CODE HERE ###
# define the path to the .config file for ssd resnet 50 v1 640x640
pipeline_config = "/content/models/research/object_detection/configs/tf2/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.config"


# Load the configuration file into a dictionary
configs = config_util.get_configs_from_pipeline_file(
    pipeline_config_path=pipeline_config)

### END CODE HERE ###
# See what configs looks like
configs

{'model': ssd {
   num_classes: 90
   image_resizer {
     fixed_shape_resizer {
       height: 640
       width: 640
     }
   }
   feature_extractor {
     type: "ssd_resnet50_v1_fpn_keras"
     depth_multiplier: 1.0
     min_depth: 16
     conv_hyperparams {
       regularizer {
         l2_regularizer {
           weight: 0.00039999998989515007
         }
       }
       initializer {
         truncated_normal_initializer {
           mean: 0.0
           stddev: 0.029999999329447746
         }
       }
       activation: RELU_6
       batch_norm {
         decay: 0.996999979019165
         scale: true
         epsilon: 0.0010000000474974513
       }
     }
     override_base_feature_extractor_hyperparams: true
     fpn {
       min_level: 3
       max_level: 7
     }
   }
   box_coder {
     faster_rcnn_box_coder {
       y_scale: 10.0
       x_scale: 10.0
       height_scale: 5.0
       width_scale: 5.0
     }
   }
   matcher {
     argmax_matcher {
       matched_threshold: 0.5
       unmatched_threshold: 0.5
       ignore_thresholds: false
       negatives_lower_than_unmatched: true
       force_match_for_each_row: true
       use_matmul_gather: true
     }
   }
   similarity_calculator {
     iou_similarity {
     }
   }
   box_predictor {
     weight_shared_convolutional_box_predictor {
       conv_hyperparams {
         regularizer {
           l2_regularizer {
             weight: 0.00039999998989515007
           }
         }
         initializer {
           random_normal_initializer {
             mean: 0.0
             stddev: 0.009999999776482582
           }
         }
         activation: RELU_6
         batch_norm {
           decay: 0.996999979019165
           scale: true
           epsilon: 0.0010000000474974513
         }
       }
       depth: 256
       num_layers_before_predictor: 4
       kernel_size: 3
       class_prediction_bias_init: -4.599999904632568
     }
   }
   anchor_generator {
     multiscale_anchor_generator {
       min_level: 3
       max_level: 7
       anchor_scale: 4.0
       aspect_ratios: 1.0
       aspect_ratios: 2.0
       aspect_ratios: 0.5
       scales_per_octave: 2
     }
   }
   post_processing {
     batch_non_max_suppression {
       score_threshold: 9.99999993922529e-09
       iou_threshold: 0.6000000238418579
       max_detections_per_class: 100
       max_total_detections: 100
     }
     score_converter: SIGMOID
   }
   normalize_loss_by_num_matches: true
   loss {
     localization_loss {
       weighted_smooth_l1 {
       }
     }
     classification_loss {
       weighted_sigmoid_focal {
         gamma: 2.0
         alpha: 0.25
       }
     }
     classification_weight: 1.0
     localization_weight: 1.0
   }
   encode_background_as_zeros: true
   normalize_loc_loss_by_codesize: true
   inplace_batchnorm_update: true
   freeze_batchnorm: false
 },
 'train_config': batch_size: 64
 data_augmentation_options {
   random_horizontal_flip {
   }
 }
 data_augmentation_options {
   random_crop_image {
     min_object_covered: 0.0
     min_aspect_ratio: 0.75
     max_aspect_ratio: 3.0
     min_area: 0.75
     max_area: 1.0
     overlap_thresh: 0.0
   }
 }
 sync_replicas: true
 optimizer {
   momentum_optimizer {
     learning_rate {
       cosine_decay_learning_rate {
         learning_rate_base: 0.03999999910593033
         total_steps: 25000
         warmup_learning_rate: 0.013333000242710114
         warmup_steps: 2000
       }
     }
     momentum_optimizer_value: 0.8999999761581421
   }
   use_moving_average: false
 }
 fine_tune_checkpoint: "PATH_TO_BE_CONFIGURED/resnet50.ckpt-1"
 num_steps: 25000
 startup_delay_steps: 0.0
 replicas_to_aggregate: 8
 max_number_of_boxes: 100
 unpad_groundtruth_tensors: false
 fine_tune_checkpoint_type: "classification"
 use_bfloat16: true
 fine_tune_checkpoint_version: V2,
 'train_input_config': label_map_path: "PATH_TO_BE_CONFIGURED/label_map.txt"
 tf_record_input_reader {
   input_path: "PATH_TO_BE_CONFIGURED/train2017-?????-of-00256.tfrecord"
 },
 'eval_config': metrics_set: "coco_detection_metrics"
 use_moving_averages: false,
 'eval_input_configs': [label_map_path: "PATH_TO_BE_CONFIGURED/label_map.txt"
 shuffle: false
 num_epochs: 1
 tf_record_input_reader {
   input_path: "PATH_TO_BE_CONFIGURED/val2017-?????-of-00032.tfrecord"
 }
 ],
 'eval_input_config': label_map_path: "PATH_TO_BE_CONFIGURED/label_map.txt"
 shuffle: false
 num_epochs: 1
 tf_record_input_reader {
   input_path: "PATH_TO_BE_CONFIGURED/val2017-?????-of-00032.tfrecord"
 }}

Exercise 5.2: Get the model configuration

model_config

• From the configs dictionary, access the object associated with the key ‘model’.
• model_config now contains an object of type object_detection.protos.model_pb2.DetectionModel.
• If you print model_config, you’ll see something like this:

ssd {
  num_classes: 90
  image_resizer {
    fixed_shape_resizer {
      height: 640
      width: 640
    }
  }
  feature_extractor {
...
...
  freeze_batchnorm: false

### START CODE HERE ###
# Read in the object stored at the key 'model' of the configs dictionary
model_config = configs["model"]

### END CODE HERE
# see what model_config looks like
model_config

ssd {
  num_classes: 90
  image_resizer {
    fixed_shape_resizer {
      height: 640
      width: 640
    }
  }
  feature_extractor {
    type: "ssd_resnet50_v1_fpn_keras"
    depth_multiplier: 1.0
    min_depth: 16
    conv_hyperparams {
      regularizer {
        l2_regularizer {
          weight: 0.00039999998989515007
        }
      }
      initializer {
        truncated_normal_initializer {
          mean: 0.0
          stddev: 0.029999999329447746
        }
      }
      activation: RELU_6
      batch_norm {
        decay: 0.996999979019165
        scale: true
        epsilon: 0.0010000000474974513
      }
    }
    override_base_feature_extractor_hyperparams: true
    fpn {
      min_level: 3
      max_level: 7
    }
  }
  box_coder {
    faster_rcnn_box_coder {
      y_scale: 10.0
      x_scale: 10.0
      height_scale: 5.0
      width_scale: 5.0
    }
  }
  matcher {
    argmax_matcher {
      matched_threshold: 0.5
      unmatched_threshold: 0.5
      ignore_thresholds: false
      negatives_lower_than_unmatched: true
      force_match_for_each_row: true
      use_matmul_gather: true
    }
  }
  similarity_calculator {
    iou_similarity {
    }
  }
  box_predictor {
    weight_shared_convolutional_box_predictor {
      conv_hyperparams {
        regularizer {
          l2_regularizer {
            weight: 0.00039999998989515007
          }
        }
        initializer {
          random_normal_initializer {
            mean: 0.0
            stddev: 0.009999999776482582
          }
        }
        activation: RELU_6
        batch_norm {
          decay: 0.996999979019165
          scale: true
          epsilon: 0.0010000000474974513
        }
      }
      depth: 256
      num_layers_before_predictor: 4
      kernel_size: 3
      class_prediction_bias_init: -4.599999904632568
    }
  }
  anchor_generator {
    multiscale_anchor_generator {
      min_level: 3
      max_level: 7
      anchor_scale: 4.0
      aspect_ratios: 1.0
      aspect_ratios: 2.0
      aspect_ratios: 0.5
      scales_per_octave: 2
    }
  }
  post_processing {
    batch_non_max_suppression {
      score_threshold: 9.99999993922529e-09
      iou_threshold: 0.6000000238418579
      max_detections_per_class: 100
      max_total_detections: 100
    }
    score_converter: SIGMOID
  }
  normalize_loss_by_num_matches: true
  loss {
    localization_loss {
      weighted_smooth_l1 {
      }
    }
    classification_loss {
      weighted_sigmoid_focal {
        gamma: 2.0
        alpha: 0.25
      }
    }
    classification_weight: 1.0
    localization_weight: 1.0
  }
  encode_background_as_zeros: true
  normalize_loc_loss_by_codesize: true
  inplace_batchnorm_update: true
  freeze_batchnorm: false
}

Exercise 5.3: Modify model_config

• Modify num_classes from the default 90 to the num_classes that you set earlier in this notebook.
  • num_classes is nested under ssd. You’ll need to use dot notation ‘obj.x’ and NOT bracket notation obj[‘x’]` to access num_classes.
• Freeze batch normalization
  • Batch normalization is not frozen in the default configuration.
  • If you inspect the model_config object, you’ll see that freeze_batchnorm is nested under ssd just like num_classes.
  • Freeze batch normalization by setting the relevant field to True.

### START CODE HERE ###
# Modify the number of classes from its default of 90
model_config.ssd.num_classes = num_classes

# Freeze batch normalization
model_config.ssd.freeze_batchnorm = True

### END CODE HERE

# See what model_config now looks like after you've customized it!
model_config

ssd {
  num_classes: 1
  image_resizer {
    fixed_shape_resizer {
      height: 640
      width: 640
    }
  }
  feature_extractor {
    type: "ssd_resnet50_v1_fpn_keras"
    depth_multiplier: 1.0
    min_depth: 16
    conv_hyperparams {
      regularizer {
        l2_regularizer {
          weight: 0.00039999998989515007
        }
      }
      initializer {
        truncated_normal_initializer {
          mean: 0.0
          stddev: 0.029999999329447746
        }
      }
      activation: RELU_6
      batch_norm {
        decay: 0.996999979019165
        scale: true
        epsilon: 0.0010000000474974513
      }
    }
    override_base_feature_extractor_hyperparams: true
    fpn {
      min_level: 3
      max_level: 7
    }
  }
  box_coder {
    faster_rcnn_box_coder {
      y_scale: 10.0
      x_scale: 10.0
      height_scale: 5.0
      width_scale: 5.0
    }
  }
  matcher {
    argmax_matcher {
      matched_threshold: 0.5
      unmatched_threshold: 0.5
      ignore_thresholds: false
      negatives_lower_than_unmatched: true
      force_match_for_each_row: true
      use_matmul_gather: true
    }
  }
  similarity_calculator {
    iou_similarity {
    }
  }
  box_predictor {
    weight_shared_convolutional_box_predictor {
      conv_hyperparams {
        regularizer {
          l2_regularizer {
            weight: 0.00039999998989515007
          }
        }
        initializer {
          random_normal_initializer {
            mean: 0.0
            stddev: 0.009999999776482582
          }
        }
        activation: RELU_6
        batch_norm {
          decay: 0.996999979019165
          scale: true
          epsilon: 0.0010000000474974513
        }
      }
      depth: 256
      num_layers_before_predictor: 4
      kernel_size: 3
      class_prediction_bias_init: -4.599999904632568
    }
  }
  anchor_generator {
    multiscale_anchor_generator {
      min_level: 3
      max_level: 7
      anchor_scale: 4.0
      aspect_ratios: 1.0
      aspect_ratios: 2.0
      aspect_ratios: 0.5
      scales_per_octave: 2
    }
  }
  post_processing {
    batch_non_max_suppression {
      score_threshold: 9.99999993922529e-09
      iou_threshold: 0.6000000238418579
      max_detections_per_class: 100
      max_total_detections: 100
    }
    score_converter: SIGMOID
  }
  normalize_loss_by_num_matches: true
  loss {
    localization_loss {
      weighted_smooth_l1 {
      }
    }
    classification_loss {
      weighted_sigmoid_focal {
        gamma: 2.0
        alpha: 0.25
      }
    }
    classification_weight: 1.0
    localization_weight: 1.0
  }
  encode_background_as_zeros: true
  normalize_loc_loss_by_codesize: true
  inplace_batchnorm_update: true
  freeze_batchnorm: true
}

Build the model

Recall that you imported model_builder.

• You’ll use model_builder to build the model according to the configurations that you have just downloaded and customized.

Exercise 5.4: Build the custom model

model_builder

model_builder has a function build:

def build(model_config, is_training, add_summaries=True):

• model_config: Set this to the model configuration that you just customized.
• is_training: Set this to True.
• You can keep the default value for the remaining parameter.
• Note that it will take some time to build the model.

### START CODE HERE (Replace instances of `None` with your code) ###
detection_model = model_builder.build(
    model_config=model_config,
    is_training=True
)
### END CODE HERE ###

print(type(detection_model))

<class 'object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch'>

Expected Output:

><class 'object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch'>

Restore weights from your checkpoint

Now, you will selectively restore weights from your checkpoint.

• Your end goal is to create a custom model which reuses parts of, but not all of the layers of RetinaNet (currently stored in the variable detection_model.)
  • The parts of RetinaNet that you want to reuse are:
    • Feature extraction layers
    • Bounding box regression prediction layer
  • The part of RetinaNet that you will not want to reuse is the classification prediction layer (since you will define and train your own classification layer specific to zombies).
  • For the parts of RetinaNet that you want to reuse, you will also restore the weights from the checkpoint that you selected.

Inspect the detection_model

First, take a look at the type of the detection_model and its Python class.

# Run this to check the type of detection_model
detection_model

<object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch at 0x7aa858af26e0>

Find the source code for detection_model

You’ll see that the type of the model is object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch. Please practice some detective work and open up the source code for this class in GitHub repository. Recall that at the start of this assignment, you cloned from this repository: TensorFlow Models.

• Navigate through these subfolders: models -> research -> object_detection.
  • If you get stuck, go to this link: object_detection
• Take a look at this ‘object_detection’ folder and look for the remaining folders to navigate based on the class type of detection_model: object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch
  • Hopefully you’ll find the meta_architectures folder, and within it you’ll notice a file named ssd_meta_arch.py.
  • Please open and view this ssd_meta_arch.py file.

View the variables in detection_model

Now, check the class variables that are in detection_model.

vars(detection_model)

{'_self_setattr_tracking': True,
 '_obj_reference_counts_dict': ObjectIdentityDictionary({<_ObjectIdentityWrapper wrapping False>: 4, <_ObjectIdentityWrapper wrapping 1>: 1, <_ObjectIdentityWrapper wrapping DictWrapper({})>: 1, <_ObjectIdentityWrapper wrapping True>: 7, <_ObjectIdentityWrapper wrapping <object_detection.anchor_generators.multiscale_grid_anchor_generator.MultiscaleGridAnchorGenerator object at 0x7aa856fae9e0>>: 1, <_ObjectIdentityWrapper wrapping <object_detection.predictors.convolutional_keras_box_predictor.WeightSharedConvolutionalBoxPredictor object at 0x7aa858af1ae0>>: 1, <_ObjectIdentityWrapper wrapping <object_detection.box_coders.faster_rcnn_box_coder.FasterRcnnBoxCoder object at 0x7aa858682230>>: 1, <_ObjectIdentityWrapper wrapping <object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor object at 0x7aa858682650>>: 1, <_ObjectIdentityWrapper wrapping 'ResNet50V1_FPN'>: 1, <_ObjectIdentityWrapper wrapping <tf.Tensor: shape=(2,), dtype=float32, numpy=array([0., 0.], dtype=float32)>>: 1, <_ObjectIdentityWrapper wrapping <object_detection.core.target_assigner.TargetAssigner object at 0x7aa858af2650>>: 1, <_ObjectIdentityWrapper wrapping <object_detection.core.losses.SigmoidFocalClassificationLoss object at 0x7aa858af2590>>: 1, <_ObjectIdentityWrapper wrapping <object_detection.core.losses.WeightedSmoothL1LocalizationLoss object at 0x7aa858af25f0>>: 1, <_ObjectIdentityWrapper wrapping 1.0>: 1, <_ObjectIdentityWrapper wrapping 1.0>: 1, <_ObjectIdentityWrapper wrapping 16>: 1, <_ObjectIdentityWrapper wrapping functools.partial(<function resize_image at 0x7aa859bdda20>, new_height=640, new_width=640, method=0)>: 1, <_ObjectIdentityWrapper wrapping functools.partial(<function batch_multiclass_non_max_suppression at 0x7aa8599aed40>, score_thresh=9.99999993922529e-09, iou_thresh=0.6000000238418579, max_size_per_class=100, max_total_size=100, use_static_shapes=False, use_class_agnostic_nms=False, max_classes_per_detection=1, soft_nms_sigma=0.0, use_partitioned_nms=False, use_combined_nms=False, change_coordinate_frame=True, use_hard_nms=False, use_cpu_nms=False)>: 1, <_ObjectIdentityWrapper wrapping <function _score_converter_fn_with_logit_scale.<locals>.score_converter_fn at 0x7aa858696b00>>: 1, <_ObjectIdentityWrapper wrapping ListWrapper([])>: 1, <_ObjectIdentityWrapper wrapping 1.0>: 1, <_ObjectIdentityWrapper wrapping EqualizationLossConfig(weight=0.0, exclude_prefixes=[])>: 1}),
 '_auto_get_config': False,
 '_num_classes': 1,
 '_self_unconditional_checkpoint_dependencies': [TrackableReference(name=_groundtruth_lists, ref={}),
  TrackableReference(name=_box_predictor, ref=<object_detection.predictors.convolutional_keras_box_predictor.WeightSharedConvolutionalBoxPredictor object at 0x7aa858af1ae0>),
  TrackableReference(name=_feature_extractor, ref=<object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor object at 0x7aa858682650>),
  TrackableReference(name=_batched_prediction_tensor_names, ref=ListWrapper([]))],
 '_self_unconditional_dependency_names': {'_groundtruth_lists': {},
  '_box_predictor': <object_detection.predictors.convolutional_keras_box_predictor.WeightSharedConvolutionalBoxPredictor at 0x7aa858af1ae0>,
  '_feature_extractor': <object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor at 0x7aa858682650>,
  '_batched_prediction_tensor_names': ListWrapper([])},
 '_self_unconditional_deferred_dependencies': {},
 '_self_update_uid': -1,
 '_self_name_based_restores': set(),
 '_self_saveable_object_factories': {},
 '_self_tracked_trackables': [{},
  <object_detection.predictors.convolutional_keras_box_predictor.WeightSharedConvolutionalBoxPredictor at 0x7aa858af1ae0>,
  <object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor at 0x7aa858682650>,
  ListWrapper([])],
 '_groundtruth_lists': {},
 '_training_step': None,
 '_instrumented_keras_api': True,
 '_instrumented_keras_layer_class': True,
 '_instrumented_keras_model_class': False,
 '_trainable': True,
 '_stateful': False,
 'built': False,
 '_input_spec': None,
 '_build_input_shape': None,
 '_saved_model_inputs_spec': None,
 '_saved_model_arg_spec': None,
 '_supports_masking': False,
 '_name': 'ssd_meta_arch',
 '_activity_regularizer': None,
 '_trainable_weights': [],
 '_non_trainable_weights': [],
 '_updates': [],
 '_thread_local': <_thread._local at 0x7aa857d0c9a0>,
 '_callable_losses': [],
 '_losses': [],
 '_metrics': [],
 '_metrics_lock': <unlocked _thread.lock object at 0x7aa856f236c0>,
 '_dtype_policy': <Policy "float32">,
 '_compute_dtype_object': tf.float32,
 '_autocast': True,
 '_inbound_nodes_value': [],
 '_outbound_nodes_value': [],
 '_call_spec': <keras.src.utils.layer_utils.CallFunctionSpec at 0x7aa858af27a0>,
 '_dynamic': False,
 '_initial_weights': None,
 '_auto_track_sub_layers': True,
 '_preserve_input_structure_in_config': False,
 '_name_scope_on_declaration': '',
 '_captured_weight_regularizer': [],
 '_is_training': True,
 '_freeze_batchnorm': True,
 '_inplace_batchnorm_update': True,
 '_anchor_generator': <object_detection.anchor_generators.multiscale_grid_anchor_generator.MultiscaleGridAnchorGenerator at 0x7aa856fae9e0>,
 '_box_predictor': <object_detection.predictors.convolutional_keras_box_predictor.WeightSharedConvolutionalBoxPredictor at 0x7aa858af1ae0>,
 '_box_coder': <object_detection.box_coders.faster_rcnn_box_coder.FasterRcnnBoxCoder at 0x7aa858682230>,
 '_feature_extractor': <object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor at 0x7aa858682650>,
 '_add_background_class': True,
 '_explicit_background_class': False,
 '_extract_features_scope': 'ResNet50V1_FPN',
 '_unmatched_class_label': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([0., 0.], dtype=float32)>,
 '_target_assigner': <object_detection.core.target_assigner.TargetAssigner at 0x7aa858af2650>,
 '_classification_loss': <object_detection.core.losses.SigmoidFocalClassificationLoss at 0x7aa858af2590>,
 '_localization_loss': <object_detection.core.losses.WeightedSmoothL1LocalizationLoss at 0x7aa858af25f0>,
 '_classification_loss_weight': 1.0,
 '_localization_loss_weight': 1.0,
 '_normalize_loss_by_num_matches': True,
 '_normalize_loc_loss_by_codesize': True,
 '_hard_example_miner': None,
 '_random_example_sampler': None,
 '_parallel_iterations': 16,
 '_image_resizer_fn': functools.partial(<function resize_image at 0x7aa859bdda20>, new_height=640, new_width=640, method=0),
 '_non_max_suppression_fn': functools.partial(<function batch_multiclass_non_max_suppression at 0x7aa8599aed40>, score_thresh=9.99999993922529e-09, iou_thresh=0.6000000238418579, max_size_per_class=100, max_total_size=100, use_static_shapes=False, use_class_agnostic_nms=False, max_classes_per_detection=1, soft_nms_sigma=0.0, use_partitioned_nms=False, use_combined_nms=False, change_coordinate_frame=True, use_hard_nms=False, use_cpu_nms=False),
 '_score_conversion_fn': <function object_detection.builders.post_processing_builder._score_converter_fn_with_logit_scale.<locals>.score_converter_fn(logits)>,
 '_anchors': None,
 '_add_summaries': True,
 '_batched_prediction_tensor_names': ListWrapper([]),
 '_expected_loss_weights_fn': None,
 '_use_confidences_as_targets': False,
 '_implicit_example_weight': 1.0,
 '_equalization_loss_config': EqualizationLossConfig(weight=0.0, exclude_prefixes=[]),
 '_return_raw_detections_during_predict': False}

You’ll see that detection_model contains several variables:

Two of these will be relevant to you:

...
_box_predictor': <object_detection.predictors.convolutional_keras_box_predictor.WeightSharedConvolutionalBoxPredictor at 0x7f5205eeb1d0>,
...
_feature_extractor': <object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor at 0x7f52040f1ef0>,

Inspect _feature_extractor

Take a look at the ssd_meta_arch.py code.

# Line 302
feature_extractor: a SSDFeatureExtractor object.

Also

# Line 380
self._feature_extractor = feature_extractor

So detection_model._feature_extractor is a feature extractor, which you will want to reuse for your zombie detector model.

Inspect _box_predictor

• View the ssd_meta_arch.py file (which is the source code for detection_model)
• Notice that in the init constructor for class SSDMetaArch(model.DetectionModel),

...
box_predictor: a box_predictor.BoxPredictor object
...
self._box_predictor = box_predictor

Inspect _box_predictor

Please take a look at the class type of detection_model._box_predictor

# view the type of _box_predictor
detection_model._box_predictor

<object_detection.predictors.convolutional_keras_box_predictor.WeightSharedConvolutionalBoxPredictor at 0x7aa858af1ae0>

You’ll see that the class type of _box_predictor is

object_detection.predictors.convolutional_keras_box_predictor.WeightSharedConvolutionalBoxPredictor

You can navigate through the GitHub repository to this path:

• objection_detection/predictors
• Notice that there is a file named convolutional_keras_box_predictor.py. Please open that file.

View variables in _box_predictor

Also view the variables contained in _box_predictor:

vars(detection_model._box_predictor)

{'_self_setattr_tracking': True,
 '_obj_reference_counts_dict': ObjectIdentityDictionary({<_ObjectIdentityWrapper wrapping False>: 5, <_ObjectIdentityWrapper wrapping True>: 3, <_ObjectIdentityWrapper wrapping 1>: 1, <_ObjectIdentityWrapper wrapping <object_detection.predictors.heads.keras_box_head.WeightSharedConvolutionalBoxHead object at 0x7aa856f93fd0>>: 1, <_ObjectIdentityWrapper wrapping DictWrapper({'class_predictions_with_background': <object_detection.predictors.heads.keras_class_head.WeightSharedConvolutionalClassHead object at 0x7aa856f72b00>})>: 1, <_ObjectIdentityWrapper wrapping ListWrapper(['class_predictions_with_background'])>: 1, <_ObjectIdentityWrapper wrapping <object_detection.builders.hyperparams_builder.KerasLayerHyperparams object at 0x7aa856faf310>>: 1, <_ObjectIdentityWrapper wrapping 256>: 1, <_ObjectIdentityWrapper wrapping 4>: 1, <_ObjectIdentityWrapper wrapping 3>: 1, <_ObjectIdentityWrapper wrapping ListWrapper([])>: 1, <_ObjectIdentityWrapper wrapping DictWrapper({'box_encodings': ListWrapper([]), 'class_predictions_with_background': ListWrapper([])})>: 1, <_ObjectIdentityWrapper wrapping DictWrapper({})>: 1}),
 '_auto_get_config': False,
 '_instrumented_keras_api': True,
 '_instrumented_keras_layer_class': True,
 '_instrumented_keras_model_class': False,
 '_trainable': True,
 '_stateful': False,
 'built': False,
 '_input_spec': None,
 '_build_input_shape': None,
 '_saved_model_inputs_spec': None,
 '_saved_model_arg_spec': None,
 '_supports_masking': False,
 '_name': 'WeightSharedConvolutionalBoxPredictor',
 '_activity_regularizer': None,
 '_trainable_weights': [],
 '_non_trainable_weights': [],
 '_updates': [],
 '_thread_local': <_thread._local at 0x7aa856efe110>,
 '_callable_losses': [],
 '_losses': [],
 '_metrics': [],
 '_metrics_lock': <unlocked _thread.lock object at 0x7aa856f87f40>,
 '_dtype_policy': <Policy "float32">,
 '_compute_dtype_object': tf.float32,
 '_autocast': True,
 '_self_tracked_trackables': [<object_detection.predictors.heads.keras_box_head.WeightSharedConvolutionalBoxHead at 0x7aa856f93fd0>,
  {'class_predictions_with_background': <object_detection.predictors.heads.keras_class_head.WeightSharedConvolutionalClassHead at 0x7aa856f72b00>},
  ListWrapper(['class_predictions_with_background']),
  ListWrapper([]),
  {'box_encodings': ListWrapper([]),
   'class_predictions_with_background': ListWrapper([])},
  {}],
 '_inbound_nodes_value': [],
 '_outbound_nodes_value': [],
 '_call_spec': <keras.src.utils.layer_utils.CallFunctionSpec at 0x7aa858af1c00>,
 '_dynamic': False,
 '_initial_weights': None,
 '_auto_track_sub_layers': True,
 '_preserve_input_structure_in_config': False,
 '_name_scope_on_declaration': '',
 '_captured_weight_regularizer': [],
 '_is_training': True,
 '_num_classes': 1,
 '_freeze_batchnorm': True,
 '_inplace_batchnorm_update': False,
 '_self_unconditional_checkpoint_dependencies': [TrackableReference(name=_box_prediction_head, ref=<object_detection.predictors.heads.keras_box_head.WeightSharedConvolutionalBoxHead object at 0x7aa856f93fd0>),
  TrackableReference(name=_prediction_heads, ref={'class_predictions_with_background': <object_detection.predictors.heads.keras_class_head.WeightSharedConvolutionalClassHead object at 0x7aa856f72b00>}),
  TrackableReference(name=_sorted_head_names, ref=ListWrapper(['class_predictions_with_background'])),
  TrackableReference(name=_additional_projection_layers, ref=ListWrapper([])),
  TrackableReference(name=_base_tower_layers_for_heads, ref={'box_encodings': ListWrapper([]), 'class_predictions_with_background': ListWrapper([])}),
  TrackableReference(name=_head_scope_conv_layers, ref={})],
 '_self_unconditional_dependency_names': {'_box_prediction_head': <object_detection.predictors.heads.keras_box_head.WeightSharedConvolutionalBoxHead at 0x7aa856f93fd0>,
  '_prediction_heads': {'class_predictions_with_background': <object_detection.predictors.heads.keras_class_head.WeightSharedConvolutionalClassHead at 0x7aa856f72b00>},
  '_sorted_head_names': ListWrapper(['class_predictions_with_background']),
  '_additional_projection_layers': ListWrapper([]),
  '_base_tower_layers_for_heads': {'box_encodings': ListWrapper([]),
   'class_predictions_with_background': ListWrapper([])},
  '_head_scope_conv_layers': {}},
 '_self_unconditional_deferred_dependencies': {},
 '_self_update_uid': -1,
 '_self_name_based_restores': set(),
 '_self_saveable_object_factories': {},
 '_box_prediction_head': <object_detection.predictors.heads.keras_box_head.WeightSharedConvolutionalBoxHead at 0x7aa856f93fd0>,
 '_prediction_heads': {'class_predictions_with_background': <object_detection.predictors.heads.keras_class_head.WeightSharedConvolutionalClassHead at 0x7aa856f72b00>},
 '_sorted_head_names': ListWrapper(['class_predictions_with_background']),
 '_conv_hyperparams': <object_detection.builders.hyperparams_builder.KerasLayerHyperparams at 0x7aa856faf310>,
 '_depth': 256,
 '_num_layers_before_predictor': 4,
 '_kernel_size': 3,
 '_apply_batch_norm': True,
 '_share_prediction_tower': False,
 '_use_depthwise': False,
 '_apply_conv_hyperparams_pointwise': False,
 '_additional_projection_layers': ListWrapper([]),
 '_base_tower_layers_for_heads': {'box_encodings': ListWrapper([]),
  'class_predictions_with_background': ListWrapper([])},
 '_head_scope_conv_layers': {}}

Among the variables listed, a few will be relevant to you:

...
_base_tower_layers_for_heads
...
_box_prediction_head
...
_prediction_heads

In the source code for convolutional_keras_box_predictor.py that you just opened, look at the source code to get a sense for what these three variables represent.

Inspect base_tower_layers_for_heads

If you look at the convolutional_keras_box_predictor.py file, you’ll notice this:

# line 302
self._base_tower_layers_for_heads = {
        BOX_ENCODINGS: [],
        CLASS_PREDICTIONS_WITH_BACKGROUND: [],
    }

• base_tower_layers_for_heads is a dictionary with two key-value pairs.
  • BOX_ENCODINGS: points to a list of layers
  • CLASS_PREDICTIONS_WITH_BACKGROUND: points to a list of layers
  • If you scan the code, you’ll see that for both of these, the lists are filled with all layers that appear BEFORE the prediction layer.

# Line 377
# Stack the base_tower_layers in the order of conv_layer, batch_norm_layer
    # and activation_layer
    base_tower_layers = []
    for i in range(self._num_layers_before_predictor):

So detection_model.box_predictor._base_tower_layers_for_heads contains:

• The layers for the prediction before the final bounding box prediction
• The layers for the prediction before the final class prediction.

You will want to use these in your model.

Inspect _box_prediction_head

If you again look at convolutional_keras_box_predictor.py file, you’ll see this

# Line 248
box_prediction_head: The head that predicts the boxes.

So detection_model.box_predictor._box_prediction_head points to the bounding box prediction layer, which you’ll want to use for your model.

Inspect _prediction_heads

If you again look at convolutional_keras_box_predictor.py file, you’ll see this

# Line 121
self._prediction_heads = {
        BOX_ENCODINGS: box_prediction_heads,
        CLASS_PREDICTIONS_WITH_BACKGROUND: class_prediction_heads,
    }

You’ll also see this docstring

# Line 83
class_prediction_heads: A list of heads that predict the classes.

So detection_model.box_predictor._prediction_heads is a dictionary that points to both prediction layers:

• The layer that predicts the bounding boxes
• The layer that predicts the class (category).

Which layers will you reuse?

Remember that you are reusing the model for its feature extraction and bounding box detection.

• You will create your own classification layer and train it on zombie images.
• So you won’t need to reuse the class prediction layer of detection_model.

Define checkpoints for desired layers

You will now isolate the layers of detection_model that you wish to reuse so that you can restore the weights to just those layers.

• First, define checkpoints for the box predictor
• Next, define checkpoints for the model, which will point to this box predictor checkpoint as well as the feature extraction layers.

Please use tf.train.Checkpoint.

As a reminder of how to use tf.train.Checkpoint:

tf.train.Checkpoint(
    **kwargs
)

Pretend that detection_model contains these variables for which you want to restore weights:

• detection_model._ice_cream_sundae
• detection_model._pies._apple_pie
• detection_model._pies._pecan_pie

Notice that the pies are nested within ._pies.

If you just want the ice cream sundae and apple pie variables (and not the pecan pie) then you can do the following:

tmp_pies_checkpoint = tf.train.Checkpoint(
  _apple_pie = detection_model._pies._apple_pie
)

Next, in order to connect these together in a node graph, do this:

tmp_model_checkpoint = tf.train.Checkpoint(
  _pies = tmp_pies_checkpoint,
  _ice_cream_sundae = detection_model._ice_cream_sundae
)

Finally, define a checkpoint that uses the key model and takes in the tmp_model_checkpoint.

checkpoint = tf.train.Checkpoint(
  model = tmp_model_checkpoint
)

You’ll then be ready to restore the weights from the checkpoint that you downloaded.

Try this out step by step!

Exercise 6.1: Define Checkpoints for the box predictor

• Please define box_predictor_checkpoint to be checkpoint for these two layers of the detection_model’s box predictor:
  • The base tower layer (the layers the precede both the class prediction and bounding box prediction layers).
  • The box prediction head (the prediction layer for bounding boxes).
• Note, you won’t include the class prediction layer.
• Important: Be careful to avoid typos in the key names for the checkpoint. For example, if there is a layer called _apple_pies and you accidentally added an extra “t” like this: tmp_pies_checkpoint = tf.train.Checkpoint( _apple_piest = detection_model._box_predictor._apple_pies ) then, when you restore the checkpoint, it will update the variable _apple_piest, instead of _apple_pies like you intended. This will likely make the model train slower in Exercise 10 later.

### START CODE HERE ###

tmp_box_predictor_checkpoint = tf.train.Checkpoint(
    _base_tower_layers_for_heads = detection_model._box_predictor._base_tower_layers_for_heads,
    _box_prediction_head = detection_model._box_predictor._box_prediction_head
)



### END CODE HERE

# Check the datatype of this checkpoint
type(tmp_box_predictor_checkpoint)

# Expected output:
# tensorflow.python.training.tracking.util.Checkpoint

tensorflow.python.checkpoint.checkpoint.Checkpoint

# Check the variables of this checkpoint
vars(tmp_box_predictor_checkpoint)

{'_root': None,
 '_self_setattr_tracking': True,
 '_self_unconditional_checkpoint_dependencies': [TrackableReference(name=_base_tower_layers_for_heads, ref={'box_encodings': ListWrapper([]), 'class_predictions_with_background': ListWrapper([])}),
  TrackableReference(name=_box_prediction_head, ref=<object_detection.predictors.heads.keras_box_head.WeightSharedConvolutionalBoxHead object at 0x7aa856f93fd0>)],
 '_self_unconditional_dependency_names': {'_base_tower_layers_for_heads': {'box_encodings': ListWrapper([]),
   'class_predictions_with_background': ListWrapper([])},
  '_box_prediction_head': <object_detection.predictors.heads.keras_box_head.WeightSharedConvolutionalBoxHead at 0x7aa856f93fd0>},
 '_self_unconditional_deferred_dependencies': {},
 '_self_update_uid': -1,
 '_self_name_based_restores': set(),
 '_self_saveable_object_factories': {},
 '_kwargs': {'_base_tower_layers_for_heads': {'box_encodings': ListWrapper([]),
   'class_predictions_with_background': ListWrapper([])},
  '_box_prediction_head': <object_detection.predictors.heads.keras_box_head.WeightSharedConvolutionalBoxHead at 0x7aa856f93fd0>},
 '_async_checkpointer_impl': None,
 '_checkpoint_options': None,
 '_save_counter': None,
 '_save_assign_op': None,
 '_base_tower_layers_for_heads': {'box_encodings': ListWrapper([]),
  'class_predictions_with_background': ListWrapper([])},
 '_box_prediction_head': <object_detection.predictors.heads.keras_box_head.WeightSharedConvolutionalBoxHead at 0x7aa856f93fd0>,
 '_saver': <tensorflow.python.checkpoint.checkpoint.TrackableSaver at 0x7aa858af31c0>,
 '_attached_dependencies': None}

Expected output

You should expect to see a list of variables that include the following:

 ...
 '_base_tower_layers_for_heads': {'box_encodings': ListWrapper([]),
  'class_predictions_with_background': ListWrapper([])},
 '_box_prediction_head': <object_detection.predictors.heads.keras_box_head.WeightSharedConvolutionalBoxHead at 0x7f49d0234450>,
 ...

Exercise 6.2: Define the temporary model checkpoint**

Now define tmp_model_checkpoint so that it points to these two layers:

• The feature extractor of the detection model.
• The temporary box predictor checkpoint that you just defined.

### START CODE HERE ###

tmp_model_checkpoint = tf.train.Checkpoint(
    _box_predictor = tmp_box_predictor_checkpoint,
    _feature_extractor = detection_model._feature_extractor
)



### END CODE HERE ###

# Check the datatype of this checkpoint
type(tmp_model_checkpoint)

# Expected output
# tensorflow.python.training.tracking.util.Checkpoint

tensorflow.python.checkpoint.checkpoint.Checkpoint

# Check the vars of this checkpoint
vars(tmp_model_checkpoint)

{'_root': None,
 '_self_setattr_tracking': True,
 '_self_unconditional_checkpoint_dependencies': [TrackableReference(name=_box_predictor, ref=<tensorflow.python.checkpoint.checkpoint.Checkpoint object at 0x7aa858af3490>),
  TrackableReference(name=_feature_extractor, ref=<object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor object at 0x7aa858682650>)],
 '_self_unconditional_dependency_names': {'_box_predictor': <tensorflow.python.checkpoint.checkpoint.Checkpoint at 0x7aa858af3490>,
  '_feature_extractor': <object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor at 0x7aa858682650>},
 '_self_unconditional_deferred_dependencies': {},
 '_self_update_uid': -1,
 '_self_name_based_restores': set(),
 '_self_saveable_object_factories': {},
 '_kwargs': {'_box_predictor': <tensorflow.python.checkpoint.checkpoint.Checkpoint at 0x7aa858af3490>,
  '_feature_extractor': <object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor at 0x7aa858682650>},
 '_async_checkpointer_impl': None,
 '_checkpoint_options': None,
 '_save_counter': None,
 '_save_assign_op': None,
 '_box_predictor': <tensorflow.python.checkpoint.checkpoint.Checkpoint at 0x7aa858af3490>,
 '_feature_extractor': <object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor at 0x7aa858682650>,
 '_saver': <tensorflow.python.checkpoint.checkpoint.TrackableSaver at 0x7aa8e2cbbb20>,
 '_attached_dependencies': None}

Expected output

Among the variables of this checkpoint, you should see:

'_box_predictor': <tensorflow.python.training.tracking.util.Checkpoint at 0x7fefac044a20>,
 '_feature_extractor': <object_detection.models.ssd_resnet_v1_fpn_keras_feature_extractor.SSDResNet50V1FpnKerasFeatureExtractor at 0x7fefac0240b8>,

Exercise 6.3: Restore the checkpoint

You can now restore the checkpoint.

First, find and set the checkpoint_path

• checkpoint_path:
  • Using the “files” browser in the left side of Colab, navigate to models -> research -> object_detection -> test_data.
  • If you completed the previous code cell that downloads and moves the checkpoint, you’ll see a subfolder named “checkpoint”.
    • The ‘checkpoint’ folder contains three files:
      • checkpoint
      • ckpt-0.data-00000-of-00001
      • ckpt-0.index
    • Please set checkpoint_path to the path to the full path models/.../ckpt-0
      • Notice that you don’t want to include a file extension after ckpt-0.
    • IMPORTANT: Please don’t set the path to include the .index extension in the checkpoint file name.
      • If you do set it to ckpt-0.index, there won’t be any immediate error message, but later during training, you’ll notice that your model’s loss doesn’t improve, which means that the pre-trained weights were not restored properly.

Next, define one last checkpoint using tf.train.Checkpoint().

• For the single keyword argument,
  • Set the key as model=
  • Set the value to your temporary model checkpoint that you just defined.
• IMPORTANT: You’ll need to set the keyword argument as model= and not something else like detection_model=.
• If you set this keyword argument to anything else, it won’t show an immmediate error, but when you train your model on the zombie images, your model loss will not decrease (your model will not learn).

Finally, call this checkpoint’s .restore() function, passing in the path to the checkpoint.

### START CODE HERE ###

checkpoint_path = "/content/models/research/object_detection/test_data/checkpoint/ckpt-0"

# Define a checkpoint that sets `model` to the temporary model checkpoint
checkpoint = tf.train.Checkpoint(
    model=tmp_model_checkpoint
)

# Restore the checkpoint to the checkpoint path
checkpoint.restore(checkpoint_path)

### END CODE HERE ###

<tensorflow.python.checkpoint.checkpoint.CheckpointLoadStatus at 0x7aa8590901f0>

Exercise 7: Run a dummy image to generate the model variables

Run a dummy image through the model so that variables are created. We need to select the trainable variables later in Exercise 9 and right now, it is still empty. Try running len(detection_model.trainable_variables) in a code cell and you will get 0. We will pass in a dummy image through the forward pass to create these variables.

Recall that detection_model is an object of type object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch

Important methods that are available in the detection_model object are:

• preprocess():

  • takes in a tensor representing an image and returns
  • returns image, shapes
  • For the dummy image, you can declare a tensor of zeros that has a shape that the preprocess() method can accept (i.e. [batch, height, width, channels]).
  • Remember that your images have dimensions 640 x 640 x 3.
  • You can pass in a batch of 1 when making the dummy image.

• predict()

  • takes in image, shapes which are created by the preprocess() function call.
  • returns a prediction in a Python dictionary
  • this will pass the dummy image through the forward pass of the network and create the model variables

• postprocess()

  • Takes in the prediction_dict and shapes
  • returns a dictionary of post-processed predictions of detected objects (“detections”).

Note: Please use the recommended variable names, which include the prefix tmp_, since these variables won’t be used later, but you’ll define similarly-named variables later for predicting on actual zombie images.

### START CODE HERE (Replace instances of `None` with your code)###

# use the detection model's `preprocess()` method and pass a dummy image
tmp_image, tmp_shapes = detection_model.preprocess(
    tf.zeros([1, 640, 640, 3])
)

# run a prediction with the preprocessed image and shapes
tmp_prediction_dict = detection_model.predict(tmp_image, tmp_shapes)

# postprocess the predictions into final detections
tmp_detections = detection_model.postprocess(tmp_prediction_dict, tmp_shapes)

### END CODE HERE ###

print('Weights restored!')

Weights restored!

# Test Code:
assert len(detection_model.trainable_variables) > 0, "Please pass in a dummy image to create the trainable variables."

print(detection_model.weights[0].shape)
print(detection_model.weights[231].shape)
print(detection_model.weights[462].shape)

(3, 3, 256, 24)
(512,)
(256,)

Expected Output:

>(3, 3, 256, 24)
(512,)
(256,)

Eager mode custom training loop

With the data and model now setup, you can now proceed to configure the training.

Exercise 8: Set training hyperparameters

Set an appropriate learning rate and optimizer for the training.

• batch_size: you can use 4
  • You can increase the batch size up to 5, since you have just 5 images for training.
• num_batches: You can use 100
  • You can increase the number of batches but the training will take longer to complete.
• learning_rate: You can use 0.01
  • When you run the training loop later, notice how the initial loss INCREASES` before decreasing.
  • You can try a lower learning rate to see if you can avoid this increased loss.
• optimizer: you can use tf.keras.optimizers.SGD
  • Set the learning rate
  • Set the momentum to 0.9

Training will be fairly quick, so we do encourage you to experiment a bit with these hyperparameters!

tf.keras.backend.set_learning_phase(True)

### START CODE HERE (Replace instances of `None` with your code)###

# set the batch_size
batch_size = 4

# set the number of batches
num_batches = 100

# Set the learning rate
learning_rate = .01

# set the optimizer and pass in the learning_rate
optimizer = tf.keras.optimizers.SGD(learning_rate=.001, momentum=.9)

### END CODE HERE ###

/usr/local/lib/python3.10/dist-packages/keras/src/backend.py:452: UserWarning: `tf.keras.backend.set_learning_phase` is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the `training` argument of the `__call__` method of your layer or model.
  warnings.warn(

Choose the layers to fine-tune

To make use of transfer learning and pre-trained weights, you will train just certain parts of the detection model, namely, the last prediction layers.

• Please take a minute to inspect the layers of detection_model.

# Inspect the layers of detection_model
for i,v in enumerate(detection_model.trainable_variables):
    print(f"i: {i} \t name: {v.name} \t shape:{v.shape} \t dtype={v.dtype}")

i: 0 	 name: WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalBoxHead/BoxPredictor/kernel:0 	 shape:(3, 3, 256, 24) 	 dtype=<dtype: 'float32'>
i: 1 	 name: WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalBoxHead/BoxPredictor/bias:0 	 shape:(24,) 	 dtype=<dtype: 'float32'>
i: 2 	 name: WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalClassHead/ClassPredictor/kernel:0 	 shape:(3, 3, 256, 12) 	 dtype=<dtype: 'float32'>
i: 3 	 name: WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalClassHead/ClassPredictor/bias:0 	 shape:(12,) 	 dtype=<dtype: 'float32'>
i: 4 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 5 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/BatchNorm/feature_0/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 6 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/BatchNorm/feature_0/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 7 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 8 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/BatchNorm/feature_0/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 9 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/BatchNorm/feature_0/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 10 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 11 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_0/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 12 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_0/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 13 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 14 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/BatchNorm/feature_0/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 15 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/BatchNorm/feature_0/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 16 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/BatchNorm/feature_1/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 17 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/BatchNorm/feature_1/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 18 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/BatchNorm/feature_1/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 19 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/BatchNorm/feature_1/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 20 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_1/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 21 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_1/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 22 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/BatchNorm/feature_1/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 23 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/BatchNorm/feature_1/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 24 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/BatchNorm/feature_2/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 25 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/BatchNorm/feature_2/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 26 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/BatchNorm/feature_2/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 27 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/BatchNorm/feature_2/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 28 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_2/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 29 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_2/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 30 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/BatchNorm/feature_2/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 31 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/BatchNorm/feature_2/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 32 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/BatchNorm/feature_3/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 33 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/BatchNorm/feature_3/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 34 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/BatchNorm/feature_3/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 35 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/BatchNorm/feature_3/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 36 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_3/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 37 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_3/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 38 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/BatchNorm/feature_3/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 39 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/BatchNorm/feature_3/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 40 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/BatchNorm/feature_4/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 41 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_0/BatchNorm/feature_4/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 42 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/BatchNorm/feature_4/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 43 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_1/BatchNorm/feature_4/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 44 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_4/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 45 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_4/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 46 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/BatchNorm/feature_4/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 47 	 name: WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_3/BatchNorm/feature_4/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 48 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 49 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/BatchNorm/feature_0/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 50 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/BatchNorm/feature_0/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 51 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 52 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/BatchNorm/feature_0/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 53 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/BatchNorm/feature_0/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 54 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 55 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/BatchNorm/feature_0/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 56 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/BatchNorm/feature_0/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 57 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 58 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/BatchNorm/feature_0/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 59 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/BatchNorm/feature_0/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 60 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/BatchNorm/feature_1/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 61 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/BatchNorm/feature_1/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 62 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/BatchNorm/feature_1/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 63 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/BatchNorm/feature_1/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 64 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/BatchNorm/feature_1/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 65 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/BatchNorm/feature_1/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 66 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/BatchNorm/feature_1/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 67 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/BatchNorm/feature_1/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 68 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/BatchNorm/feature_2/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 69 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/BatchNorm/feature_2/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 70 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/BatchNorm/feature_2/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 71 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/BatchNorm/feature_2/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 72 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/BatchNorm/feature_2/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 73 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/BatchNorm/feature_2/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 74 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/BatchNorm/feature_2/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 75 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/BatchNorm/feature_2/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 76 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/BatchNorm/feature_3/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 77 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/BatchNorm/feature_3/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 78 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/BatchNorm/feature_3/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 79 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/BatchNorm/feature_3/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 80 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/BatchNorm/feature_3/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 81 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/BatchNorm/feature_3/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 82 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/BatchNorm/feature_3/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 83 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/BatchNorm/feature_3/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 84 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/BatchNorm/feature_4/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 85 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_0/BatchNorm/feature_4/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 86 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/BatchNorm/feature_4/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 87 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_1/BatchNorm/feature_4/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 88 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/BatchNorm/feature_4/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 89 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_2/BatchNorm/feature_4/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 90 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/BatchNorm/feature_4/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 91 	 name: WeightSharedConvolutionalBoxPredictor/ClassPredictionTower/conv2d_3/BatchNorm/feature_4/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 92 	 name: ResNet50V1_FPN/bottom_up_block5_conv/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 93 	 name: ResNet50V1_FPN/bottom_up_block5_batchnorm/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 94 	 name: ResNet50V1_FPN/bottom_up_block5_batchnorm/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 95 	 name: ResNet50V1_FPN/bottom_up_block6_conv/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 96 	 name: ResNet50V1_FPN/bottom_up_block6_batchnorm/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 97 	 name: ResNet50V1_FPN/bottom_up_block6_batchnorm/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 98 	 name: conv1_conv/kernel:0 	 shape:(7, 7, 3, 64) 	 dtype=<dtype: 'float32'>
i: 99 	 name: conv1_bn/gamma:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 100 	 name: conv1_bn/beta:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 101 	 name: conv2_block1_1_conv/kernel:0 	 shape:(1, 1, 64, 64) 	 dtype=<dtype: 'float32'>
i: 102 	 name: conv2_block1_1_bn/gamma:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 103 	 name: conv2_block1_1_bn/beta:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 104 	 name: conv2_block1_2_conv/kernel:0 	 shape:(3, 3, 64, 64) 	 dtype=<dtype: 'float32'>
i: 105 	 name: conv2_block1_2_bn/gamma:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 106 	 name: conv2_block1_2_bn/beta:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 107 	 name: conv2_block1_0_conv/kernel:0 	 shape:(1, 1, 64, 256) 	 dtype=<dtype: 'float32'>
i: 108 	 name: conv2_block1_3_conv/kernel:0 	 shape:(1, 1, 64, 256) 	 dtype=<dtype: 'float32'>
i: 109 	 name: conv2_block1_0_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 110 	 name: conv2_block1_0_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 111 	 name: conv2_block1_3_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 112 	 name: conv2_block1_3_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 113 	 name: conv2_block2_1_conv/kernel:0 	 shape:(1, 1, 256, 64) 	 dtype=<dtype: 'float32'>
i: 114 	 name: conv2_block2_1_bn/gamma:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 115 	 name: conv2_block2_1_bn/beta:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 116 	 name: conv2_block2_2_conv/kernel:0 	 shape:(3, 3, 64, 64) 	 dtype=<dtype: 'float32'>
i: 117 	 name: conv2_block2_2_bn/gamma:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 118 	 name: conv2_block2_2_bn/beta:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 119 	 name: conv2_block2_3_conv/kernel:0 	 shape:(1, 1, 64, 256) 	 dtype=<dtype: 'float32'>
i: 120 	 name: conv2_block2_3_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 121 	 name: conv2_block2_3_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 122 	 name: conv2_block3_1_conv/kernel:0 	 shape:(1, 1, 256, 64) 	 dtype=<dtype: 'float32'>
i: 123 	 name: conv2_block3_1_bn/gamma:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 124 	 name: conv2_block3_1_bn/beta:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 125 	 name: conv2_block3_2_conv/kernel:0 	 shape:(3, 3, 64, 64) 	 dtype=<dtype: 'float32'>
i: 126 	 name: conv2_block3_2_bn/gamma:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 127 	 name: conv2_block3_2_bn/beta:0 	 shape:(64,) 	 dtype=<dtype: 'float32'>
i: 128 	 name: conv2_block3_3_conv/kernel:0 	 shape:(1, 1, 64, 256) 	 dtype=<dtype: 'float32'>
i: 129 	 name: conv2_block3_3_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 130 	 name: conv2_block3_3_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 131 	 name: conv3_block1_1_conv/kernel:0 	 shape:(1, 1, 256, 128) 	 dtype=<dtype: 'float32'>
i: 132 	 name: conv3_block1_1_bn/gamma:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 133 	 name: conv3_block1_1_bn/beta:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 134 	 name: conv3_block1_2_conv/kernel:0 	 shape:(3, 3, 128, 128) 	 dtype=<dtype: 'float32'>
i: 135 	 name: conv3_block1_2_bn/gamma:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 136 	 name: conv3_block1_2_bn/beta:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 137 	 name: conv3_block1_0_conv/kernel:0 	 shape:(1, 1, 256, 512) 	 dtype=<dtype: 'float32'>
i: 138 	 name: conv3_block1_3_conv/kernel:0 	 shape:(1, 1, 128, 512) 	 dtype=<dtype: 'float32'>
i: 139 	 name: conv3_block1_0_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 140 	 name: conv3_block1_0_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 141 	 name: conv3_block1_3_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 142 	 name: conv3_block1_3_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 143 	 name: conv3_block2_1_conv/kernel:0 	 shape:(1, 1, 512, 128) 	 dtype=<dtype: 'float32'>
i: 144 	 name: conv3_block2_1_bn/gamma:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 145 	 name: conv3_block2_1_bn/beta:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 146 	 name: conv3_block2_2_conv/kernel:0 	 shape:(3, 3, 128, 128) 	 dtype=<dtype: 'float32'>
i: 147 	 name: conv3_block2_2_bn/gamma:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 148 	 name: conv3_block2_2_bn/beta:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 149 	 name: conv3_block2_3_conv/kernel:0 	 shape:(1, 1, 128, 512) 	 dtype=<dtype: 'float32'>
i: 150 	 name: conv3_block2_3_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 151 	 name: conv3_block2_3_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 152 	 name: conv3_block3_1_conv/kernel:0 	 shape:(1, 1, 512, 128) 	 dtype=<dtype: 'float32'>
i: 153 	 name: conv3_block3_1_bn/gamma:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 154 	 name: conv3_block3_1_bn/beta:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 155 	 name: conv3_block3_2_conv/kernel:0 	 shape:(3, 3, 128, 128) 	 dtype=<dtype: 'float32'>
i: 156 	 name: conv3_block3_2_bn/gamma:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 157 	 name: conv3_block3_2_bn/beta:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 158 	 name: conv3_block3_3_conv/kernel:0 	 shape:(1, 1, 128, 512) 	 dtype=<dtype: 'float32'>
i: 159 	 name: conv3_block3_3_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 160 	 name: conv3_block3_3_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 161 	 name: conv3_block4_1_conv/kernel:0 	 shape:(1, 1, 512, 128) 	 dtype=<dtype: 'float32'>
i: 162 	 name: conv3_block4_1_bn/gamma:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 163 	 name: conv3_block4_1_bn/beta:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 164 	 name: conv3_block4_2_conv/kernel:0 	 shape:(3, 3, 128, 128) 	 dtype=<dtype: 'float32'>
i: 165 	 name: conv3_block4_2_bn/gamma:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 166 	 name: conv3_block4_2_bn/beta:0 	 shape:(128,) 	 dtype=<dtype: 'float32'>
i: 167 	 name: conv3_block4_3_conv/kernel:0 	 shape:(1, 1, 128, 512) 	 dtype=<dtype: 'float32'>
i: 168 	 name: conv3_block4_3_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 169 	 name: conv3_block4_3_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 170 	 name: conv4_block1_1_conv/kernel:0 	 shape:(1, 1, 512, 256) 	 dtype=<dtype: 'float32'>
i: 171 	 name: conv4_block1_1_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 172 	 name: conv4_block1_1_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 173 	 name: conv4_block1_2_conv/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 174 	 name: conv4_block1_2_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 175 	 name: conv4_block1_2_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 176 	 name: conv4_block1_0_conv/kernel:0 	 shape:(1, 1, 512, 1024) 	 dtype=<dtype: 'float32'>
i: 177 	 name: conv4_block1_3_conv/kernel:0 	 shape:(1, 1, 256, 1024) 	 dtype=<dtype: 'float32'>
i: 178 	 name: conv4_block1_0_bn/gamma:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 179 	 name: conv4_block1_0_bn/beta:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 180 	 name: conv4_block1_3_bn/gamma:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 181 	 name: conv4_block1_3_bn/beta:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 182 	 name: conv4_block2_1_conv/kernel:0 	 shape:(1, 1, 1024, 256) 	 dtype=<dtype: 'float32'>
i: 183 	 name: conv4_block2_1_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 184 	 name: conv4_block2_1_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 185 	 name: conv4_block2_2_conv/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 186 	 name: conv4_block2_2_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 187 	 name: conv4_block2_2_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 188 	 name: conv4_block2_3_conv/kernel:0 	 shape:(1, 1, 256, 1024) 	 dtype=<dtype: 'float32'>
i: 189 	 name: conv4_block2_3_bn/gamma:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 190 	 name: conv4_block2_3_bn/beta:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 191 	 name: conv4_block3_1_conv/kernel:0 	 shape:(1, 1, 1024, 256) 	 dtype=<dtype: 'float32'>
i: 192 	 name: conv4_block3_1_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 193 	 name: conv4_block3_1_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 194 	 name: conv4_block3_2_conv/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 195 	 name: conv4_block3_2_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 196 	 name: conv4_block3_2_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 197 	 name: conv4_block3_3_conv/kernel:0 	 shape:(1, 1, 256, 1024) 	 dtype=<dtype: 'float32'>
i: 198 	 name: conv4_block3_3_bn/gamma:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 199 	 name: conv4_block3_3_bn/beta:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 200 	 name: conv4_block4_1_conv/kernel:0 	 shape:(1, 1, 1024, 256) 	 dtype=<dtype: 'float32'>
i: 201 	 name: conv4_block4_1_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 202 	 name: conv4_block4_1_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 203 	 name: conv4_block4_2_conv/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 204 	 name: conv4_block4_2_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 205 	 name: conv4_block4_2_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 206 	 name: conv4_block4_3_conv/kernel:0 	 shape:(1, 1, 256, 1024) 	 dtype=<dtype: 'float32'>
i: 207 	 name: conv4_block4_3_bn/gamma:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 208 	 name: conv4_block4_3_bn/beta:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 209 	 name: conv4_block5_1_conv/kernel:0 	 shape:(1, 1, 1024, 256) 	 dtype=<dtype: 'float32'>
i: 210 	 name: conv4_block5_1_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 211 	 name: conv4_block5_1_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 212 	 name: conv4_block5_2_conv/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 213 	 name: conv4_block5_2_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 214 	 name: conv4_block5_2_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 215 	 name: conv4_block5_3_conv/kernel:0 	 shape:(1, 1, 256, 1024) 	 dtype=<dtype: 'float32'>
i: 216 	 name: conv4_block5_3_bn/gamma:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 217 	 name: conv4_block5_3_bn/beta:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 218 	 name: conv4_block6_1_conv/kernel:0 	 shape:(1, 1, 1024, 256) 	 dtype=<dtype: 'float32'>
i: 219 	 name: conv4_block6_1_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 220 	 name: conv4_block6_1_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 221 	 name: conv4_block6_2_conv/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 222 	 name: conv4_block6_2_bn/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 223 	 name: conv4_block6_2_bn/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 224 	 name: conv4_block6_3_conv/kernel:0 	 shape:(1, 1, 256, 1024) 	 dtype=<dtype: 'float32'>
i: 225 	 name: conv4_block6_3_bn/gamma:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 226 	 name: conv4_block6_3_bn/beta:0 	 shape:(1024,) 	 dtype=<dtype: 'float32'>
i: 227 	 name: conv5_block1_1_conv/kernel:0 	 shape:(1, 1, 1024, 512) 	 dtype=<dtype: 'float32'>
i: 228 	 name: conv5_block1_1_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 229 	 name: conv5_block1_1_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 230 	 name: conv5_block1_2_conv/kernel:0 	 shape:(3, 3, 512, 512) 	 dtype=<dtype: 'float32'>
i: 231 	 name: conv5_block1_2_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 232 	 name: conv5_block1_2_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 233 	 name: conv5_block1_0_conv/kernel:0 	 shape:(1, 1, 1024, 2048) 	 dtype=<dtype: 'float32'>
i: 234 	 name: conv5_block1_3_conv/kernel:0 	 shape:(1, 1, 512, 2048) 	 dtype=<dtype: 'float32'>
i: 235 	 name: conv5_block1_0_bn/gamma:0 	 shape:(2048,) 	 dtype=<dtype: 'float32'>
i: 236 	 name: conv5_block1_0_bn/beta:0 	 shape:(2048,) 	 dtype=<dtype: 'float32'>
i: 237 	 name: conv5_block1_3_bn/gamma:0 	 shape:(2048,) 	 dtype=<dtype: 'float32'>
i: 238 	 name: conv5_block1_3_bn/beta:0 	 shape:(2048,) 	 dtype=<dtype: 'float32'>
i: 239 	 name: conv5_block2_1_conv/kernel:0 	 shape:(1, 1, 2048, 512) 	 dtype=<dtype: 'float32'>
i: 240 	 name: conv5_block2_1_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 241 	 name: conv5_block2_1_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 242 	 name: conv5_block2_2_conv/kernel:0 	 shape:(3, 3, 512, 512) 	 dtype=<dtype: 'float32'>
i: 243 	 name: conv5_block2_2_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 244 	 name: conv5_block2_2_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 245 	 name: conv5_block2_3_conv/kernel:0 	 shape:(1, 1, 512, 2048) 	 dtype=<dtype: 'float32'>
i: 246 	 name: conv5_block2_3_bn/gamma:0 	 shape:(2048,) 	 dtype=<dtype: 'float32'>
i: 247 	 name: conv5_block2_3_bn/beta:0 	 shape:(2048,) 	 dtype=<dtype: 'float32'>
i: 248 	 name: conv5_block3_1_conv/kernel:0 	 shape:(1, 1, 2048, 512) 	 dtype=<dtype: 'float32'>
i: 249 	 name: conv5_block3_1_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 250 	 name: conv5_block3_1_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 251 	 name: conv5_block3_2_conv/kernel:0 	 shape:(3, 3, 512, 512) 	 dtype=<dtype: 'float32'>
i: 252 	 name: conv5_block3_2_bn/gamma:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 253 	 name: conv5_block3_2_bn/beta:0 	 shape:(512,) 	 dtype=<dtype: 'float32'>
i: 254 	 name: conv5_block3_3_conv/kernel:0 	 shape:(1, 1, 512, 2048) 	 dtype=<dtype: 'float32'>
i: 255 	 name: conv5_block3_3_bn/gamma:0 	 shape:(2048,) 	 dtype=<dtype: 'float32'>
i: 256 	 name: conv5_block3_3_bn/beta:0 	 shape:(2048,) 	 dtype=<dtype: 'float32'>
i: 257 	 name: ResNet50V1_FPN/FeatureMaps/top_down/projection_3/kernel:0 	 shape:(1, 1, 2048, 256) 	 dtype=<dtype: 'float32'>
i: 258 	 name: ResNet50V1_FPN/FeatureMaps/top_down/projection_3/bias:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 259 	 name: ResNet50V1_FPN/FeatureMaps/top_down/projection_2/kernel:0 	 shape:(1, 1, 1024, 256) 	 dtype=<dtype: 'float32'>
i: 260 	 name: ResNet50V1_FPN/FeatureMaps/top_down/projection_2/bias:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 261 	 name: ResNet50V1_FPN/FeatureMaps/top_down/projection_1/kernel:0 	 shape:(1, 1, 512, 256) 	 dtype=<dtype: 'float32'>
i: 262 	 name: ResNet50V1_FPN/FeatureMaps/top_down/projection_1/bias:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 263 	 name: ResNet50V1_FPN/FeatureMaps/top_down/smoothing_2_conv/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 264 	 name: ResNet50V1_FPN/FeatureMaps/top_down/smoothing_2_batchnorm/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 265 	 name: ResNet50V1_FPN/FeatureMaps/top_down/smoothing_2_batchnorm/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 266 	 name: ResNet50V1_FPN/FeatureMaps/top_down/smoothing_1_conv/kernel:0 	 shape:(3, 3, 256, 256) 	 dtype=<dtype: 'float32'>
i: 267 	 name: ResNet50V1_FPN/FeatureMaps/top_down/smoothing_1_batchnorm/gamma:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>
i: 268 	 name: ResNet50V1_FPN/FeatureMaps/top_down/smoothing_1_batchnorm/beta:0 	 shape:(256,) 	 dtype=<dtype: 'float32'>

Notice that there are some layers whose names are prefixed with the following:

WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalBoxHead
...
WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalClassHead
...
WeightSharedConvolutionalBoxPredictor/BoxPredictionTower
...
WeightSharedConvolutionalBoxPredictor/ClassPredictionTower
...

Among these, which do you think are the prediction layers at the “end” of the model?

• Recall that when inspecting the source code to restore the checkpoints (convolutional_keras_box_predictor.py) you noticed that:
  • _base_tower_layers_for_heads: refers to the layers that are placed right before the prediction layer
  • _box_prediction_head refers to the prediction layer for the bounding boxes
  • _prediction_heads: refers to the set of prediction layers (both for classification and for bounding boxes)

So you can see that in the source code for this model, “tower” refers to layers that are before the prediction layer, and “head” refers to the prediction layers.

Exercise 9: Select the prediction layer variables

Based on inspecting the detection_model.trainable_variables, please select the prediction layer variables that you will fine tune:

• The bounding box head variables (which predict bounding box coordinates)
• The class head variables (which predict the class/category)

You have a few options for doing this:

• You can access them by their list index:

detection_model.trainable_variables[92]

• Alternatively, you can use string matching to select the variables:

tmp_list = []
for v in detection_model.trainable_variables:
  if v.name.startswith('ResNet50V1_FPN/bottom_up_block5'):
    tmp_list.append(v)

Hint: There are a total of four variables that you want to fine tune.

### START CODE HERE (Replace instances of `None` with your code) ###

# define a list that contains the layers that you wish to fine tune
to_fine_tune = []
for v in detection_model.trainable_variables:
    if v.name.startswith("WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutional"):
        to_fine_tune.append(v)



### END CODE HERE

# Test Code:

print(to_fine_tune[0].name)
print(to_fine_tune[2].name)

WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalBoxHead/BoxPredictor/kernel:0
WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalClassHead/ClassPredictor/kernel:0

Expected Output:

>WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalBoxHead/BoxPredictor/kernel:0
WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalClassHead/ClassPredictor/kernel:0

Train your model

You’ll define a function that handles training for one batch, which you’ll later use in your training loop.

First, walk through these code cells to learn how you’ll perform training using this model.

# Get a batch of your training images
g_images_list = train_image_tensors[0:2]

The detection_model is of class SSDMetaArch, and its source code shows that is has this function preprocess.

• This preprocesses the images so that they can be passed into the model (for training or prediction):

  def preprocess(self, inputs):
    """Feature-extractor specific preprocessing.
    ...
    Args:
      inputs: a [batch, height_in, width_in, channels] float tensor representing
        a batch of images with values between 0 and 255.0.
    Returns:
      preprocessed_inputs: a [batch, height_out, width_out, channels] float
        tensor representing a batch of images.
        
      true_image_shapes: int32 tensor of shape [batch, 3] where each row is
        of the form [height, width, channels] indicating the shapes
        of true images in the resized images, as resized images can be padded
        with zeros.

# Use .preprocess to preprocess an image
g_preprocessed_image = detection_model.preprocess(g_images_list[0])
print(f"g_preprocessed_image type: {type(g_preprocessed_image)}")
print(f"g_preprocessed_image length: {len(g_preprocessed_image)}")
print(f"index 0 has the preprocessed image of shape {g_preprocessed_image[0].shape}")
print(f"index 1 has information about the image's true shape excluding padding: {g_preprocessed_image[1]}")

g_preprocessed_image type: <class 'tuple'>
g_preprocessed_image length: 2
index 0 has the preprocessed image of shape (1, 640, 640, 3)
index 1 has information about the image's true shape excluding padding: [[640 640   3]]

You can pre-process each image and save their outputs into two separate lists

• One list of the preprocessed images
• One list of the true shape for each preprocessed image

preprocessed_image_list = []
true_shape_list = []

for img in g_images_list:
    processed_img, true_shape = detection_model.preprocess(img)
    preprocessed_image_list.append(processed_img)
    true_shape_list.append(true_shape)

print(f"preprocessed_image_list is of type {type(preprocessed_image_list)}")
print(f"preprocessed_image_list has length {len(preprocessed_image_list)}")
print()
print(f"true_shape_list is of type {type(true_shape_list)}")
print(f"true_shape_list has length {len(true_shape_list)}")

preprocessed_image_list is of type <class 'list'>
preprocessed_image_list has length 2

true_shape_list is of type <class 'list'>
true_shape_list has length 2

Make a prediction

The detection_model also has a .predict function. According to the source code for predict

  def predict(self, preprocessed_inputs, true_image_shapes):
    """Predicts unpostprocessed tensors from input tensor.
    This function takes an input batch of images and runs it through the forward
    pass of the network to yield unpostprocessesed predictions.
...
    Args:
      preprocessed_inputs: a [batch, height, width, channels] image tensor.
      
      true_image_shapes: int32 tensor of shape [batch, 3] where each row is
        of the form [height, width, channels] indicating the shapes
        of true images in the resized images, as resized images can be padded
        with zeros.
        
    Returns:
      prediction_dict: a dictionary holding "raw" prediction tensors:
        1) preprocessed_inputs: the [batch, height, width, channels] image
          tensor.
        2) box_encodings: 4-D float tensor of shape [batch_size, num_anchors,
          box_code_dimension] containing predicted boxes.
        3) class_predictions_with_background: 3-D float tensor of shape
          [batch_size, num_anchors, num_classes+1] containing class predictions
          (logits) for each of the anchors.  Note that this tensor *includes*
          background class predictions (at class index 0).
        4) feature_maps: a list of tensors where the ith tensor has shape
          [batch, height_i, width_i, depth_i].
        5) anchors: 2-D float tensor of shape [num_anchors, 4] containing
          the generated anchors in normalized coordinates.
        6) final_anchors: 3-D float tensor of shape [batch_size, num_anchors, 4]
          containing the generated anchors in normalized coordinates.
        If self._return_raw_detections_during_predict is True, the dictionary
        will also contain:
        7) raw_detection_boxes: a 4-D float32 tensor with shape
          [batch_size, self.max_num_proposals, 4] in normalized coordinates.
        8) raw_detection_feature_map_indices: a 3-D int32 tensor with shape
          [batch_size, self.max_num_proposals].
    """

Notice that .predict takes its inputs as tensors. If you tried to pass in the preprocessed images and true shapes, you’ll get an error.

# Try to call `predict` and pass in lists; look at the error message
try:
    detection_model.predict(preprocessed_image_list, true_shape_list)
except AttributeError as e:
    print("Error message:", e)

Error message: Exception encountered when calling layer 'ResNet50V1_FPN' (type SSDResNet50V1FpnKerasFeatureExtractor).

'list' object has no attribute 'get_shape'

Call arguments received by layer 'ResNet50V1_FPN' (type SSDResNet50V1FpnKerasFeatureExtractor):
  • inputs=['tf.Tensor(shape=(1, 640, 640, 3), dtype=float32)', 'tf.Tensor(shape=(1, 640, 640, 3), dtype=float32)']
  • kwargs={'training': 'True'}

But don’t worry! You can check how to properly use predict:

• Notice that the source code documentation says that preprocessed_inputs and true_image_shapes are expected to be tensors and not lists of tensors.
• One way to turn a list of tensors into a tensor is to use tf.concat

tf.concat(
    values, axis, name='concat'
)

# Turn a list of tensors into a tensor
preprocessed_image_tensor = tf.concat(preprocessed_image_list, axis=0)
true_shape_tensor = tf.concat(true_shape_list, axis=0)

print(f"preprocessed_image_tensor shape: {preprocessed_image_tensor.shape}")
print(f"true_shape_tensor shape: {true_shape_tensor.shape}")

preprocessed_image_tensor shape: (2, 640, 640, 3)
true_shape_tensor shape: (2, 3)

Now you can make predictions for the images. According to the source code, predict returns a dictionary containing the prediction information, including:

• The bounding box predictions
• The class predictions

# Make predictions on the images
prediction_dict = detection_model.predict(preprocessed_image_tensor, true_shape_tensor)

print("keys in prediction_dict:")
for key in prediction_dict.keys():
    print(key)

keys in prediction_dict:
preprocessed_inputs
feature_maps
anchors
final_anchors
box_encodings
class_predictions_with_background

Calculate loss

Now that your model has made its prediction, you want to compare it to the ground truth in order to calculate a loss.

• The detection_model has a loss function.

>  def loss(self, prediction_dict, true_image_shapes, scope=None):
    """Compute scalar loss tensors with respect to provided groundtruth.
    Calling this function requires that groundtruth tensors have been
    provided via the provide_groundtruth function.
    Args:
      prediction_dict: a dictionary holding prediction tensors with
        1) box_encodings: 3-D float tensor of shape [batch_size, num_anchors,
          box_code_dimension] containing predicted boxes.
        2) class_predictions_with_background: 3-D float tensor of shape
          [batch_size, num_anchors, num_classes+1] containing class predictions
          (logits) for each of the anchors. Note that this tensor *includes*
          background class predictions.
      true_image_shapes: int32 tensor of shape [batch, 3] where each row is
        of the form [height, width, channels] indicating the shapes
        of true images in the resized images, as resized images can be padded
        with zeros.
      scope: Optional scope name.
    Returns:
      a dictionary mapping loss keys (`localization_loss` and
        `classification_loss`) to scalar tensors representing corresponding loss
        values.
    """

It takes in:

• The prediction dictionary that comes from your call to .predict().
• the true images shape that comes from your call to .preprocess() followed by the conversion from a list to a tensor.

Try calling .loss. You’ll see an error message that you’ll addres in order to run the .loss function.

try:
    losses_dict = detection_model.loss(prediction_dict, true_shape_tensor)
except RuntimeError as e:
    print(e)

Groundtruth tensor boxes has not been provided

This is giving an error about groundtruth_classes_list:

The graph tensor has name: groundtruth_classes_list:0

Notice in the docstring for loss (shown above), it says:

Calling this function requires that groundtruth tensors have been
    provided via the provide_groundtruth function.

So you’ll first want to set the ground truth (true labels and true bounding boxes) before you calculate the loss.

• This makes sense, since the loss is comparing the prediction to the ground truth, and so the loss function needs to know the ground truth.

Provide the ground truth

The source code for providing the ground truth is located in the parent class of SSDMetaArch, model.DetectionModel.

• Here is the link to the code for provide_ground_truth

>def provide_groundtruth(
      self,
      groundtruth_boxes_list,
      groundtruth_classes_list,
... # more parameters not show here
"""
    Args:
      groundtruth_boxes_list: a list of 2-D tf.float32 tensors of shape
        [num_boxes, 4] containing coordinates of the groundtruth boxes.
          Groundtruth boxes are provided in [y_min, x_min, y_max, x_max]
          format and assumed to be normalized and clipped
          relative to the image window with y_min <= y_max and x_min <= x_max.
      groundtruth_classes_list: a list of 2-D tf.float32 one-hot (or k-hot)
        tensors of shape [num_boxes, num_classes] containing the class targets
        with the 0th index assumed to map to the first non-background class.
"""

You’ll set two parameters in provide_ground_truth:

• The true bounding boxes
• The true classes

# Get the ground truth bounding boxes
gt_boxes_list = gt_box_tensors[0:2]

# Get the ground truth class labels
gt_classes_list = gt_classes_one_hot_tensors[0:2]

# Provide the ground truth to the model
detection_model.provide_groundtruth(
            groundtruth_boxes_list=gt_boxes_list,
            groundtruth_classes_list=gt_classes_list)

Now you can calculate the loss

# Calculate the loss after you've provided the ground truth
losses_dict = detection_model.loss(prediction_dict, true_shape_tensor)

# View the loss dictionary
losses_dict = detection_model.loss(prediction_dict, true_shape_tensor)
print(f"loss dictionary keys: {losses_dict.keys()}")
print(f"localization loss {losses_dict['Loss/localization_loss']:.8f}")
print(f"classification loss {losses_dict['Loss/classification_loss']:.8f}")

loss dictionary keys: dict_keys(['Loss/localization_loss', 'Loss/classification_loss'])
localization loss 0.08388442
classification loss 1.16436803

You can now calculate the gradient and optimize the variables that you selected to fine tune.

• Use tf.GradientTape

>with tf.GradientTape() as tape:
    # Make the prediction
    
    # calculate the loss
        
    # calculate the gradient of each model variable with respect to each loss
    gradients = tape.gradient([some loss], variables to fine tune)
    
    # apply the gradients to update these model variables
    optimizer.apply_gradients(zip(gradients, variables to fine tune))

# Let's just reset the model so that you can practice setting it up yourself!
detection_model.provide_groundtruth(groundtruth_boxes_list=[], groundtruth_classes_list=[])

Exercise 10: Define the training step

Please complete the function below to set up one training step.

• Preprocess the images
• Make a prediction
• Calculate the loss (and make sure the loss function has the ground truth to compare with the prediction)
• Calculate the total loss:
  • total_loss = localization_loss + classification_loss
  • Note: this is different than the example code that you saw above
• Calculate gradients with respect to the variables you selected to train.
• Optimize the model’s variables

# decorate with @tf.function for faster training (remember, graph mode!)
@tf.function
def train_step_fn(image_list,
                groundtruth_boxes_list,
                groundtruth_classes_list,
                model,
                optimizer,
                vars_to_fine_tune):
    """A single training iteration.

    Args:
      image_list: A list of [1, height, width, 3] Tensor of type tf.float32.
        Note that the height and width can vary across images, as they are
        reshaped within this function to be 640x640.
      groundtruth_boxes_list: A list of Tensors of shape [N_i, 4] with type
        tf.float32 representing groundtruth boxes for each image in the batch.
      groundtruth_classes_list: A list of Tensors of shape [N_i, num_classes]
        with type tf.float32 representing groundtruth boxes for each image in
        the batch.

    Returns:
      A scalar tensor representing the total loss for the input batch.
    """

    model.provide_groundtruth(
        groundtruth_boxes_list=groundtruth_boxes_list,
        groundtruth_classes_list=groundtruth_classes_list)


    with tf.GradientTape() as tape:
    ### START CODE HERE (Replace instances of `None` with your code) ###

        # Preprocess the images
        preprocessed_image_list = []
        true_shape_list = []

        for img in image_list:
            processed_img, true_shape = detection_model.preprocess(img)
            preprocessed_image_list.append(processed_img)
            true_shape_list.append(true_shape)

        preprocessed_image_tensor = tf.concat(preprocessed_image_list, axis=0)
        true_shape_tensor = tf.concat(true_shape_list, axis=0)

        # Make a prediction
        prediction_dict = model.predict(preprocessed_image_tensor, true_shape_tensor)

        # Calculate the total loss (sum of both losses)

        loss_dict = model.loss(prediction_dict, true_shape_tensor)

        total_loss = loss_dict["Loss/localization_loss"] + loss_dict["Loss/classification_loss"]

        # Calculate the gradients
        gradients = tape.gradient([total_loss], vars_to_fine_tune)

        # Optimize the model's selected variables
        optimizer.apply_gradients(zip(gradients, vars_to_fine_tune))

        ### END CODE HERE ###

    return total_loss

Run the training loop

Run the training loop using the training step function that you just defined.

print('Start fine-tuning!', flush=True)

for idx in range(num_batches):
    # Grab keys for a random subset of examples
    all_keys = list(range(len(train_images_np)))
    random.shuffle(all_keys)
    example_keys = all_keys[:batch_size]

    # Get the ground truth
    gt_boxes_list = [gt_box_tensors[key] for key in example_keys]
    gt_classes_list = [gt_classes_one_hot_tensors[key] for key in example_keys]

    # get the images
    image_tensors = [train_image_tensors[key] for key in example_keys]

    # Training step (forward pass + backwards pass)
    total_loss = train_step_fn(image_tensors,
                               gt_boxes_list,
                               gt_classes_list,
                               detection_model,
                               optimizer,
                               to_fine_tune
                              )

    if idx % 10 == 0:
        print('batch ' + str(idx) + ' of ' + str(num_batches)
        + ', loss=' +  str(total_loss.numpy()), flush=True)

print('Done fine-tuning!')

Start fine-tuning!
batch 0 of 100, loss=1.2259816
batch 10 of 100, loss=0.29560208
batch 20 of 100, loss=0.101838745
batch 30 of 100, loss=0.050350092
batch 40 of 100, loss=0.03305868
batch 50 of 100, loss=0.023922784
batch 60 of 100, loss=0.020205157
batch 70 of 100, loss=0.020926852
batch 80 of 100, loss=0.01674586
batch 90 of 100, loss=0.014692143
Done fine-tuning!

Expected Output:

Total loss should be decreasing and should be less than 1 after fine tuning. For example:

>Start fine-tuning!
batch 0 of 100, loss=1.2559178
batch 10 of 100, loss=16.067217
batch 20 of 100, loss=8.094654
batch 30 of 100, loss=0.34514275
batch 40 of 100, loss=0.033170983
batch 50 of 100, loss=0.0024622646
batch 60 of 100, loss=0.00074224477
batch 70 of 100, loss=0.0006149876
batch 80 of 100, loss=0.00046916265
batch 90 of 100, loss=0.0004159231
Done fine-tuning!

Load test images and run inference with new model!

You can now test your model on a new set of images. The cell below downloads 237 images of a walking zombie and stores them in a results/ directory.

# uncomment if you want to delete existing files
!rm zombie-walk-frames.zip
!rm -rf ./zombie-walk
!rm -rf ./results

# download test images
!wget --no-check-certificate \
    https://storage.googleapis.com/tensorflow-3-public/datasets/zombie-walk-frames.zip \
    -O zombie-walk-frames.zip

# unzip test images
local_zip = './zombie-walk-frames.zip'
zip_ref = zipfile.ZipFile(local_zip, 'r')
zip_ref.extractall('./results')
zip_ref.close()

rm: cannot remove 'zombie-walk-frames.zip': No such file or directory
--2023-10-04 10:52:34--  https://storage.googleapis.com/tensorflow-3-public/datasets/zombie-walk-frames.zip
Resolving storage.googleapis.com (storage.googleapis.com)... 74.125.197.207, 74.125.135.207, 74.125.142.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|74.125.197.207|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 94778747 (90M) [application/zip]
Saving to: ‘zombie-walk-frames.zip’

zombie-walk-frames. 100%[===================>]  90.39M   144MB/s    in 0.6s    

2023-10-04 10:52:35 (144 MB/s) - ‘zombie-walk-frames.zip’ saved [94778747/94778747]

You will load these images into numpy arrays to prepare it for inference.

test_image_dir = './results/'
test_images_np = []

# load images into a numpy array. this will take a few minutes to complete.
for i in range(0, 237):
    image_path = os.path.join(test_image_dir, 'zombie-walk' + "{0:04}".format(i) + '.jpg')
    print(image_path)
    test_images_np.append(np.expand_dims(
      load_image_into_numpy_array(image_path), axis=0))

./results/zombie-walk0227.jpg
./results/zombie-walk0228.jpg
./results/zombie-walk0229.jpg
./results/zombie-walk0230.jpg
./results/zombie-walk0231.jpg
./results/zombie-walk0232.jpg
./results/zombie-walk0233.jpg
./results/zombie-walk0234.jpg
./results/zombie-walk0235.jpg
./results/zombie-walk0236.jpg

Exercise 11: Preprocess, predict, and post process an image

Define a function that returns the detection boxes, classes, and scores.

# Again, uncomment this decorator if you want to run inference eagerly
@tf.function
def detect(input_tensor):
    """Run detection on an input image.

    Args:
    input_tensor: A [1, height, width, 3] Tensor of type tf.float32.
      Note that height and width can be anything since the image will be
      immediately resized according to the needs of the model within this
      function.

    Returns:
    A dict containing 3 Tensors (`detection_boxes`, `detection_classes`,
      and `detection_scores`).
    """
    preprocessed_image, shapes = detection_model.preprocess(input_tensor)
    prediction_dict = detection_model.predict(preprocessed_image, shapes)

    ### START CODE HERE (Replace instances of `None` with your code) ###
    # use the detection model's postprocess() method to get the the final detections
    detections = detection_model.postprocess(prediction_dict, shapes)
    ### END CODE HERE ###

    return detections

You can now loop through the test images and get the detection scores and bounding boxes to overlay in the original image. We will save each result in a results dictionary and the autograder will use this to evaluate your results.

# Note that the first frame will trigger tracing of the tf.function, which will
# take some time, after which inference should be fast.

label_id_offset = 1
results = {'boxes': [], 'scores': []}

for i in range(len(test_images_np)):
    input_tensor = tf.convert_to_tensor(test_images_np[i], dtype=tf.float32)
    detections = detect(input_tensor)
    plot_detections(
      test_images_np[i][0],
      detections['detection_boxes'][0].numpy(),
      detections['detection_classes'][0].numpy().astype(np.uint32)
      + label_id_offset,
      detections['detection_scores'][0].numpy(),
      category_index, figsize=(15, 20), image_name="./results/gif_frame_" + ('%03d' % i) + ".jpg")
    results['boxes'].append(detections['detection_boxes'][0][0].numpy())
    results['scores'].append(detections['detection_scores'][0][0].numpy())

# TEST CODE

print(len(results['boxes']))
print(results['boxes'][0].shape)
print()

# compare with expected bounding boxes
print(np.allclose(results['boxes'][0], [0.28838485, 0.06830047, 0.7213766 , 0.19833465], rtol=0.18))
print(np.allclose(results['boxes'][5], [0.29168868, 0.07529271, 0.72504973, 0.20099735], rtol=0.18))
print(np.allclose(results['boxes'][10], [0.29548776, 0.07994056, 0.7238164 , 0.20778716], rtol=0.18))

237
(4,)

True
True
True

Expected Output: Ideally the three boolean values at the bottom should be True. But if you only get two, you can still try submitting. This compares your resulting bounding boxes for each zombie image to some preloaded coordinates (i.e. the hardcoded values in the test cell above). Depending on how you annotated the training images,it’s possible that some of your results differ for these three frames but still get good results overall when all images are examined by the grader. If two or all are False, please try annotating the images again with a tighter bounding box or use the predefined gt_boxes list.

>237
(4,)

True
True
True

You can also check if the model detects a zombie class in the images by examining the scores key of the results dictionary. You should get higher than 88.0 here.

x = np.array(results['scores'])

# percent of frames where a zombie is detected
zombie_detected = (np.where(x > 0.9, 1, 0).sum())/237*100
print(zombie_detected)

42.19409282700422

You can also display some still frames and inspect visually. If you don’t see a bounding box around the zombie, please consider re-annotating the ground truth or use the predefined gt_boxes here

print('Frame 0')
display(IPyImage('./results/gif_frame_000.jpg'))
print()
print('Frame 5')
display(IPyImage('./results/gif_frame_005.jpg'))
print()
print('Frame 10')
display(IPyImage('./results/gif_frame_010.jpg'))

Frame 0

Frame 5

Frame 10

Create a zip of the zombie-walk images.

You can download this if you like to create your own animations

zipf = zipfile.ZipFile('./zombie.zip', 'w', zipfile.ZIP_DEFLATED)

filenames = glob.glob('./results/gif_frame_*.jpg')
filenames = sorted(filenames)

for filename in filenames:
    zipf.write(filename)

zipf.close()

Create Zombie animation

imageio.plugins.freeimage.download()

!rm -rf ./results/zombie-anim.gif

anim_file = './zombie-anim.gif'

filenames = glob.glob('./results/gif_frame_*.jpg')
filenames = sorted(filenames)
last = -1
images = []

for filename in filenames:
    image = imageio.imread(filename)
    images.append(image)

imageio.mimsave(anim_file, images, 'GIF-FI', fps=10)

Imageio: 'libfreeimage-3.16.0-linux64.so' was not found on your computer; downloading it now.
Try 1. Download from https://github.com/imageio/imageio-binaries/raw/master/freeimage/libfreeimage-3.16.0-linux64.so (4.6 MB)
Downloading: 8192/4830080 bytes (0.2%)4830080/4830080 bytes (100.0%)
  Done
File saved as /root/.imageio/freeimage/libfreeimage-3.16.0-linux64.so.


<ipython-input-63-c631f4650915>:13: DeprecationWarning: Starting with ImageIO v3 the behavior of this function will switch to that of iio.v3.imread. To keep the current behavior (and make this warning disappear) use `import imageio.v2 as imageio` or call `imageio.v2.imread` directly.
  image = imageio.imread(filename)

Unfortunately, using IPyImage in the notebook (as you’ve done in the rubber ducky detection tutorial) for the large gif generated will disconnect the runtime. To view the animation, you can instead use the Files pane on the left and double-click on zombie-anim.gif. That will open a preview page on the right. It will take 2 to 3 minutes to load and see the walking zombie.

Save results file for grading

Run the cell below to save your results. Download the results.data file and upload it to the grader in the classroom.

import pickle

# remove file if it exists
!rm results.data

# write results to binary file. upload for grading.
with open('results.data', 'wb') as filehandle:
    pickle.dump(results['boxes'], filehandle)

print('Done saving! Please download `results.data` from the Files tab\n' \
      'on the left and submit for grading.\nYou can also use the next cell as a shortcut for downloading.')

rm: cannot remove 'results.data': No such file or directory
Done saving! Please download `results.data` from the Files tab
on the left and submit for grading.
You can also use the next cell as a shortcut for downloading.

from google.colab import files

files.download('results.data')

<IPython.core.display.Javascript object>



<IPython.core.display.Javascript object>

Congratulations on completing this assignment! Please go back to the Coursera classroom and upload results.data to the Graded Lab item for Week 2.