Coursera

Ungraded Lab: Model Analysis with TFX Evaluator

Now that you’ve used TFMA as a standalone library in the previous lab, you will now see how it is used by TFX with its Evaluator component. This component comes after your Trainer run and it checks if your trained model meets the minimum required metrics and also compares it with previously generated models.

You will go through a TFX pipeline that prepares and trains the same model architecture you used in the previous lab. As a reminder, this is a binary classifier to be trained on the Census Income dataset. Since you’re already familiar with the earlier TFX components, we will just go over them quickly but we’ve placed notes on where you can modify code if you want to practice or produce a better result.

Let’s begin!

Credits: Some of the code and discussions are based on the TensorFlow team’s official tutorial.

Setup

Imports

import os
import pprint

import tensorflow as tf
import tensorflow_model_analysis as tfma

from tfx import v1 as tfx
from absl import logging as absl_logging
from apache_beam import logging as beam_logging

from tfx.orchestration.experimental.interactive.interactive_context import InteractiveContext

tf.get_logger().propagate = False
tf.get_logger().setLevel('ERROR')
absl_logging.set_verbosity('ERROR')
beam_logging.getLogger().setLevel('ERROR')
pp = pprint.PrettyPrinter()

Set up pipeline paths

# Location of the pipeline metadata store
_pipeline_root = './pipeline/'

# Directory of the raw data files
_data_root = './data/census'

_data_filepath = os.path.join(_data_root, "data.csv")

# Create the TFX pipeline files directory (ignore if it exists already)
!mkdir {_pipeline_root}

Preview the dataset

You will use the training split of the Census Income Dataset. This is twice as large as the test dataset you used in the previous lab. Take a quick look at the first few rows.

# Preview dataset
!head {_data_filepath}

age,workclass,fnlwgt,education,education-num,marital-status,occupation,relationship,race,sex,capital-gain,capital-loss,hours-per-week,native-country,label
39,State-gov,77516,Bachelors,13,Never-married,Adm-clerical,Not-in-family,White,Male,2174,0,40,United-States,0
50,Self-emp-not-inc,83311,Bachelors,13,Married-civ-spouse,Exec-managerial,Husband,White,Male,0,0,13,United-States,0
38,Private,215646,HS-grad,9,Divorced,Handlers-cleaners,Not-in-family,White,Male,0,0,40,United-States,0
53,Private,234721,11th,7,Married-civ-spouse,Handlers-cleaners,Husband,Black,Male,0,0,40,United-States,0
28,Private,338409,Bachelors,13,Married-civ-spouse,Prof-specialty,Wife,Black,Female,0,0,40,Cuba,0
37,Private,284582,Masters,14,Married-civ-spouse,Exec-managerial,Wife,White,Female,0,0,40,United-States,0
49,Private,160187,9th,5,Married-spouse-absent,Other-service,Not-in-family,Black,Female,0,0,16,Jamaica,0
52,Self-emp-not-inc,209642,HS-grad,9,Married-civ-spouse,Exec-managerial,Husband,White,Male,0,0,45,United-States,1
31,Private,45781,Masters,14,Never-married,Prof-specialty,Not-in-family,White,Female,14084,0,50,United-States,1

TFX Pipeline

Create the InteractiveContext

As usual, you will initialize the pipeline and use a local SQLite file for the metadata store.

# Initialize InteractiveContext
context = InteractiveContext(pipeline_root=_pipeline_root)

ExampleGen

You will start by ingesting the data through CsvExampleGen. The code below uses the default 2:1 train-eval split (i.e. 33% of the data goes to eval) but feel free to modify if you want. You can review splitting techniques here.

# NOTE: Uncomment and run this if you get an error saying there are different
# headers in the dataset. This is usually because of the notebook checkpoints saved in
# that folder.
!rm -rf ./data/.ipynb_checkpoints
!rm -rf ./data/census/.ipynb_checkpoints

# Run CsvExampleGen
example_gen = tfx.components.CsvExampleGen(input_base=_data_root)
context.run(example_gen)

ExecutionResult at 0x7f299812d890

.execution_id

.component

CsvExampleGen at 0x7f287b0d2d50

.inputs

{}

.outputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f299812d490

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/CsvExampleGen/examples/1) at 0x7f2998168150

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/CsvExampleGen/examples/1
.span	0
.split_names	["train", "eval"]
.version	0

.exec_properties

['input_base']	./data/census
['input_config']	{ "splits": [ { "name": "single_split", "pattern": "*" } ] }
['output_config']	{ "split_config": { "splits": [ { "hash_buckets": 2, "name": "train" }, { "hash_buckets": 1, "name": "eval" } ] } }
['output_data_format']	6
['output_file_format']	5
['custom_config']	None
['range_config']	None
['span']	0
['version']	None
['input_fingerprint']	split:single_split,num_files:1,total_bytes:3396202,xor_checksum:1686600210,sum_checksum:1686600210

.component.inputs

{}

.component.outputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f299812d490

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/CsvExampleGen/examples/1) at 0x7f2998168150

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/CsvExampleGen/examples/1
.span	0
.split_names	["train", "eval"]
.version	0

# Print split names and URI
artifact = example_gen.outputs['examples'].get()[0]
print(artifact.split_names, artifact.uri)

["train", "eval"] ./pipeline/CsvExampleGen/examples/1

StatisticsGen

You will then compute the statistics so it can be used by the next components.

# Run StatisticsGen
statistics_gen = tfx.components.StatisticsGen(
    examples=example_gen.outputs['examples'])
context.run(statistics_gen)

ExecutionResult at 0x7f287b0d2d10

.execution_id

.component

StatisticsGen at 0x7f287b0c4350

.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f299812d490

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/CsvExampleGen/examples/1) at 0x7f2998168150

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/CsvExampleGen/examples/1
.span	0
.split_names	["train", "eval"]
.version	0

.outputs

['statistics']

Channel of type 'ExampleStatistics' (1 artifact) at 0x7f287b0c4dd0

.type_name

ExampleStatistics

._artifacts

[0]

Artifact of type 'ExampleStatistics' (uri: ./pipeline/StatisticsGen/statistics/2) at 0x7f29415b3790

.type	<class 'tfx.types.standard_artifacts.ExampleStatistics'>
.uri	./pipeline/StatisticsGen/statistics/2
.span	0
.split_names	["train", "eval"]

.exec_properties

['stats_options_json']	None
['exclude_splits']	[]

.component.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f299812d490

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/CsvExampleGen/examples/1) at 0x7f2998168150

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/CsvExampleGen/examples/1
.span	0
.split_names	["train", "eval"]
.version	0

.component.outputs

['statistics']

Channel of type 'ExampleStatistics' (1 artifact) at 0x7f287b0c4dd0

.type_name

ExampleStatistics

._artifacts

[0]

Artifact of type 'ExampleStatistics' (uri: ./pipeline/StatisticsGen/statistics/2) at 0x7f29415b3790

.type	<class 'tfx.types.standard_artifacts.ExampleStatistics'>
.uri	./pipeline/StatisticsGen/statistics/2
.span	0
.split_names	["train", "eval"]

You can look at the visualizations below if you want to explore the data some more.

# Visualize statistics
context.show(statistics_gen.outputs['statistics'])

Artifact at ./pipeline/StatisticsGen/statistics/2

'train' split:

'eval' split:

SchemaGen

You can then infer the dataset schema with SchemaGen. This will be used to validate incoming data to ensure that it is formatted correctly.

# Run SchemaGen
schema_gen = tfx.components.SchemaGen(
    statistics=statistics_gen.outputs['statistics'])
context.run(schema_gen)

ExecutionResult at 0x7f287b0d2510

.execution_id

.component

SchemaGen at 0x7f293e7f5990

.inputs

['statistics']

Channel of type 'ExampleStatistics' (1 artifact) at 0x7f287b0c4dd0

.type_name

ExampleStatistics

._artifacts

[0]

Artifact of type 'ExampleStatistics' (uri: ./pipeline/StatisticsGen/statistics/2) at 0x7f29415b3790

.type	<class 'tfx.types.standard_artifacts.ExampleStatistics'>
.uri	./pipeline/StatisticsGen/statistics/2
.span	0
.split_names	["train", "eval"]

.outputs

['schema']

Channel of type 'Schema' (1 artifact) at 0x7f293e7f50d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/SchemaGen/schema/3) at 0x7f293e7d9290

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/SchemaGen/schema/3

.exec_properties

['infer_feature_shape']	1
['exclude_splits']	[]

.component.inputs

['statistics']

Channel of type 'ExampleStatistics' (1 artifact) at 0x7f287b0c4dd0

.type_name

ExampleStatistics

._artifacts

[0]

Artifact of type 'ExampleStatistics' (uri: ./pipeline/StatisticsGen/statistics/2) at 0x7f29415b3790

.type	<class 'tfx.types.standard_artifacts.ExampleStatistics'>
.uri	./pipeline/StatisticsGen/statistics/2
.span	0
.split_names	["train", "eval"]

.component.outputs

['schema']

Channel of type 'Schema' (1 artifact) at 0x7f293e7f50d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/SchemaGen/schema/3) at 0x7f293e7d9290

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/SchemaGen/schema/3

For simplicity, you will just accept the inferred schema but feel free to modify with the TFDV API if you want.

# Visualize the inferred Schema
context.show(schema_gen.outputs['schema'])

Artifact at ./pipeline/SchemaGen/schema/3

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	Type	Presence	Domain
Feature name
'age'	INT	required	-
'capital-gain'	INT	required	-
'capital-loss'	INT	required	-
'education'	STRING	required	'education'
'education-num'	INT	required	-
'fnlwgt'	INT	required	-
'hours-per-week'	INT	required	-
'label'	INT	required	-
'marital-status'	STRING	required	'marital-status'
'native-country'	STRING	required	'native-country'
'occupation'	STRING	required	'occupation'
'race'	STRING	required	'race'
'relationship'	STRING	required	'relationship'
'sex'	STRING	required	'sex'
'workclass'	STRING	required	'workclass'

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	Values
Domain
'education'	'10th', '11th', '12th', '1st-4th', '5th-6th', '7th-8th', '9th', 'Assoc-acdm', 'Assoc-voc', 'Bachelors', 'Doctorate', 'HS-grad', 'Masters', 'Preschool', 'Prof-school', 'Some-college'
'marital-status'	'Divorced', 'Married-AF-spouse', 'Married-civ-spouse', 'Married-spouse-absent', 'Never-married', 'Separated', 'Widowed'
'native-country'	'?', 'Cambodia', 'Canada', 'China', 'Columbia', 'Cuba', 'Dominican-Republic', 'Ecuador', 'El-Salvador', 'England', 'France', 'Germany', 'Greece', 'Guatemala', 'Haiti', 'Holand-Netherlands', 'Honduras', 'Hong', 'Hungary', 'India', 'Iran', 'Ireland', 'Italy', 'Jamaica', 'Japan', 'Laos', 'Mexico', 'Nicaragua', 'Outlying-US(Guam-USVI-etc)', 'Peru', 'Philippines', 'Poland', 'Portugal', 'Puerto-Rico', 'Scotland', 'South', 'Taiwan', 'Thailand', 'Trinadad&Tobago', 'United-States', 'Vietnam', 'Yugoslavia'
'occupation'	'?', 'Adm-clerical', 'Armed-Forces', 'Craft-repair', 'Exec-managerial', 'Farming-fishing', 'Handlers-cleaners', 'Machine-op-inspct', 'Other-service', 'Priv-house-serv', 'Prof-specialty', 'Protective-serv', 'Sales', 'Tech-support', 'Transport-moving'
'race'	'Amer-Indian-Eskimo', 'Asian-Pac-Islander', 'Black', 'Other', 'White'
'relationship'	'Husband', 'Not-in-family', 'Other-relative', 'Own-child', 'Unmarried', 'Wife'
'sex'	'Female', 'Male'
'workclass'	'?', 'Federal-gov', 'Local-gov', 'Never-worked', 'Private', 'Self-emp-inc', 'Self-emp-not-inc', 'State-gov', 'Without-pay'

ExampleValidator

Next, run ExampleValidator to check if there are anomalies in the data.

# Run ExampleValidator
example_validator = tfx.components.ExampleValidator(
    statistics=statistics_gen.outputs['statistics'],
    schema=schema_gen.outputs['schema'])
context.run(example_validator)

ExecutionResult at 0x7f2940aeb7d0

.execution_id

.component

ExampleValidator at 0x7f293e7da0d0

.inputs

['statistics']

Channel of type 'ExampleStatistics' (1 artifact) at 0x7f287b0c4dd0

.type_name

ExampleStatistics

._artifacts

[0]

Artifact of type 'ExampleStatistics' (uri: ./pipeline/StatisticsGen/statistics/2) at 0x7f29415b3790

.type	<class 'tfx.types.standard_artifacts.ExampleStatistics'>
.uri	./pipeline/StatisticsGen/statistics/2
.span	0
.split_names	["train", "eval"]

['schema']

Channel of type 'Schema' (1 artifact) at 0x7f293e7f50d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/SchemaGen/schema/3) at 0x7f293e7d9290

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/SchemaGen/schema/3

.outputs

['anomalies']

Channel of type 'ExampleAnomalies' (1 artifact) at 0x7f293e7dab50

.type_name

ExampleAnomalies

._artifacts

[0]

Artifact of type 'ExampleAnomalies' (uri: ./pipeline/ExampleValidator/anomalies/4) at 0x7f293e7dad10

.type	<class 'tfx.types.standard_artifacts.ExampleAnomalies'>
.uri	./pipeline/ExampleValidator/anomalies/4
.span	0
.split_names	["train", "eval"]

.exec_properties

['exclude_splits']	[]
['custom_validation_config']	None

.component.inputs

['statistics']

Channel of type 'ExampleStatistics' (1 artifact) at 0x7f287b0c4dd0

.type_name

ExampleStatistics

._artifacts

[0]

Artifact of type 'ExampleStatistics' (uri: ./pipeline/StatisticsGen/statistics/2) at 0x7f29415b3790

.type	<class 'tfx.types.standard_artifacts.ExampleStatistics'>
.uri	./pipeline/StatisticsGen/statistics/2
.span	0
.split_names	["train", "eval"]

['schema']

Channel of type 'Schema' (1 artifact) at 0x7f293e7f50d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/SchemaGen/schema/3) at 0x7f293e7d9290

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/SchemaGen/schema/3

.component.outputs

['anomalies']

Channel of type 'ExampleAnomalies' (1 artifact) at 0x7f293e7dab50

.type_name

ExampleAnomalies

._artifacts

[0]

Artifact of type 'ExampleAnomalies' (uri: ./pipeline/ExampleValidator/anomalies/4) at 0x7f293e7dad10

.type	<class 'tfx.types.standard_artifacts.ExampleAnomalies'>
.uri	./pipeline/ExampleValidator/anomalies/4
.span	0
.split_names	["train", "eval"]

If you just used the inferred schema, then there should not be any anomalies detected. If you modified the schema, then there might be some results here and you can again use TFDV to modify or relax constraints as needed.

In actual deployments, this component will also help you understand how your data evolves over time and identify data errors. For example, the first batches of data that you get from your users might conform to the schema but it might not be the case after several months. This component will detect that and let you know that your model might need to be updated.

# Check results
context.show(example_validator.outputs['anomalies'])

Artifact at ./pipeline/ExampleValidator/anomalies/4

'train' split:

No anomalies found.

'eval' split:

No anomalies found.

Transform

Now you will perform feature engineering on the training data. As shown when you previewed the CSV earlier, the data is still in raw format and cannot be consumed by the model just yet. The transform code in the following cells will take care of scaling your numeric features and one-hot encoding your categorical variables.

Note: If you’re running this exercise for the first time, we advise that you leave the transformation code as is. After you’ve gone through the entire notebook, then you can modify these for practice and see what results you get. Just make sure that your model builder code in the Trainer component will also reflect those changes if needed. For example, removing a feature here should also remove an input layer for that feature in the model.

# Set the constants module filename
_census_constants_module_file = 'census_constants.py'

%%writefile {_census_constants_module_file}

# Features with string data types that will be converted to indices
VOCAB_FEATURE_DICT = {
    'education': 16, 'marital-status': 7, 'occupation': 15, 'race': 5, 
    'relationship': 6, 'workclass': 9, 'sex': 2, 'native-country': 42
}

# Numerical features that are marked as continuous
NUMERIC_FEATURE_KEYS = ['fnlwgt', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']

# Feature that can be grouped into buckets
BUCKET_FEATURE_DICT = {'age': 4}

# Number of out-of-vocabulary buckets
NUM_OOV_BUCKETS = 5

# Feature that the model will predict
LABEL_KEY = 'label'

Writing census_constants.py

# Set the transform module filename
_census_transform_module_file = 'census_transform.py'

%%writefile {_census_transform_module_file}

import tensorflow as tf
import tensorflow_transform as tft

# import constants from cells above
import census_constants

# Unpack the contents of the constants module
_NUMERIC_FEATURE_KEYS = census_constants.NUMERIC_FEATURE_KEYS
_VOCAB_FEATURE_DICT = census_constants.VOCAB_FEATURE_DICT
_BUCKET_FEATURE_DICT = census_constants.BUCKET_FEATURE_DICT
_NUM_OOV_BUCKETS = census_constants.NUM_OOV_BUCKETS
_LABEL_KEY = census_constants.LABEL_KEY

# Define the transformations
def preprocessing_fn(inputs):
    """tf.transform's callback function for preprocessing inputs.
    Args:
        inputs: map from feature keys to raw not-yet-transformed features.
    Returns:
        Map from string feature key to transformed feature operations.
    """

    # Initialize outputs dictionary
    outputs = {}

    # Scale these features to the range [0,1]
    for key in _NUMERIC_FEATURE_KEYS:
        scaled = tft.scale_to_0_1(inputs[key])
        outputs[key] = tf.reshape(scaled, [-1])

    # Convert strings to indices and convert to one-hot vectors
    for key, vocab_size in _VOCAB_FEATURE_DICT.items():
        indices = tft.compute_and_apply_vocabulary(inputs[key], num_oov_buckets=_NUM_OOV_BUCKETS)
        one_hot = tf.one_hot(indices, vocab_size + _NUM_OOV_BUCKETS)
        outputs[key] = tf.reshape(one_hot, [-1, vocab_size + _NUM_OOV_BUCKETS])

    # Bucketize this feature and convert to one-hot vectors
    for key, num_buckets in _BUCKET_FEATURE_DICT.items():
        indices = tft.bucketize(inputs[key], num_buckets)
        one_hot = tf.one_hot(indices, num_buckets)
        outputs[key] = tf.reshape(one_hot, [-1, num_buckets])

    # Cast label to float
    outputs[_LABEL_KEY] = tf.cast(inputs[_LABEL_KEY], tf.float32)

    return outputs

Writing census_transform.py

Now, we pass in this feature engineering code to the Transform component and run it to transform your data.

# Run the Transform component
transform = tfx.components.Transform(
    examples=example_gen.outputs['examples'],
    schema=schema_gen.outputs['schema'],
    module_file=os.path.abspath(_census_transform_module_file))
context.run(transform, enable_cache=False)

ExecutionResult at 0x7f2940da5c90

.execution_id

.component

Transform at 0x7f2940ac7110

.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f299812d490

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/CsvExampleGen/examples/1) at 0x7f2998168150

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/CsvExampleGen/examples/1
.span	0
.split_names	["train", "eval"]
.version	0

['schema']

Channel of type 'Schema' (1 artifact) at 0x7f293e7f50d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/SchemaGen/schema/3) at 0x7f293e7d9290

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/SchemaGen/schema/3

.outputs

['transform_graph']

Channel of type 'TransformGraph' (1 artifact) at 0x7f2940acf690

.type_name

TransformGraph

._artifacts

[0]

Artifact of type 'TransformGraph' (uri: ./pipeline/Transform/transform_graph/5) at 0x7f2940acf890

.type	<class 'tfx.types.standard_artifacts.TransformGraph'>
.uri	./pipeline/Transform/transform_graph/5

['transformed_examples']

Channel of type 'Examples' (1 artifact) at 0x7f2940acf550

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/Transform/transformed_examples/5) at 0x7f2940ac2d50

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/Transform/transformed_examples/5
.span	0
.split_names	["train", "eval"]
.version	0

['updated_analyzer_cache']

Channel of type 'TransformCache' (1 artifact) at 0x7f2940acfd90

.type_name

TransformCache

._artifacts

[0]

Artifact of type 'TransformCache' (uri: ./pipeline/Transform/updated_analyzer_cache/5) at 0x7f2940acf910

.type	<class 'tfx.types.standard_artifacts.TransformCache'>
.uri	./pipeline/Transform/updated_analyzer_cache/5

['pre_transform_schema']

Channel of type 'Schema' (1 artifact) at 0x7f2940acf1d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/Transform/pre_transform_schema/5) at 0x7f2940acffd0

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/Transform/pre_transform_schema/5

['pre_transform_stats']

Channel of type 'ExampleStatistics' (1 artifact) at 0x7f2940acf650

.type_name

ExampleStatistics

._artifacts

[0]

Artifact of type 'ExampleStatistics' (uri: ./pipeline/Transform/pre_transform_stats/5) at 0x7f2940acfad0

.type	<class 'tfx.types.standard_artifacts.ExampleStatistics'>
.uri	./pipeline/Transform/pre_transform_stats/5
.span	0
.split_names

['post_transform_schema']

Channel of type 'Schema' (1 artifact) at 0x7f2940acfc10

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/Transform/post_transform_schema/5) at 0x7f293e7c43d0

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/Transform/post_transform_schema/5

['post_transform_stats']

Channel of type 'ExampleStatistics' (1 artifact) at 0x7f2940acf790

.type_name

ExampleStatistics

._artifacts

[0]

Artifact of type 'ExampleStatistics' (uri: ./pipeline/Transform/post_transform_stats/5) at 0x7f293e7c4550

.type	<class 'tfx.types.standard_artifacts.ExampleStatistics'>
.uri	./pipeline/Transform/post_transform_stats/5
.span	0
.split_names

['post_transform_anomalies']

Channel of type 'ExampleAnomalies' (1 artifact) at 0x7f2940acf110

.type_name

ExampleAnomalies

._artifacts

[0]

Artifact of type 'ExampleAnomalies' (uri: ./pipeline/Transform/post_transform_anomalies/5) at 0x7f293e7c4dd0

.type	<class 'tfx.types.standard_artifacts.ExampleAnomalies'>
.uri	./pipeline/Transform/post_transform_anomalies/5
.span	0
.split_names

.exec_properties

['module_file']	None
['preprocessing_fn']	None
['stats_options_updater_fn']	None
['force_tf_compat_v1']	0
['custom_config']	null
['splits_config']	None
['disable_statistics']	0
['module_path']	census_transform@./pipeline/_wheels/tfx_user_code_Transform-0.0+b4cdffd259c88473f714b412cc62df1dc81614e72429a198acc51340f3a1f977-py3-none-any.whl

.component.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f299812d490

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/CsvExampleGen/examples/1) at 0x7f2998168150

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/CsvExampleGen/examples/1
.span	0
.split_names	["train", "eval"]
.version	0

['schema']

Channel of type 'Schema' (1 artifact) at 0x7f293e7f50d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/SchemaGen/schema/3) at 0x7f293e7d9290

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/SchemaGen/schema/3

.component.outputs

['transform_graph']

Channel of type 'TransformGraph' (1 artifact) at 0x7f2940acf690

.type_name

TransformGraph

._artifacts

[0]

Artifact of type 'TransformGraph' (uri: ./pipeline/Transform/transform_graph/5) at 0x7f2940acf890

.type	<class 'tfx.types.standard_artifacts.TransformGraph'>
.uri	./pipeline/Transform/transform_graph/5

['transformed_examples']

Channel of type 'Examples' (1 artifact) at 0x7f2940acf550

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/Transform/transformed_examples/5) at 0x7f2940ac2d50

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/Transform/transformed_examples/5
.span	0
.split_names	["train", "eval"]
.version	0

['updated_analyzer_cache']

Channel of type 'TransformCache' (1 artifact) at 0x7f2940acfd90

.type_name

TransformCache

._artifacts

[0]

Artifact of type 'TransformCache' (uri: ./pipeline/Transform/updated_analyzer_cache/5) at 0x7f2940acf910

.type	<class 'tfx.types.standard_artifacts.TransformCache'>
.uri	./pipeline/Transform/updated_analyzer_cache/5

['pre_transform_schema']

Channel of type 'Schema' (1 artifact) at 0x7f2940acf1d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/Transform/pre_transform_schema/5) at 0x7f2940acffd0

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/Transform/pre_transform_schema/5

['pre_transform_stats']

Channel of type 'ExampleStatistics' (1 artifact) at 0x7f2940acf650

.type_name

ExampleStatistics

._artifacts

[0]

Artifact of type 'ExampleStatistics' (uri: ./pipeline/Transform/pre_transform_stats/5) at 0x7f2940acfad0

.type	<class 'tfx.types.standard_artifacts.ExampleStatistics'>
.uri	./pipeline/Transform/pre_transform_stats/5
.span	0
.split_names

['post_transform_schema']

Channel of type 'Schema' (1 artifact) at 0x7f2940acfc10

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/Transform/post_transform_schema/5) at 0x7f293e7c43d0

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/Transform/post_transform_schema/5

['post_transform_stats']

Channel of type 'ExampleStatistics' (1 artifact) at 0x7f2940acf790

.type_name

ExampleStatistics

._artifacts

[0]

Artifact of type 'ExampleStatistics' (uri: ./pipeline/Transform/post_transform_stats/5) at 0x7f293e7c4550

.type	<class 'tfx.types.standard_artifacts.ExampleStatistics'>
.uri	./pipeline/Transform/post_transform_stats/5
.span	0
.split_names

['post_transform_anomalies']

Channel of type 'ExampleAnomalies' (1 artifact) at 0x7f2940acf110

.type_name

ExampleAnomalies

._artifacts

[0]

Artifact of type 'ExampleAnomalies' (uri: ./pipeline/Transform/post_transform_anomalies/5) at 0x7f293e7c4dd0

.type	<class 'tfx.types.standard_artifacts.ExampleAnomalies'>
.uri	./pipeline/Transform/post_transform_anomalies/5
.span	0
.split_names

You can see a sample result for one row with the code below. Notice that the numeric features are indeed scaled and the categorical features are now one-hot encoded.

# Get the URI of the output artifact representing the transformed examples
train_uri = os.path.join(transform.outputs['transformed_examples'].get()[0].uri, 'Split-train')

# Get the list of files in this directory (all compressed TFRecord files)
tfrecord_filenames = [os.path.join(train_uri, name)
                      for name in os.listdir(train_uri)]

# Create a `TFRecordDataset` to read these files
dataset = tf.data.TFRecordDataset(tfrecord_filenames, compression_type="GZIP")

# Decode the first record and print output
for tfrecord in dataset.take(1):
  serialized_example = tfrecord.numpy()
  example = tf.train.Example()
  example.ParseFromString(serialized_example)
  pp.pprint(example)

features {
  feature {
    key: "age"
    value {
      float_list {
        value: 0.0
        value: 0.0
        value: 1.0
        value: 0.0
      }
    }
  }
  feature {
    key: "capital-gain"
    value {
      float_list {
        value: 0.02174021676182747
      }
    }
  }
  feature {
    key: "capital-loss"
    value {
      float_list {
        value: 0.0
      }
    }
  }
  feature {
    key: "education"
    value {
      float_list {
        value: 0.0
        value: 0.0
        value: 1.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
      }
    }
  }
  feature {
    key: "education-num"
    value {
      float_list {
        value: 0.800000011920929
      }
    }
  }
  feature {
    key: "fnlwgt"
    value {
      float_list {
        value: 0.044301897287368774
      }
    }
  }
  feature {
    key: "hours-per-week"
    value {
      float_list {
        value: 0.3979591727256775
      }
    }
  }
  feature {
    key: "label"
    value {
      float_list {
        value: 0.0
      }
    }
  }
  feature {
    key: "marital-status"
    value {
      float_list {
        value: 0.0
        value: 1.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
      }
    }
  }
  feature {
    key: "native-country"
    value {
      float_list {
        value: 1.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
      }
    }
  }
  feature {
    key: "occupation"
    value {
      float_list {
        value: 0.0
        value: 0.0
        value: 0.0
        value: 1.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
      }
    }
  }
  feature {
    key: "race"
    value {
      float_list {
        value: 1.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
      }
    }
  }
  feature {
    key: "relationship"
    value {
      float_list {
        value: 0.0
        value: 1.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
      }
    }
  }
  feature {
    key: "sex"
    value {
      float_list {
        value: 1.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
      }
    }
  }
  feature {
    key: "workclass"
    value {
      float_list {
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 1.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
        value: 0.0
      }
    }
  }
}

As you already know, the Transform component not only outputs the transformed examples but also the transformation graph. This should be used on all inputs when your model is deployed to ensure that it is transformed the same way as your training data. Else, it can produce training-serving skew which leads to noisy predictions.

The Transform component stores related files in its transform_graph output and it would be good to quickly review its contents before we move on to the next component. As shown below, the URI of this output points to a directory containing three subdirectories.

# Get URI and list subdirectories
graph_uri = transform.outputs['transform_graph'].get()[0].uri
os.listdir(graph_uri)

['metadata', 'transformed_metadata', 'transform_fn']

The transformed_metadata subdirectory contains the schema of the preprocessed data.
The transform_fn subdirectory contains the actual preprocessing graph.
The metadata subdirectory contains the schema of the original data.

Trainer

Next, you will now build the model to make your predictions. As mentioned earlier, this is a binary classifier where the label is 1 if a person earns more than 50k USD and 0 if less than or equal. The model used here uses the wide and deep model as reference but feel free to modify after you’ve completed the exercise. Also for simplicity, the hyperparameters (e.g. number of hidden units) have been hardcoded but feel free to use a Tuner component as you did in Week 1 if you want to get some practice.

As a reminder, it is best to start from run_fn() when you’re reviewing the module file below. The Trainer component looks for that function first. All other functions defined in the module are just utility functions for run_fn().

Another thing you will notice below is the _get_tf_serving_examples_signature() function. This is tied to the serving_default signature which makes it possible for you to just pass in raw features when the model is served for inference. You have seen this in action in the previous lab. This is done by decorating the enclosing function, serve_tf_examples_fn(), with tf.function. This indicates that the computation will be done by first tracing a Tensorflow graph. You will notice that this function uses model.tft_layer_inference which comes from transform_graph output. Now when you call the .get_concrete_function() method on this tf.function in run_fn(), you are creating the graph that will be used in later computations. This graph is used whenever you pass in an examples argument pointing to a Tensor with tf.string dtype. That matches the format of the serialized batches of data you used in the previous lab.

# Declare trainer module file
_census_trainer_module_file = 'census_trainer.py'

%%writefile {_census_trainer_module_file}

from typing import List, Text

import tensorflow as tf
import tensorflow_transform as tft
from tensorflow_transform.tf_metadata import schema_utils

from tfx.components.trainer.fn_args_utils import DataAccessor, FnArgs
from tfx_bsl.public.tfxio import TensorFlowDatasetOptions

# import same constants from transform module
import census_constants

# Unpack the contents of the constants module
_NUMERIC_FEATURE_KEYS = census_constants.NUMERIC_FEATURE_KEYS
_VOCAB_FEATURE_DICT = census_constants.VOCAB_FEATURE_DICT
_BUCKET_FEATURE_DICT = census_constants.BUCKET_FEATURE_DICT
_NUM_OOV_BUCKETS = census_constants.NUM_OOV_BUCKETS
_LABEL_KEY = census_constants.LABEL_KEY


def _gzip_reader_fn(filenames):
  '''Load compressed dataset
  
  Args:
    filenames - filenames of TFRecords to load

  Returns:
    TFRecordDataset loaded from the filenames
  '''

  # Load the dataset. Specify the compression type since it is saved as `.gz`
  return tf.data.TFRecordDataset(filenames, compression_type='GZIP')


def _input_fn(file_pattern,
              tf_transform_output,
              num_epochs=None,
              batch_size=32) -> tf.data.Dataset:
  '''Create batches of features and labels from TF Records

  Args:
    file_pattern - List of files or patterns of file paths containing Example records.
    tf_transform_output - transform output graph
    num_epochs - Integer specifying the number of times to read through the dataset. 
            If None, cycles through the dataset forever.
    batch_size - An int representing the number of records to combine in a single batch.

  Returns:
    A dataset of dict elements, (or a tuple of dict elements and label). 
    Each dict maps feature keys to Tensor or SparseTensor objects.
  '''

  # Get post-transfrom feature spec
  transformed_feature_spec = (
      tf_transform_output.transformed_feature_spec().copy())
  
  # Create batches of data
  dataset = tf.data.experimental.make_batched_features_dataset(
      file_pattern=file_pattern,
      batch_size=batch_size,
      features=transformed_feature_spec,
      reader=_gzip_reader_fn,
      num_epochs=num_epochs,
      label_key=_LABEL_KEY
      )
  
  return dataset


def _get_tf_examples_serving_signature(model, tf_transform_output):
  """Returns a serving signature that accepts `tensorflow.Example`."""

  # Get the transformation graph
  model.tft_layer_inference = tf_transform_output.transform_features_layer()

  @tf.function(input_signature=[
      tf.TensorSpec(shape=[None], dtype=tf.string, name='examples')
  ])
  def serve_tf_examples_fn(serialized_tf_example):
    """Returns the output to be used in the serving signature."""
    
    # Get pre-transform feature spec
    raw_feature_spec = tf_transform_output.raw_feature_spec()
    
    # Remove label feature since these will not be present at serving time.
    raw_feature_spec.pop(_LABEL_KEY)
    
    # Parse raw examples into a dictionary of tensors matching the feature spec
    raw_features = tf.io.parse_example(serialized_tf_example, raw_feature_spec)
    
    # Transform the raw examples using the transform graph
    transformed_features = model.tft_layer_inference(raw_features)

    # Get predictions using the transformed features
    outputs = model(transformed_features)

    return {'outputs': outputs}

  return serve_tf_examples_fn


def export_serving_model(tf_transform_output, model, output_dir):
  """Exports a keras model for serving.
  Args:
    tf_transform_output: Wrapper around output of tf.Transform.
    model: A keras model to export for serving.
    output_dir: A directory where the model will be exported to.
  """
  # The layer has to be saved to the model for keras tracking purpases.
  model.tft_layer = tf_transform_output.transform_features_layer()

  signatures = {
      'serving_default':
          _get_tf_examples_serving_signature(model, tf_transform_output)
  }

  model.save(output_dir, save_format='tf', signatures=signatures)


def _build_keras_model(hidden_units: List[int] = None) -> tf.keras.Model:
  """Creates a DNN Keras model for classifying income data.

  Args:
    hidden_units: [int], the layer sizes of the DNN (input layer first).

  Returns:
    A keras Model.
  """

  # Use helper function to create the model
  model = _wide_and_deep_classifier(
      dnn_hidden_units=hidden_units or [100, 70, 50, 25])
  
  return model


def _wide_and_deep_classifier(dnn_hidden_units):
  """Build a simple keras wide and deep model using the Functional API.

  Args:
    wide_columns: Feature columns wrapped in indicator_column for wide (linear)
      part of the model.
    deep_columns: Feature columns for deep part of the model.
    dnn_hidden_units: [int], the layer sizes of the hidden DNN.

  Returns:
    A Wide and Deep Keras model
  """

  # Define input layers for numeric keys
  input_numeric = [
      tf.keras.layers.Input(name=colname, shape=(1,), dtype=tf.float32)
      for colname in _NUMERIC_FEATURE_KEYS
  ]

  # Define input layers for vocab keys
  input_categorical = [
      tf.keras.layers.Input(name=colname, shape=(vocab_size + _NUM_OOV_BUCKETS,), dtype=tf.float32)
      for colname, vocab_size in _VOCAB_FEATURE_DICT.items()
  ]

  # Define input layers for bucket key
  input_categorical += [
      tf.keras.layers.Input(name=colname, shape=(num_buckets,), dtype=tf.float32)
      for colname, num_buckets in _BUCKET_FEATURE_DICT.items()
  ]

  # Concatenate numeric inputs
  deep = tf.keras.layers.concatenate(input_numeric)

  # Create deep dense network for numeric inputs
  for numnodes in dnn_hidden_units:
    deep = tf.keras.layers.Dense(numnodes)(deep)

  # Concatenate categorical inputs
  wide = tf.keras.layers.concatenate(input_categorical)

  # Create shallow dense network for categorical inputs
  wide = tf.keras.layers.Dense(128, activation='relu')(wide)

  # Combine wide and deep networks
  combined = tf.keras.layers.concatenate([deep, wide])
                                              
  # Define output of binary classifier
  output = tf.keras.layers.Dense(
      1, activation='sigmoid')(combined)

  # Setup combined input
  input_layers = input_numeric + input_categorical

  # Create the Keras model
  model = tf.keras.Model(input_layers, output)

  # Define training parameters
  model.compile(
      loss='binary_crossentropy',
      optimizer=tf.keras.optimizers.Adam(lr=0.001),
      metrics='binary_accuracy')
  
  # Print model summary
  model.summary()

  return model


# TFX Trainer will call this function.
def run_fn(fn_args: FnArgs):
  """Defines and trains the model.
  
  Args:
    fn_args: Holds args as name/value pairs. Refer here for the complete attributes: 
    https://www.tensorflow.org/tfx/api_docs/python/tfx/components/trainer/fn_args_utils/FnArgs#attributes
  """

  # Number of nodes in the first layer of the DNN
  first_dnn_layer_size = 100
  num_dnn_layers = 4
  dnn_decay_factor = 0.7

  # Get transform output (i.e. transform graph) wrapper
  tf_transform_output = tft.TFTransformOutput(fn_args.transform_output)

  # Create batches of train and eval sets
  train_dataset = _input_fn(fn_args.train_files[0], tf_transform_output, 10)
  eval_dataset = _input_fn(fn_args.eval_files[0], tf_transform_output, 10)

  # Build the model
  model = _build_keras_model(
      # Construct layers sizes with exponential decay
      hidden_units=[
          max(2, int(first_dnn_layer_size * dnn_decay_factor**i))
          for i in range(num_dnn_layers)
      ])
  
  # Callback for TensorBoard
  tensorboard_callback = tf.keras.callbacks.TensorBoard(
      log_dir=fn_args.model_run_dir, update_freq='batch')


  # Train the model
  model.fit(
      train_dataset,
      steps_per_epoch=fn_args.train_steps,
      validation_data=eval_dataset,
      validation_steps=fn_args.eval_steps,
      callbacks=[tensorboard_callback])
  

  # Export the model.
  export_serving_model(tf_transform_output, model, fn_args.serving_model_dir)

Writing census_trainer.py

Now, we pass in this model code to the Trainer component and run it to train the model.

trainer = tfx.components.Trainer(
    module_file=os.path.abspath(_census_trainer_module_file),
    examples=transform.outputs['transformed_examples'],
    transform_graph=transform.outputs['transform_graph'],
    schema=schema_gen.outputs['schema'],
    train_args=tfx.proto.TrainArgs(num_steps=50),
    eval_args=tfx.proto.EvalArgs(num_steps=50))
context.run(trainer, enable_cache=False)

Model: "model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 fnlwgt (InputLayer)            [(None, 1)]          0           []                               
                                                                                                  
 education-num (InputLayer)     [(None, 1)]          0           []                               
                                                                                                  
 capital-gain (InputLayer)      [(None, 1)]          0           []                               
                                                                                                  
 capital-loss (InputLayer)      [(None, 1)]          0           []                               
                                                                                                  
 hours-per-week (InputLayer)    [(None, 1)]          0           []                               
                                                                                                  
 concatenate (Concatenate)      (None, 5)            0           ['fnlwgt[0][0]',                 
                                                                  'education-num[0][0]',          
                                                                  'capital-gain[0][0]',           
                                                                  'capital-loss[0][0]',           
                                                                  'hours-per-week[0][0]']         
                                                                                                  
 dense (Dense)                  (None, 100)          600         ['concatenate[0][0]']            
                                                                                                  
 dense_1 (Dense)                (None, 70)           7070        ['dense[0][0]']                  
                                                                                                  
 education (InputLayer)         [(None, 21)]         0           []                               
                                                                                                  
 marital-status (InputLayer)    [(None, 12)]         0           []                               
                                                                                                  
 occupation (InputLayer)        [(None, 20)]         0           []                               
                                                                                                  
 race (InputLayer)              [(None, 10)]         0           []                               
                                                                                                  
 relationship (InputLayer)      [(None, 11)]         0           []                               
                                                                                                  
 workclass (InputLayer)         [(None, 14)]         0           []                               
                                                                                                  
 sex (InputLayer)               [(None, 7)]          0           []                               
                                                                                                  
 native-country (InputLayer)    [(None, 47)]         0           []                               
                                                                                                  
 age (InputLayer)               [(None, 4)]          0           []                               
                                                                                                  
 dense_2 (Dense)                (None, 48)           3408        ['dense_1[0][0]']                
                                                                                                  
 concatenate_1 (Concatenate)    (None, 146)          0           ['education[0][0]',              
                                                                  'marital-status[0][0]',         
                                                                  'occupation[0][0]',             
                                                                  'race[0][0]',                   
                                                                  'relationship[0][0]',           
                                                                  'workclass[0][0]',              
                                                                  'sex[0][0]',                    
                                                                  'native-country[0][0]',         
                                                                  'age[0][0]']                    
                                                                                                  
 dense_3 (Dense)                (None, 34)           1666        ['dense_2[0][0]']                
                                                                                                  
 dense_4 (Dense)                (None, 128)          18816       ['concatenate_1[0][0]']          
                                                                                                  
 concatenate_2 (Concatenate)    (None, 162)          0           ['dense_3[0][0]',                
                                                                  'dense_4[0][0]']                
                                                                                                  
 dense_5 (Dense)                (None, 1)            163         ['concatenate_2[0][0]']          
                                                                                                  
==================================================================================================
Total params: 31,723
Trainable params: 31,723
Non-trainable params: 0
__________________________________________________________________________________________________
50/50 [==============================] - 2s 11ms/step - loss: 0.5378 - binary_accuracy: 0.7406 - val_loss: 0.4277 - val_binary_accuracy: 0.8056

ExecutionResult at 0x7f2765c735d0

.execution_id

.component

Trainer at 0x7f2940ab0ad0

.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f2940acf550

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/Transform/transformed_examples/5) at 0x7f2940ac2d50

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/Transform/transformed_examples/5
.span	0
.split_names	["train", "eval"]
.version	0

['transform_graph']

Channel of type 'TransformGraph' (1 artifact) at 0x7f2940acf690

.type_name

TransformGraph

._artifacts

[0]

Artifact of type 'TransformGraph' (uri: ./pipeline/Transform/transform_graph/5) at 0x7f2940acf890

.type	<class 'tfx.types.standard_artifacts.TransformGraph'>
.uri	./pipeline/Transform/transform_graph/5

['schema']

Channel of type 'Schema' (1 artifact) at 0x7f293e7f50d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/SchemaGen/schema/3) at 0x7f293e7d9290

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/SchemaGen/schema/3

.outputs

['model']

Channel of type 'Model' (1 artifact) at 0x7f2765a9c7d0

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/6) at 0x7f2765ac18d0

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/6

['model_run']

Channel of type 'ModelRun' (1 artifact) at 0x7f2765a9cb10

.type_name

ModelRun

._artifacts

[0]

Artifact of type 'ModelRun' (uri: ./pipeline/Trainer/model_run/6) at 0x7f2940ad4b90

.type	<class 'tfx.types.standard_artifacts.ModelRun'>
.uri	./pipeline/Trainer/model_run/6

.exec_properties

['train_args']	{ "num_steps": 50 }
['eval_args']	{ "num_steps": 50 }
['module_file']	None
['run_fn']	None
['trainer_fn']	None
['custom_config']	null
['module_path']	census_trainer@./pipeline/_wheels/tfx_user_code_Trainer-0.0+48cbd699b9b98d6808f806735a8a67817df89a5fd43fd5a9e489201454f3a0c9-py3-none-any.whl

.component.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f2940acf550

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/Transform/transformed_examples/5) at 0x7f2940ac2d50

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/Transform/transformed_examples/5
.span	0
.split_names	["train", "eval"]
.version	0

['transform_graph']

Channel of type 'TransformGraph' (1 artifact) at 0x7f2940acf690

.type_name

TransformGraph

._artifacts

[0]

Artifact of type 'TransformGraph' (uri: ./pipeline/Transform/transform_graph/5) at 0x7f2940acf890

.type	<class 'tfx.types.standard_artifacts.TransformGraph'>
.uri	./pipeline/Transform/transform_graph/5

['schema']

Channel of type 'Schema' (1 artifact) at 0x7f293e7f50d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/SchemaGen/schema/3) at 0x7f293e7d9290

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/SchemaGen/schema/3

.component.outputs

['model']

Channel of type 'Model' (1 artifact) at 0x7f2765a9c7d0

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/6) at 0x7f2765ac18d0

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/6

['model_run']

Channel of type 'ModelRun' (1 artifact) at 0x7f2765a9cb10

.type_name

ModelRun

._artifacts

[0]

Artifact of type 'ModelRun' (uri: ./pipeline/Trainer/model_run/6) at 0x7f2940ad4b90

.type	<class 'tfx.types.standard_artifacts.ModelRun'>
.uri	./pipeline/Trainer/model_run/6

Let’s review the outputs of this component. The model output points to the model output itself.

# Get `model` output of the component
model_artifact_dir = trainer.outputs['model'].get()[0].uri

# List top-level directory
pp.pprint(os.listdir(model_artifact_dir))

# Get model directory
model_dir = os.path.join(model_artifact_dir, 'Format-Serving')

# List subdirectories
pp.pprint(os.listdir(model_dir))

['Format-Serving']
['fingerprint.pb', 'keras_metadata.pb', 'variables', 'saved_model.pb', 'assets']

The model_run output acts as the working directory and can be used to output non-model related output (e.g., TensorBoard logs). You can see below that it has separate directories for the training and validation output.

# Get `model_run` output URI
model_run_artifact_dir = trainer.outputs['model_run'].get()[0].uri

# # List subdirectories
pp.pprint(os.listdir(model_run_artifact_dir))

['train', 'validation']

Evaluator

The Evaluator component computes model performance metrics over the evaluation set using the TensorFlow Model Analysis library. The Evaluator can also optionally validate that a newly trained model is better than the previous model. This is useful in a production pipeline setting where you may automatically train and validate a model every day.

There’s a few steps needed to setup this component and you will do it in the next cells.

Define the EvalConfig

First, you will define the EvalConfig as you did in the previous lab. You can also set thresholds so you can compare subsequent models to it. The module below should look familiar. One minor difference is you don’t have to define the candidate and baseline model names in the model_specs. That is automatically detected.

eval_config = tfma.EvalConfig(
    model_specs=[
        # This assumes a serving model with signature 'serving_default'.
        tfma.ModelSpec(
            signature_name='serving_default',
            label_key='label'
            )
        ],
    metrics_specs=[
        tfma.MetricsSpec(
            # The metrics added here are in addition to those saved with the
            # model (assuming either a keras model or EvalSavedModel is used).
            # Any metrics added into the saved model (for example using
            # model.compile(..., metrics=[...]), etc) will be computed
            # automatically.
            # To add validation thresholds for metrics saved with the model,
            # add them keyed by metric name to the thresholds map.
            metrics=[
                tfma.MetricConfig(class_name='ExampleCount'),
                tfma.MetricConfig(class_name='BinaryAccuracy',
                      threshold=tfma.MetricThreshold(
                          value_threshold=tfma.GenericValueThreshold(
                              lower_bound={'value': 0.5}),
                              # Change threshold will be ignored if there is no
                              # baseline model resolved from MLMD (first run).
                              change_threshold=tfma.GenericChangeThreshold(
                                  direction=tfma.MetricDirection.HIGHER_IS_BETTER,
                                  absolute={'value': -1e-10}
                                  )
                              )
                      )
            ]
        )
    ],
    slicing_specs=[
        # An empty slice spec means the overall slice, i.e. the whole dataset.
        tfma.SlicingSpec(),
        # Data can be sliced along a feature column.
        tfma.SlicingSpec(
            feature_keys=['sex']),
        tfma.SlicingSpec(
            feature_keys=['race']),
    ])

Resolve the latest blessed model

If you remember in the last lab, you were able to validate a candidate model against a baseline by modifying the EvalConfig and EvalSharedModel definitions. That is also possible using the Evaluator component and you will see how it is done in this section.

First thing to note is that the Evaluator marks a model as BLESSED if it meets the metrics thresholds you set in the eval config module. You can load the latest blessed model by using the LatestBlessedModelStrategy with the Resolver component. This component takes care of finding the latest blessed model for you so you don’t have to remember it manually. The syntax is shown below.

# Setup the Resolver node to find the latest blessed model
model_resolver = tfx.dsl.Resolver(
      strategy_class=tfx.dsl.experimental.LatestBlessedModelStrategy,
      model=tfx.dsl.Channel(type=tfx.types.standard_artifacts.Model),
      model_blessing=tfx.dsl.Channel(
          type=tfx.types.standard_artifacts.ModelBlessing)).with_id(
              'latest_blessed_model_resolver')

# Run the Resolver node
context.run(model_resolver)

ExecutionResult at 0x7f2765b3be90

.execution_id

.component

<tfx.dsl.components.common.resolver.Resolver object at 0x7f27677eab50>

.component.inputs

['model']	<tfx.types.resolved_channel.ResolvedChannel object at 0x7f276745e3d0>
['model_blessing']	<tfx.types.resolved_channel.ResolvedChannel object at 0x7f287b0b87d0>

.component.outputs

['model']

Channel of type 'Model' (0 artifacts) at 0x7f27677eaa10

.type_name	Model
._artifacts	[]

['model_blessing']

Channel of type 'ModelBlessing' (0 artifacts) at 0x7f27677ea050

.type_name	ModelBlessing
._artifacts	[]

As expected, the search yielded 0 artifacts because you haven’t evaluated any models yet. You will run this component again in later parts of this notebook and you will see a different result.

# Load Resolver outputs
model_resolver.outputs['model']

Channel of type 'Model' (0 artifacts) at 0x7f27677eaa10

.type_name	Model
._artifacts	[]

Run the Evaluator component

With the EvalConfig defined and code to load the latest blessed model available, you can proceed to run the Evaluator component.

You will notice that two models are passed into the component. The Trainer output will serve as the candidate model while the output of the Resolver will be the baseline model. While you can indeed run the Evaluator without comparing two models, it would likely be required in production environments so it’s best to include it. Since the Resolver doesn’t have any results yet, the Evaluator will just mark the candidate model as BLESSED in this run.

Aside from the eval config and models (i.e. Trainer and Resolver output), you will also pass in the raw examples from ExampleGen. By default, the component will look for the eval split of these examples and since you’ve defined the serving signature, these will be transformed internally before feeding to the model inputs.

# Setup and run the Evaluator component
evaluator = tfx.components.Evaluator(
    examples=example_gen.outputs['examples'],
    model=trainer.outputs['model'],
    baseline_model=model_resolver.outputs['model'],
    eval_config=eval_config)
context.run(evaluator, enable_cache=False)

ExecutionResult at 0x7f27677dfdd0

.execution_id

.component

Evaluator at 0x7f2767601710

.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f299812d490

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/CsvExampleGen/examples/1) at 0x7f2998168150

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/CsvExampleGen/examples/1
.span	0
.split_names	["train", "eval"]
.version	0

['model']

Channel of type 'Model' (1 artifact) at 0x7f2765a9c7d0

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/6) at 0x7f2765ac18d0

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/6

['baseline_model']

Channel of type 'Model' (0 artifacts) at 0x7f27677eaa10

.type_name	Model
._artifacts	[]

.outputs

['evaluation']

Channel of type 'ModelEvaluation' (1 artifact) at 0x7f27666bdc10

.type_name

ModelEvaluation

._artifacts

[0]

Artifact of type 'ModelEvaluation' (uri: ./pipeline/Evaluator/evaluation/8) at 0x7f2766d964d0

.type	<class 'tfx.types.standard_artifacts.ModelEvaluation'>
.uri	./pipeline/Evaluator/evaluation/8

['blessing']

Channel of type 'ModelBlessing' (1 artifact) at 0x7f27666bda50

.type_name

ModelBlessing

._artifacts

[0]

Artifact of type 'ModelBlessing' (uri: ./pipeline/Evaluator/blessing/8) at 0x7f2766d96410

.type	<class 'tfx.types.standard_artifacts.ModelBlessing'>
.uri	./pipeline/Evaluator/blessing/8

.exec_properties

['eval_config']	{ "metrics_specs": [ { "metrics": [ { "class_name": "ExampleCount" }, { "class_name": "BinaryAccuracy", "threshold": { "change_threshold": { "absolute": -1e-10, "direction": "HIGHER_IS_BETTER" }, "value_threshold": { "lower_bound": 0.5 } } } ] } ], "model_specs": [ { "label_key": "label", "signature_name": "serving_default" } ], "slicing_specs": [ {}, { "feature_keys": [ "sex" ] }, { "feature_keys": [ "race" ] } ] }
['feature_slicing_spec']	None
['fairness_indicator_thresholds']	null
['example_splits']	null
['module_file']	None
['module_path']	None

.component.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f299812d490

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/CsvExampleGen/examples/1) at 0x7f2998168150

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/CsvExampleGen/examples/1
.span	0
.split_names	["train", "eval"]
.version	0

['model']

Channel of type 'Model' (1 artifact) at 0x7f2765a9c7d0

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/6) at 0x7f2765ac18d0

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/6

['baseline_model']

Channel of type 'Model' (0 artifacts) at 0x7f27677eaa10

.type_name	Model
._artifacts	[]

.component.outputs

['evaluation']

Channel of type 'ModelEvaluation' (1 artifact) at 0x7f27666bdc10

.type_name

ModelEvaluation

._artifacts

[0]

Artifact of type 'ModelEvaluation' (uri: ./pipeline/Evaluator/evaluation/8) at 0x7f2766d964d0

.type	<class 'tfx.types.standard_artifacts.ModelEvaluation'>
.uri	./pipeline/Evaluator/evaluation/8

['blessing']

Channel of type 'ModelBlessing' (1 artifact) at 0x7f27666bda50

.type_name

ModelBlessing

._artifacts

[0]

Artifact of type 'ModelBlessing' (uri: ./pipeline/Evaluator/blessing/8) at 0x7f2766d96410

.type	<class 'tfx.types.standard_artifacts.ModelBlessing'>
.uri	./pipeline/Evaluator/blessing/8

Now let’s examine the output artifacts of Evaluator.

# Print component output keys
evaluator.outputs.keys()

dict_keys(['evaluation', 'blessing'])

The blessing output simply states if the candidate model was blessed. The artifact URI will have a BLESSED or NOT_BLESSED file depending on the result. As mentioned earlier, this first run will pass the evaluation because there is no baseline model yet.

# Get `Evaluator` blessing output URI
blessing_uri = evaluator.outputs['blessing'].get()[0].uri

# List files under URI
os.listdir(blessing_uri)

['BLESSED']

The evaluation output, on the other hand, contains the evaluation logs and can be used to visualize the global metrics on the entire evaluation set. You can do that with TFMA as you did in the previous lab.

# Get the TFMA output result path and load the result.
# NOTE: If you don't see the visualization on the first try, simply re-run this cell to generate it again.
PATH_TO_RESULT = evaluator.outputs['evaluation'].get()[0].uri
tfma_result = tfma.load_eval_result(PATH_TO_RESULT)

# Show global meterics
tfma.view.render_slicing_metrics(
    tfma_result)

SlicingMetricsViewer(config={'weightedExamplesColumn': 'example_count'}, data=[{'slice': 'Overall', 'metrics':…

You can also see the individual slices as you did before.

# Show data sliced along a feature column
tfma.view.render_slicing_metrics(
    tfma_result, slicing_column='sex')

SlicingMetricsViewer(config={'weightedExamplesColumn': 'example_count'}, data=[{'slice': 'sex:Male', 'metrics'…

You can also load the validation results by specifying the output URI of the evaluation output. This would be more useful if your model was not blessed because you can see the metric failure prompts. Try to simulate this later by training with fewer epochs (or raising the threshold) and see the results you get here.

# Get `evaluation` output URI
PATH_TO_RESULT = evaluator.outputs['evaluation'].get()[0].uri

# Print validation result
print(tfma.load_validation_result(PATH_TO_RESULT))

validation_ok: true
validation_details {
  slicing_details {
    slicing_spec {
    }
    num_matching_slices: 8
  }
}

Now that your Evaluator has finished running, the Resolver component should be able to detect the latest blessed model. Let’s run the component again.

# Re-run the Resolver component
context.run(model_resolver)

ExecutionResult at 0x7f27538a1f10

.execution_id

.component

<tfx.dsl.components.common.resolver.Resolver object at 0x7f27677eab50>

.component.inputs

['model']	<tfx.types.resolved_channel.ResolvedChannel object at 0x7f276745e3d0>
['model_blessing']	<tfx.types.resolved_channel.ResolvedChannel object at 0x7f287b0b87d0>

.component.outputs

['model']

Channel of type 'Model' (1 artifact) at 0x7f27677eaa10

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/6) at 0x7f27538d2750

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/6

['model_blessing']

Channel of type 'ModelBlessing' (1 artifact) at 0x7f27677ea050

.type_name

ModelBlessing

._artifacts

[0]

Artifact of type 'ModelBlessing' (uri: ./pipeline/Evaluator/blessing/8) at 0x7f27538d2910

.type	<class 'tfx.types.standard_artifacts.ModelBlessing'>
.uri	./pipeline/Evaluator/blessing/8

You should now see an artifact in the component outputs. You can also get it programmatically as shown below.

# Get path to latest blessed model
model_resolver.outputs['model'].get()[0].uri

'./pipeline/Trainer/model/6'

Comparing two models

Now let’s see how Evaluator compares two models. You will train the same model with more epochs and this should hopefully result in higher accuracy and overall metrics.

# Setup trainer to train with more epochs
trainer = tfx.components.Trainer(
    module_file=os.path.abspath(_census_trainer_module_file),
    examples=transform.outputs['transformed_examples'],
    transform_graph=transform.outputs['transform_graph'],
    schema=schema_gen.outputs['schema'],
    train_args=tfx.proto.TrainArgs(num_steps=500),
    eval_args=tfx.proto.EvalArgs(num_steps=200))

# Run trainer
context.run(trainer, enable_cache=False)

Model: "model_1"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 fnlwgt (InputLayer)            [(None, 1)]          0           []                               
                                                                                                  
 education-num (InputLayer)     [(None, 1)]          0           []                               
                                                                                                  
 capital-gain (InputLayer)      [(None, 1)]          0           []                               
                                                                                                  
 capital-loss (InputLayer)      [(None, 1)]          0           []                               
                                                                                                  
 hours-per-week (InputLayer)    [(None, 1)]          0           []                               
                                                                                                  
 concatenate_3 (Concatenate)    (None, 5)            0           ['fnlwgt[0][0]',                 
                                                                  'education-num[0][0]',          
                                                                  'capital-gain[0][0]',           
                                                                  'capital-loss[0][0]',           
                                                                  'hours-per-week[0][0]']         
                                                                                                  
 dense_6 (Dense)                (None, 100)          600         ['concatenate_3[0][0]']          
                                                                                                  
 dense_7 (Dense)                (None, 70)           7070        ['dense_6[0][0]']                
                                                                                                  
 education (InputLayer)         [(None, 21)]         0           []                               
                                                                                                  
 marital-status (InputLayer)    [(None, 12)]         0           []                               
                                                                                                  
 occupation (InputLayer)        [(None, 20)]         0           []                               
                                                                                                  
 race (InputLayer)              [(None, 10)]         0           []                               
                                                                                                  
 relationship (InputLayer)      [(None, 11)]         0           []                               
                                                                                                  
 workclass (InputLayer)         [(None, 14)]         0           []                               
                                                                                                  
 sex (InputLayer)               [(None, 7)]          0           []                               
                                                                                                  
 native-country (InputLayer)    [(None, 47)]         0           []                               
                                                                                                  
 age (InputLayer)               [(None, 4)]          0           []                               
                                                                                                  
 dense_8 (Dense)                (None, 48)           3408        ['dense_7[0][0]']                
                                                                                                  
 concatenate_4 (Concatenate)    (None, 146)          0           ['education[0][0]',              
                                                                  'marital-status[0][0]',         
                                                                  'occupation[0][0]',             
                                                                  'race[0][0]',                   
                                                                  'relationship[0][0]',           
                                                                  'workclass[0][0]',              
                                                                  'sex[0][0]',                    
                                                                  'native-country[0][0]',         
                                                                  'age[0][0]']                    
                                                                                                  
 dense_9 (Dense)                (None, 34)           1666        ['dense_8[0][0]']                
                                                                                                  
 dense_10 (Dense)               (None, 128)          18816       ['concatenate_4[0][0]']          
                                                                                                  
 concatenate_5 (Concatenate)    (None, 162)          0           ['dense_9[0][0]',                
                                                                  'dense_10[0][0]']               
                                                                                                  
 dense_11 (Dense)               (None, 1)            163         ['concatenate_5[0][0]']          
                                                                                                  
==================================================================================================
Total params: 31,723
Trainable params: 31,723
Non-trainable params: 0
__________________________________________________________________________________________________
500/500 [==============================] - 3s 3ms/step - loss: 0.3611 - binary_accuracy: 0.8299 - val_loss: 0.3169 - val_binary_accuracy: 0.8491

ExecutionResult at 0x7f2936a9c610

.execution_id

.component

Trainer at 0x7f27538f5e10

.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f2940acf550

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/Transform/transformed_examples/5) at 0x7f2940ac2d50

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/Transform/transformed_examples/5
.span	0
.split_names	["train", "eval"]
.version	0

['transform_graph']

Channel of type 'TransformGraph' (1 artifact) at 0x7f2940acf690

.type_name

TransformGraph

._artifacts

[0]

Artifact of type 'TransformGraph' (uri: ./pipeline/Transform/transform_graph/5) at 0x7f2940acf890

.type	<class 'tfx.types.standard_artifacts.TransformGraph'>
.uri	./pipeline/Transform/transform_graph/5

['schema']

Channel of type 'Schema' (1 artifact) at 0x7f293e7f50d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/SchemaGen/schema/3) at 0x7f293e7d9290

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/SchemaGen/schema/3

.outputs

['model']

Channel of type 'Model' (1 artifact) at 0x7f2753880f10

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/10) at 0x7f27538d2190

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/10

['model_run']

Channel of type 'ModelRun' (1 artifact) at 0x7f2753880c10

.type_name

ModelRun

._artifacts

[0]

Artifact of type 'ModelRun' (uri: ./pipeline/Trainer/model_run/10) at 0x7f27538d2d90

.type	<class 'tfx.types.standard_artifacts.ModelRun'>
.uri	./pipeline/Trainer/model_run/10

.exec_properties

['train_args']	{ "num_steps": 500 }
['eval_args']	{ "num_steps": 200 }
['module_file']	None
['run_fn']	None
['trainer_fn']	None
['custom_config']	null
['module_path']	census_trainer@./pipeline/_wheels/tfx_user_code_Trainer-0.0+48cbd699b9b98d6808f806735a8a67817df89a5fd43fd5a9e489201454f3a0c9-py3-none-any.whl

.component.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f2940acf550

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/Transform/transformed_examples/5) at 0x7f2940ac2d50

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/Transform/transformed_examples/5
.span	0
.split_names	["train", "eval"]
.version	0

['transform_graph']

Channel of type 'TransformGraph' (1 artifact) at 0x7f2940acf690

.type_name

TransformGraph

._artifacts

[0]

Artifact of type 'TransformGraph' (uri: ./pipeline/Transform/transform_graph/5) at 0x7f2940acf890

.type	<class 'tfx.types.standard_artifacts.TransformGraph'>
.uri	./pipeline/Transform/transform_graph/5

['schema']

Channel of type 'Schema' (1 artifact) at 0x7f293e7f50d0

.type_name

Schema

._artifacts

[0]

Artifact of type 'Schema' (uri: ./pipeline/SchemaGen/schema/3) at 0x7f293e7d9290

.type	<class 'tfx.types.standard_artifacts.Schema'>
.uri	./pipeline/SchemaGen/schema/3

.component.outputs

['model']

Channel of type 'Model' (1 artifact) at 0x7f2753880f10

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/10) at 0x7f27538d2190

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/10

['model_run']

Channel of type 'ModelRun' (1 artifact) at 0x7f2753880c10

.type_name

ModelRun

._artifacts

[0]

Artifact of type 'ModelRun' (uri: ./pipeline/Trainer/model_run/10) at 0x7f27538d2d90

.type	<class 'tfx.types.standard_artifacts.ModelRun'>
.uri	./pipeline/Trainer/model_run/10

You will re-run the evaluator but you will specify the latest trainer output as the candidate model. The baseline is automatically found with the Resolver node.

# Setup and run the Evaluator component
evaluator = tfx.components.Evaluator(
    examples=example_gen.outputs['examples'],
    model=trainer.outputs['model'],
    baseline_model=model_resolver.outputs['model'],
    eval_config=eval_config)
context.run(evaluator, enable_cache=False)

ExecutionResult at 0x7f293e5836d0

.execution_id

.component

Evaluator at 0x7f2940dd13d0

.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f299812d490

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/CsvExampleGen/examples/1) at 0x7f2998168150

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/CsvExampleGen/examples/1
.span	0
.split_names	["train", "eval"]
.version	0

['model']

Channel of type 'Model' (1 artifact) at 0x7f2753880f10

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/10) at 0x7f27538d2190

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/10

['baseline_model']

Channel of type 'Model' (1 artifact) at 0x7f27677eaa10

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/6) at 0x7f27538d2750

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/6

.outputs

['evaluation']

Channel of type 'ModelEvaluation' (1 artifact) at 0x7f293baa3dd0

.type_name

ModelEvaluation

._artifacts

[0]

Artifact of type 'ModelEvaluation' (uri: ./pipeline/Evaluator/evaluation/11) at 0x7f293e6b95d0

.type	<class 'tfx.types.standard_artifacts.ModelEvaluation'>
.uri	./pipeline/Evaluator/evaluation/11

['blessing']

Channel of type 'ModelBlessing' (1 artifact) at 0x7f2878089e10

.type_name

ModelBlessing

._artifacts

[0]

Artifact of type 'ModelBlessing' (uri: ./pipeline/Evaluator/blessing/11) at 0x7f27538f5910

.type	<class 'tfx.types.standard_artifacts.ModelBlessing'>
.uri	./pipeline/Evaluator/blessing/11

.exec_properties

['eval_config']	{ "metrics_specs": [ { "metrics": [ { "class_name": "ExampleCount" }, { "class_name": "BinaryAccuracy", "threshold": { "change_threshold": { "absolute": -1e-10, "direction": "HIGHER_IS_BETTER" }, "value_threshold": { "lower_bound": 0.5 } } } ] } ], "model_specs": [ { "label_key": "label", "signature_name": "serving_default" } ], "slicing_specs": [ {}, { "feature_keys": [ "sex" ] }, { "feature_keys": [ "race" ] } ] }
['feature_slicing_spec']	None
['fairness_indicator_thresholds']	null
['example_splits']	null
['module_file']	None
['module_path']	None

.component.inputs

['examples']

Channel of type 'Examples' (1 artifact) at 0x7f299812d490

.type_name

Examples

._artifacts

[0]

Artifact of type 'Examples' (uri: ./pipeline/CsvExampleGen/examples/1) at 0x7f2998168150

.type	<class 'tfx.types.standard_artifacts.Examples'>
.uri	./pipeline/CsvExampleGen/examples/1
.span	0
.split_names	["train", "eval"]
.version	0

['model']

Channel of type 'Model' (1 artifact) at 0x7f2753880f10

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/10) at 0x7f27538d2190

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/10

['baseline_model']

Channel of type 'Model' (1 artifact) at 0x7f27677eaa10

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/6) at 0x7f27538d2750

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/6

.component.outputs

['evaluation']

Channel of type 'ModelEvaluation' (1 artifact) at 0x7f293baa3dd0

.type_name

ModelEvaluation

._artifacts

[0]

Artifact of type 'ModelEvaluation' (uri: ./pipeline/Evaluator/evaluation/11) at 0x7f293e6b95d0

.type	<class 'tfx.types.standard_artifacts.ModelEvaluation'>
.uri	./pipeline/Evaluator/evaluation/11

['blessing']

Channel of type 'ModelBlessing' (1 artifact) at 0x7f2878089e10

.type_name

ModelBlessing

._artifacts

[0]

Artifact of type 'ModelBlessing' (uri: ./pipeline/Evaluator/blessing/11) at 0x7f27538f5910

.type	<class 'tfx.types.standard_artifacts.ModelBlessing'>
.uri	./pipeline/Evaluator/blessing/11

Depending on the result, the Resolver should reflect the latest blessed model. Since you trained with more epochs, it is most likely that your candidate model will pass the thresholds you set in the eval config. If so, the artifact URI should be different here compared to your earlier runs.

# Re-run the resolver
context.run(model_resolver, enable_cache=False)

ExecutionResult at 0x7f27880f7290

.execution_id

.component

<tfx.dsl.components.common.resolver.Resolver object at 0x7f27677eab50>

.component.inputs

['model']	<tfx.types.resolved_channel.ResolvedChannel object at 0x7f276745e3d0>
['model_blessing']	<tfx.types.resolved_channel.ResolvedChannel object at 0x7f287b0b87d0>

.component.outputs

['model']

Channel of type 'Model' (1 artifact) at 0x7f27677eaa10

.type_name

Model

._artifacts

[0]

Artifact of type 'Model' (uri: ./pipeline/Trainer/model/10) at 0x7f2750438610

.type	<class 'tfx.types.standard_artifacts.Model'>
.uri	./pipeline/Trainer/model/10

['model_blessing']

Channel of type 'ModelBlessing' (1 artifact) at 0x7f27677ea050

.type_name

ModelBlessing

._artifacts

[0]

Artifact of type 'ModelBlessing' (uri: ./pipeline/Evaluator/blessing/11) at 0x7f2767fcefd0

.type	<class 'tfx.types.standard_artifacts.ModelBlessing'>
.uri	./pipeline/Evaluator/blessing/11

# Get path to latest blessed model
model_resolver.outputs['model'].get()[0].uri

'./pipeline/Trainer/model/10'

Finally, the evaluation output of the Evaluator component will produce the diff results you saw in the previous lab. This will signify if the metrics from the candidate model has indeed improved compared to the baseline. Unlike the previous lab, visualizing this will just show the results for the candidate (i.e. only one row instead of two in the tabular output in the graph below).

# Get the TFMA output result path and load the result.
# NOTE: If you don't see the visualization on the first try, simply re-run this cell to generate it again.
PATH_TO_RESULT = evaluator.outputs['evaluation'].get()[0].uri
tfma_result = tfma.load_eval_result(PATH_TO_RESULT)

# Show global meterics
tfma.view.render_slicing_metrics(
    tfma_result)

SlicingMetricsViewer(config={'weightedExamplesColumn': 'example_count'}, data=[{'slice': 'Overall', 'metrics':…

Congratulations! You can now successfully evaluate your models in a TFX pipeline! This is a critical part of production ML because you want to make sure that subsequent deployments are indeed improving your results. Moreover, you can extract the evaluation results from your pipeline directory for further investigation with TFMA. In the next sections, you will continue to study techniques related to model evaluation and ensuring fairness.