文档建议反馈控制台
首页
学习
活动
专区
圈层
工具
MCP广场
文章/答案/技术大牛
发布
社区首页 >问答首页 >将MobileNet与Tensorflow联合使用时，无法序列化协议缓冲区

问将MobileNet与Tensorflow联合使用时，无法序列化协议缓冲区
EN

Stack Overflow用户
提问于 2019-05-03 04:32:30
回答 1查看 283关注 0票数 2

我正在使用来自Keras的预训练的MobileNet，并希望使用TensorFlows联邦学习来训练它，但我总是得到一个错误，即协议缓冲区无法序列化，因为超过了2 2GB的限制。我的输入是224x224RGB图像。

编辑:我有一个包含1000个图像(每个客户端500个图像)的数据集，每个数据集使用10个时期。

我创建和训练Keras模型的代码如下：

代码语言：javascript
复制
def create_compiled_keras_model():
    base_model = MobileNet(weights='imagenet', include_top=False)
    x = base_model.output
    x = GlobalAveragePooling2D()(x)
    x = Dense(1024, activation='relu')(x)
    x = Dense(1024, activation='relu')(x)
    x = Dense(512, activation='relu')(x)
    preds = Dense(47, activation='softmax')(x)

    # specify input & output
    model = Model(inputs=base_model.input, outputs=preds)

    # set the first 20 layers of the network to be non-trainable
    for layer in model.layers[:20]:
        layer.trainable = False
    for layer in model.layers[20:]:
        layer.trainable = True

    def loss_fn(y_true, y_pred):
        return tf.reduce_mean(tf.keras.losses.sparse_categorical_crossentropy(
        y_true, y_pred))

    model.compile(
    loss=loss_fn,
    optimizer=gradient_descent.SGD(learning_rate=0.02),
    metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])

    return model


def model_fn():
    keras_model = create_compiled_keras_model()
    return tff.learning.from_compiled_keras_model(keras_model,  sample_batch)


# @test {"output": "ignore"}
iterative_process =   tff.learning.build_federated_averaging_process(model_fn)
state = iterative_process.initialize()

# @test {"timeout": 600, "output": "ignore"}

state, metrics = iterative_process.next(state, federated_train_data)
print('round  1, metrics={}'.format(metrics))

我收到的错误：

代码语言：javascript
复制
Traceback (most recent call last):
  File "/Users/recep/Documents/federated_learning.py", line 128, in <module>
    state, metrics = iterative_process.next(state, federated_train_data)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/function_utils.py", line 600, in __call__
    return context.invoke(self, arg)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 700, in invoke
    result = computed_comp.value(computed_arg)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 843, in <lambda>
    return ComputedValue(lambda x: self._compute(comp.result, _wrap(x)),
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 746, in _compute
    return self._compute_tuple(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 802, in _compute_tuple
    computed_v = self._compute(v, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 744, in _compute
    return self._compute_call(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 784, in _compute_call
    computed_arg = self._compute(comp.argument, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 746, in _compute
    return self._compute_tuple(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 802, in _compute_tuple
    computed_v = self._compute(v, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 744, in _compute
    return self._compute_call(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 784, in _compute_call
    computed_arg = self._compute(comp.argument, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 746, in _compute
    return self._compute_tuple(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 802, in _compute_tuple
    computed_v = self._compute(v, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 744, in _compute
    return self._compute_call(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 784, in _compute_call
    computed_arg = self._compute(comp.argument, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 746, in _compute
    return self._compute_tuple(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 802, in _compute_tuple
    computed_v = self._compute(v, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 744, in _compute
    return self._compute_call(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 784, in _compute_call
    computed_arg = self._compute(comp.argument, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 746, in _compute
    return self._compute_tuple(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 802, in _compute_tuple
    computed_v = self._compute(v, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 744, in _compute
    return self._compute_call(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 784, in _compute_call
    computed_arg = self._compute(comp.argument, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 746, in _compute
    return self._compute_tuple(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 802, in _compute_tuple
    computed_v = self._compute(v, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 744, in _compute
    return self._compute_call(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 784, in _compute_call
    computed_arg = self._compute(comp.argument, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 746, in _compute
    return self._compute_tuple(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 802, in _compute_tuple
    computed_v = self._compute(v, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 744, in _compute
    return self._compute_call(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 784, in _compute_call
    computed_arg = self._compute(comp.argument, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 746, in _compute
    return self._compute_tuple(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 802, in _compute_tuple
    computed_v = self._compute(v, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 744, in _compute
    return self._compute_call(comp, context)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 791, in _compute_call
    result = computed_fn.value(computed_arg)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 871, in <lambda>
    lambda x: my_method(fit_argument(x, arg_type, context)),
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 914, in _federated_map
    fn(ComputedValue(x, mapping_type.parameter)).value for x in arg.value[1]
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 914, in <listcomp>
    fn(ComputedValue(x, mapping_type.parameter)).value for x in arg.value[1]
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 775, in <lambda>
    return ComputedValue(lambda x: run_tensorflow(comp, x),
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/reference_executor.py", line 340, in run_tensorflow
    comp.proto, stamped_arg, graph))
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/tensorflow_deserialization.py", line 90, in deserialize_and_call_tf_computation
    arg_type, arg_binding = graph_utils.capture_result_from_graph(arg, graph)
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/graph_utils.py", line 244, in capture_result_from_graph
    for k, v in name_value_pairs
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/graph_utils.py", line 244, in <listcomp>
    for k, v in name_value_pairs
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow_federated/python/core/impl/graph_utils.py", line 265, in capture_result_from_graph
    handle_name = result.make_one_shot_iterator().string_handle().name
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1358, in make_one_shot_iterator
    _make_dataset.add_to_graph(ops.get_default_graph())
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 490, in add_to_graph
    self._create_definition_if_needed()
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 341, in _create_definition_if_needed
    self._create_definition_if_needed_impl()
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 417, in _create_definition_if_needed_impl
    self._op_def = self.definition.signature
  File "/Users/recep/anaconda3/envs/python37/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 289, in definition
    c_api.TF_FunctionToFunctionDef(self._c_func.func, buf)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot serialize protocol buffer of type tensorflow.FunctionDef as the serialized size (3004868250bytes) would be larger than the limit (2147483647 bytes)
python
tensorflow
keras
tensorflow-federated
mobilenet
EN

回答 1

Stack Overflow用户

发布于 2020-02-14 12:55:50

在0.12.0版本中，TensorFlow联邦在默认情况下开始使用新的急切模式执行器，这可能会显着改善这一点；有报告称，完成了对ResNet模型的培训。

票数 1
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：

https://stackoverflow.com/questions/55959848

复制
相关文章

相似问题

领券

腾讯云开发者

扫码关注腾讯云开发者

扫码关注腾讯云开发者

领取腾讯云代金券

热门产品

热门推荐

更多推荐

Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 粤公网安备44030502008569号

腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号

问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档