首页
学习
活动
专区
圈层
工具
发布
社区首页 >问答首页 >StreamingFileSink无法重命名进程中的文件

StreamingFileSink无法重命名进程中的文件
EN

Stack Overflow用户
提问于 2019-11-12 09:34:00
回答 1查看 176关注 0票数 0

当StreamingFileSink尝试重命名进程中的文件时,我偶尔会遇到一个问题。请参见下面的异常:

文件系统挂载在Google Cloud Storage buckets上。它在大多数情况下都工作得很好,但突然失败了。

我已经验证了inprogress文件是否存在。任何提示和建议都将非常有用--谢谢!

代码语言:javascript
复制
2019-11-12 01:26:44.908 [flink-akka.actor.default-dispatcher-77] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph - Job kafka-ingestion-avro-enriched-structured-trace-event-staging (b2f5e85f57049e0841a88498fcda868d) switched from state RUNNING to FAILING.
java.nio.file.NoSuchFileException: /var/data/rawdata/2019-11-11--19/.part-0-0.inprogress.a99cb596-8e96-4b88-ae60-e5b64875bb74 -> /var/data/rawdata/2019-11-11--19/part-0-0
    at sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
    at sun.nio.fs.UnixCopyFile.move(Unknown Source) ~[?:?]
    at sun.nio.fs.UnixFileSystemProvider.move(Unknown Source) ~[?:?]
    at java.nio.file.Files.move(Unknown Source) ~[?:?]
    at org.apache.flink.core.fs.local.LocalRecoverableFsDataOutputStream$LocalCommitter.commitAfterRecovery(LocalRecoverableFsDataOutputStream.java:181) ~[flink-core-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.commitRecoveredPendingFiles(Bucket.java:137) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.<init>(Bucket.java:119) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.restore(Bucket.java:346) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.api.functions.sink.filesystem.DefaultBucketFactoryImpl.restoreBucket(DefaultBucketFactoryImpl.java:64) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.handleRestoredBucketState(Buckets.java:177) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.initializeActiveBuckets(Buckets.java:165) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.initializeState(Buckets.java:149) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:334) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289) ~[flink-streaming-java_2.11-1.7.0.jar:1.7.0]
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704) ~[flink-runtime_2.11-1.7.0.jar:1.7.0]
EN

回答 1

Stack Overflow用户

发布于 2020-03-30 00:22:30

遇到同样的问题,我打赌这与最终的一致性或使用对象存储的文件系统模拟有关。我使用了基于gcs://的路径,现在它似乎运行得很好。

票数 0
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/58810779

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档