[Tensorflow] 에러 (tensorflow Fail to find the dnn implementation)

 

에러 내용

2022-08-20 21:35:04.723160: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
Epoch 1/20
2022-08-20 21:35:10.323280: E tensorflow/stream_executor/cuda/cuda_dnn.cc:374] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2022-08-20 21:35:10.323540: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at cudnn_rnn_ops.cc:1555 : Unknown: Fail to find the dnn implementation.
Traceback (most recent call last):
  File "E:\RnD\Tensorflow\tk2001\kerasStockLSTM_basic.py", line 65, in <module>
    model.fit(x_train, y_train, batch_size=10, epochs=20) # 20번동안 반복
  File "C:\Users\admin\.conda\envs\env001\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1193, in fit
    tmp_logs = self.train_function(iterator)
  File "C:\Users\admin\.conda\envs\env001\lib\site-packages\tensorflow\python\eager\def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\admin\.conda\envs\env001\lib\site-packages\tensorflow\python\eager\def_function.py", line 950, in _call
    return self._stateless_fn(*args, **kwds)
  File "C:\Users\admin\.conda\envs\env001\lib\site-packages\tensorflow\python\eager\function.py", line 3039, in __call__
    return graph_function._call_flat(
  File "C:\Users\admin\.conda\envs\env001\lib\site-packages\tensorflow\python\eager\function.py", line 1963, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\admin\.conda\envs\env001\lib\site-packages\tensorflow\python\eager\function.py", line 591, in call
    outputs = execute.execute(
  File "C:\Users\admin\.conda\envs\env001\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnknownError:    Fail to find the dnn implementation.
         [[{{node CudnnRNN}}]]
         [[sequential/lstm/PartitionedCall]] [Op:__inference_train_function_5392]

 

해결

- 다른 tensorflow 프로그램이 실행되고 있는 상황에서 메모리를 Full로 요청하기 때문에 발생하는 것으로 보임.

아래와 같이 메모리 사용을 설정해주어서 해결 됨.

tf.compat.v1.disable_eager_execution()
gpus = tf.config.experimental.list_physical_devices('GPU')
if len(gpus) > 0:
    print(f'GPUs {gpus}')
    try:
        tf.config.experimental.set_visible_devices(gpus[0], 'GPU')
        tf.config.experimental.set_memory_growth(gpus[0], True)
    except RuntimeError:
        pass