17/07/19 08:46:11 main INFO CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT] 17/07/19 08:46:12 main INFO SecurityManager: Changing view acls to: yarn,ec2-user 17/07/19 08:46:12 main INFO SecurityManager: Changing modify acls to: yarn,ec2-user 17/07/19 08:46:12 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, ec2-user); users with modify permissions: Set(yarn, ec2-user) 17/07/19 08:46:12 main INFO SecurityManager: Changing view acls to: yarn,ec2-user 17/07/19 08:46:12 main INFO SecurityManager: Changing modify acls to: yarn,ec2-user 17/07/19 08:46:12 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, ec2-user); users with modify permissions: Set(yarn, ec2-user) 17/07/19 08:46:13 sparkExecutorActorSystem-akka.actor.default-dispatcher-2 INFO Slf4jLogger: Slf4jLogger started 17/07/19 08:46:13 sparkExecutorActorSystem-akka.actor.default-dispatcher-2 INFO Remoting: Starting remoting 17/07/19 08:46:13 sparkExecutorActorSystem-akka.actor.default-dispatcher-2 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutorActorSystem@ip-10-153-214-122.ec2.internal:43077] 17/07/19 08:46:13 main INFO Utils: Successfully started service 'sparkExecutorActorSystem' on port 43077. 17/07/19 08:46:13 main INFO DiskBlockManager: Created local directory at /media/ephemeral0/yarn/local/usercache/ec2-user/appcache/application_1500440834352_0004/blockmgr-830adcc2-4d77-4c66-9ebc-9154ccbea960 17/07/19 08:46:13 main INFO MemoryStore: MemoryStore started with capacity 14.2 GB 17/07/19 08:46:13 dispatcher-event-loop-1 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@10.153.214.122:36433 17/07/19 08:46:13 dispatcher-event-loop-2 INFO CoarseGrainedExecutorBackend: Successfully registered with driver 17/07/19 08:46:13 dispatcher-event-loop-2 INFO Executor: Starting executor ID 3 on host ip-10-153-214-122.ec2.internal 17/07/19 08:46:13 dispatcher-event-loop-2 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42141. 17/07/19 08:46:13 dispatcher-event-loop-2 INFO NettyBlockTransferService: Server created on 42141 17/07/19 08:46:13 dispatcher-event-loop-2 INFO BlockManager: external shuffle service port = 7337 17/07/19 08:46:13 dispatcher-event-loop-2 INFO BlockManagerMaster: Trying to register BlockManager 17/07/19 08:46:13 dispatcher-event-loop-2 INFO BlockManagerMaster: Registered BlockManager 17/07/19 08:46:13 dispatcher-event-loop-2 INFO BlockManager: Registering executor with local external shuffle service. 17/07/19 08:46:15 dispatcher-event-loop-1 INFO CoarseGrainedExecutorBackend: Got assigned task 3 17/07/19 08:46:15 Executor task launch worker-0 INFO Executor: Running task 3.0 in stage 0.0 (TID 3) 17/07/19 08:46:16 Executor task launch worker-0 INFO TorrentBroadcast: Started reading broadcast variable 2 17/07/19 08:46:16 Executor task launch worker-0 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 7.6 KB, free 7.6 KB) 17/07/19 08:46:16 Executor task launch worker-0 INFO TorrentBroadcast: Reading broadcast variable 2 took 105 ms 17/07/19 08:46:16 Executor task launch worker-0 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 11.5 KB, free 19.1 KB) 2017-07-19 08:46:16,538 INFO (MainThread-14523) connected to server at ('ip-10-153-214-122', 33927) 2017-07-19 08:46:16,539 INFO (MainThread-14523) TFSparkNode.reserve: {'authkey': '\x8f\x1a\xb4tA\xc2G\xfb\x94\xd1\xe2g\x03\xba\xf5\x84', 'worker_num': 3, 'host': 'ip-10-153-214-122', 'tb_port': 0, 'addr': '/tmp/pymp-4n8JKO/listener-RJ0DGB', 'ppid': 14517, 'task_index': 2, 'job_name': 'worker', 'tb_pid': 0, 'port': 39217} 2017-07-19 08:46:18,543 INFO (MainThread-14523) node: {'addr': ('ip-10-111-178-35', 41949), 'task_index': 0, 'job_name': 'ps', 'authkey': '\x12\xa5\x18$$\xb8O\x85\xa6nO\x97\x81+\xd4C', 'worker_num': 0, 'host': 'ip-10-111-178-35', 'ppid': 14643, 'port': 37951, 'tb_pid': 0, 'tb_port': 0} 2017-07-19 08:46:18,543 INFO (MainThread-14523) node: {'addr': '/tmp/pymp-8T8aOT/listener-8c7obu', 'task_index': 0, 'job_name': 'worker', 'authkey': '\xce\xb9 \xd2\xec\x97J>\x84\x15\x08\x1fY\xf9vE', 'worker_num': 1, 'host': 'ip-10-156-14-210', 'ppid': 14602, 'port': 35991, 'tb_pid': 0, 'tb_port': 0} 2017-07-19 08:46:18,544 INFO (MainThread-14523) node: {'addr': '/tmp/pymp-jI4zyL/listener-Ucfnew', 'task_index': 1, 'job_name': 'worker', 'authkey': "\xa9\xc83L\x17\xacN\xc5\xab'\x01\xef]\xb3\xd48", 'worker_num': 2, 'host': 'ip-10-167-218-220', 'ppid': 13976, 'port': 35779, 'tb_pid': 0, 'tb_port': 0} 2017-07-19 08:46:18,544 INFO (MainThread-14523) node: {'addr': '/tmp/pymp-4n8JKO/listener-RJ0DGB', 'task_index': 2, 'job_name': 'worker', 'authkey': '\x8f\x1a\xb4tA\xc2G\xfb\x94\xd1\xe2g\x03\xba\xf5\x84', 'worker_num': 3, 'host': 'ip-10-153-214-122', 'ppid': 14517, 'port': 39217, 'tb_pid': 0, 'tb_port': 0} 2017-07-19 08:46:18,680 INFO (MainThread-14523) Starting TensorFlow worker:2 on cluster node 3 on background process 17/07/19 08:46:18 Executor task launch worker-0 INFO PythonRunner: Times: total = 2413, boot = 226, init = 35, finish = 2152 17/07/19 08:46:18 Executor task launch worker-0 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 1016 bytes result sent to driver 17/07/19 08:46:18 dispatcher-event-loop-3 INFO CoarseGrainedExecutorBackend: Got assigned task 4 17/07/19 08:46:18 Executor task launch worker-0 INFO Executor: Running task 0.0 in stage 1.0 (TID 4) 17/07/19 08:46:18 Executor task launch worker-0 INFO TorrentBroadcast: Started reading broadcast variable 3 17/07/19 08:46:18 Executor task launch worker-0 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 7.0 KB, free 26.1 KB) 17/07/19 08:46:18 Executor task launch worker-0 INFO TorrentBroadcast: Reading broadcast variable 3 took 17 ms 17/07/19 08:46:18 Executor task launch worker-0 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 14.2 KB, free 40.3 KB) 17/07/19 08:46:18 Executor task launch worker-0 INFO HadoopRDD: Input split: hdfs://ec2-54-87-28-131.compute-1.amazonaws.com:9000/user/ec2-user/mnist/csv/train/images/part-00000:0+9338236 17/07/19 08:46:18 Executor task launch worker-0 INFO TorrentBroadcast: Started reading broadcast variable 0 17/07/19 08:46:18 Executor task launch worker-0 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 24.8 KB, free 65.1 KB) 17/07/19 08:46:18 Executor task launch worker-0 INFO TorrentBroadcast: Reading broadcast variable 0 took 14 ms 17/07/19 08:46:18 Executor task launch worker-0 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 360.6 KB, free 425.7 KB) 17/07/19 08:46:18 Executor task launch worker-0 INFO deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS 17/07/19 08:46:20 Executor task launch worker-0 INFO GPLNativeCodeLoader: Loaded native gpl library 17/07/19 08:46:20 Executor task launch worker-0 INFO LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8] 17/07/19 08:46:20 Executor task launch worker-0 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id 17/07/19 08:46:20 Executor task launch worker-0 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 17/07/19 08:46:20 Executor task launch worker-0 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap 17/07/19 08:46:20 Executor task launch worker-0 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition 17/07/19 08:46:20 Executor task launch worker-0 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id 2017-07-19 08:46:20,113 INFO (MainThread-14561) 3: ======== worker:2 ======== 2017-07-19 08:46:20,113 INFO (MainThread-14561) 3: Cluster spec: {'ps': ['ip-10-111-178-35:37951'], 'worker': ['ip-10-156-14-210:35991', 'ip-10-167-218-220:35779', 'ip-10-153-214-122:39217']} 2017-07-19 08:46:20,113 INFO (MainThread-14561) 3: Using CPU 2017-07-19 08:46:20.113876: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2017-07-19 08:46:20.113896: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2017-07-19 08:46:20.113903: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2017-07-19 08:46:20.121832: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> ip-10-111-178-35:37951} 2017-07-19 08:46:20.121859: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> ip-10-156-14-210:35991, 1 -> ip-10-167-218-220:35779, 2 -> localhost:39217} 2017-07-19 08:46:20.122266: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:39217 tensorflow model path: hdfs://ec2-54-87-28-131.compute-1.amazonaws.com:9000/user/ec2-user/mnist_model 17/07/19 08:46:20 Executor task launch worker-0 INFO HadoopRDD: Input split: hdfs://ec2-54-87-28-131.compute-1.amazonaws.com:9000/user/ec2-user/mnist/csv/train/labels/part-00000:0+204800 17/07/19 08:46:20 Executor task launch worker-0 INFO TorrentBroadcast: Started reading broadcast variable 1 17/07/19 08:46:20 Executor task launch worker-0 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 24.8 KB, free 450.5 KB) 17/07/19 08:46:20 Executor task launch worker-0 INFO TorrentBroadcast: Reading broadcast variable 1 took 14 ms 17/07/19 08:46:20 Executor task launch worker-0 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 360.6 KB, free 811.1 KB) 17/07/19 08:46:20 Executor task launch worker-0 INFO deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS 2017-07-19 08:46:20,768 INFO (MainThread-14591) Connected to TFSparkNode.mgr on ip-10-153-214-122, ppid=14517, state='running' 2017-07-19 08:46:20,778 INFO (MainThread-14591) mgr.state='running' 2017-07-19 08:46:20,778 INFO (MainThread-14591) Feeding partition into input queue 17/07/19 08:46:22 stdout writer for Python/bin/python INFO PythonRunner: Times: total = 2601, boot = -1495, init = 1611, finish = 2485 17/07/19 08:46:22 stdout writer for Python/bin/python INFO PythonRunner: Times: total = 195, boot = 2, init = 58, finish = 135 2017-07-19 08:46:26.153934: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 779008aae0a739ec with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:46:26,165 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:46:56.176294: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session dae6ba2839f4045a with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:46:56,187 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:47:26.219498: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 0d5299827c404a8d with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:47:26,230 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:47:56.249074: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session ca926f7cc4bad694 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:47:56,259 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:48:26.289086: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session d20fb050700204b7 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:48:26,300 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:48:56.336018: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 25fe00763f21967e with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:48:56,346 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:49:26.384845: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 93a84a0ecc355760 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:49:26,395 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:49:56.406491: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 8ce3272cb5e16960 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:49:56,417 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:50:26.456076: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session d94b60bab60a60fd with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:50:26,466 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:50:56.505748: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 875799497f7e7a44 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:50:56,516 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:51:26.540140: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session f1eb41ba54b6c9c4 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:51:26,551 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:51:56.566035: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 784e991cfbff3184 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:51:56,576 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:52:26.613374: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 995c148acc93a283 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:52:26,624 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:52:56.649742: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 290bfdc22d4d8d69 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:52:56,660 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:53:26.699637: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 5b6deff7308a1f98 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:53:26,710 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:53:56.737786: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 6a3b21d73be1fbc1 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:53:56,748 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:54:26.763732: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 0aad9989adde5a50 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:54:26,774 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:54:56.813762: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 5211997795692958 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:54:56,824 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:55:26.856791: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 45034a2bed4e8e1b with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:55:26,867 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:55:56.906564: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session f54e3a6af96b97ed with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:55:56,917 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:56:26.949444: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 34d4818ad66d45f9 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:56:26,960 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:56:56.985064: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session a28d75ef9f4e7b9f with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:56:56,995 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:57:27.034866: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session ed1d13a5ec39c62b with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:57:27,045 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:57:57.058529: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 1ee8d6a1a2d71ea4 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:57:57,069 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:58:27.101392: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 6d07a9e54e007709 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:58:27,112 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:58:57.134116: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 757b095d99db3831 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:58:57,144 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:59:27.183710: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session a6b0cac6172f8f62 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:59:27,194 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:59:57.228241: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session d761bc556b07c8ef with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 08:59:57,238 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 09:00:27.276457: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session a904a2cb8047edd9 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 09:00:27,287 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 09:00:57.308670: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 2ab09dfe47bded19 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 09:00:57,319 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 09:01:27.336428: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 7400797690f2b4e5 with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 09:01:27,347 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 09:01:57.358818: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 2c01837aa43ee14e with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 09:01:57,370 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 09:02:27.408167: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session f18dc62030fbe0ce with config: INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad 2017-07-19 09:02:27,418 INFO (MainThread-14561) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad