URLError массовой загрузки данных в App Engine: «Сброс соединения по пиру»

Question

URLError массовой загрузки данных в App Engine: «Сброс соединения по пиру»

1

Я пытаюсь загрузить некоторые записи в локальное хранилище данных с помощью appcfg.py

На самом деле вставляется только небольшое количество записей, и я получаю следующий вывод (с тонны ошибок):

       $ appcfg.py upload_data --config_file=bulkloader.yaml --filename=/output.csv --kind=AutoCompleteIndex --url=http://localhost:8084/remote_api .
        Application: theapp1; version: dev.
        Uploading data records.
        [INFO    ] Logging to bulkloader-log-20100808.170223
        [INFO    ] Throttling transfers:
        [INFO    ] Bandwidth: 250000 bytes/second
        [INFO    ] HTTP connections: 8/second
        [INFO    ] Entities inserted/fetched/modified: 20/second
        [INFO    ] Batch Size: 10
        [INFO    ] Opening database: bulkloader-progress-20100808.170223.sql3
        Please enter login credentials for localhost
        Email: kjuyg
        Password for kjuyg: 
        [INFO    ] Connecting to localhost:8084/remote_api
        [INFO    ] Starting import; maximum 10 entities per post
        .......................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        ................................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .............[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        ................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .......................................................................................................................................................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put

.... etc .....
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C..[INFO    ] An error occurred. Shutting down...
        ..........
        [INFO    ] 19580 entites total, 0 previously transferred
        [INFO    ] 18570 entities (3593152 bytes) transferred in 225.1 seconds
        [INFO    ] Some entities not successfully transferred

Любая идея, что может быть проблемой?

Matthew H 08 авг. 2010, в 18:56

Источник

Теги:

python

google-app-engine

google-cloud-datastore

bigtable

urlfetch

2 ответа

2

У меня была такая же проблема на моем Mac. Я решил это, создав прокси, поместив следующее в мой httpd-vhost.conf файл (в etc/apache2/extra/):

<Location /remote_api>
  ProxyPass http://localhost:8888/remote_api
</Location>

если ваш тестовый сервер работает на порту 8888.

Затем я запускаю:

sudo apachectl restart

чтобы перезапустить сервер Apache.

Теперь эта команда ( "Обзор" является одним из элементов данных):

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/appcfg.py upload_data /
--kind=Survey --file=Survey.csv /
--url=http://localhost/remote_api [email protected] /
--config_file=config.yml

дает этот результат:

[INFO    ] Logging to bulkloader-log-20121114.100837
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Password for [email protected]: 
[INFO    ] Opening database: bulkloader-progress-20121114.100837.sql3
[INFO    ] Connecting to localhost/remote_api
[INFO    ] Starting import; maximum 10 entities per post
....
[INFO    ] 36 entities total, 0 previously transferred
[INFO    ] 36 entities (15402 bytes) transferred in 1.7 seconds
[INFO    ] All entities successfully transferred

Как и должно быть.

MTWestra 14 нояб. 2012, в 07:49

Ещё вопросы

Travis Webb · Accepted Answer · 2010-08-09T02-01-00.000Z

Посмотрите на журналы своего сервера. Вы получаете превышение размера кучи или исключение OutOfMemmory? Это произойдет на локальном dev-сервере, если ваша загрузка большой, потому что сервер dev хранит хранилище данных в основной памяти. Сервер немного повиснет, а затем вы получите соединение reset. У меня на сервере dev закончилась нехватка памяти всего за 10 Мбайт данных хранилища данных, так как для ее создания и хранения метаданных используется много дополнительной ОЗУ. Вы можете увеличить размер кучи с помощью переключателя -Xmx.

Куда нам обратиться, чтобы установить ключ -Xmx?