Django IOError & associated meltdown
Info
Django will randomly start crashing with this error, and I'll need to restart the server. It can happen when Django is trying to open the database, when I refresh the page, when an AJAX call is made, etc. It sounds like there's some file descriptor leak (some code not properly closing file descriptors), but I don't know.
I thought I fixed it with ulimit -n 2048 (increasing the file descriptor limit to 2048), but recently it's been happening again. There's no reliable way to reproduce it, or reliable timing - I may not have it for a few days, or get it several times an hour. It's happened over the last couple months (started maybe around Christmas break? Don't remember exactly).
I develop on my own laptop on Arch Linux.
Raw data
Django Error page: django_io_error.html
Traceback 1:
Traceback (most recent call last):
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 64, in __call__
return super(StaticFilesHandler, self).__call__(environ, start_response)
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 177, in __call__
response = self.get_response(request)
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 54, in get_response
return self.serve(request)
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 47, in serve
return serve(request, self.file_path(request.path), insecure=True)
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/staticfiles/views.py", line 40, in serve
return static.serve(request, path, document_root=document_root, **kwargs)
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/views/static.py", line 68, in serve
response = FileResponse(open(fullpath, 'rb'), content_type=content_type)
IOError: [Errno 24] Too many open files: u'/home/blackcap/Code/beat.web/beat/web/reports/static/reports/app/data/table_itemcontent.json'
[23/Feb/2017 15:10:37] "GET /static/reports/app/data/table_itemcontent.json?v1.1.5 HTTP/1.1" 500 59Traceback 2 (right after 1):
Internal Server Error: /api/v1/plotters/
Traceback (most recent call last):
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 123, in get_response
File "/home/blackcap/Code/beat.web/beat/web/navigation/middleware.py", line 62, in process_request
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/utils/functional.py", line 204, in inner
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/utils/functional.py", line 351, in _setup
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/auth/middleware.py", line 22, in <lambda>
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/auth/middleware.py", line 10, in get_user
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 167, in get_user
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 59, in _get_user_session_key
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py", line 48, in __getitem__
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py", line 198, in _get_session
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/contrib/sessions/backends/cached_db.py", line 45, in load
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/models/manager.py", line 122, in manager_method
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/models/query.py", line 381, in get
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/models/query.py", line 240, in __len__
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/models/query.py", line 52, in __iter__
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 846, in execute_sql
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/backends/base/base.py", line 231, in cursor
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/backends/base/base.py", line 204, in _cursor
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/utils.py", line 95, in __exit__
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/backends/base/base.py", line 171, in connect
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 207, in get_new_connection
OperationalError: unable to open database file
[23/Feb/2017 15:10:38] "GET /api/v1/plotters/ HTTP/1.1" 500 64611Traceback 3:
Internal Server Error: /reports/user/test/
Traceback (most recent call last):
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 149, in get_response
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response
File "/home/blackcap/Code/beat.web/beat/web/reports/views.py", line 81, in for_author
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/shortcuts.py", line 45, in render_to_response
using=using)
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/template/loader.py", line 114, in render_to_string
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/template/engine.py", line 231, in render_to_string
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/template/engine.py", line 190, in get_template
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/template/engine.py", line 157, in find_template
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 40, in get_template
File "/home/blackcap/Code/beat.env.web/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 24, in get_contents
IOError: [Errno 24] Too many open files: '/home/blackcap/Code/beat.web/beat/web/templates/reports/report.html'
[23/Feb/2017 15:10:52] "GET /reports/user/test/ HTTP/1.1" 500 64762