Databases
StackOverflow2013
Postsdbo
POPyPI is slow. How do I run my own server?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPyPI is slow. How do I run my own server?
    primarykey
    data
    text
    <p>When a new developer joins the team, or Jenkins runs a complete build, I need to create a fresh virtualenv. I often find that setting up a virtualenv with Pip and a large number (more than 10) of requirements takes a very long time to install everything from PyPI. Often it fails altogether with:</p> <pre><code>Downloading/unpacking Django==1.4.5 (from -r requirements.pip (line 1)) Exception: Traceback (most recent call last): File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/basecommand.py", line 107, in main status = self.run(options, args) File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/commands/install.py", line 256, in run requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/req.py", line 1018, in prepare_files self.unpack_url(url, location, self.is_download) File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/req.py", line 1142, in unpack_url retval = unpack_http_url(link, location, self.download_cache, self.download_dir) File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/download.py", line 463, in unpack_http_url download_hash = _download_url(resp, link, temp_location) File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/download.py", line 380, in _download_url chunk = resp.read(4096) File "/usr/lib64/python2.6/socket.py", line 353, in read data = self._sock.recv(left) File "/usr/lib64/python2.6/httplib.py", line 538, in read s = self.fp.read(amt) File "/usr/lib64/python2.6/socket.py", line 353, in read data = self._sock.recv(left) timeout: timed out </code></pre> <p>I'm aware of Pip's <code>--use-mirrors</code> flag, and sometimes people on my team have worked around by using <code>--index-url http://f.pypi.python.org/simple</code> (or another mirror) until they have a mirror that responds in a timely fashion. We're in the UK, but there's a PyPI mirror in Germany, and we don't have issues downloading data from other sites.</p> <p>So, I'm looking at ways to mirror PyPI internally for our team.</p> <p>The options I've looked at are:</p> <ol> <li><p>Running my own PyPI instance. There's the official PyPI implementation: <a href="https://bitbucket.org/loewis/pypi" rel="nofollow noreferrer">CheeseShop</a> as well as several third party implementations, such as: <a href="https://github.com/benliles/djangopypi" rel="nofollow noreferrer">djangopypi</a> and <a href="https://github.com/schmir/pypiserver" rel="nofollow noreferrer">pypiserver</a> (see footnote)</p> <p>The problem with this approach is that I'm not interested in full PyPI functionality with file upload, I just want to mirror the content it provides.</p></li> <li><p>Running a PyPI mirror with <a href="https://bitbucket.org/loewis/pep381client/" rel="nofollow noreferrer">pep381client</a> or <a href="https://github.com/mviera/pypi-mirror" rel="nofollow noreferrer">pypi-mirror</a>.</p> <p>This looks like it could work, but it requires my mirror to download everything from PyPI first. I've set up a test instance of pep381client, but my download speed varies between 5 Kb/s and 200 Kb/s (bits, not bytes). Unless there's a copy of the full PyPI archive somewhere, it will take me weeks to have a useful mirror.</p></li> <li><p>Using a PyPI round-robin proxy such as <a href="https://code.google.com/p/yopypi/" rel="nofollow noreferrer">yopypi</a>. </p> <p>This is irrelevant now that <a href="http://pypi.python.org" rel="nofollow noreferrer">http://pypi.python.org</a> itself consists of <a href="http://www.pypi-mirrors.org/" rel="nofollow noreferrer">several geographically distinct servers</a>.</p></li> <li><p>Copying around a virtualenv between developers, or hosting a <a href="http://honza.ca/2012/02/how-not-to-depend-on-pypi" rel="nofollow noreferrer">folder of the current project's dependencies</a>.</p> <p>This doesn't scale: we have several different Python projects whose dependencies change (slowly) over time. As soon as the dependencies of any project change, this central folder must be updated to add the new dependencies. Copying the virtualenv is worse than copying the packages though, since any Python packages with C modules need to be compiled for the target system. Our team has both Linux and OS X users.</p> <p>(This still looks like the best option of a bad bunch.)</p></li> <li><p>Using an intelligent PyPI caching proxy: <a href="https://github.com/camptocamp/collective.eggproxy" rel="nofollow noreferrer">collective.eggproxy</a></p> <p>This seems like it would be a very good solution, but <a href="https://pypi.python.org/pypi/collective.eggproxy" rel="nofollow noreferrer">the last version on PyPI is dated 2009</a> and discusses mod_python.</p></li> </ol> <p>What do other large Python teams do? What's the best solution to quickly install the same set of python packages?</p> <p>Footnotes:</p> <ul> <li>I've seen the question <a href="https://stackoverflow.com/questions/1235331/how-to-roll-my-own-pypi/12874682#12874682">How to roll my own PyPI?</a>, but that question relates to hosting private code.</li> <li>The Python wiki <a href="http://wiki.python.org/moin/PyPiImplementations" rel="nofollow noreferrer">lists alternative PyPI implementations</a></li> <li>I've also recently discovered <a href="https://crate.io/" rel="nofollow noreferrer">Crate.io</a> but I don't believe that helps me when using Pip.</li> <li>There's a website monitoring <a href="http://www.pypi-mirrors.org/" rel="nofollow noreferrer">PyPI mirror status</a></li> <li><a href="http://reinout.vanrees.org/weblog/2013/02/28/pypi-discussion.html" rel="nofollow noreferrer">Some packages on PyPI have their files hosted elsewhere</a> so even a perfect mirror won't help all dependencies</li> </ul>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USWilfred Hughes
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPyPI is slow. How do I run my own server?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POPyPI is slow. How do I run my own server?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POPyPI is slow. How do I run my own server?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COPersonally when I was in a situation where I needed to script the (repeated) creation of a virtualenv, I just built .egg files for the packages I wanted and checked them into the repo along with the script to create the virtualenv. I don't claim to be part of large python team, however.
      singulars
      1. POPyPI is slow. How do I run my own server?
      1. USFatalError
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload