Combining conda environment.yml with pip requirements.txt

python pip anaconda conda

I work with conda environments and need some pip packages as well, e.g. pre-compiled wheels from ~gohlke.

At the moment I have two files: environment.yml for conda with:

# run: conda env create --file environment.yml
name: test-env
dependencies:
- python>=3.5
- anaconda

and requirements.txt for pip which can be used after activating above conda environment:

# run: pip install -i requirements.txt
docx
gooey
http://www.lfd.uci.edu/~gohlke/pythonlibs/bofhrmxk/opencv_python-3.1.0-cp35-none-win_amd64.whl

Is there a possibility to combine them in one file (for conda)?

There is a GitHub pull request for using requirements.txt in conda. github.com/conda/conda-env/pull/172

conda env export will generate the text for a .yml based on the currently installed packages.

from inspecting my env.yml file it seems that conda env export > environment.yml already includes the pip packages I need. That's great!

Steven C. Howell

Pip dependencies can be included in the environment.yml file like this (docs):

# run: conda env create --file environment.yml
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- numpy=1.13.3  # pin version for conda
- pip:
  # works for regular pip packages
  - docx
  - gooey
  - matplotlib==2.0.0  # pin version for pip
  # and for wheels
  - http://www.lfd.uci.edu/~gohlke/pythonlibs/bofhrmxk/opencv_python-3.1.0-cp35-none-win_amd64.whl

It also works for .whl files in the same directory (see Dengar's answer) as well as with common pip packages.

It did well! conda env create -f environment.yml, and environment.yml file has to be in the current directory, otherwise, it will be thrown oddly errors.

is there a way to generate the yaml file automatically? e.g. I used to do pip freeze > requirements4pip.txt and conda list --export > requirements4conda.txt

from inspecting my env.yml file it seems that conda env export > environment.yml already includes the pip packages I need. That's great!

@bastelflp What happens with the dependencies of these items (docx, gooey)? Is there an implied --no-deps here for pip, with the dependencies picked up by pip or is there the usual danger that pip will try to assume the provision of dependencies?

conda env create -f environment.yml does not include pip packages installed using local code such as pip install -e ., but the requirements.txt file generated using pip has these packages included. So I think the best option is to use -r requirements.txt in the yaml file.

merv

One can also use the requirements.txt directly in the YAML. For example,

name: test-env
dependencies:
  - python>=3.5
  - anaconda
  - pip
  - pip:
    - -r requirements.txt

Basically, any option you can run with pip install you can run in a YAML. See the Advanced Pip Example for a showcase of other capabilities.

Important Note

A previous version of this answer (and Conda's Advanced Pip Example) used a substandard file URI syntax:

    - -r file:requirements.txt

Pip v21.2.1 introduced stricter behavior for URI parsing and no longer supports this. See this answer for details.

What does the -r mean? Looking at pip's command line options (here) there isn't a -r so it would be useful to know where that's from (and if there is a -- long-form equivalent)

Found it, I should have looked at the pip install options here. -r is for the requirement file and the longer equivalent is --requirement

I tried combining - -r requirements.txt -e . on one line, but it seemed not to work. Using two lines, - -r requirements.txt followed by - -e . seems to work.

is there a way to create all the requirement for my project (that uses pip and conda) automatically? Right now I infer from your answer the best way to do this (manually) is to first create the conda environment file witn conda env export > environment.yml or conda env create --file environment.yml and then add a pip line like you did pointing to the requirements.txt file I create with conda. Right? Is there a way to do this all with 1 command or in the command line like conda create_all_env_from_pip_and_conda envfile.yml or something?

@merv done. Just for completness I post the link here ;) stackoverflow.com/questions/68571543/…

Dengar

Just want to add that adding a wheel in the directory also works. I was getting this error when using the entire URL:

HTTP error 404 while getting http://www.lfd.uci.edu/~gohlke/pythonlibs/f9r7rmd8/opencv_python-3.1.0-cp35-none-win_amd64.whl

Ended up downloading the wheel and saving it into the same directory as the yml file.

name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- pip:
  - opencv_python-3.1.0-cp35-none-win_amd64.whl

I mentioned this in the answer above.

Charlie Parker

If you want to do it automatically it seems that if you do:

conda env export > environment.yml`

already has the pip things you need. No need to run pip freeze > requirements4pip.txt separately for me or include it as an

  - pip:
    - -r file:requirements.txt

as another answer mentioned.

See my yml file:

$ cat environment.yml
name: myenv
channels:
  - pytorch
  - dglteam
  - defaults
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.12.0=py38h06a4308_0
  - aiohttp=3.7.4=py38h27cfd23_1
  - async-timeout=3.0.1=py38h06a4308_0
  - attrs=20.3.0=pyhd3eb1b0_0
  - beautifulsoup4=4.9.3=pyha847dfd_0
  - blas=1.0=mkl
  - blinker=1.4=py38h06a4308_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.4.13=h06a4308_1
  - cachetools=4.2.1=pyhd3eb1b0_0
  - cairo=1.14.12=h8948797_3
  - certifi=2020.12.5=py38h06a4308_0
  - cffi=1.14.0=py38h2e261b9_0
  - chardet=3.0.4=py38h06a4308_1003
  - click=7.1.2=pyhd3eb1b0_0
  - conda=4.10.1=py38h06a4308_1
  - conda-build=3.21.4=py38h06a4308_0
  - conda-package-handling=1.7.3=py38h27cfd23_1
  - coverage=5.5=py38h27cfd23_2
  - cryptography=3.4.7=py38hd23ed53_0
  - cudatoolkit=11.0.221=h6bb024c_0
  - cycler=0.10.0=py38_0
  - cython=0.29.23=py38h2531618_0
  - dbus=1.13.18=hb2f20db_0
  - decorator=4.4.2=pyhd3eb1b0_0
  - dgl-cuda11.0=0.6.1=py38_0
  - dill=0.3.3=pyhd3eb1b0_0
  - expat=2.3.0=h2531618_2
  - filelock=3.0.12=pyhd3eb1b0_1
  - fontconfig=2.13.1=h6c09931_0
  - freetype=2.10.4=h7ca028e_0
  - fribidi=1.0.10=h7b6447c_0
  - gettext=0.21.0=hf68c758_0
  - glib=2.66.3=h58526e2_0
  - glob2=0.7=pyhd3eb1b0_0
  - google-auth=1.29.0=pyhd3eb1b0_0
  - google-auth-oauthlib=0.4.4=pyhd3eb1b0_0
  - graphite2=1.3.14=h23475e2_0
  - graphviz=2.40.1=h21bd128_2
  - grpcio=1.36.1=py38h2157cd5_1
  - gst-plugins-base=1.14.0=h8213a91_2
  - gstreamer=1.14.0=h28cd5cc_2
  - harfbuzz=1.8.8=hffaf4a1_0
  - icu=58.2=he6710b0_3
  - idna=2.10=pyhd3eb1b0_0
  - importlib-metadata=3.10.0=py38h06a4308_0
  - intel-openmp=2021.2.0=h06a4308_610
  - jinja2=2.11.3=pyhd3eb1b0_0
  - joblib=1.0.1=pyhd3eb1b0_0
  - jpeg=9b=h024ee3a_2
  - kiwisolver=1.3.1=py38h2531618_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libarchive=3.4.2=h62408e4_0
  - libffi=3.2.1=hf484d3e_1007
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libglib=2.66.3=hbe7bbb4_0
  - libiconv=1.16=h516909a_0
  - liblief=0.10.1=he6710b0_0
  - libpng=1.6.37=h21135ba_2
  - libprotobuf=3.14.0=h8c45485_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_1
  - libuuid=1.0.3=h1bed415_2
  - libuv=1.40.0=h7b6447c_0
  - libxcb=1.14=h7b6447c_0
  - libxml2=2.9.10=hb55368b_3
  - lz4-c=1.9.2=he1b5a44_3
  - markdown=3.3.4=py38h06a4308_0
  - markupsafe=1.1.1=py38h7b6447c_0
  - matplotlib=3.3.4=py38h06a4308_0
  - matplotlib-base=3.3.4=py38h62a2d02_0
  - mkl=2020.2=256
  - mkl-service=2.3.0=py38h1e0a361_2
  - mkl_fft=1.3.0=py38h54f3939_0
  - mkl_random=1.2.0=py38hc5bc63f_1
  - multidict=5.1.0=py38h27cfd23_2
  - ncurses=6.2=he6710b0_1
  - networkx=2.5.1=pyhd3eb1b0_0
  - ninja=1.10.2=hff7bd54_1
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=pyh9f0ad1d_1
  - openssl=1.1.1k=h27cfd23_0
  - pandas=1.2.4=py38h2531618_0
  - pango=1.42.4=h049681c_0
  - patchelf=0.12=h2531618_1
  - pcre=8.44=he6710b0_0
  - pillow=8.2.0=py38he98fc37_0
  - pip=21.0.1=py38h06a4308_0
  - pixman=0.40.0=h7b6447c_0
  - pkginfo=1.7.0=py38h06a4308_0
  - protobuf=3.14.0=py38h2531618_1
  - psutil=5.8.0=py38h27cfd23_1
  - py-lief=0.10.1=py38h403a769_0
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycosat=0.6.3=py38h7b6447c_1
  - pycparser=2.20=py_2
  - pyjwt=2.0.1=pyhd8ed1ab_1
  - pyopenssl=20.0.1=pyhd3eb1b0_1
  - pyparsing=2.4.7=pyhd3eb1b0_0
  - pyqt=5.9.2=py38h05f1152_4
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.2=hcf32534_0
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - python-libarchive-c=2.9=pyhd3eb1b0_1
  - python_abi=3.8=1_cp38
  - pytorch=1.7.1=py3.8_cuda11.0.221_cudnn8.0.5_0
  - pytz=2021.1=pyhd3eb1b0_0
  - pyyaml=5.4.1=py38h27cfd23_1
  - qt=5.9.7=h5867ecd_1
  - readline=8.1=h27cfd23_0
  - requests=2.25.1=pyhd3eb1b0_0
  - requests-oauthlib=1.3.0=py_0
  - ripgrep=12.1.1=0
  - rsa=4.7.2=pyhd3eb1b0_1
  - ruamel_yaml=0.15.100=py38h27cfd23_0
  - scikit-learn=0.24.1=py38ha9443f7_0
  - scipy=1.6.2=py38h91f5cce_0
  - setuptools=52.0.0=py38h06a4308_0
  - sip=4.19.13=py38he6710b0_0
  - six=1.15.0=pyh9f0ad1d_0
  - soupsieve=2.2.1=pyhd3eb1b0_0
  - sqlite=3.35.4=hdfb4753_0
  - tensorboard=2.4.0=pyhc547734_0
  - tensorboard-plugin-wit=1.6.0=py_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tk=8.6.10=hbc83047_0
  - torchaudio=0.7.2=py38
  - torchtext=0.8.1=py38
  - torchvision=0.8.2=py38_cu110
  - tornado=6.1=py38h27cfd23_0
  - typing-extensions=3.7.4.3=0
  - typing_extensions=3.7.4.3=py_0
  - urllib3=1.26.4=pyhd3eb1b0_0
  - werkzeug=1.0.1=pyhd3eb1b0_0
  - wheel=0.36.2=pyhd3eb1b0_0
  - xz=5.2.5=h7b6447c_0
  - yaml=0.2.5=h7b6447c_0
  - yarl=1.6.3=py38h27cfd23_0
  - zipp=3.4.1=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
  - pip:
    - aioconsole==0.3.1
    - lark-parser==0.6.5
    - lmdb==0.94
    - pexpect==4.6.0
    - progressbar2==3.39.3
    - ptyprocess==0.7.0
    - pycapnp==1.0.0
    - python-utils==2.5.6
    - sexpdata==0.0.3
    - tqdm==4.56.0
prefix: /home/miranda9/miniconda3/envs/myenv

note that at the time of this writing doing conda env create --file environment.yml to create the yml env results in an error:

$ conda env create --file environment.yml

CondaValueError: prefix already exists: /home/miranda9/miniconda3/envs/myenv

Combining conda environment.yml with pip requirements.txt

Follow WeChat

Want to stay one step ahead of the latest teleworks?

相似问题

Platform

Support

Contact US