Scrapy Error On Cryptography Under MacOS

Possible Problem When Install Scrapy Using Pip

In MacOS, if Scrapy package is installed using pip, the following warning maybe generated when trying to run scrapy:

cffi.ffiplatform.VerificationError: importing '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/cryptography/': dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/cryptography/, 2): Symbol not found: _CRYPTO_malloc_debug_init

Solution 1

This is because cryptography package needs pyOpenSSL. If the openssl is installed using MacPort, the cryptography package need to have correct location flag for openssl library:

1 sudo port install openssl
2 sudo env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/opt/local/lib" CFLAGS="-I/opt/local/include/openssl" 

Now, reinstall the cryptography package using pip:

1 sudo pip uninstall cryptography
2 sudo pip install --upgrade cryptography

Solution 2

Another simple way to solve this problem is to install cryptography package using MacPort as well:

1 sudo port install openssl
2 // The original cryptography and its dependencies need to uninstall using pip
3 sudo port install py27-cryptography