Python issues using numpy and h5py

General topics about Trelis/Cubit, such as localization, GUI, etc.

Python issues using numpy and h5py

Postby Fabian » Thu Mar 16, 2017 11:06 am

Hello to the list,
we just bought Trelis and have some issues with Python:

I want to use numpy and h5py (Linux)

According to sys.version Trelis comes with Python 2.7.3.
I tried to link numpy and h5py from my /usr/lib64/python2.7/site-packages to
/opt/Trelis-16.3/bin/lib/python2.7/site-packages

Then
import numpy

Fails with an multiarray issue which seems to be caused by incompatible versions.
My local python2 is Python 2.7.13

I asked someone who is using Trelis with numpy and h5py about his numpy installation.
When I copy it to
/opt/Trelis-16.3/bin/lib/python2.7/site-packages/

I get
[code]import numpy
...
from . import multiarray
ImportError: /opt/Trelis-16.3/bin/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: _PyUnicodeUCS4_IsWhitespace[/quote]

The same error occured on another Linux system (Ubuntu, I use openSUSE)

What can I do?

I see the following options:
* make Trelis to use my System Python 2.7
* compile Python 2.7.3 manually and then build numpy manually. However this failed in the first attempt. And when I build Paraview 5.3 with Python I also have this Unicode issue.

So I prefer using System Python.

Thanks a lot,

Fabian
Fabian
 
Posts: 8
Joined: Wed Mar 15, 2017 4:24 am

Re: Python issues using numpy and h5py

Postby Fabian » Thu Mar 16, 2017 11:25 am

Another idea could be to install numpy based on the python provided by Trelis. But I don't know how to do this ...
Fabian
 
Posts: 8
Joined: Wed Mar 15, 2017 4:24 am

Re: Python issues using numpy and h5py

Postby Fabian » Thu Mar 16, 2017 2:52 pm

I put more work on the issue. I could life without h5py by adapting a python mesh exporter we already have, however it would be very annoying to have to life without numpy.

Numpy is build via python, so I assume I have to build numpy with a python 2.7.3 interpreter.

I really could not find a python interpreter binary in the Trelis distribution (Linux rpm) I could use manually.

I failed to build python 2.7.3 locally. build and install works but when I execute the binary I get
./python
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Traceback (most recent call last):
File "/home/fwein/tmp/lib/python2.7/site.py", line 563, in <module>
main()
File "/home/fwein/tmp/lib/python2.7/site.py", line 545, in main
known_paths = addusersitepackages(known_paths)
File "/home/fwein/tmp/lib/python2.7/site.py", line 278, in addusersitepackages
user_site = getusersitepackages()
File "/home/fwein/tmp/lib/python2.7/site.py", line 253, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/home/fwein/tmp/lib/python2.7/site.py", line 243, in getuserbase
USER_BASE = get_config_var('userbase')
File "/home/fwein/tmp/lib/python2.7/sysconfig.py", line 521, in get_config_var
return get_config_vars().get(name)
File "/home/fwein/tmp/lib/python2.7/sysconfig.py", line 420, in get_config_vars
_init_posix(_CONFIG_VARS)
File "/home/fwein/tmp/lib/python2.7/sysconfig.py", line 288, in _init_posix
raise IOError(msg)
IOError: invalid Python installation: unable to open /home/fwein/tmp/lib/python2.7/config/Makefile (No such file or directory)

But even building numpy with my local python failed for me.

Is there any change that Trelis simply extends its Python to numpy? That would really be the best! :))
Fabian
 
Posts: 8
Joined: Wed Mar 15, 2017 4:24 am

Re: Python issues using numpy and h5py

Postby csimsoft Moderator » Fri Mar 17, 2017 9:57 am

Based on the error, I suspect that our version of Python and your version of Python were built using different versions of Unicode. Here is one way to check in Python 2.7 which version of Unicode it was built with:

http://stackoverflow.com/questions/1446 ... 2-or-ucs-4

Also, from the Python docs (https://docs.python.org/2/c-api/unicode.html) it says "Note that UCS2 and UCS4 Python builds are not binary compatible", which would explain why you can't import numpy into our Python.

We are reviewing the license for numpy for possibly adding it to our package.
csimsoft
csimsoft Moderator
 
Posts: 246
Joined: Tue Feb 14, 2012 5:59 pm
Location: American Fork, UT

Re: Python issues using numpy and h5py

Postby Fabian » Fri Mar 17, 2017 10:43 am

There are two issues: When I link /usr/lib64/python2.7/site-packages/numpy
for my Python 2.7.13 I get the error

import numpy
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/Trelis-16.3/bin/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module>
from . import add_newdocs
File "/opt/Trelis-16.3/bin/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "/opt/Trelis-16.3/bin/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>
from .type_check import *
File "/opt/Trelis-16.3/bin/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "/opt/Trelis-16.3/bin/lib/python2.7/site-packages/numpy/core/__init__.py", line 24, in <module>
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.

As as I understand this is because Trelis' Python is too old.
print(sys.version)
2.7.3 (default, Dec 6 2012, 13:01:46)
[GCC 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)]

But you are right, even with equal version I would have been bitten by the unicode (thanks for the link!)
Trelis: print(sys.maxunicode) -> 65535
my system: print(sys.maxunicode) -> 1114111
Whyever 2^16 is not enough and one wants more???

So adding numpy with Trelis would be really nice. BTW, Paraview adds numpy.
Fabian
 
Posts: 8
Joined: Wed Mar 15, 2017 4:24 am

Re: Python issues using numpy and h5py

Postby csimsoft Moderator » Fri Mar 17, 2017 1:54 pm

Here is something else you can try.

I did an experiment and removed python from my Trelis installation so that it would fall back to the system python. That is, I removed bin/libpython* and bin/lib/. With that change, Trelis starts up fine, which tells me that Trelis is not sensitive to the ucs2 vs. ucs4 difference.

I tried alternatives to force Trelis to use the system python, without modifying the Trelis installation, but ran into difficulty with that. I assume its because of something we do which sets PYTHONHOME and other things.

It looks like we will add numpy to Trelis. Thanks for your comments.
csimsoft
csimsoft Moderator
 
Posts: 246
Joined: Tue Feb 14, 2012 5:59 pm
Location: American Fork, UT

Re: Python issues using numpy and h5py

Postby Fabian » Fri Mar 17, 2017 3:51 pm

Thank you very much: For the outlook for adding numpy and for the idea how to force Trelis using system python.

Will you upgrade python and switch to 32bit unicode?
Fabian
 
Posts: 8
Joined: Wed Mar 15, 2017 4:24 am

Re: Python issues using numpy and h5py

Postby Fabian » Wed Mar 29, 2017 6:31 am

csimsoft Moderator wrote:I did an experiment and removed python from my Trelis installation so that it would fall back to the system python. That is, I removed bin/libpython* and bin/lib/. With that change, Trelis starts up fine, which tells me that Trelis is not sensitive to the ucs2 vs. ucs4 difference.

From what I understand, having no python by itself is the situation for the macOS version and it indeed works fine there (import numpy and import h5py without issues).
Fabian
 
Posts: 8
Joined: Wed Mar 15, 2017 4:24 am


Return to General Topics

Who is online

Users browsing this forum: No registered users and 1 guest

cron

Search / Mark

User Control Panel

Login

Who is online

In total there is 1 user online :: 0 registered, 0 hidden and 1 guest (based on users active over the past 5 minutes)
Most users ever online was 58 on Mon Jun 26, 2017 4:19 pm

Users browsing this forum: No registered users and 1 guest