Compiling the NFINDR cython version on WindowsΒΆ

The endmembers extraction algorithm NFINDR exist in two different versions. The first version is a pure Python implementation. It leave in the _nfindr.py file. The second one is a cython implementation, it leave in the _nfindr.pyx file and is compiled as _nfindr.pyd. Both have the same class interface defined in the file umix.py.

There is tree main differences between these versions. The first one is speed, on my data set the cython version is 2x faster than the pure python version. The second one is the use of float64 by the cython version for the volume expansion calculation giving a more precise result. The pure python version use float32.

But it is at the implementation that they differ the most. As usual, the pure python version is to be run everywhere. And, the goal of the cython version is speed improvement. The NFINDR algorithm is a good candidate for speed improvement. We can attack this problem by tree fronts: memory management, vector operations with the sse2 SIMD slates on an Intel processor and use a parallelism inherent to the NFINDR algorithm.

For now, a tight memory management is implemented. The idea is to avoid the frenzy buffers creation and deletion associated to the lapack determinant function call. The solution is to reuse the memory allocated here. And another thing that I wanted to totally avoid is a patch to the scipy librairy to allow the reallocation of the buffers. Well, I rewrote the determinant and LU decomposition from scratch in cython and in a way that give the chance to reallocate the memory. It give a near 2x speedup when we compare to a direct lapack call (I used numpy-MKL for the benchmarks). Lapack is not very good at small matrix and a zillion calls.


Following is a step by step tutorial on how to compile NFINDR on Windows 7 (it may be right for vista and Windows 8).


Installing the C/C++ compiler on Windows.

I use Python 2.7 on windows, the 32 bits and 64 bits versions. The very first step is to install the C/C++ compiler. The official Python 2.7 was compiled with MSVC 2008. This is the version of the C/C++ compiler to install for sure if we want things to work. A good overview is at the page http://wiki.cython.org/64BitCythonExtensionsOnWindows. It is the information that I have used. Note that this installation can build binaries for 32 and 64 bits modules extension.

You have to install two packages. Here some comments that can be helpful:

  • To find the “Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1” start Google and type: “GRMSDK_EN_DVD.iso” and not GRMSDKX_EN_DVD.iso. With chance, you can grab the first link and go to the good Microsoft download page. The name of the file is truly GRMSDK_EN_DVD.iso; take this one.

  • I am not sure if MSVC 2008 needs to be installed or not. It is on my computer. I have done the setup last fall and I do not remember if it is mandatory. To find it Google “Visual C++ Express Edition 2008” and take some main software distribution sites, like www.01net.com.

  • About finding this two packages an interesting post on stackoverflow is: http://stackoverflow.com/questions/15318560/visual-c-2008-express-download-link-dead

  • Don’t install the MSVC 2008 with SP1

  • Set the variable DISTUTILS_USE_SDK to 1

  • To open the Windows SDK Command Prompt window: Click Start, point to All Programs, point to Microsoft Windows SDK , and then click SDK Command Prompt or CMD Shell.

  • If you want to compile a 32-bits extension, with the yellow text command prompt, go to the folder where the cython code is and execute the standard command to compile your Cython code (python setup.py build_ext –inplace). Note that you can use a vanilla command prompt here.

  • If you want to compile a 64-bits extension, execute in the yellow text command prompt the two lines:

    set DISTUTILS_USE_SDK=1

    setenv /x64 /release

Avoid the first line if you have permanently set DISTUTILS_USE_SDK

The text turn to green and you can proceed to the compilation.

I tested Cython+openmp on this setup and it work fine with both 32 and 64 bits.

And finally!

The last step is to compile the _nfindr.pyx file.

cd to the directory where _nfindr.pyx live and call setu_nfindr.py like this: python setup_nfindr.py build_ext –inplace

You get the _nfindr.pyd file.