Large array memory problem.

[Carolina]

Hi,

I am running IDL 5.5 in a computer that has 2GB of RAM and a paging
file size of 4GB. However it seems that IDL will not give more than
about 1.2GB for array storage per IDL session. Unfortunately, I need to
do an array multiplication where the arrays are double precision
complex 11000 by 11000 square arrays, and IDL will not even give me
enough memory for a single dcomplexarr(6500,6500) array!

Am I being too ambitious? Could anyone please tell me if it is possible
to store such a large array?

I though IDL was the best language for large array manipulations, is it
true? or are there other languages that are better suited for this?

Many thanks for your time,

Carolina.

[Paolo Grigis]

Is this on windows? There was a thread here dealing about memory
fragmentation etc. some time ago, if I remember correctly the
conclusion was that there is not so much one can do in windows,
but a few tricks could help a bit, and that linux manages the
memory better. But anyway with idl 6.0 or less there is no chance
to allocate a very large array because of the "array has too many
elements" limit anyway.

Can you allocate two double (real) array instead of a complex one?

Ciao,
Paolo

Post by Carolina
Hi,
I am running IDL 5.5 in a computer that has 2GB of RAM and a paging
file size of 4GB. However it seems that IDL will not give more than
about 1.2GB for array storage per IDL session. Unfortunately, I need to
do an array multiplication where the arrays are double precision
complex 11000 by 11000 square arrays, and IDL will not even give me
enough memory for a single dcomplexarr(6500,6500) array!
Am I being too ambitious? Could anyone please tell me if it is possible
to store such a large array?
I though IDL was the best language for large array manipulations, is it
true? or are there other languages that are better suited for this?
Many thanks for your time,
Carolina.

[Carolina]

Yes, I was using Windows. I remember seeing some articles in this
websites that mentioned how to get a couple of mb more of memory, but
that is definetly not good enough for what I want to do. I didn't think
it was related to memory fragmentation, since this problem is present
even with a new IDL session, without any other arrays defined
previously. I am currently working in a unix machine and I can now
access around 4gb of memory, and I can allocate 3 complex square arrays
with dimensions 11000 by 11000, as long as they aren't double precision
(these occupied 2.9gb all together). It wouldn't let me allocate 2
double precision ones (these would take about 4gb).

Thanks alot for your quick reply!

Carolina.

[Mark Hadfield]

Post by Carolina
Yes, I was using Windows. I remember seeing some articles in this
websites that mentioned how to get a couple of mb more of memory, but
that is definetly not good enough for what I want to do. I didn't think
it was related to memory fragmentation, since this problem is present
even with a new IDL session, without any other arrays defined
previously. I am currently working in a unix machine and I can now
access around 4gb of memory, and I can allocate 3 complex square arrays
with dimensions 11000 by 11000, as long as they aren't double precision
(these occupied 2.9gb all together). It wouldn't let me allocate 2
double precision ones (these would take about 4gb).

There is an IDL procedure called memtest (or mem_test) that will report
on the areas of *contiguous* memory available to IDL. You can get it here:

http://www.rsinc.com/services/techtip.asp?ttid=3441

You might might want to look here as well:

http://www.rsinc.com/services/techtip.asp?ttid=3512

On my system (Windows 2000 Service Pack 4, 1 GiB RAM, a few GiB of swap
space), when IDL is freshly started, memtest reports the following

Memory block # 1: 1033 Mb (total: 1033 Mb)
Memory block # 2: 388 Mb (total: 1421 Mb)
Memory block # 3: 203 Mb (total: 1624 Mb)
Memory block # 4: 61 Mb (total: 1685 Mb)
Memory block # 5: 58 Mb (total: 1743 Mb)
Memory block # 6: 48 Mb (total: 1791 Mb)
Memory block # 7: 33 Mb (total: 1824 Mb)
Memory block # 8: 21 Mb (total: 1845 Mb)
Memory block # 9: 20 Mb (total: 1865 Mb)
Memory block #10: 17 Mb (total: 1882 Mb)

(where the "Mb"s are supposed to be "MB"s, ie megabytes rather than
megabits).

I doubt that you're goin gto do much better than this in Windows.

[Rick Towler]

On a related note, is anyone using IDL x86-64 for linux?

-Rick

Post by Carolina
Hi,
I am running IDL 5.5 in a computer that has 2GB of RAM and a paging
file size of 4GB. However it seems that IDL will not give more than
about 1.2GB for array storage per IDL session. Unfortunately, I need to
do an array multiplication where the arrays are double precision
complex 11000 by 11000 square arrays, and IDL will not even give me
enough memory for a single dcomplexarr(6500,6500) array!
Am I being too ambitious? Could anyone please tell me if it is possible
to store such a large array?
I though IDL was the best language for large array manipulations, is it
true? or are there other languages that are better suited for this?
Many thanks for your time,
Carolina.

[Greg Hennessy]

Post by Rick Towler
On a related note, is anyone using IDL x86-64 for linux?

Yes. I can hold the entire 2MASS catalog in memory. :)

[R.G. Stockwell]

Post by Rick Towler
On a related note, is anyone using IDL x86-64 for linux?
-Rick

Ours has just come online. I have not actually used it, but I
have snooped around.
Anything in particular you want to see?


Here are the results of memtest:
IDL> memtest
% Compiled module: MEMTEST.
Memory block # 1: 2047 Mb (total: 2047 Mb)
Memory block # 2: 2047 Mb (total: 4094 Mb)
Memory block # 3: 2047 Mb (total: 2045 Mb)
Memory block # 4: 2047 Mb (total: 4092 Mb)
Memory block # 5: 2047 Mb (total: 2043 Mb)
Memory block # 6: 2047 Mb (total: 4090 Mb)
Memory block # 7: 2047 Mb (total: 2041 Mb)
Memory block # 8: 2047 Mb (total: 4088 Mb)
Memory block # 9: 2047 Mb (total: 2039 Mb)
Memory block #10: 2047 Mb (total: 4086 Mb)


IDL> help,/memory
heap memory used: 1056207, max: 21465408521, gets: 743, frees:
381


Cheers,
bob

[Rick Towler]

"Rick Towler" wrote...

Post by Rick Towler
On a related note, is anyone using IDL x86-64 for linux?

Ours has just come online. I have not actually used it, but I
have snooped around.
Anything in particular you want to see?

A WinXP-64 port ;)

Actually I rarely have the need to address huge amounts of memory. I'm
just interested in seeing IDL move forward in this area as I know it is
a concern of colleagues and some in this newsgroup. With the death of
the Itanium near, Intel's dream of pushing IA-64 into the low end HPC
market is dead. Itanium has already killed PA-RISC and the beloved
Alpha. uSparc? That's comedy... Even if Fujitsu delivers it isn't an
especially compelling platform. Apple+PPC? We know where that
love-fest is headed. IBM can deliver some excitement with Power but it
will cost you (and that isn't counting the dozens of doughnuts you'll
need to bribe your IT department with to run AIX in their server room).
So those of us with modest budgets and large problems are left with
x86-64, warts and all, for the foreseeable future.

So I guess what I wanted to see was just what I am seeing. Interest in
the linux x86-64 version which will hopefully fuel a WinXP-64 port.

-Rick

[Nigel Wade]

Post by Rick Towler
On a related note, is anyone using IDL x86-64 for linux?
-Rick

We have it here. I've only just got the 6.2 license manager up and running, so
no-one has tested it except me as yet (I need to impose ulimits first).

IDL> bigarr=bindgen(1024,1024,1024,14)
IDL> help,bigarr,/memory
heap memory used: 15033176469, max: 15033176469, gets: 414, frees: 110

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14482 nmw 20 0 14.1g 11g 4776 R 43.2 73.7 0:24.93 idl

It takes a bit of CPU time to initialise 14Gigs of memory...

[Reimar Bauer]

Post by Rick Towler
On a related note, is anyone using IDL x86-64 for linux?
-Rick

Yes,
we have several dual opterons using it in 64 bit.

By one of the last security updates of the kernel I believe we can't start
idl with the -32 option on these machines.
But this doesn't matter. 64 bit is great.


cheers
Reimar

Post by Rick Towler

Post by Carolina
Hi,
I am running IDL 5.5 in a computer that has 2GB of RAM and a paging
file size of 4GB. However it seems that IDL will not give more than
about 1.2GB for array storage per IDL session. Unfortunately, I need to
do an array multiplication where the arrays are double precision
complex 11000 by 11000 square arrays, and IDL will not even give me
enough memory for a single dcomplexarr(6500,6500) array!
Am I being too ambitious? Could anyone please tell me if it is possible
to store such a large array?
I though IDL was the best language for large array manipulations, is it
true? or are there other languages that are better suited for this?
Many thanks for your time,
Carolina.

[Nigel Wade]

Post by Reimar Bauer

Post by Rick Towler
On a related note, is anyone using IDL x86-64 for linux?
-Rick

Yes,
we have several dual opterons using it in 64 bit.
By one of the last security updates of the kernel I believe we can't start
idl with the -32 option on these machines.
But this doesn't matter. 64 bit is great.
cheers
Reimar

Which kernel/security update?

We require 32bit IDL (for the time being, until some extensive external code is
made 64-bit safe).

[Marius Hagen]

Post by Carolina
Unfortunately, I need to
do an array multiplication where the arrays are double precision
complex 11000 by 11000 square arrays, and IDL will not even give me
enough memory for a single dcomplexarr(6500,6500) array!

I often find myself working with large arrays. Usually, these are
sparse, and so I can make use of sparse storage techniques to make
matrix multiplication and the like tractable. However, if the matrices
are not sparse, then one method I've made use of in the past is to
break the matrix multiplication up into pieces. That is, rather than
store the matrices in full, you can store in IDL just a single
row/column vector of each of the matrices, then apply the
multiplication, save the result, and loop over the next row/column.
It's a little slow, but there are quite a number of operations taking
place in each iteration of the loop, so it's faster than you might
expect. If accessing the next row/column vector requires reading a
large file over and over again, you might see the loop taking an
enormous amount of time. On the other hand, if you can actually
generate the matrix elements on the fly, then you can generate the row
vector and column vector inside the matrix multiplication loop.

Post by Carolina
I though IDL was the best language for large array manipulations, is it
true? or are there other languages that are better suited for this?

Of course, C/C++ can do this stuff a lot better than IDL, but you have
to deal with a lot of programming hassles and a rather unfriendly
programming environment (compared to IDL) that really slows down the
process of writing routines. I have some experience doing this in
Matlab and Mathematica, but Matlab generally is more limited than IDL
as far as speed of operation and array size limitations go. Mathematica
is even more so, and can sometimes crash with surprisingly small
matrices. Not to mention Mathematica's well-known tendency to corrupt
datasets and program files.

- Marius

[Carolina]

Post by Marius Hagen
I often find myself working with large arrays. Usually, these are
sparse, and so I can make use of sparse storage techniques to make
matrix multiplication and the like tractable.

Does anyone know a clever trick to compute inverse of large sparse
matrices?

Thanks for all the replies,

Carolina.

[R.G. Stockwell]

Post by Carolina

Post by Marius Hagen
I often find myself working with large arrays. Usually, these are
sparse, and so I can make use of sparse storage techniques to make
matrix multiplication and the like tractable.

Does anyone know a clever trick to compute inverse of large sparse
matrices?
Thanks for all the replies,
Carolina.

IDL has some functions for sparse arrays.
For example see the IDL help:
http://idlastro.gsfc.nasa.gov/idl_html_help/mathematics15.html

Depending on your situation you may need to look at some other
code (netlib.org for example)
http://netlib.org/sparse/index.html
or I've heard matlab has some good sparse array routines as well.


Cheers,
bob

[Marius Hagen]

Post by Carolina
Does anyone know a clever trick to compute inverse of large sparse
matrices?

While I'm no expert on the subject, I do spend a lot of time working
with large matrices, so maybe I can say a few things that might help
here. In most instances getting the inverse of a large matrix is a *bad
idea*, since, in floating-point math, you can easily wind up amplifying
very small errors in the input into HUGE errors in the output. For an
array with 10^8 elements, even in double-precision, most direct matrix
inverses are so corrupted with these amplified errors as to be almost
useless. Instead, what most people try is to use either a direct solver
or an indirect solver of the *forward* matrix problem.

That is, if you are trying to calculate Ax = b (A=huge matrix, x and b
long vectors), then rather than solving this as x = A^-1 b, you can use
QR decomposition to change the form of A such that it is easy to solve
by back-substitution. This can be rather slow, so an alternative is to
obtain an approximate solution for x by use of an iterative solver.
These are much faster, and typically they can obtain as much accuracy
as you want in a fraction of the time that a direct solver requires.
And these techniques do not have the extreme amplification of small
errors.

Having said that, my own work is entirely in matrices which do not have
an inverse at all, and which have 10^15 elements or so, and so
iterative solvers are my only choice. Choosing which algorithm to use
here is an art --- generally each application wants something
different. But if you have access to Matlab, its support for sparse
matrix routines and for direct or indirect matrix solvers is
impressive.

- Marius