hdf5 hdf5oct

[jb]

I'm trying to build hdf5 support in Octave, building 4.4.1 from source. The
documentation says hdf5 support is enabled by default, without being
specific as to whether that support exists only in the form of the octave
load -hdf5 and save -hdf5 commands. Meanwhile, hdf5oct package promises
implementations of h5create and h5write. That package builds in my octave
4.4.1 with pkg install, and then with autoload from the .oct file I can get
h5create to show up with a help command, but when I try to use it I get the
message "error: h5create: support for HDF5 IO was unavailable or disabled
when Octave was built". OK, so maybe the hdf5 has to be configured in the
octave build. For this I used --with-hdf5-libdir="/usr/local/hdf5/lib"
--with-hdf5-includedir="/usr/local/hdf5/include" in the configure command.
Still no luck though with h5create from the hdf5oct package. Although the
internet is littered with hundreds of references to hdf5, hdf5oct, and
octave, I can't seem to put the pieces together. Any thoughts appreciated!



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

[Kai Torben Ohlhus]

Post by jb
I'm trying to build hdf5 support in Octave, building 4.4.1 from source. The
documentation says hdf5 support is enabled by default, without being
specific as to whether that support exists only in the form of the octave
load -hdf5 and save -hdf5 commands. Meanwhile, hdf5oct package promises
implementations of h5create and h5write. That package builds in my octave
4.4.1 with pkg install, and then with autoload from the .oct file I can get
h5create to show up with a help command, but when I try to use it I get the
message "error: h5create: support for HDF5 IO was unavailable or disabled
when Octave was built". OK, so maybe the hdf5 has to be configured in the
octave build. For this I used --with-hdf5-libdir="/usr/local/hdf5/lib"
--with-hdf5-includedir="/usr/local/hdf5/include" in the configure command.
Still no luck though with h5create from the hdf5oct package. Although the
internet is littered with hundreds of references to hdf5, hdf5oct, and
octave, I can't seem to put the pieces together. Any thoughts appreciated!

Do you refer to that package https://github.com/tmullins/hdf5oct Octave
Forge does not offer such a package, right?

Best,
Kai

[jb]

Yes that's the one, it has been available there on github for a few years and
people have evidently been using it. I'm not aware of any similar package on
octave forge. I'm not even assuming hdf5oct will definitely work, because it
hasn't been updated in a while. But I'n not able to get to the point of
finding out, because it seems to be octave that is creating the error in my
case.



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

[Michael Martins]

Post by jb
Yes that's the one, it has been available there on github for a few years and
people have evidently been using it. I'm not aware of any similar package on
octave forge. I'm not even assuming hdf5oct will definitely work, because it
hasn't been updated in a while. But I'n not able to get to the point of
finding out, because it seems to be octave that is creating the error in my
case.

I am using this package for some time and it is working for me. At least
the function I am using.
It would be very nice, if this package could be added some how to octave
forge, as in my opinion this is a very important package, hdf5 is a very
popular data format. Unfortunately, many hdf5 files are much too large
to read it just by the "load -hdf5" command.

[Kai Torben Ohlhus]

Post by jb

Post by jb
Yes that's the one, it has been available there on github for a few

years and

Post by jb
people have evidently been using it. I'm not aware of any similar

package on

Post by jb
octave forge. I'm not even assuming hdf5oct will definitely work,

because it

Post by jb
hasn't been updated in a while. But I'n not able to get to the point of
finding out, because it seems to be octave that is creating the error in

my

Post by jb
case.

I am using this package for some time and it is working for me. At least
the function I am using.
It would be very nice, if this package could be added some how to octave
forge, as in my opinion this is a very important package, hdf5 is a very
popular data format. Unfortunately, many hdf5 files are much too large
to read it just by the "load -hdf5" command.
--
|\ /| PD Dr. Michael Martins, Universitaet Hamburg
| \/ | Institut fuer Experimentalphysik Tel.: [phone]
| |\/| Luruper Chaussee 149

@Michael Martins: What version of Octave do you use for "hdf5oct"? I did
not get it to work with 4.4.1 either. So I proposed the following PR for
them [1] maybe you can test it. The major problem is, that [1]
tmullins/hdf5oct does not check itself for the existence of HDF5
development libraries (using autotools) and queries the "HAVE_HDF5" and
"HAVE_HDF5_18" defines, that are not exposed to package builds in
"octave-config.h" since 4.4 as far as I know.

@jb: If you want to take advantage of [1], just replace h5read.cc by my
modified version [2]. Run "make" and "make install", then you can use it
from Octave with "pkg load hdf5oct". But be warned, the attached test
"h5test." does not finish without errors, see [1] for details.

Alternatively, there is a pending patch on the tracker [3]. If you jb
compile from source anyway, you might want to include this patch as well?

[1] https://github.com/tmullins/hdf5oct/pull/15
[2]
https://raw.githubusercontent.com/tmullins/hdf5oct/2c4b1c412f67cc8b11923054756e001c4161db39/h5read.cc

[3] https://savannah.gnu.org/patch/?8607

[PhilipNienhuis]

On Thu, Nov 15, 2018 at 12:57 PM Michael Martins <
>

Post by jb

Post by jb
Yes that's the one, it has been available there on github for a few

years and

Post by jb
people have evidently been using it. I'm not aware of any similar

package on

Post by jb
octave forge. I'm not even assuming hdf5oct will definitely work,

because it

Post by jb
hasn't been updated in a while. But I'n not able to get to the point of
finding out, because it seems to be octave that is creating the error

in
my

Post by jb
case.

I am using this package for some time and it is working for me. At least
the function I am using.
It would be very nice, if this package could be added some how to octave
forge, as in my opinion this is a very important package, hdf5 is a very
popular data format. Unfortunately, many hdf5 files are much too large
to read it just by the "load -hdf5" command.

<snip>
Alternatively, there is a pending patch on the tracker [3]. If you jb
compile from source anyway, you might want to include this patch as well?
[3] https://savannah.gnu.org/patch/?8607

That patch still applies to the development version and I occasionally use
it. I build mingw binaries for dev Octave ~twice a week in which that patch
is included.

Phili




--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

[jb]

@siko1056 (Kai): thank you for that modified h5read. It works! From the
message I thought it was octave with the mislaid hdf5 libraries, and did not
have the wherewithal to track it down in h5read.cc. Cheers!
Jonathan



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

[Kai Torben Ohlhus]

Post by jb
@siko1056 (Kai): thank you for that modified h5read. It works! From the
message I thought it was octave with the mislaid hdf5 libraries, and did not
have the wherewithal to track it down in h5read.cc. Cheers!
Jonathan

@Jonathan: Thanks for the feedback. If you are more familiar to hdf5,
maybe you can help me, why the tests

https://github.com/tmullins/hdf5oct/blob/master/test/h5test.m

do not finish?

Best,
Kai

[jb]

I'm not too knowledge about hdf5 as yet and do not have any insight to the
h5test problem right now, but now that h5write is working, I've immediately
discovered other hdf5 problems in my code (it's a matlab program I'm trying
to run in octave). I'll keep the h5test in mind as I work though it and add
whatever I can.
Jonathan



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

[jb]

OK took a closer look at h5test.m. There's certainly some problem with the
creation of a subgroup, and the creation of a complex dataset. I think these
both could probably be made to work if I could get h5read.cc running in gdb,
but failing that what I added to get the rest of the test to run was just
this:

line 120. pre-creating the dataset seemed to be necessary for the following
h5write call.
h5create("test.h5",location,size(evalin("caller",data)));

lnes 144.145. again, pre-creating these seemed to be necessary
h5create("test.h5","/rangetest",[10,10],'Datatype','int8'); // the [10 10]
is just an arbitrary size that's big enough
h5create("test.h5","/rangetest2",[10,10],'Datatype','single');

Also I renamed the variable range as it is a reserved word in octave.

However, I'm not knowledgeable to say specifically what is going on here, a
C developer might see through this stuff quite directly. At any rate my
matlab codebase which utilizes a 32 rank hdf file with no subgroups and no
complex numbers is now running in octave so the functionality if a bit
limited is still quite good. Thanks again for that tip.



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

[PhilipNienhuis]

Post by jb
OK took a closer look at h5test.m. There's certainly some problem with the
creation of a subgroup, and the creation of a complex dataset. I think these
both could probably be made to work if I could get h5read.cc running in gdb,
but failing that what I added to get the rest of the test to run was just
line 120. pre-creating the dataset seemed to be necessary for the following
h5write call.
h5create("test.h5",location,size(evalin("caller",data)));
lnes 144.145. again, pre-creating these seemed to be necessary
h5create("test.h5","/rangetest",[10,10],'Datatype','int8'); // the [10 10]
is just an arbitrary size that's big enough
h5create("test.h5","/rangetest2",[10,10],'Datatype','single');
Also I renamed the variable range as it is a reserved word in octave.And
for some reason I had to load the various h5xxxx functions with explicit
autoload statements, the pkg add hdf5oct statement by itself didn't work.
However, I'm not knowledgeable to say specifically what is going on here, a
C developer might see through this stuff quite directly. At any rate my
matlab codebase which utilizes a 32 rank hdf file with no subgroups and no
complex numbers is now running in octave so the functionality if a bit
limited is still quite good. Thanks again for that tip.

Looking better at the patch tracker entry [(patch #8607, [1]), only now I
see that it's originally based on that github repo [2].

AFAICS the patch on the patch tracker seems to have been kept more
up-to-date (July 2018) than the github repo (2015). Would it be better to
keep the path on the tracker current or update the github repo?
In that patch tracker entry JWE commented that incorporating h5ead/h5wite
would be a possibility at some point to replace Octave's internal hfd5
functions; see comments #5 and #7.

I've kept the patch reasonably current so that it compiles in the
development version of Octave but actually I don't need it anymore (that is,
ATM). BTW, I think that patch version of July also compiles with
Octave-4.4.1.
Like you I noted the "ranks" keyword shadowing but AFAICS it doesn't hurt at
run time so didn't bother delving into it further.
If you don't mind I'll probably add your changes to the patch in a next
build / cross-build and update it on the tracker and credit you.

Philip

[1] https://savannah.gnu.org/patch/?8607
[2] https://github.com/tmullins/hdf5oct/pull/15




--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

[jb]

Philip, I'm afraid patches are a bit beyond my typical use of software so
whatever is best is about all I can say. On that note however there was one
more thing in h5read.cc that I changed without being sure of whether or not
it is an actual problem. On line 979 I changed

int rank = ov_data.dims ().numel ();

to read

int rank = ov_data.ndims ();

because it looked like the length of one dimension of a dataset was being
interpreted as the rank of the entire dataset. ie any dimension of length >
32 will throw an error. Since I'm new to hdf5 I'll defer to anyone with some
more knowledge but otherwise If you agree maybe this can be part of the
patch as well? Cheers,
Jonathan



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html