davix
Davix Documentation
Author
Devresse Adrien ( adrie.nosp@m.n.de.nosp@m.vress.nosp@m.e@ce.nosp@m.rn.ch )

Developped at CERN (IT-SDC-ID)

Official WebSite: Here

User Documentation: Here

Mailing list : davix.nosp@m.-dev.nosp@m.el@ce.nosp@m.rn.c.nosp@m.h

davix API :

file / object store API : Davix::DavFile
posix compatiblity layer : Davix::DavPosix
http request layer: Davix::HttpRequest
main header: davix.hpp

What is davix ?

davix is a library and a set of tools for remote I/O on resources with HTTP based protocols. It aims to be a simple, performant and portable I/O layer for Cloud and Grid Storages services.

Davix supports:

  • HTTP, WebDav, Amazon S3
  • SSL/TLS
  • X509 client auth with proxy credential support
  • Vector operations (Partial reads, multi-range, single range)
  • Metalinks
  • Redirections caching
  • Webdav parsing
  • Data management operations ( mkdir, rm, stat )

The Davix targets to:

  • Be simple to use for simple use cases
  • Provide the needed features for High Performance I/O use cases
  • Be a data management swiss knife for HTTP based data stores

Examples

Query basic file metadata

StatInfo infos;
DavFile file(context, "http://my.webdav.server.org/myfolder/myfile");
file.statInfo(NULL, infos);
std::cout << "my file is " << infos.size << " bytes large " << std::endl;

Create a directory

DavFile file(context, "http://my.webdav.server.org/myfoldier/newfolder");
// creat directory
file.makeCollection(NULL);

Get a full file content

DavFile f(context, "http://mysite.org/file");
int fd = open("/tmp/local_file", O_WRONLY | O_CREAT);
// get full file
file.getToFd(NULL,fd, NULL) < 0)

Execute a partial GET

char buffer[255] = {0};
DavFile file(context, "http://mysite.org/file");
// get 100 bytes from http://mysite.org/file after an offset of 200 bytes
file.readPartial(NULL, buffer, 100, 200);

Execute a Vector Operation

char buffer[255] = {0}
DavFile file(context, "http://mysite.org/file");
DavIOVecInput in[3];
DavIOVecOutput ou[3];
// setup vector operations parameters
// --------
// execute query
file.readPartialBufferVec(NULL, in, out , 3 , NULL);

Random I/O in posix mode

// read ops
fd= p.open(NULL, "https://mywebdav-server.org/myfile.jpg", O_RDONLY, NULL);
p.read(fd, buffer, size, NULL);
p.pread(fd, buffer, size2, offset, NULL);
p.close(fd);
//

Manual HTTP query:

Davix::HttpRequest req("https://restapi-server.org/rest")
req.addHeaderField(...)
req.setRequestMethod("PUT")
// .. configure ....
//
//
// execute your request
req.executeRequest(...);

How to compile

  • Davix Dependencies :
    • openssl
    • libxml-2.0
    • Doxygen ( optional, for documentation generation )
  • Davix Portability :
    • Target any POSIX OS
    • Ported on Linux > 2.6 , Windows with Cygwin, OSX > 10.2, AIX.
    • Packaged on Debian > 6, Ubuntu > 13.04, Fedora > 18, SL > 5,
  • Compile :
    • " 1. git clone http://git.cern.ch/pub/davix "
    • " 2. cd davix "
    • " 3. mkdir build; cd build"
    • " 4. cmake ../"
    • " 5. make "
  • Generate doc :
    • make doc
  • Compile and run unit tests :
    • cmake ../
    • make
    • make test
  • Compile & execute func tests : warning : functionals test needs davserver and a valid credential
    • " 4. cmake -DFUNCTIONAL_TESTS=TRUE ../ "
    • " 5. . ../test/setup_test_env.sh - " 5. make; make test"
  • make RPMS :
    • ./packaging/make-srpm.sh

Play with davix command line tool :

davix has a set of command line tools for testing purpose and demonstration

-> davix-ls: file listing
-> davix-get: download operations
-> davix-put: upload operations
-> davix-http: low level query composition

TODO in Davix

- WebHDFS support
- S3 ACL support
- CDMI support
- X-stream mode support ( metalink multi source download )

For any contribution please contact us on davix-devel@cern.ch
Davix::DavPosix::pread
ssize_t pread(DAVIX_FD *fd, void *buffer, size_t count, off_t offset, DavixError **err)
do a partial read of a file in a POSIX-like approach with HTTP(S).
Davix::DavPosix
POSIX-like API of Davix.
Definition: davposix.hpp:62
Davix::DavPosix::open
DAVIX_FD * open(const RequestParams *params, const std::string &url, int flags, DavixError **err)
open a file for read/write operation in a POSIX-like approach.
Davix::DavPosix::close
int close(DAVIX_FD *fd, DavixError **err)
close a existing file descriptor.
Davix::DavPosix::read
ssize_t read(DAVIX_FD *fd, void *buffer, size_t count, DavixError **err)
read a file in a POSIX-like approach with HTTP(S).
Davix::HttpRequest
Http low level request interface.
Definition: httprequest.hpp:86