DPMbox

Hi everybody! I’m delighted ‘cause DPMbox received an award at the ninth edition of Concurso Universitario de Software Libre whose closing event was held in Zaragoza last Thursday and Friday.

The “Concurso Universitario de Software Libre” (CUSL, Free Software University Contest), is a contest of free software/hardware development and free technical documentation. It is an initiative similar to the Google Summer of Code, but specifically aimed to the spanish university and high school students, and organized by a group of free software university offices.

Its main objective is to encourage the creation and contribute to the consolidation of the free software community at university. After nine editions, more than 700 projects and a thousand students have took part through the years.

In this edition DPMbox was considered the Best Web Project by the evaluation committee, I’m really pleased!

I want to thank the organization for its work on CUSL through all its nine editions, the selection committee for considering DPMbox the best web project, and the people from Osluz for giving us a charming stay in Zaragoza.

¡¡Viva el software libre!! =)

Hey! The week from April 20th to 26th, DPMbox will be tested on site on CERN facilities in Geneva and presented to the IT-SDC-ID section, which is reponsible of DPM development.

I will update this entry with all the progress achieved and the feedbaack received there.

I’m excited, cheers!

Hello there! I’ll write today a brief review of how DPMbox manage the file uploads, and how we solved a small problem that had complicated this feature.

So, the steps to upload a file to the Disk Pool Manager would be as folllows:

In the DPM architecture one entity handles the namespace with the metadata, and other ones store the files with the actual data. This means that any request must go through a server hosting the namespace first, and then redirect to the disk node storing the physical file. We execute a PUT call and the file won’t be yet uploaded but DPM will answer back, then our web application has to try a second time. The URL needed to actually do the PUT would look like this, you can see the specific DPM node where the file will be uploaded, including a token validating the transaction:

http://lxfsra04a04.cern.ch/srv/dpm/01/nogroup/2015-03-02/XHR.js.zip.2900489.0?sfn=%2Fdpm%2Fcern.ch%2Fhome%2Fdteam%2Faalvarez%2Fpublic%2Fcollection_d%2FXHR.js.zip&dpmtoken=dbf5fcbe-7c31-4849-aa57-b07c6b474f5e&token=ojkU%2FdgyPqTYEvu4TfZAyueVDqQ%3D%401425288602%401

User agents commonly apply same-origin restrictions to network requests. These restrictions prevent a client-side Web application as DPMbox running from one origin from obtaining data retrieved from another origin, and also limit unsafe HTTP requests that can be automatically launched toward destinations that differ from the running application’s origin. In user agents that follow this pattern, network requests typically include user credentials with cross-origin requests, including HTTP authentication and cookie information. As DPMbox relies heavily on XHR calls this has to be handled carefully or we’ll face a lot of error messages like this:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://lxfsra04a04.cern.ch/dpm/cern.ch/home/dteam/aalvarez/public/test5.file. (Reason: CORS header 'Access-Control-Allow-Origin' missing).

You can check a bit of history of XHR and CORS and why is this happening here [2].

Back in our development, the DPM initial behaviour was to answer with a 302 code indicating redirection, and then the browser will try transparently to reach that new location. That would work it out, but the thing is that 30x responses are treated as error by CORS specification [1]:

This is the actual request. Apply the make a request steps and observe the request rules below while making the request.

If the response has an HTTP status code of 301, 302, 303, 307, or 308 Apply the cache and network error steps.

(…)

Whenever the network error steps are applied, terminate the algorithm that invoked this set of steps and set the cross-origin request status to network error.

Note: This has no effect on setting of user credentials. I.e. if the block cookies flag is unset, cookies will be set by the response.

Whenever the cache and network error steps are applied, follow these steps:

Remove the entries in the preflight result cache where origin field value is a case-sensitive match for source origin and url field value is a case-sensitive match for request URL.

Apply the network error steps acting as if the algorithm that invoked the cache and network error steps invoked the network error steps instead.

So with ths configuration it would be impossible to get any information of the response, not any header at a JavaScript level, specifically the Location header. Finally, the solution given was manage the uploads though a first empty PUT including a header X-No-Redirect and the server will answer with a 202 Accepted response. We can extract any header from this type of response as it’s not treated as error for XHR CORS requests.

After this we have to add this new header information to our server CORS security settings and add it to the directive Access-Control-Expose-Headers. There’s a lot of tutorials for different servers CORS configuration, for example you can check this one for Apache [2]. But in the end, in the server side we would need to configurate it as to response with this headers:

Access-Control-Allow-Origin: UrlWhereDPMboxIsHosted.host Access-Control-Allow-Credentials: true Access-Control-Allow-Methods: ACL, CANCELUPLOAD, CHECKIN, CHECKOUT, COPY, DELETE, GET, HEAD, LOCK, MKCALENDAR, MKCOL, MOVE, OPTIONS, POST, PROPFIND, PROPPATCH, PUT, REPORT, SEARCH, UNCHECKOUT, UNLOCK, UPDATE, VERSION-CONTROL Access-Control-Allow-Headers: Authorization, Overwrite, Destination, Content-Type, Depth, User-Agent, Translate, Range, Content-Range, Timeout, X-File-Size, X-Requested-With, Accept, Accept-Version, If-Modified-Since, X-File-Name, Cache-Control, Location, Lock-Token, If, X-No-Redirect Access-Control-Expose-Headers: DAV, content-length, Allow, Location

At first, find a solution to this problem was tough since I’m not very familiar with this Cross-Origin Resource Sharing HTML5 new settings, but after all it was properly (and happily) solved. Good night!

References:

[1] W3 CORS specification w3.org/TR/cors

[2] Unleash your AJAX requests dev.housetrip.com/2014/04/17/unleash-your-ajax-requests-with-cors

[3] CORS and complex AJAX requests danieldemmel.me/blog/2013/01/18/getting-cross-origin-resource-sharing-with-complex-jquery-ajax-requests

Hi! Today I’d like to show you how the jQuery.dpm library works and how it communicates with the user interface.

The library is responsible af all communication between DPMbox and the DPM server through its WebDAV interface. So basically it acts as an API exposing the needed WebDAV methods, lying on the JavaScript XMLHttpRequest object through the jQuery Ajax method.

Besides taking care of global data as XML headers or defining some additional functions, what the library offers is (method with correspoding WebDAV call in brackets):

Main methods

• get (GET)
• post (POST)
• head (HEAD)
• mkcol, createFolder (MKCOL)
• put, createFile (PUT)
• remove (DELETE)
• report (REPORT)
• getVersionTreeReport (REPORT)
• checkout (CHECKOUT)
• uncheckout (UNCHEKCOUT)
• checkin (CHECKIN)
• versionControl (VERSION-CONTROL)
• lock (LOCK)
• unlock (UNLOCK)
• propfind, getProperty (PROPFIND)
• proppatch, setProperty, removeProperty (PROPPATCH)

And here’s a diagram represeting how a jquery.dpm method works. As you see by the prepare function we configurate accordingly an XHR that then is sent to the server:

Additional methods

• isCollection: Returns whether an element is a collection or not by looking for a collection XML tag inside a resourcetype property. For a DPM server there’s no need to use it since a DPM response includes a specific property iscollection to check this, but the method is needed for standard WebDAV server support.

• getNodesByTag: Works exactly like the standard getElementsByTagName, however introduces the ability to filter those results by namespace, which is important for handling WebDAV results.

• seekToNode: Sets resource variable to first node matched via getNodesByTag, and returns this, allowing further processing.

• eachNode: Executes a function on each element.

• nodeText: Get the text contained by a node.

• nodeName: Returns the name of a node (the tag name, essentially).

• extendbeforesend: jQuery ajax option beforeSend may be set to a function reference, and this function will fire prior to the request being sent. It gets the XHR object, so this is the point you will want to do things like set headers, for example.

• prepare: Prepares the WebDAV call. Here we want to ensure integrity of call object, verify WebDAV method requested, set any authorization information (if necesssary), and return the modified call object. WebDAV servers usually respond in XML but some WebDAV methods will not return anything at all, or return an empty response. This matters to jQuery in that the success method of a jQuery.ajax options object won’t fire if the reponse could not be parsed (although the complete method will). Is needed to be aware of what dataType of your ajax response will be, and set it appropriately.

An XML response of the DPM server looks like this:

<D:multistatus xmlns:D="DAV:" xmlns:ns0="DAV:"> <D:response xmlns:lcgdm="LCGDM:" xmlns:lp3="LCGDM:" xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/"> <D:href>/dpm/cern.ch/home/dteam/aalvarez/public/test_collection/</D:href> <D:propstat> <D:prop> <lcgdm:type>0</lcgdm:type> <lp1:resourcetype> <D:collection></D:collection> </lp1:resourcetype> <lp1:creationdate>2015-04-04T09:51:59Z</lp1:creationdate> <lp1:getlastmodified>Fri, 27 Feb 2015 18:57:50 GMT</lp1:getlastmodified> <lp3:lastaccessed>Sat, 04 Apr 2015 09:51:59 GMT</lp3:lastaccessed> <lp1:getetag>5c317-54f0be2e</lp1:getetag> <lp1:getcontentlength>0</lp1:getcontentlength> <lp1:displayname>test_collection</lp1:displayname> <lp1:iscollection>1</lp1:iscollection> <lp3:guid></lp3:guid> <lp3:mode>040775</lp3:mode> <lp3:sumtype></lp3:sumtype> <lp3:sumvalue></lp3:sumvalue> <lp3:fileid>377623</lp3:fileid> <lp3:status>-</lp3:status> <lp3:xattr>{"type": 0, "normPath": "\/dpm\/cern.ch\/home\/dteam\/aalvarez\/public\/test_collection"}</lp3:xattr> <lp1:owner>3</lp1:owner> <lp1:group>102</lp1:group> </D:prop> <D:status>HTTP/1.1 200 OK</D:status> </D:propstat> </D:response> </D:multistatus>

Depending on the context in the interface and the w2ui library, the XML data is parsed in JSON format and adapted with a specific dpmFilters method. And considering that is a petition on the network, the way to use it is by taking advantage of jQuery ajax callbacks, like for example success.

/* * A function to refresh the grid content */ function refreshContent(directory_route){ $.dpm(server + directory_route).readFolder({ success: function(dat) { w2ui.grid.clear(); w2ui.grid.add(dat); }, dataFilter: $.dpmFilters.filesDPM }); }

And that’s all. I hope that with this post you have now a clear vision of how DPMbox communicates with a DPM server.