How to reuse existing Amazon S3 structure in Liferay

Posted 11 years ago by janhaj

I described the way how to migrate data from Liferay out to Amazon S3 in a previous article.

This article is about the opposite case. Are you using file storage solution with Amazon S3 and want to import documents to Liferay? Is it possible to move existing files and folders from Amazon S3 to Liferay Portal? Yes, it is!

Liferay, as you might know, uses it’s own file structure. It implies that we have to get the files and folders into Liferay’s structure. One of the ways to do it is via multiple downloads from Amazon S3 (using s3cmd tool) and multiple uploads via Web-DAV or Liferay Sync access to Liferay. I’m going to describe using s3cmd & Web-DAV below.

Keep in mind

Files are downloaded and then copied (not moved) to Liferay. That implies two important things. Firstly, files you downloaded will stay present on Amazon S3 servers and you have to pay for the space until you delete them. Secondly, if you move many GBs, you will have to pay adequately for the data traffic.

Multiple downloads from Amazon S3

The web interface of Amazon S3 doesn’t support downloads of multiple files at once, folders can’t be downloaded at all. How could it be done? You can use S3 Tools for this purpose. It’s a package of powerful open source tools that (as you might expect) work with Amazon S3.

You don’t need to download the whole package, only s3cmd is needed. s3cmd is a command-line tool with capability to list buckets, folders and files, download them, delete them, upload new and it all can be done recursively. You can download it from sourceforge.

This tutorial is based on version 1.5.0-alpha3. Unpack the archive, when the download is finished. We have to set up s3cmd before using. Run

./s3cmd --configure

and follow the steps and add necessary information:

• Access key
• Secret key
• Password for transferring files to S3
• Path to GPG program (if the path is different, put the right path)
• Use HTTPS protocol: default No, optionally Yes
• To test your credentials press Y (after prompt), you should see something like: Success. Your access key and secret key worked fine in that case.
• The last prompt is to save settings – press y

The output is saved in the ~/.s3cfg file.

Commands for s3cmd

List buckets

./s3cmd ls

List folders and files inside bucket or folder in bucket

./s3cmd ls s3://BUCKET_NAME
./s3cmd ls s3://BUCKET_NAME/FOLDER

Dowload file

./s3cmd get s3://BUCKET_NAME/FILE

Download folder

./s3cmd get --recursive s3://BUCKET_NAME/FOLDER

Download entire bucket

./s3cmd get --recursive s3://BUCKET_NAME

Now you know, how to use basic features of s3cmd. So let’s go, download whatever you need. If you are interested in s3cmd, go to http://s3tools.org/s3cmd where you can find more commands and tips.

Downloads are saved in the directory that you are starting s3cmd from (E.g.: If you are at ~ and you start s3cmd by ./s3cmd_tool/s3cmd, downloads are placed in ~).

You can remove the ~/.s3cfg file now or make sure that only administrator is able to access it. This is for security reasons, as it stores your secret key in plain text.

Uploading multiple files and folders to Liferay

Once you have downloaded your content from Amazon S3, you can upload it to Liferay using Web-DAV or Liferay Sync. I’m going to describe how you can setup Web-DAV in order to upload the whole Amazon S3 content (and more) to Liferay’s local storage.

Upload to Liferay using Web-DAV

Caution
Make sure, that all files you want to upload to Liferay are smaller than the upload limit in Liferay. Otherwise you have to increase the file upload limit in Control Panel in section Server Administration.

How to set up Web-DAV access? Open your Liferay portal in a web browser and navigate to Documents and Media portlet or section Documents and Media in Control panel. There should be a Home in the Menu, a small arrow is shown at mouse-over. Choose Access from desktop from the sub-menu. This is the Web-DAV url. It shows you you the entire file structure of your pages. Switch to a different folder, if you don’t need the whole Home structure.

The last but one step is to set up Web-DAV to your computer.

• In Linux OS, open file manager (nautilus, nemo, caja or any different) paste the Web-DAV url in the location bar or look for “Connect to a server” in the menu. If you choose the first option, replace “http” by “dav” (or “webdav” in KDE). If the second one, remove http://.
• In Microsoft Windows OS, look for “Map Network Drive” or “Add Network Place” and follow the steps.
  
• In Mac OS X, select the menu Go and Connect to Server in Finder and paste the Web-DAV url.Enter your Liferay credentials when prompted. You can use username or username@maildomain.

Liferay’s Document’s library is mounted as another hard drive partition. Changes we made in the mounted directory are immediately saved in Liferay Portal. We can copy downloaded files to the folder (within the mounted directory), where we want to have them.

Posts

After setting up Web-DAV, copy files that you want to upload to Liferay’s local storage to the Web-DAV mounted folder and files will be automatically uploaded to Liferay’s document’s library and accessible from Documents and Media Portlet.

Tests

• Open your Liferay Portal, go to documents library and check that new files (and folders) are under the same tree, where you placed them on the mounted drive.
• Check that correct files and folders are inside the bucket on Amazon S3 console that we have configured in Liferay. It’s good to know Liferay’s file structure, it’s well described at the official Liferay guide in section “Using the File System store” (the same structure is used on Amazon S3 servers).
• Advanced users should check logs of the application server, where Liferay is deployed to see potential errors coming.

Finish

You have imported your documents from Amazon S3 to Liferay now. Also you know, how to use Web-DAV which might be very useful.

Used resources

s3cmd manual
Accessing the Document Library with Web-DAV on Liferay Wiki

janhaj