Setup Amazon S3 to Liferay with data migration

We slightly introduced Amazon S3 and its basic integration with a clean Liferay installation in a previous article. What should you do, if you have already uploaded files in Liferay and you want to move them from local storage to Amazon S3? Files that are uploaded to Liferay’s local storage will become inaccessible, if you just perform the steps described in the previous article. That’s because the procedure configures Liferay to use Amazon S3 instead of its local storage without moving files that were originally uploaded to the local storage. Attempts to access those files will result to an Internal Server Error. Is it possible to move these files? Yes, it is. Does data migration affects their settings? No, file settings stay untouched. Here is a short intro on how you can migrate files using the Data Migration Tool.

Prerequisites

Have an account on AWS (Amazon Web Services) and basic knowledge of Liferay installation and configuration.

Migration log

Tomcat users have nothing to do. If you are running Jboss 6.1.1 and want to see Liferay log, you have to append

JAVA_OPTS="$JAVA_OPTS -Dorg.jboss.as.logging.per-deployment=false"

to file JBOSS-HOME/bin/standalone.conf.

Prepare for data migration

Liferay has a functionality to migrate already uploaded documents. We have to tell Liferay, where to migrate files. Liferay needs an access key, secret key and a bucket name. Navigate to our previous article, where we described, how to obtain these.

Open (or create) the file named portal-ext.properties in

Tomcat: TOMCAT-HOME/webapps/ROOT/WEB-INF/classes/
Jboss: JBOSS-HOME/standalone/deployments/ROOT.war/WEB-INF/classes

or wherever you have installed your Liferay and paste in:

dl.store.s3.access.key=
dl.store.s3.secret.key=
dl.store.s3.bucket.name=

Paste valid values, save changes and restart your application server (Tomcat, Jboss, ..). The properties file isn’t loaded without the restart and the migration failes.

Migration

We have everything prepared now, migration can start. Open your web browser at your Liferay installation and navigate to Control Panel. From Server menu choose Server administration and go to Data Migration.

Drop-down menu in section “Migrate documents from one repository to another” change to

com.liferay.portlet.documentlibrary.store.S3Store

and Execute.

Liferay Portal temporarily changes itself to maintenance mode, when the migration starts. You can see, how many files are in your document library. All of these files are moved to Amazon S3 server.

Redirection to the main page of your Liferay Portal means operation completed, files are already moved and Amazon S3 is set as file storage. Document previews and thumbnails aren’t moved. Instead new will be generated automatically in place.

Completion of migration

Liferay uses Amazon S3 as storage only until the first restart of Liferay. To make the  usage of Amazon S3 permanent, we have to tell Liferay to start with Amazon S3 by default. Add to portal-ext.properties (where you also saved login credentials and bucket name):

dl.store.impl=com.liferay.portlet.documentlibrary.store.S3Store

Tests

• Probably the easiest test of successful migration is trying to access to originally uploaded file(s) in documents library. Choose one randomly and download it. If download works fine, Amazon S3 should be configured correctly in Liferay and data migration seems to be successful.
• Check that correct files and folders are inside the bucket on Amazon S3 console that we have configured in Liferay. It’s good to know Liferay’s file structure, it’s well described at official Liferay guide in section “Using the File System store” (the same structure is used on Amazon S3 servers).
• Advanced users should check logs of application server on which Liferay is deployed to see potential errors coming.Log file is located at:

  Tomcat: TOMCAT-HOME/logs/catalina.out
  Jboss: JBOSS-HOME/standalone/log/server.log

Caution

If you don’t set correct values for the S3 connection or if you don’t restart the application server before of the start migration, the migration fails. Navigation through file structure will work but all files will be unaccessible until Liferay restarts. If it happens to you, you have to delete again

dl.store.impl=com.liferay.portlet.documentlibrary.store.S3Store

or change it to your previous correct settings. If you don’t change it, Liferay will try to access these files with new storage settings and files remain inaccessible even after restart! See details at bug no. 36105.

Finish

From this moment, your Liferay installation uses Amazon S3 cloud storage and files that were originally uploaded to Liferay’s local storage are already moved to S3 servers too.

Used resources

Official Liferay user guide, section Server administration: Data Migration