{"id":2559,"date":"2015-04-21T07:09:22","date_gmt":"2015-04-21T11:09:22","guid":{"rendered":"http:\/\/antipaucity.com\/?p=2559"},"modified":"2015-04-21T07:09:22","modified_gmt":"2015-04-21T11:09:22","slug":"create-your-own-clustered-cloud-storage-system-with-moosefs-and-pydio","status":"publish","type":"post","link":"https:\/\/merikebi.warrenmyers.com\/?p=2559","title":{"rendered":"create your own clustered cloud storage system with moosefs and pydio"},"content":{"rendered":"<p>This started-off as a how-to on installing <a href=\"http:\/\/owncloud.org\">ownCloud<\/a>. But their own <a href=\"https:\/\/doc.owncloud.org\/server\/8.0\/admin_manual\/installation\/index.html\">installation<\/a> procedures don&#8217;t work for the 8.0x release and {{CentOS}} 6.<\/p>\n<p>Most of you know I&#8217;ve been interested in <a href=\"http:\/\/antipaucity.com\/2013\/10\/07\/on-demand-secure-distributed-storage\/#.VSwszBPF8nQ\">distributed<\/a>\u00a0\/ <a href=\"http:\/\/antipaucity.com\/tag\/cloud\/#.VSwtHxPF8nR\">cloud<\/a> <a href=\"http:\/\/antipaucity.com\/tag\/storage\/#.VSws9RPF8nQ\">storage<\/a> for quite some <a href=\"http:\/\/serverfault.com\/q\/89269\/2321\">time<\/a>.<\/p>\n<p>And that I find <a href=\"http:\/\/moosefs.org\">MooseFS<\/a> to be <a href=\"http:\/\/antipaucity.com\/2013\/10\/07\/on-demand-secure-distributed-storage\/#.VSwszBPF8nQ\">fascinating<\/a>. As of <a href=\"http:\/\/www.moosefs.org\/news-reader\/items\/moosefs-20-released.html\">2.0, MooseFS<\/a> comes in two flavors &#8211; the Community Edition, and the Professional Edition. This how-to uses the CE flavor, but it&#8217;d work with the Pro version, too.<\/p>\n<p>I started with the MooseFS <a href=\"https:\/\/moosefs.com\/Content\/Downloads\/moosefs-installation.pdf\">install guide<\/a> (pdf) and the Pydio <a href=\"https:\/\/pyd.io\/docs\/v6\/getting-started\/quick-start\/\">quick start<\/a>\u00a0steps. And, as usual, I used <a href=\"https:\/\/www.digitalocean.com\/?refcode=d197a961987a\">Digital Ocean<\/a> to host the cluster while I built it out. Of course, this will work with\u00a0<em>any<\/em> hosting provider (even internal to your data center using something like <a href=\"http:\/\/www.backblaze.com\/partner\/af9240\">Backblaze<\/a>\u00a0<a href=\"https:\/\/www.backblaze.com\/blog\/backblaze-storage-pod-4\">storage<\/a>\u00a0<a href=\"http:\/\/antipaucity.com\/2009\/09\/02\/backblaze-storage-pods\/#.VSwvVxPF8nQ\">pods<\/a>\u00a0&#8211; I chose <a href=\"https:\/\/www.digitalocean.com\/?refcode=d197a961987a\">Digital Ocean<\/a> because they have hourly pricing; <a href=\"https:\/\/chunkhost.com\/r\/wmadg\">Chunk Host<\/a>\u00a0is a &#8220;better&#8221; deal if you don&#8217;t care about hourly pricing). In many ways, this how-to is in response to my rather <a href=\"http:\/\/serverfault.com\/a\/129878\/2321\">hackish<\/a> (though quite functional) need to offer file storage in an otherwise-overloaded lab several years back. Make sure you have &#8220;private networking&#8221; (or equivalent) enabled for your VMs &#8211; don&#8217;t want to be sharing-out your MooseFS storage to just\u00a0<em>anyone<\/em> \ud83d\ude42<\/p>\n<p>Also, as I&#8217;ve done in other <a href=\"http:\/\/antipaucity.com\/tag\/how-to\/#.VSwufxPF8nQ\">how-tos<\/a> on this blog, I&#8217;m using {{CentOS}} {{Linux}} for my distro of choice (because I&#8217;m an {{RHEL}} guy, and it shortens\u00a0<em>my<\/em>\u00a0learning curve).<\/p>\n<h3>With the introduction\u00a0out of the way, here&#8217;s what I did &#8211; and what you can do, too:<\/h3>\n<h4>Preliminaries<\/h4>\n<ul>\n<li>spin-up at least 3 (4 would be better) systems (for purposes of the how-to, low-resource (512M RAM, 20G storage) machines were used; use the biggest [storage] machines you can for Chunk Servers, and the biggest [RAM] machine(s) you can for the Master(s))\n<ul>\n<li>1 for the MooseFS Master Server (if using Pro, you want at least 2)<\/li>\n<li>(1 or more for metaloggers &#8211; only for the Community edition, and not required)<\/li>\n<li>2+ for MooseFS Chunk Servers (minimum required to ensure data is available in the event of a Chunk failure)<\/li>\n<li>1 for ownCloud (while this might be able to co-reside with the MooseFS Master &#8211; this tutorial uses a fully-separate \/ tiered approach)<\/li>\n<\/ul>\n<\/li>\n<li>make sure the servers are either all in the same data center, or that you&#8217;re not paying for inter-DC traffic<\/li>\n<li>make sure you have &#8220;private networking&#8221; (or equivalent) enabled so you do not share your MooseFS mounts to the world<\/li>\n<li>make sure you have some <a href=\"http:\/\/antipaucity.com\/2011\/08\/08\/why-technical-intricacies-matter\/#.VS0guRPF8nQ\">swap<\/a> space on every server (may not matter, but I prefer &#8220;safe&#8221; to &#8220;sorry&#8221;) &#8211; I covered how to do this in the <a href=\"http:\/\/antipaucity.com\/2014\/05\/08\/setting-up-etherpad-in-centos-6\/#.VS0g3RPF8nQ\">etherpad tutorial<\/a><\/li>\n<\/ul>\n<h4>MooseFS Master<\/h4>\n<ul>\n<li>install MooseFS master\n<ul>\n<li>curl &#8220;http:\/\/ppa.moosefs.com\/RPM-GPG-KEY-MooseFS&#8221; &gt; \/etc\/pki\/rpm-gpg\/RPM-GPG-KEY-MooseFS &amp;&amp; curl &#8220;http:\/\/ppa.moosefs.com\/MooseFS-stable-rhsysv.repo&#8221; &gt; \/etc\/yum.repos.d\/MooseFS.repo &amp;&amp; yum -y install moosefs-master moosefs-cli<\/li>\n<\/ul>\n<\/li>\n<li>make changes to \/etc\/mfs\/mfsexports.cfg\n<ul>\n<li><span style=\"line-height: 1.5;\"># Allow everything but &#8220;meta&#8221;.<\/span><\/li>\n<li>#* \/ rw,alldirs,maproot=0<\/li>\n<li>10.132.0.0\/16 \/ rw,alldirs,maproot=0<\/li>\n<\/ul>\n<\/li>\n<li>add hostname entry to \/etc\/hosts\n<ul>\n<li>10.132.41.59 mfsmaster<\/li>\n<\/ul>\n<\/li>\n<li>start master\n<ul>\n<li>service moosefs-master start<\/li>\n<\/ul>\n<\/li>\n<li>see how much space is available to you (none to start)\n<ul>\n<li>mfscli -SIN<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h4>MooseFS Chunk(s)<\/h4>\n<ul>\n<li>install MooseFS chunk\n<ul>\n<li>curl &#8220;http:\/\/ppa.moosefs.com\/RPM-GPG-KEY-MooseFS&#8221; &gt; \/etc\/pki\/rpm-gpg\/RPM-GPG-KEY-MooseFS &amp;&amp; curl &#8220;http:\/\/ppa.moosefs.com\/MooseFS-stable-rhsysv.repo&#8221; &gt; \/etc\/yum.repos.d\/MooseFS.repo &amp;&amp; yum -y install moosefs-chunkserver<\/li>\n<\/ul>\n<\/li>\n<li>add the mfsmaster line from previous steps to \/etc\/hosts\n<ul>\n<li>cat &gt;&gt; \/etc\/hosts<\/li>\n<li>10.132.41.59 mfsmaster<\/li>\n<li>&lt;ctrl&gt;-d<\/li>\n<\/ul>\n<\/li>\n<li>make your share directory\n<ul>\n<li>mkdir \/mnt\/mfschunks<\/li>\n<\/ul>\n<\/li>\n<li>add your freshly-made directory to the end of \/etc\/mfshdd.cfg, with a size you want to share\n<ul>\n<li>\/mnt\/mfschunks 15GiB<\/li>\n<\/ul>\n<\/li>\n<li>start the chunk\n<ul>\n<li>service moosefs-chunkserver start<\/li>\n<\/ul>\n<\/li>\n<li>on the MooseFS master, make sure your new space has become available\n<ul>\n<li>mfscli -SIN<\/li>\n<\/ul>\n<\/li>\n<li>repeat for as many chunks as you want to have<\/li>\n<\/ul>\n<h4>Pydio \/ MooseFS Client<\/h4>\n<ul>\n<li>install MooseFS client\n<ul>\n<li>curl &#8220;http:\/\/ppa.moosefs.com\/RPM-GPG-KEY-MooseFS&#8221; &gt; \/etc\/pki\/rpm-gpg\/RPM-GPG-KEY-MooseFS &amp;&amp; curl &#8220;http:\/\/ppa.moosefs.com\/MooseFS-stable-rhsysv.repo&#8221; &gt; \/etc\/yum.repos.d\/MooseFS.repo &amp;&amp; yum -y install moosefs-client<\/li>\n<\/ul>\n<\/li>\n<li>add the mfsmaster line from previous steps to \/etc\/hosts\n<ul>\n<li>cat &gt;&gt; \/etc\/hosts<\/li>\n<li>10.132.41.59 mfsmaster<\/li>\n<li>&lt;ctrl&gt;-d<\/li>\n<\/ul>\n<\/li>\n<li>mount MooseFS share somewhere where Pydio will be able to get to it later (we&#8217;ll use a <a href=\"http:\/\/antipaucity.com\/2012\/02\/27\/binding-your-mounts\/#.VTVphq1VhBc\">bind<\/a> mount for that in a while)\n<ul>\n<li>mfsmount \/mnt\/mfs -H mfsmaster<\/li>\n<\/ul>\n<\/li>\n<li>install {{Apache}} and {{PHP}}\n<ul>\n<li>yum -y install httpd<\/li>\n<li>yum -y install php-common\n<ul>\n<li>you need more than this, and hopefully Apache grabs it for you &#8211; I installed {{Nginx}} then uninstalled it, which brought-in all the PHP stuff I needed (and probably stuff I didn&#8217;t)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>modify php.ini to support large files (Pydio is exclusively a webapp for now)\n<ul>\n<li>memory_limit = 384M<\/li>\n<li>post_max_size = 256M<\/li>\n<li>upload_max_filesize = 200M<\/li>\n<\/ul>\n<\/li>\n<li>grab Pydio\n<ul>\n<li>you can use either the yum method, or the manual &#8211; I picked manual<\/li>\n<li>curl http:\/\/hivelocity.dl.sourceforge.net\/project\/ajaxplorer\/pydio\/stable-channel\/6.0.6\/pydio-core-6.0.6.tar.gz\n<ul>\n<li>URL correct as of publish date of this blog post<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>extract Pydio tgz to \/var\/www\/html<\/li>\n<li>move everything in \/var\/www\/html\/data to \/mnt\/moosefs<\/li>\n<li><a href=\"http:\/\/antipaucity.com\/2012\/02\/27\/binding-your-mounts\/#.VTVwqa1VhBc\">bind<\/a> mount \/mnt\/moosefs to \/var\/www\/html\/data\n<ul>\n<li>mount &#8211;bind \/mnt\/moosefs \/var\/www\/html\/data<\/li>\n<\/ul>\n<\/li>\n<li>set ownership of all Pydio files to apache:apache\n<ul>\n<li>cd \/var\/www\/html &amp;&amp; chown -R apache:apache *<\/li>\n<li>note &#8211; this will give an error such as the following screen:<\/li>\n<li><a href=\"https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-16.32.48.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2567 size-medium\" src=\"http:\/\/antipaucity.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-16.32.48-184x300.png\" alt=\"Screen Shot 2015-04-20 at 16.32.48\" width=\"184\" height=\"300\" srcset=\"https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-16.32.48-184x300.png 184w, https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-16.32.48-768x1253.png 768w, https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-16.32.48-627x1024.png 627w, https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-16.32.48.png 1060w\" sizes=\"auto, (max-width: 184px) 100vw, 184px\" \/><\/a>this is &#8220;ok&#8221; &#8211; but don&#8217;t leave it like this (good enough for a how-to, not production)<\/li>\n<\/ul>\n<\/li>\n<li>start Pydio <a href=\"https:\/\/pyd.io\/docs\/v6\/getting-started\/quick-start\">wizard<\/a><\/li>\n<li>fill-in forms as they say they should be (admin, etc)\n<ul>\n<li>I picked &#8220;No DB&#8221; for this tutorial &#8211; you should use a database if you want to roll this out &#8220;for real&#8221;<\/li>\n<\/ul>\n<\/li>\n<li>login and starting using it<\/li>\n<\/ul>\n<h3><a href=\"https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-17.07.51.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-2568\" src=\"http:\/\/antipaucity.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-17.07.51-300x216.png\" alt=\"Screen Shot 2015-04-20 at 17.07.51\" width=\"300\" height=\"216\" srcset=\"https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-17.07.51-300x216.png 300w, https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-17.07.51-768x552.png 768w, https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-17.07.51-1024x736.png 1024w, https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-17.07.51-1568x1128.png 1568w, https:\/\/merikebi.warrenmyers.com\/wp-content\/uploads\/2015\/04\/Screen-Shot-2015-04-20-at-17.07.51.png 1866w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/h3>\n<h3>Now what?<\/h3>\n<p>Why would you want to do this? Maybe you need an in-house shared\/shareable storage environment for your company \/ organization \/ school \/ etc. Maybe you&#8217;re just a geek who likes to play with new things. Or maybe you want to get into the {{reselling}} business, and being able to offer a redundant, clustered, {{cloud}}, on-demand type storage service is something you, or your customers, would find profitable.<\/p>\n<h3>Caveats of the above how-to:<\/h3>\n<ul>\n<li>nothing about this example is &#8220;production-level&#8221; in any manner (I used <a href=\"https:\/\/www.digitalocean.com\/?refcode=d197a961987a\">Digital Ocean<\/a> droplets at the very small end of the spectrum (512M memory, 20G storage, 1 CPU))\n<ul>\n<li>there is a [somewhat outdated] sizing guide for <a href=\"https:\/\/owncloud.com\/wp-content\/uploads\/2013\/07\/owncloud-hardware-sizing.pdf\">ownCloud<\/a>\u00a0(pdf) that shows just how much it\u00a0wants for resources in anything other than a toy deployment<\/li>\n<li>Pydio is pretty light on its basic <a href=\"https:\/\/pyd.io\/docs\/v6\/getting-started\/requirements\">requirements<\/a>\u00a0&#8211; which also helped this how-to out<\/li>\n<li>while MooseFS is <a href=\"http:\/\/www.moosefs.org\/moosefs-faq.html#sort\">leaner<\/a> when it comes to system requirements, it still shouldn&#8217;t be nerfed by being stuck on small machines<\/li>\n<\/ul>\n<\/li>\n<li>you shouldn&#8217;t be managing hostnames via \/etc\/hosts &#8211; you should be using DNS\n<ul>\n<li>DNS settings are far more than I wanted to deal with in this tutorial<\/li>\n<\/ul>\n<\/li>\n<li>security has, intentionally, been ignored in this how-to\n<ul>\n<li>just like verifying your inputs is ignored in the vast majority of programming classes, I ignored security considerations (other than putting the MooseFS servers on non-public-facing IPs)<\/li>\n<li>don&#8217;t be dumb about security &#8211; it&#8217;s a real issue, and one you need to plan-in from the very start\n<ul>\n<li>DO encrypt your file systems<\/li>\n<li>DO ensure your passwords are complex (and used rarely)<\/li>\n<li>DO use key-based authentication wherever possible<\/li>\n<li>DON&#8217;T be naive<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>you should be on the mailing list for <a href=\"http:\/\/www.moosefs.org\/contact.html\">MooseFS<\/a> and Pydio <a href=\"https:\/\/pyd.io\/f\">forum<\/a>.\n<ul>\n<li>the communities are excellent, and have been extremely helpful to me, even as a <a href=\"http:\/\/serverfault.com\/q\/589475\/2321\">lurker<\/a><\/li>\n<\/ul>\n<\/li>\n<li>I cannot answer more than basic questions about any of the tools used herein<\/li>\n<li>why I picked what I picked and did it the way I did\n<ul>\n<li>I picked MooseFS because it\u00a0<em>seems<\/em> the easiest to run<\/li>\n<li>I picked Pydio because the ownCloud docs were borked for the 8.0x release on CentOS 6 &#8211; and it\u00a0<em>seems\u00a0<\/em>better than alternatives I could find (<a href=\"http:\/\/seafile.com\">Seafile<\/a>, etc)\u00a0<em>for this tutorial<\/em><\/li>\n<li>I wanted to use {{ownCloud}} because it has clients for everywhere ({{iOS}}, {{Android}}, web, etc)<\/li>\n<li>I have no affiliation with either MooseFS or {{Pydio}} beyond thinking they&#8217;re cool\n<ul>\n<li>there are other filesystems you could use &#8211; {{Gluster}}, {{Ceph}}, {{XtreemFS}}, <a href=\"http:\/\/lizardfs.com\/download\">LizardFS<\/a>, <a href=\"http:\/\/beegfs.com\">BeeGFS<\/a>, and more<\/li>\n<li>there are other cloud storage tools you could use &#8211; <a href=\"http:\/\/sparkleshare.org\">SparkleShare<\/a>, <a href=\"http:\/\/seafile.com\">Seafile<\/a>, etc<\/li>\n<\/ul>\n<\/li>\n<li>I like learning new things and showing them off to others<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3>Final thoughts<\/h3>\n<p>Please go make this better and show-off what you did that was smarter, more efficient, cheaper, faster, etc. Turn it into something you could deploy as an AMID on {{AWS}}. Or {{Docker}} containers. Or something I couldn&#8217;t imagine.\u00a0<a href=\"http:\/\/antipaucity.com\/about\/#.VS0qJRPF8nQ\">Everything<\/a> on this site is licensed under the <a href=\"http:\/\/creativecommons.org\/licenses\/by\/3.0\">CC BY 3.0<\/a>\u00a0&#8211; have fun with what you find, make it awesomer, and then tell everyone else about it.<\/p>\n<p>I think I&#8217;ll give LizardFS a try next time &#8211; their architecture is, diagrammatically, identical to the &#8220;pro&#8221; edition of MooseFS. And it&#8217;d be fun to have experience with more than one solution.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This started-off as a how-to on installing ownCloud. But their own installation procedures don&#8217;t work for the 8.0x release and {{CentOS}} 6. Most of you know I&#8217;ve been interested in distributed\u00a0\/ cloud storage for quite some time. And that I find MooseFS to be fascinating. As of 2.0, MooseFS comes in two flavors &#8211; the &hellip;<br \/><a href=\"https:\/\/merikebi.warrenmyers.com\/?p=2559\" class=\"more-link pen_button pen_element_default pen_icon_arrow_double\">Continue reading <span class=\"screen-reader-text\">create your own clustered cloud storage system with moosefs and pydio<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[31,41,45],"tags":[178,179,246,390,498,540,605,725,789],"keyring_services":[],"class_list":["post-2559","post","type-post","status-publish","format-standard","hentry","category-personal","category-technical","category-tutorial","tag-cloud","tag-cluster","tag-distributed","tag-how-to","tag-moosfs","tag-owncloud","tag-pydio","tag-storage","tag-tutorial"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/2559","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2559"}],"version-history":[{"count":0,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/2559\/revisions"}],"wp:attachment":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2559"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2559"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2559"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fkeyring_services&post=2559"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}