How do you migrate a 20GB site?

Abe Olandres Editor-in-chief January 15, 2006 · 1 min read

Listen to article

iOS 27 Adds Native Baybayin Keyboard to iPhone: Here is How to Enable It AMD Radeon RX 9070 GRE announced GPD MicroPC 2 with Intel Core 3 N350 announced Nintendo Direct to unveil Switch 2 games for second half of 2026 iOS 27 Beta Is Out, But You'll Need to Join a Waitlist to Try the New Siri AI

I’m moving over the entire Motorcycle Philippines site into this server. The problem is, it’s over 20GB worth of files, databasess and whatnots (17GB when tarred/gzip-ed.

The usual cPanel server to server transfer just wouldn’t work this time (times out just before it finishes). FTP cannot handle files over 2GB, scp and rsync could take hours and hours. Before this, the largest site I ever handled in this kind of situation was houseonahill.net at just about 1.8GB. This one is 10 times bigger.

Been at this since 3pm this afternoon and I’m on strike 3 already (splitting up the files into 17 1GB parts).

React to this article:

Abe Olandres

Editor-in-chief

Abe is the founder and Editor-in-Chief of YugaTech with over 20 years of experience in the technology industry. He is one of the pioneers of blogging in the country and is considered by many as the Father of Tech Blogging in the Philippines.

8,007 Articles

View all posts by Abe Olandres →

22 Comments

Load Philippines · 16 years ago

I just adore stumbleupon for being able to show me this web page. instant fan now. I have viewed some definitely poor websites on stumble and when I discovered this, I immediately knew it was excellent! This blogpost is informative specifically for somebody like me haha. I’ve been attempting to read a lot more these days and being able to discover a web page like this can be excellent for my education. Love what you are doing, keep it up!

hoop · 20 years ago

Ah the ever reliable SSH, nice to know it can handle that much data without any corruption or termination… this is a tip that will come in handy in the future

bimbo · 20 years ago

well it certainly took its time! ;)

Abe Olandres Editor-in-chief · 20 years ago

What we did was to backup the whole site via ssh (./scripts/pkgacct) which resulted into a tarball of about 17GB file. Copying took about 5 hours.

Once in the ne server, we did a manual restore via SSH again (./scripts/restorepkg) and the un-packaging took about 2 hours and the restore another 4 hours.

During this period, if your own net connection blips for one second, the process aborts, which happened once.

markku · 20 years ago

There are FTP clients that can do FTP-to-FTP transfers. I’m sure that would take some time, but possibly the best solution right now, without requiring you to work on the site immediately.

hoop · 20 years ago

I usually run optimize table in phpmyadmin to defragment the tables and reclaim some unused disk space. This might help to bring that DB down further

Maybe you should work with the smaller tables first and see how much it brings down the file size…

I’d suggest preventing access to the DB while optimizing if incase you go down this path to prevent corruption of the DB.

then once you’ve uploaded the DB to the new server you can dump the old DB so that all you have to worry about is the 1gb classifieds and 1.5 photofiles

kzap · 20 years ago

how bout clean it up first, im sure theres a lot of unneeded stuff there

demonhale · 20 years ago

Hmmm… that is a problem really, post about your solutions will ya?

Anyways heres an idea, I worked with a big site before (I just dont want to mention their URL coz their a private internal site) .. Which had their own hosting, but the Outer parts for public visitors are hosted by an external host located cross-country away. I issued an order to the dept. to send a request to the external host to send us a collection of the backup files written in discs and labeled appropriately, boxed and sent via a freight service. We paid for the overhead cost as well as service fee (affordable considering all the work that needs to be done) …

Abe Olandres Editor-in-chief · 20 years ago

Here’s a quick breakdown: 500MB DB, 4GB forum files, 1GB classified ads files, 1.5GB photo gallery files, then there are the attachments and other core site files.