How do you migrate a 20GB site?

I’m moving over the entire Motorcycle Philippines site into this server. The problem is, it’s over 20GB worth of files, databasess and whatnots (17GB when tarred/gzip-ed.

The usual cPanel server to server transfer just wouldn’t work this time (times out just before it finishes). FTP cannot handle files over 2GB, scp and rsync could take hours and hours. Before this, the largest site I ever handled in this kind of situation was houseonahill.net at just about 1.8GB. This one is 10 times bigger.

Been at this since 3pm this afternoon and I’m on strike 3 already (splitting up the files into 17 1GB parts).

Abe Olandres

Abe is the founder and Editor-in-Chief of YugaTech with over 20 years of experience in the technology industry. He is one of the pioneers of blogging in the country and considered by many as the Father of Tech Blogging in the Philippines. He is also a technology consultant, a tech columnist with several national publications, resource speaker and mentor/advisor to several start-up companies.

https://www.yugatech.com

11 Comments

Bong says:

19 years ago

all the best and good luck!
- Reply
hoop says:

19 years ago

1. maybe you can try exporting the tablespaces piecemeal for that particular DB via FTP if it’s below 2 GB per table.

2. Use phpmyadmin to export the tables individually

unless it’s not the tablespaces/DB that comprise the entire 20GB

:D
- Reply
Abe Olandres says:

19 years ago

Here’s a quick breakdown: 500MB DB, 4GB forum files, 1GB classified ads files, 1.5GB photo gallery files, then there are the attachments and other core site files.
- Reply
demonhale says:

19 years ago

Hmmm… that is a problem really, post about your solutions will ya?

Anyways heres an idea, I worked with a big site before (I just dont want to mention their URL coz their a private internal site) .. Which had their own hosting, but the Outer parts for public visitors are hosted by an external host located cross-country away. I issued an order to the dept. to send a request to the external host to send us a collection of the backup files written in discs and labeled appropriately, boxed and sent via a freight service. We paid for the overhead cost as well as service fee (affordable considering all the work that needs to be done) …
- Reply
kzap says:

19 years ago

how bout clean it up first, im sure theres a lot of unneeded stuff there
- Reply
hoop says:

19 years ago

I usually run optimize table in phpmyadmin to defragment the tables and reclaim some unused disk space. This might help to bring that DB down further

Maybe you should work with the smaller tables first and see how much it brings down the file size…

I’d suggest preventing access to the DB while optimizing if incase you go down this path to prevent corruption of the DB.

then once you’ve uploaded the DB to the new server you can dump the old DB so that all you have to worry about is the 1gb classifieds and 1.5 photofiles

:D
- Reply
markku says:

19 years ago

There are FTP clients that can do FTP-to-FTP transfers. I’m sure that would take some time, but possibly the best solution right now, without requiring you to work on the site immediately.
- Reply
Abe Olandres says:

19 years ago

What we did was to backup the whole site via ssh (./scripts/pkgacct) which resulted into a tarball of about 17GB file. Copying took about 5 hours.

Once in the ne server, we did a manual restore via SSH again (./scripts/restorepkg) and the un-packaging took about 2 hours and the restore another 4 hours.

During this period, if your own net connection blips for one second, the process aborts, which happened once.
- Reply
bimbo says:

19 years ago

well it certainly took its time! ;)
- Reply
hoop says:

19 years ago

Ah the ever reliable SSH, nice to know it can handle that much data without any corruption or termination… this is a tip that will come in handy in the future

:D
- Reply
Load Philippines says:

15 years ago

I just adore stumbleupon for being able to show me this web page. instant fan now. I have viewed some definitely poor websites on stumble and when I discovered this, I immediately knew it was excellent! This blogpost is informative specifically for somebody like me haha. I’ve been attempting to read a lot more these days and being able to discover a web page like this can be excellent for my education. Love what you are doing, keep it up!
- Reply