I decided to do a test migration of one of my project websites onto an AWS EC2 instance in July of 2019. I was thinking about migrating it, and wanted to see just how much work it would take. I’m sharing the information here, because I found it pretty difficult to get a quick estimate for how long it would all take and how much trouble it might be.
How to create an EC2 Instance
- Sign in to Amazon AWS console. Don’t have an account? You will need to create one!
- Go to the Launch Instance Wizard.
- Choose Amazon Linux 2 AMI (HVM), SSD Volume Type – ami-0b898040803850657 (64-bit x86). You may want to choose something different depending upon your application properties. It is pretty low risk, since you can always delete an instance, and create a new one.
- Pick the one you want from a list of possible instances. I chose “Type” t2.small. Click the “Review and Launch” button at the lower right of screen.
- When you click Launch, you see a banner on the next page which says “Your instance configuration is not eligible for the free usage tier”. Okay, not free! Maybe you want a free one; you can go back to the previous page and change your setting to something else.
- On the current page, you can edit security groups. But, I didn’t want this yet. I viewed my instance details, and they looked good with the default settings. The “purchasing option” was “on demand”. There was no “termination protection”, which sounds a bit risky, but my own application was experimental, so I didn’t care. By default, the “Storage size” is 8GiB. I guess that’s okay for me.
- Click the “Launch” button in the lower right corner. A popup appears, reading “Select an existing key pair or create a new key pair”. I elected to create a new key pair. I named it fullstackoasis_key, then clicked “Download Key Pair”. This downloads a file with the .pem suffix, fullstackoasis_key.pem.
- The instructions say “For Linux AMIs, the private key file allows you to securely SSH into your instance.” Great, since I really need that to set up my environment!
- After downloading the .pem file, I see a screen which says the instance is being launched, and I shouldn’t close it. In just a few seconds, the instance is launched. I see a message like: “The following instance launches have been initiated: i-15543e0a907162afe”
Total time to do this: 20 minutes.
How to connect to the instance
My instance is set up, and I need to connect!
I want to connect with a Linux SSH client.
The documentation gives this example:
ssh -i /path/my-key-pair.pem ec2-user@ec2-198-51-100-1.compute-1.amazonaws.com
- You must find your “Public DNS” to set the correct string when using ssh to connect.
- I went to my console (click Services in upper left corner, click EC2). There are a lot of services listed.
- I have “1 running instance”. I clicked that link.
- The list shows my instance, and the Public DNS (IPv4). I clicked that, because it wasn’t shown in full. Now, I could see the full string at the bottom of the page. I copied it to a file. Now I could use it with the ssh command as shown in the example above:
ssh -i /path/to/my/key/fullstackoasis_key.pem ec2-user@ec2-XX-XXX-XX-XX.compute-1.amazonaws.com
Don’t just copy my little snippet! Make sure you use the correct Public DNS string.
Now I got this error (after answering ‘yes’ to the question and hitting enter):
The authenticity of host 'ec2-XX-XXX-XX-XX.compute-1.amazonaws.com (XX-XXX-XX-XX)' can't be established.<br>ECDSA key fingerprint is …<br>Are you sure you want to continue connecting (yes/no)? yes<br>Warning: Permanently added 'ec2-XX-XXX-XX-XX.compute-1.amazonaws.com,XX-XXX-XX-XX' (ECDSA) to the list of known hosts.<br> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@<br> @ WARNING: UNPROTECTED PRIVATE KEY FILE! @<br> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@<br>Permissions 0664 for '/path/to/my/key/fullstackoasis_key.pem' are too open.<br>It is required that your private key files are NOT accessible by others.<br>This private key will be ignored.<br>Load key "/path/to/my/key/fullstackoasis_key.pem": bad permissions<br>Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Oops!
To fix the problem, I did this:
chmod o-r /path/to/my/key/fullstackoasis_key.pem
chmod g-rw /path/to/my/key/fullstackoasis_key.pem
ls -lairt fullstackoasis_key.pem
6965735 -rw------- 1 fullstackdev fullstackoasis 1696 Jul 20 12:42 fullstackoasis_key.pem
Now, I could sign in. I repeated my ssh command, and successfully connected to my EC2 instance via ssh.
Total time to do this: 20 minutes.
Install PHP 5.x
Apply all updates:
sudo yum update
I went down a rabbithole trying to install PHP 5.6.
mkdir installs
cd installs
sudo yum -y update
sudo yum install –y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo wget https://centos7.iuscommunity.org/ius-release.rpm
sudo rpm -Uvh ius-release*.rpm
sudo yum -y update
sudo yum -y install php56u php56u-opcache php56u-xml php56u-mcrypt php56u-gd php56u-devel php56u-mysql php56u-intl php56u-mbstring php56u-bcmath php56u-soap
output:
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
189 packages excluded due to repository priority protections
No package php56u available.
No package php56u-opcache available.
No package php56u-xml available.
No package php56u-mcrypt available.
No package php56u-gd available.
No package php56u-devel available.
No package php56u-mysql available.
No package php56u-intl available.
No package php56u-mbstring available.
No package php56u-bcmath available.
No package php56u-soap available.
Error: Nothing to do
Nothing to do?? Then I did this:
php --version
PHP 5.6.40 (cli) (built: Jan 11 2019 10:27:04)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
Guess I should have done that to begin with. It would appear that PHP 5.6 was preinstalled, something I did not expect.
I “wasted” about 20 minutes on this task.
Install MySQL
First, see if MySQL is installed:
rpm -qa | grep -i mysql
There was no output from this command, so I assumed that MySQL was not installed.
This StackOverflow instruction did not work.
sudo yum install mysql56-server
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core | 2.4 kB 00:00:00
189 packages excluded due to repository priority protections
No package mysql56-server available.
Error: Nothing to do
Nothing to do?? This worked:
wget http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.23-1.el7.x86_64.rpm-bundle.tar
tar -xvf MySQL-5.6.23-1.el7.x86_64.rpm-bundle.tar
sudo yum -y install MySQL-client-5.6.23-1.el7.x86_64.rpm
sudo yum install MySQL-shared-compat-5.6.23-1.el7.x86_64.rpm
...
Installed:
MySQL-shared-compat.x86_64 0:5.6.23-1.el7
Dependency Installed:
mariadb101u-common.x86_64 1:10.1.35-1.ius.el7 mariadb101u-config.x86_64 1:10.1.35-1.ius.el7 mariadb101u-libs.x86_64 1:10.1.35-1.ius.el7
Replaced:
mariadb-libs.x86_64 1:5.5.62-1.amzn2.0.1
Complete!
sudo yum install MySQL-server-5.6.23-1.el7.x86_64.rpm
...
Running transaction
Installing : MySQL-server-5.6.23-1.el7.x86_64 1/1
warning: user mysql does not exist - using root
warning: group mysql does not exist - using root
FATAL ERROR: please install the following Perl modules before executing /usr/bin/mysql_install_db:
Data::Dumper
Verifying : MySQL-server-5.6.23-1.el7.x86_64 1/1
Installed:
MySQL-server.x86_64 0:5.6.23-1.el7
Complete!
Oops, notice that error! It comes back to bite me later. If you want to avoid it, install that module before doing anything MySQL related.
sudo service mysqld start
Redirecting to /bin/systemctl start mysqld.service
Failed to start mysqld.service: Unit not found.
It doesn’t work… maybe because of the Perl issue, so let’s install that.
sudo yum install perl perl-Data-Dumper
sudo yum remove MySQL-server-5.6.23-1.el7.x86_64
Then, again,
sudo yum install MySQL-server-5.6.23-1.el7.x86_64.rpm
...
sudo yum install MySQL-server-5.6.23-1.el7.x86_64.rpm
...
Installed:
MySQL-server.x86_64 0:5.6.23-1.el7
sudo mysqld_safe<br>190720 17:34:47 mysqld_safe Logging to '/var/lib/mysql/ip-XXX-XX-XX-XXX.ec2.internal.err'.<br>190720 17:34:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql<br>190720 17:34:49 mysqld_safe mysqld from pid file /var/lib/mysql/ip-XXX-XX-XX-XXX.ec2.internal.pid ended
I looked at the log file and saw this:
2019-07-20 17:34:49 9458 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist
So then I ran this:
sudo mysql_install_db
Then, I saw this in the log file:
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h ip-XXX-XX-XX-XXX.ec2.internal password 'new-password'
...
You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &
So I tried it:
cd /usr ; sudo /usr/bin/mysqld_safe &
[1] 9802
[ec2-user@ip-XXX-XX-XX-XXX usr]$ 190720 17:43:07 mysqld_safe Logging to '/var/lib/mysql/ip-XXX-XX-XX-XXX.ec2.internal.err'.
190720 17:43:07 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190720 17:43:07 mysqld_safe mysqld from pid file /var/lib/mysql/ip-XXX-XX-XX-XXX.ec2.internal.pid ended
[1]+ Done sudo /usr/bin/mysqld_safe
The log file showed:
....
/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13 - Permission denied)
2019-07-20 17:43:07 9891 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
...
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
...
2019-07-20 17:43:07 9891 [ERROR] InnoDB: Could not find a valid tablespace file for 'mysql/innodb_index_stats'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue.
...
190720 17:43:07 mysqld_safe mysqld from pid file /var/lib/mysql/ip-XXX-XX-XX-XXX.ec2.internal.pid ended
Solution: I needed to change permissions to mysql:mysql for all mysql files in /var/lib/mysql and also in performance_schema directory as follows:
sudo mysqladmin -p shutdown
sudo chown -R mysql:mysql /var/lib/mysql/mysql
sudo chown -R mysql:mysql /var/lib/mysql/performance_schema
Then, I did this again:
cd /usr ; sudo /usr/bin/mysqld_safe &
/usr/bin/mysqladmin -u root password 'MyReallyGoodPassword'
/usr/bin/mysqladmin -u root -h ip-XXX-XX-XX-XXX.ec2.internal password 'MyReallyGoodPassword'
Success! At this point, I saw no more errors in the logs.
Total time to do get MySQL running: 1 hour.
Install and Run Apache
I used the instructions at this gist.
sudo service httpd stop
Redirecting to /bin/systemctl stop httpd.service
I checked the logs, /var/log/httpd/error_log. Apache is using PHP 5.6.
[Sat Jul 20 18:11:39.277716 2019] [mpm_prefork:notice] [pid 10429] AH00163: Apache/2.4.39 (IUS) PHP/5.6.40 configured -- resuming normal operations
I also read the Amazon page on setting up a web server, which said:
“To set file permissions for the Apache web server
Add the www group to your EC2 instance with the following command.”
sudo groupadd www
sudo usermod -a -G www ec2-user
exit
You have to sign out and ssh back in again to see the permissions take place. I followed the Amazon tutorial and did as follows:
groups
The output was ec2-user adm wheel systemd-journal www which looked correct. Then I did this:
sudo chgrp www /var/www<br>sudo chmod 2775 /var/www<br>find /var/www -type d -exec sudo chmod 2775 {} +<br>find /var/www -type f -exec sudo chmod 0664 {} +
Now I need to set up port 80 access to Apache.
Under my AWS console instance panel, I clicked “Network & Security”
I found the group my instance is a part of (the most recent one).
I clicked the “Actions” dropdown, to edit “inbound rules”. I selected the following: HTTP, TCP, 80, Custom, 0.0.0.0/0. Description can be empty. Make sure not to overwrite the existing SSH rule!
Do the same to add a rule for HTTPS.
Then click the Save button.
Now my Apache server works! I can surf to this page in a browser: http://my.public.ip.address/index.html
(If this doesn’t work for you, make sure you’ve added a file called index.html to the /var/www/html directory.)
Total time to get Apache running: 20 minutes.
Install and Run Apache with PHP
I added a file named test.php to /var/www/html/test.php:
<?php
echo "PHP is beautiful"
?>
I can access it immediately by surfing to http://my.public.ip.address/test.php
Time: negligible.
Import old MySQL database
I took my existing MySQL database from my other server, and migrated it to AWS as follows.
scp -i /path/to/my/key/fullstackoasis_key.pem /tmp/amysql_dump_file.dump ec2-user@ec2-XX-XXX-XX-XX.compute-1.amazonaws.com:~/
mysqladmin -u root -p create my_old_dbname
mysql -u root -p my_old_dbname < amysql_dump_file.dump
mysql -u root -p my_old_dbname
Enter password:
...
Server version: 5.6.23 MySQL Community Server (GPL)
mysql> show tables;
+--------------------------+
| Tables_in_my_old_dbname |
+--------------------------+
| abc |
| def |
| ..... |
| tableblah |
+--------------------------+
GRANT ALL PRIVILEGES ON *.* TO 'oldusername'@'localhost' IDENTIFIED BY 'oldWonderfulPassword';
Total time to import the old database: 20 minutes.
Migrate existing PHP code to current html directory
I just copied this over to my EC2:
scp -i /path/to/my/key/fullstackoasis_key.pem -pr ~/MyLocalProjectDir ec2-user@ec2-XX-XXX-XX-XX.compute-1.amazonaws.com:~/MyRemoteProjectDir
I wanted to symlink the code in my home directory to the web server:
cd /var/www/html
sudo ln -s /home/ec2-user/MyRemoteProjectDir oldsite
sudo chown -h ec2-user:www /var/www/html/oldsite
However, after I ran this, I got a 403 for my “oldsite” code, unlike the test page I had accessed earlier. One of my pages, let’s call it demo.php, in my oldsite dir was not accessible.
I found advice to use the namei command, like this:
namei -m /home/ec2-user/MyRemoteProjectDir
f: /home/ec2-user/MyRemoteProjectDir
dr-xr-xr-x /
drwxr-xr-x home
drwx------ ec2-user
drwxrwxr-x MyRemoteProjectDir
So this was a permissions issue; I needed to open up ec2-user to the world. I don’t necessarily recommend doing this. You maybe want to set up a different user which “hosts” the web application code. But in my case, it didn’t matter.
chmod g+x /home/ec2-user/
chmod o+x /home/ec2-user/
namei -m /home/ec2-user/MyRemoteProjectDir
f: /home/ec2-user/MyRemoteProjectDir
dr-xr-xr-x /
drwxr-xr-x home
drwx--x--x ec2-user
drwxrwxr-x MyRemoteProjectDir
After that, I was able to surf to my test page, http://my.public.ip.address/oldsite/test1.php
At that point, I found that my old code wouldn’t function. Errors appeared in the Apache log /var/log/httpd/error_log like this:
PHP Fatal error: Call to undefined function mysql_connect() in /home/ec2-user/oldsite/dbfunc.php on line 3
This was a real puzzler. (Aside: mysql_* functions are deprecated, so my site needs a revamp!). I added this code to a test php file:
<?php phpinfo(); ?>
I could see in the output that the web server had MySQL and PHP installed, with entries for mysqli, something called mysqlnd, and so on.
I tried switching my database code to use mysqli commands, but that didn’t work either; I still got complaints in the log like Call to undefined function mysqli_connect(). I thought, maybe there’s something wrong with the PHP install? I tried running yum:
yum search php56
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
189 packages excluded due to repository priority protections
============================================================================================ N/S matched: php56 ============================================================================================
php56u.x86_64 : PHP scripting language for creating dynamic web sites
php56u-cli.x86_64 : Command-line interface for PHP
php56u-common.x86_64 : Common files for PHP
php56u-pear.noarch : PHP Extension and Application Repository framework
php56u-pecl-jsonc.x86_64 : Support for JSON serialization
php56u-process.x86_64 : Modules for PHP script using system process interfaces
php56u-xml.x86_64 : A module for PHP applications which use XML
I did not understand what was going on at this point.
I decided to try undoing what I’d done earlier.
sudo yum remove php56u php56u-opcache php56u-xml php56u-mcrypt php56u-gd php56u-devel php56u-mysql php56u-intl php56u-mbstring php56u-bcmath php56u-soap
The output indicated that nothing happened, which was super confusing.
sudo yum remove MySQL-server-5.6.23-1.el7.x86_64
Then I tried adding some packages I’d seen in earlier output from yum:
sudo yum -y install php-mysql.x86_64
sudo yum -y install php.x86_64
Strangely, when I tested PHP after this, I found the server was running PHP 5.4. I don’t understand the downgrade. But I was hoping that the php-mysql package would just make PHP work with MySQL. I stopped and started Apache:
sudo service httpd stop
sudo service httpd start
After that, I got a page that connected to the database from my old website to work, success!
http://my.public.ip.address/oldsite/myoldpage.php
That must mean that my previous attempts to install PHP+MySQL didn’t work.
Total time to migrate migrate existing PHP code: 1.5 hours.
I didn’t time everything I did… the total time to set everything up was about 4 hours, from start to finish.
Things I didn’t do
I didn’t register a domain name or set up a DNS for my old site on AWS. My domain name is registered at my managed hosting service for a very reasonable cost.
I didn’t add an SSL certificate; without a domain name, what’s the point? My website is on a managed hosting service, so I don’t have to worry about that. I just tell them to add HTTPS, and it’s done in a few minutes. In the past, I’ve worked with LetsEncrypt to generate SSL certificates automatically. So if I needed to do that here, I might try that. I’m not sure if it’s really worth the trouble, since SSL certificates are fairly cheap.
I didn’t set up a backup and recovery plan. I am not worried about what happens if the instance goes down, since this was just an experiment.
Warning
I’m just messing around with AWS EC2, and I’m not an expert in it at all. It’s possible that I did some things wrong. If you notice something and want to make a comment, just email me at fullstackdev@fullstackoasis.com – I appreciate comments!
One of the things I noticed when I did this work was that the website I was going to port is “kind of out-of-date” :). So I’ll take a little time to get that application in better shape.