Mounting remote directories to Scientific Linux / CentOS /RedHat 6 via SSH using Fuse

Posted on September 1, 2011 by William Jamieson

Often I need to mount a directory from a remote machine onto my Linux server so that I can dump backups or perform other tasks. You can certainly use samba to mount windows shares but for Mac and Linux shares SSHFS shares are more consistently supported, especially since Apple’s change’s to Samba in Mac OS X 10.7 Lion.

This tutorial will step you through how to mount a remote directory onto your Scientific Linux / CentOS /RedHat 6 machine.

For a mount on demand solution

  1. Login to your linux server as root

  2. Install fuse-sshfs

            #yum install fuse-sshfs
  3. Create a mount directory

            #mkdir /mnt/sshfsMount
  4. Mount the remote directory

            #sshfs user@192.168.1.1:/mySharePath /mnt/sshfsMount/

    SSHFS will then ask you to authenticate with the password for the user account you supplied

  5. To unmount the drive:

            #umount /sshfs/sshfsMount/

To automate the process so that the server automatically mounts

  1. Setup SSH keys so that the linux machine can automatically login to the remote machine via SSH
    To set this up you can see our tutorial on setting up automatic SSH logins without using a password

  2. Setup SSH keys so that the linux machine can automatically login to the remote machine via SSH
    To set this up you can see our tutorial on setting up automatic SSH logins without using a password

    Now you should be able to use the following command without being asked for a password

                    #sshfs user@192.168.1.1:/mySharePath /mnt/sshfsMount/
  3. Install autofs

                    #yum install autofs
  4. Get the user and group id of the user you wish to control the share.
    In this example I’m going to use the root account

                    #cat /etc/passwd | grep root
                root:x:0:0:root:/root:/bin/bash

    The user id is the first number, and the coup id is the second number (0 and 0)

  5. Edit the /etc/auto.master file and add the following line under the line for /misc, substituting your chosen user and group id’s
    This allows us to mount to any single directory of our choosing under root ( / ) as the local mounting point without having to use sub directories.

                    /-              /etc/auto.sshfs  uid=0,gid=0,--timeout=30,--ghost

    Also comment out the following line by placing a # in front of it to avoid nsswitch errors

                    +auto.master
  6. Now create the file /etc/auto.sshfs and add the following line to it

                    /mnt/sshfsMount/ -fstype=fuse,rw,nodev,nonempty,noatime,allow_other,max_read=65536  :sshfs\#user@192.168.1.1\:/mySharePath

    Now this remote directory will mount into the folder /mnt/sshfsMount every time we access that folder. If the folder is not being used for more than 30 seconds, it will automatically be unmounted.

  7. Restart autos:

                    #service autofs restart
  8. test that the directory is mounted by listing its contents:

                    #ls /mnt/sshfsMount
  9. To unmount the drive:

                    #umount /mnt/sshfsMount/
Posted in Development | Leave a comment

Installing VMWare Tools on a terminal only Scientific Linux / CentOS /RedHat 6 Machine

Posted on September 1, 2011 by William Jamieson

VMWare Tools can greatly improve the speed, efficiency and manageability of your virtual environment as well as provide a number of other key benefits.

This tutorial will step you through how to install VMWare tools on a Scientific Linux / CentOS /RedHat 6 virtual machine with no GUI.

  1. Start your virtual machine instance and log in as root using the actual VMWare window rather than a remote terminal session.

  2. Go to the /tmp directory

            #cd /tmp
  3. On the parent host system, select the menu option: Virtual Machine > Install VMWare Tools

    VMWare will then download the latest version of VMWare Tools and ask you to authenticate using a valid admin account on the parent host system.
    A dialog box will appear asking you to confirm the installation, click ‘Install’.

  4. Mount the virtual CD-ROM and copy the files to /tmp

            #mount -o ro /dev/cdrom /mnt
        #cp /mnt/* .
  5. Extract the files from the archive

            #tar xvfz VMwareTools*
        #cd vmware-tools-distrib
  6. Run the installation script

            #perl vmware-install.pl

    Press Enter at all of the prompts to accept the script default settings

  7. Reboot the virtual machine

            #reboot

    Then re-login as root when it comes back up

  8. Check that the installation was successful

            #vmware-checkvm
        VMware software version 6 (good)

You will need to reinstall VMWare tools each time you upgrade/change the kernel on the virtual machine

Posted in Development | Leave a comment

How to upgrade Scientific Linux / CentOS /RedHat From 6.0 to 6.1

Posted on August 31, 2011 by William Jamieson

Problem

Simply running the following command won’t upgrade your system from 6.0 to 6.1 as it will only draw updated packages from the repository associated with it’s current release (6.0):

#yum update

Solution

First let’s check the version we are currently using:

#cat /etc/redhat-release
Scientific Linux release 6.0 (Carbon)

All we need to do is tell yum the version number you’d like to source updates from as follows:

#yum --releasever=6.1 update

Now lets check that the update worked:

#cat /etc/redhat-release
Scientific Linux release 6.1 (Carbon)

Simple as that!

Posted in Linux, Technology | Tagged , , , , , , | Leave a comment

Setting up a self-signed SSL certificate using ModSSL, Apache 2 and Scientific / CentOS / RedHat

Posted on July 25, 2011 by William Jamieson

Self-signed certificates are very handy for helping two known machines talk to each other in a secure (encrypted) manner over https. Self signed certificates are simply certificates that you vouch for (or trust) yourself, without having to be externally verified by a third party such as Thawte or Verisign.

Here’s how to set it up using ModSSL, Apache 2, and Scientific, CentOS, or RedHat Linux…

Configure the Server

  1. Install mod_ssl if you haven’t already: 
    yum install mod_ssl openssl openssl-devel
  2. To generate a self signed certificate, login to the server as root and generate the private key, replacing www.mysite.com with the domain you require: 
    cd ~/
openssl genrsa -out www.mysite.com.key 1024
  3. Now generate the certificate signing request (CSR) and answer the questions is asks you as guided: 
    openssl req -new -key www.mysite.com.key -out www.mysite.com.csr

---------------------------------------------------------------------------------------
Country Name (2 letter code) [GB]: <Your 2 Character Country Code>
State or Province Name (full name) [Berkshire]: <Your State>
Locality Name (eg, city) [Newbury]: <Your City>
Organization Name (eg, company) [My Company Ltd]: <Your Company Name>
Organizational Unit Name (eg, section) []: <Leave Empty Unless Required>
Common Name (eg, your name or your server's hostname) []:<Your Full Domain Name To Use With SSL>
Email Address []:<Generic Organisation Email Address.  eg. info@mysite.com>

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []: <Leave Empty Unless Required>
An optional company name []: <Leave Empty Unless Required>
---------------------------------------------------------------------------------------
  4. Now use the CSR and key to create a self-signed certificate: 
    openssl x509 -req -in www.mysite.com.csr -signkey www.mysite.com.key -out www.mysite.com.crt
  5. Create the apache SSL certificate and key directories, copy the required files and set permissions: 
    mkdir -p /etc/httpd/conf/ssl.crt
mkdir -p /etc/httpd/conf/ssl.key
cat ~/www.envision-systems.com.au.crt > /etc/httpd/conf/ssl.crt/www.envision-systems.com.au.crt
cat ~/www.envision-systems.com.au.key > /etc/httpd/conf/ssl.key/www.envision-systems.com.au.key
chmod -R 600 /etc/httpd/conf/ssl.key
chmod -R 600 /etc/httpd/conf/ssl.crt
  6. Configure apache SSL: 
    vim /etc/httpd/conf.d/ssl.conf

    <VirtualHost _default_:443>
        DocumentRoot /var/www/html/www.mysite.com
        DirectoryIndex index.php
        ServerName etc/httpd

        SSLEngine on
        SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

        SSLCertificateFile /etc/httpd/conf/ssl.crt/www.mysite.com.crt
        SSLCertificateKeyFile /etc/httpd/conf/ssl.key/www.mysite.com.key

        <Files ~ "\.(cgi|shtml|phtml|php3?)$">
            SSLOptions +StdEnvVars
        </Files>
        <Directory "/usr/local/apache/cgi-bin">
            SSLOptions +StdEnvVars
        </Directory>

        SetEnvIf User-Agent ".*MSIE.*" \
                 nokeepalive ssl-unclean-shutdown \
                 downgrade-1.0 force-response-1.0
        CustomLog /usr/local/apache/logs/ssl_request_log \
                  "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>

    Ensure that your vim /etc/httpd/conf/httpd.conf contains a NameVirtualHost entry for port 443:

    NameVirtualHost *:443
  7. Test apache configuration and restart if ok: 
    apachectl configtest
apachectl restart

Configure the Client

  1. Login to your client machine as root, and copy the www.mysite.com.crt file to your client machine then set permissions: 
    cd ~/
chmod 600 www.envision-systems.com.au.crt
  2. Install wget if you haven’t already: 
    yum install wget

    Test the certificate using wget:

    wget --ca-certificate=www.mysite.com.crt https://www.mysite.com/

    If all went well you should see something like this:

    --2011-07-25 10:36:56--  https://www.mysite.com/
Resolving www.mysite.com... 29.141.129.5
Connecting to www.mysite.com|29.141.129.5|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified 1
Saving to: `index.html.1'
    [ <=> ] 13,355      --.-K/s   in 0.001s
2011-07-25 10:36:56 (17.1 MB/s) - `index.html.1' saved [13355]
Posted in Apache, Linux, Networking, Security | Tagged , , , , , , , , , , , , , , , , | Leave a comment

Get a list of ranked client IP addresses from your apache log

Posted on July 15, 2011 by William Jamieson

Most of the software I write is for businesses that have multiple office locations. Even though the software is web based, my clients often want to lock access down so that the software can only be accessed from within one of their offices. We do this by getting the public IP address of each office and allowing only that traffic through. Because I block requests from non-registered IP addresses using PHP, apache will still handle the request. Therefore all user IP addresses will be registered within the /var/log/http/access_log whether they are registered IP addresses or not.

Sometimes you need to see if a specific IP address is trying to access your server. Here’s a handy little bash command that will list all of the IP addresses contained in the current apache log and rank them by the number of times they appear (# of requests they have made).

cat /var/log/http/access_log | awk '{print $1}' | sort | uniq -c | sort -n
Posted in Development, Linux, Networking, PHP, Technology | Leave a comment

Sending SMTP mail to gmail using PHP

Posted on August 20, 2010 by William Jamieson

Sending mail from your web server is a very common thing that many of us need to do. To do this, the most common approach is to use the following PHP code:

mail($to, $subject, $message, $headers);

This method sends a copy of the email, directly to the recipients mail server from the web server. However generally speaking, the web server is not the real mail server for the domain you’re trying to send from which can cause major issues.

For example, lets say I have a website at www.myweb.com and when someone registers it sends them an email from the address website@myweb.com. The recipients mail servers spam filtering facility may do an MX lookup on the myweb.com domain and notice that the real mail server for the myweb.com has a different IP address to the address where the email is originating from (the web servers IP address). For this reason web server emails are often flagged as spam and thus never received by the intended recipient.

To avoid this, we can relay the mail through the domains real mail server using SMTP. Using this method the mail will come to the recipients mail server from the real mail server for the senders domain as seen in its MX records. This then avoids the problematic spam filter rule.

To achieve this I use the Mail.php class as found in the PEAR::Mail library located at http://pear.php.net/package/Mail. If you’re using CentOS like me, you can easily install this package using the following command

#yum install php-pear-Mail

In this example I’m connecting to a gmail server which requires the connection to be over SSL:

 

<?php
require_once('Mail.php');
$host = 'ssl://smtp.gmail.com';
$port = '465';
$username = '<your email address>';
$password = '<yourPassword>';
$subject = 'my subject';
$to = '<to email address>';

$from = $username;
$message = 'test';
$headers = array ('From' => $from,
   'To' => $to,
   'Subject' => $subject);
$smtp = Mail::factory('smtp',
   array ('host' => $host,
     'port' =>; $port,
     'auth' => true,
     'username' => $username,
     'password' => $password));
$mail = $smtp->send($to, $headers, $message);
if (PEAR::isError($mail)) {
        echo('<p>'.$mail->getMessage().'</p>');
}
?>

If SSL wasn’t required I’d just need to remove the ‘ssl://’ from in front of the host declaration and change the port to 25 like this:

$host = 'mail.myweb.com';
$port = '25';

Obviously you’ll need to ensure that your firewall allows your web server to make connections to your smtp host over port 25 or 465 for SSL.

Posted in Development, Linux, Networking, PHP | Tagged , , , , , , , , , , , , , , , , , , , , , , , , , , , | 2 Comments

Performance tuning a CentOS LAMP web server for high traffic volumes

Posted on August 17, 2010 by William Jamieson

In August 2010 I was contracted to performance tune a LAMP server to handle approximately 70 full page loads per second which equated to 4,250 concurrent virtual users. We ended up doubling this expectation to 140 full page loads per second without striking issue. If this speed was maintained for 24 hours it would equate to over 12 million hits per day. This article will let you know how we achieved it.

The load tests were conducted using the HP performance center; a technology that HP obtained as part of its acquisition of Mercury for approximately USD$4.5 billion in 2006.

To find out more about the load testing software visit http://en.wikipedia.org/wiki/HP_LoadRunner

Goal:
Handle 4,250 concurrent users generating approximately 70 full page loads per second.

1 full page load consisted of:
- 1 dynamically generated PHP file using MySQL
- 4 JavaScript files
- 7 CSS files
- 8 image files

Original starting environment:
- ServerModel: Dell R300
- RAM: 2GB (2 x 1GB chips)
- Operating System: CentOS release 5.5 (Final)
- Apache: v2.2.3 (running in prefork mode)
- MySQL: v5.0.77
- PHP: v5.1.6 (as an apache module)
- eAccelerator: v0.9.5.3
- 120Mbits of bandwidth

 

Round 1: Initial Test

Round 1: Configuration

At the start of the process we were pretty much using the default configurations for the entire lamp stack. Linux was running iptables and ip6tables in its default configuration. eAccelerator was operating with 32MB of memory with optimization and caching enabled.

Apache (/etc/httpd/conf/httpd.conf):
For more info on variables for Apache 2.0.x go to: http://httpd.apache.org/docs/2.0/mod/mpm_common.html

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

MySQL (/etc/my.cnf):
For more info on variables for MySQL 5.0.x go to: http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html

[mysqld]
max_connections = 100
max_user_connections = 0
max_connect_errors = 10
max_allowed_packet = 1M
table_cache = 64
sort_buffer_size = 2M
read_buffer_size = 131072
read_rnd_buffer_size = 262144
myisam_sort_buffer_size = 8M
thread_cache_size = 0
query_cache_size= 0
thread_concurrency = 10

Round 1: Results

With these settings we got up to 30 page loads per second which was 42% of our target. Interestingly, we were only operating at about 8% CPU and about 50% of our memory capacity when we hit this limit.

Round 1: Review

Looking at the apache error logs we were getting a large number of MySQL errors:

mysql_connect() [<a href='function.mysql-connect'>function.mysql-connect</a>]: Too many connections in xxx.php on line 15

So the MySQL configuration seemed to be our bottleneck:

 

Round 2

Round 2: Configuration

We did our first major review of the Apache and MySQL performance settings and adjusted them accordingly. We doubled the Apache settings and used the ‘huge’ configuration as supplied with mysql (/usr/share/doc/mysql-server-5.0.77/my-huge.cnf).

Apache (/etc/httpd/conf/httpd.conf):
For more info on variables for Apache 2.0.x go to: http://httpd.apache.org/docs/2.0/mod/mpm_common.html

<IfModule prefork.c>
StartServers       16
MinSpareServers    10
MaxSpareServers   40
ServerLimit      512
MaxClients       512
MaxRequestsPerChild  8000
</IfModule>

MySQL (/etc/my.cnf):
For more info on variables for MySQL 5.0.x go to: http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html

[mysqld]
# Memory usage
skip-locking
max_connections = 500
max_user_connections = 500
max_connect_errors = 999999
key_buffer = 384M
max_allowed_packet = 1M
table_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 32M
# Try number of CPU's*2 for thread_concurrency (eHound has 4 CPU's)
thread_concurrency = 8

# Disable Federated by default
skip-federated

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

As an extra precaution we locked the network card in the server to use 1Gbit:

#ethtool -s eth0 speed 1000 duplex full

Edit the configuration for the network card:

#vim /etc/sysconfig/network-scripts/ifcfg-eth0

Add the following line:

ETHTOOL_OPTS='autoneg on speed 1000 duplex full'

Restart the network:

#service network restart

Round 2: Results

With these settings we got up to 58 full page loads per second which was 59% of our target. Interestingly, we were still only operating at about 10% CPU capacity when we hit this limit but we were using approximately 70-80% of our memory.

Our MySQL errors had disappeared and there were no more errors in the Apache logs.

Round 2: Review

We were concerned that the system was starting to use swap memory which was slowing the server to a halt.

 

Round 3

Round 3: Configuration

We added an additional 2GB of RAM to the server so it now contained 4 x 1GB chips.

Round 3: Results

With the new RAM we still only got up to 58 full page loads per second which was 59% of our target. We were still only operating at about 10% CPU capacity but now we were only using about 40% of our memory.

Round 3: Review

Still no errors in the Apache logs and the load test farm was not receiving Apache errors. In fact it was reporting that it could not even connect to the server. This led us to believe that it was either a lack of bandwidth or a NIC/network/firewall configuration issue. After checking with our datacenter, we found that there were no inhibiting factors that would cause the problem described.

We increased the Apache & MySQL Limits and ran a different style of test.

 

Round 4

Round 4: Configuration

In this test we only loaded the dynamic components of the page as generated by PHP and MySQL and served by Apache. This meant that we told the load test farm not to download static content such as images, CSS or JavaScript files.

Also we increased the MySQL and Apache limits as follows:

Apache (/etc/httpd/conf/httpd.conf):
For more info on variables for Apache 2.0.x go to: http://httpd.apache.org/docs/2.0/mod/mpm_common.html

<IfModule prefork.c>
StartServers     280
MinSpareServers   100
MaxSpareServers   300
ServerLimit      1536
MaxClients       1536
MaxRequestsPerChild  32000
</IfModule>

MySQL (/etc/my.cnf):
For more info on variables for MySQL 5.0.x go to: http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html

[mysqld]
# Memory usage
skip-locking
max_connections = 764
max_user_connections = 764
max_connect_errors = 999999
key_buffer = 256M
max_allowed_packet = 1M
table_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
# Try number of CPU's*2 for thread_concurrency (eHound has 4 CPU's)
thread_concurrency = 8

# Disable Federated by default
skip-federated

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

Round 4: Results

The results of this test were very interesting. We got up to 263 page loads without any issue. This consumed a lot more bandwidth than test 3 so we knew that bandwidth was not the issue. However the number of connections that both tests started to fail at were very similar.

Round 4: Review

So we knew we had a connection limit issue.

We also knew that the eAccelerator optcode cache was not dying at these high volumes, nor was MySQL, PHP or Apache.

We reviewing the kernel messages and found thousands of the following messages that were logged at the time of testing:

#cat /var/log/messages* | grep 'Aug 15'
...
Aug 15 01:04:27 localhost kernel: printk: 1395 messages suppressed.
Aug 15 01:04:27 localhost kernel: ip_conntrack: table full, dropping packet.
Aug 15 01:04:32 localhost kernel: printk: 1561 messages suppressed.
Aug 15 01:04:32 localhost kernel: ip_conntrack: table full, dropping packet.
Aug 15 01:04:37 localhost kernel: printk: 1274 messages suppressed.
Aug 15 01:04:37 localhost kernel: ip_conntrack: table full, dropping packet.
Aug 15 01:04:42 localhost kernel: printk: 1412 messages suppressed.
...

Further investigation revealed that the iptables/ip6tables was activated and limiting the number of connections to the box because its table was full. Ordinarily when I set up a linux server I turn iptables off because I place hardware firewalls in front of the servers. However I didn’t have the opportunity to setup this box initially, so they were still activated. I however didn’t need them, so I deactivated them.

If you still need to keep iptables running you can simply adjust the following settings:
Check the current connections limit (only works if iptables is running):

#sysctl net.ipv4.netfilter.ip_conntrack_max
65536

Change the connections limit:

#vim /etc/sysctl.conf

Add the following lines:

# conntrack limits
#inet.ipv4.netfilter.ip_conntrack_max = 65536
net.ipv4.netfilter.ip_conntrack_max = 196608

Reload the config file:

#sysctl -p

Check the new connections limit:

#sysctl net.ipv4.netfilter.ip_conntrack_max
196608

Check the current buckets limit (only works if iptables is running):

#cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
8192

To change the buckets limit:

#vim /etc/modprobe.conf

Add the following lines:

options ip_conntrack hashsize=32768

Reboot the server:

#shutdown -r now

Check the new buckets limit:

#cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
24576

–
Alternatively if you don’t need iptables like me, you can just disable them:

#service iptables stop
#service ip6tables stop
#chkconfig iptables off
#chkconfig ip6tables off

 

Round 5

Round 5: Configuration

This test used exactly the same configuration with iptables disabled.

Round 5: Results

Success!!! We got to 4,250 concurrent users which is about 70 pages per second (loading all additional image, CSS and JavaScript files also) with zero errors and a 0.7 second average response time. This used about 120Mbits worth of bandwidth pipe. The datacenter ended up running out of pipe before the server had any issues.

At this rate we were running at about:
- 15% CPU utilisation
- 30% Memory usage (with 4GB RAM installed)
- 400 apache threads
- 100% Bandwidth

Round 5: Review

Key findings:
- Increase your Apache and MySQL limits
- Turn off iptables
- Ensure that you have enough RAM
- Ensure that you are checking logs from MySQL, Apache, and the kernel to pick up any errors and give you clues as to how to best solve them

 

Round 6

Round 6: Configuration

This test used exactly the same configuration as round 5 with 250Mbit pipe instead of a 120Mbit pipe.

Round 6: Results

Success!!! We got to 140 full page loads per second (including additional images, CSS and JavaScript files also) with zero errors and still a stable 0.7 second average response time. This used the full 250Mbits worth of bandwidth pipe. The datacenter ended up running out of pipe again before the server had any issues.

At this rate we were running at about:
- 30% CPU utilisation
- 40% Memory usage (with 4GB RAM installed)
- 800 apache threads
- 100% Bandwidth

Round 6: Review

Key findings:
- Even with 250Mbits of pipe, bandwidth is still the bottleneck in this configuration.

 

Round 7

Round 7: Configuration

Even though our server was performing fine, we were given another server to experiment on with much higher specs.

It was a Dell R710 with 48GB of RAM and 8 2.53MHz Xeon processors running in hyper-threading mode (essentially making 16 processors).

We also had this box connected to a dedicated 4Gbit optical internet feed to give it as much bandwidth as it needed.

Everything on the box was configured the same except for Apache and MySQL (which we took the last settings and multipled them by 4) and sysctl.

Apache (/etc/httpd/conf/httpd.conf):
For more info on variables for Apache 2.0.x go to: http://httpd.apache.org/docs/2.0/mod/mpm_common.html

<IfModule prefork.c>
StartServers     1120
MinSpareServers   400
MaxSpareServers   1200
ServerLimit      6144
MaxClients       6144
MaxRequestsPerChild  128000
</IfModule>

MySQL (/etc/my.cnf):
For more info on variables for MySQL 5.0.x go to: http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html

[mysqld]
# Memory usage
skip-locking
max_connections = 3056
max_user_connections = 3056
max_connect_errors = 999999
key_buffer = 1024M
max_allowed_packet = 4M
table_cache = 1024
sort_buffer_size = 4M
read_buffer_size = 4M
read_rnd_buffer_size = 16M
myisam_sort_buffer_size = 256M
thread_cache_size = 32
query_cache_size= 64M
# Try number of CPU's*2 for thread_concurrency (eHound has 4 CPU's)
thread_concurrency = 32

# Disable Federated by default
skip-federated

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[mysqldump]
quick
max_allowed_packet = 64M

[mysql]
no-auto-rehash

[isamchk]
key_buffer = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M

[myisamchk]
key_buffer = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M

[mysqlhotcopy]
interactive-timeout

We also added the following lines to sysctl:
ip_conntrack_max = 196608
net.ipv4.ip_local_port_range = 1025 65535   
net.ipv4.tcp_max_tw_buckets = 1000000
net.core.somaxconn = 10000
net.ipv4.tcp_max_syn_backlog = 2000
net.ipv4.tcp_fin_timeout = 30

Round 7: Results

We got to 200 full page loads per second (including additional images, CSS and JavaScript files also) with zero errors and still a stable 0.8 second average response time. This test used 330Mbits or about 8% worth of the bandwidth available. We stopped the test simply because we didn’t need to go any higher, but potentially could have gone much higher.

At this rate we were running at about:
- 16% CPU utilisation
- 6% Memory usage (with 48GB RAM installed)
- 1227 apache threads
- 8% Bandwidth

Round 7: Review

Key findings:
- Bandwidth seem to be a much bigger bottleneck than server capability.

 

Summary

A Dell R300 with good specs can be acquired for about AUD$4,000-$4,500 and should handle over 2800 hits per second on a 20/1 dynamic/static file ratio or 140 full page loads per second in our case if it is configured correctly.

You don’t need a $25,000 server to get good performance, you just need to take some time to make sure that you are getting the most out of your hardware and ensure that your available bandwidth can handle the load you require.

Posted in Architecture, Linux, MySQL, Networking, Performance Tuning, PHP | Tagged , , , , , , , , , , , , , , , , | 5 Comments

Monitoring CentOS server performance using sar

Posted on August 17, 2010 by William Jamieson

Sar can be used to record a large number of server resource utilisation statistics in a binary log format which you can then analyse.

To install sar:

#yum install sysstat

To run sar and save the binary output to a file:

#sar -A -o /var/log/performanceTests/sarLog 5 0

sar parameters:
-A: records ALL server stats
-o: elects the binary output file Path
5 0: [interval count] in seconds: take stats every five seconds indefinitely

You can then use the sar tool to read and display the stats you require. I’ll be writing another article on how to do this soon.

Posted in Linux, Performance Tuning | Tagged , , , , , , , , | Leave a comment

Count lines of code in a linux web project

Posted on August 3, 2010 by William Jamieson

Go into your directory containing your web code:

#cd /var/www/html/myVirtualHost

Use the following command to count the lines of code in php, inc, js and css files:

#find . -type f -name '*.js' -o -name '*.css' -o -name '*.inc' -o -name '*.php'   | xargs wc -l | tail -n1
31065 total
Posted in Development, Linux, PHP | Tagged , , , , , , , , , , | Leave a comment