HOW TO SETUP LGI
----------------
In this document a howto is presented on the setup of LGI. The document
is divided into four parts. Part I will show you how to setup a resource
within an existing LGI project, part II will show you how to setup a new
LGI project with a project server. Part III will show you how to add an
extra project server into an existing LGI project. The last part part IV
will detail on the maintenance of your LGI project servers.
NOTE: For RHEL 5 and 6 based system, .spec files are present in the specs
directory to build .rpm files.
Part I : Howto setup a resource
-------------------------------
NOTE: For RHEL 5 and 6 based system, .spec files are present in the specs
directory to build .rpm files. The rpm make installing a resource easier!
Setting up a resource within an LGI project is rather easy. Just follow the
following steps:
- Untar or checkout the source tree in '~/LGI'.
- Get in contact with the project administrator to get the X.509 certificate
and a private key for your resource.
- Copy the key and certificate files into the '~/LGI/privatekeys' and
'~/LGI/certificates' directories resp.
- Make sure that libcurl is installed. Visit http://curl.haxx.se/libcurl/ if
you do not have this yet.
- Go to the '~/LGI/src' directory and invoke make. If you have a different
compiler installed than GNU's g++, or you have libcurl installed in a
different location, adjust the first few lines of the file Makefile to
your needs.
- If you want, you can also invoke make with the 'make install' command. If
you do that the directory '~/LGI/bin' will be created and you can include
that in your PATH for convencience.
- Edit the '~/LGI/daemon/LGI.cfg' example configuration file. It has been
setup to run the hello_world application by just simple forking scripts.
Other example scripts on how to use the 'hello_world' example using Torque
/ PBS or LoadLeveler are included to in the corresponding '~/LGI/daemon/
hello_world_XX_scripts' subdirectories. Please make sure you refer to
the right LGI project server, use the correct project name and use the
correct private key and certificate files from the '~/LGI/privatekeys' and
'~/LGI/certificates' directories. Also be sure to use absolute paths in
the configuration file so it is possible to invoke the daemon from any
other directory. Finally set the run directory correctly if you do not
want it to be the default.
- Start the daemon by invoking the daemon like:
~/LGI/daemon/bin/LGI_daemon -d -l ~/LGI/daemon/LGI.log ~/LGI/daemon/LGI.cfg
You can now inspect the log file '~/LGI/daemon/LGI.log' and see if everything
was setup correctly.
- You can gracefully stop the dameon by sending it a 'kill' signal:
killall LGI_daemon
While the daemon is running, you can edit the configuration file without any
problems. You can also edit the scripts in the mean time, only newly spawned
jobs will be affected by these changes.
Keep in mind for a project administrator, you should also include the new resource
into the projects database. See part IV for an example.
NOTE: For RHEL 5 and 6 based system, .spec files are present in the specs
directory to build .rpm files.
Part II : Howto setup a project server
--------------------------------------
Setting up your own project is not too hard either. You can decide to be
a fully independent project by setting up your own X.509 Certificate
Authority, or you can decide to be a sub project and thus be an X.509 sub-CA
of the LGI-CA. In this part, it is shown how to become an independant project
and be your own X.509 CA.
NOTE: One perhaps should use TinyCA (http://tinyca.sm-zone.net/) for managing
certificates and keys for your project.
NOTE: For RHEL 5 and 6 based system, .spec files are present in the specs
directory to build .rpm files. The rpm automatically installs a project LGI
and gives a working setup but excludes the MySQL and PHP APC optimizations
detailed below!
- Make sure libcurl, Perl, PHP, MySQL, Apache and OpenSSL are installed on
your system.
- Make sure MySQL is configured correctly and mysqld is running.
- Make sure PHP is configured correctly. It is highly recommended that you
also correctly instal PHP PEAR so that you can install the APC PHP PECL
module too. It will enable caching of PHP bytecode from the ZEND engine
and enhance the PHP performance significantly.
- Switch to the user apache and make sure you install the source tree
into the home of user apache where it will be served:
sudo su -l -s /bin/bash apache
tar -zxf LGI.tar.gz -C /var/www/html
A better approach is to create a user LGI on the project server and make the
user apache also member of the LGI group. In this case all files should be
group readable, the repository directory should be 'sticky' and all
directories should be group executable. The default settings in the .tar.gz
file take care of this. The .rpm installation also takes care of this,
uses the LGI username and also implements init scripts.
- Use OpenSSL to create a CA private key and certificate with which you can
create and sign your project server, resource and client certificates:
Start by creating a private key for your CA:
openssl genrsa -out /var/www/html/LGI/privatekeys/exampleCA.key 4096
Then create an X.509 selfsigned certificate for it:
openssl req -new -x509 -days 365 -set_serial 0 -extensions v3_ca
-key /var/www/html/LGI/privatekeys/exampleCA.key
-out /var/www/html/LGI/certificates/exampleCA.crt
and create a CA serial number file:
echo "0100" > /var/www/html/LGI/privatekeys/exampleCA.srl
Now create a private key for your server:
openssl genrsa -out /var/www/html/LGI/privatekeys/exampleserver.key 4096
Then create a sign request for it:
openssl req -new -key /var/www/html/LGI/privatekeys/exampleserver.key
-out /var/www/html/LGI/certificates/exampleserver.crs
and in filling in the commonname of the server certificate, be sure to use the
correct LGI format 'apache@exampleserver.somewhere.org; exampleprojectname'.
If you decided to use a special LGI username; make sure you use the correct
commonname accordingly.
Now create the CA signed server certificate from this request:
openssl x509 -req -in /var/www/html/LGI/certificates/exampleserver.crs
-days 365 -CA /var/www/html/LGI/certificates/exampleCA.crt
-CAkey /var/www/html/LGI/privatekeys/exampleCA.key
-CAserial /var/www/html/LGI/privatekeys/exampleCA.srl
-out /var/www/html/LGI/certificates/exampleserver.crt
You should now safely store the CA private key file somewere not on
this server computer that is conected to the internet. Or, at least, if
you do not do that, encrypt the file using a password protection:
openssl aes-256-cbc -e -in /var/www/html/LGI/privatekeys/exampleCA.key
-out /var/www/html/LGI/privatekeys/exampleCA.key.aes; rm
/var/www/html/LGI/privatekeys/exampleCA.key
- Next create a certificate for Apache itself. The previously generated
certificate is used by the scheduler running as apache or LGI on the project
server. Apache itself also needs a certificate to identify itself to resources
and interfacing communicating with the project server. Keep in mind in this
Apache certificate (the 'exampleserver_apache' files in this manual below) the
commonname should be the fully qualified hostname. It can also be signed
by the CA setup in the previous step. If you choose not to use another
certificate for your Apache with the correct fully quallified hostname as
the commonname, be aware that you need to relax the checkings of resources
with their daemons and the command line interface utilities through the '-W'
option.
- Make sure Apache is setup correctly to use client and server certificates
for https and check that the following options have been set correctly into
the https (virtual host) configuration:
SSLSessionCache shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout 5
SSLCertificateFile /var/www/html/LGI/certificates/exampleserver_apache.crt
SSLCertificateKeyFile /var/www/html/LGI/privatekeys/exampleserver_apache.key
SSLCertificateChainFile /var/www/html/LGI/certificates/exampleCA.crt
SSLCACertificateFile /var/www/html/LGI/certificates/exampleCA.crt
SSLVerifyClient require
SSLVerifyDepth 5
SSLOptions +ExportCertData
SSLOptions +StdEnvVars
Also make sure the AccessFileName has been set correctly in the Apache main
configuration so that the LGI document tree can be protected by .htaccess
files (if desired), make sure the important directories are carefully
protected and for performance reasons, make sure KeepAlive is turned on,
KeepAliveTimeOut is set low and MaxKeepAliveRequests set high in the
main server configuration of Apache. Also make sure the Timeout is not to
high and no DNS queries are perfomed for each request:
DocumentRoot "/var/www/html"
AccessFileName .htaccess
HostnameLookups off
Timeout 15
KeepAlive On
KeepAliveTimeout 1
MaxKeepAliveRequests 100
Options FollowSymLinks
AllowOverride None
AllowOverride All
AllowOverride None
Options -ExecCGI
php_flag engine off
SSLRequireSSL
SSLRequire ( %{SSL_CLIENT_VERIFY} == "SUCCESS" )
Script PUT /LGI/repository/put.cgi
Script DELETE /LGI/repository/delete.cgi
Allow from all
Deny from all
SSLRequireSSL
SSLRequire ( %{SSL_CLIENT_VERIFY} == "SUCCESS" )
Allow from all
SSLRequireSSL
SSLRequire ( %{SSL_CLIENT_VERIFY} == "SUCCESS" )
Allow from all
Options +ExecCGI
AddHandler cgi-script .cgi
SSLRequireSSL
SSLRequire ( %{SSL_CLIENT_VERIFY} == "SUCCESS" )
Allow from all
Options +ExecCGI
AddHandler cgi-script .cgi
Deny from all
Deny from all
Deny from all
Deny from all
Deny from all
Deny from all
Deny from all
Deny from all
Deny from all
php_value upload_max_filesize 16M
php_value post_max_size 16M
php_value upload_max_filesize 16M
php_value post_max_size 16M
php_value upload_max_filesize 16M
php_value post_max_size 16M
Make sure you have an .htaccess file in the '/var/www/html/LGI/privatekeys',
'/var/www/html/LGI/inc', '/var/www/html/LGI/tools', '/var/www/html/LGI/daemon',
'/var/www/html/LGI/scheduler', '/var/www/html/LGI/src', '/var/www/html/LGI/bin',
'/var/www/html/LGI/specs' and '/var/www/html/LGI/python' directories with the
content:
Deny from all
Make sure you have an .htaccess file in the '/var/www/html/LGI/repository'
directory with the content:
Deny from all
SSLRequireSSL
SSLRequire ( %{SSL_CLIENT_VERIFY} == "SUCCESS" )
Allow from all
SSLRequireSSL
SSLRequire ( %{SSL_CLIENT_VERIFY} == "SUCCESS" )
Allow from all
Options +ExecCGI
AddHandler cgi-script .cgi
SSLRequireSSL
SSLRequire ( %{SSL_CLIENT_VERIFY} == "SUCCESS" )
Allow from all
Options +ExecCGI
AddHandler cgi-script .cgi
Make sure you have an .htaccess file in the directories
'/var/www/html/LGI/basic_interface', '/var/www/html/LGI/interfaces' and
'/var/www/html/LGI/resources' with the following content:
php_value upload_max_filesize 16M
php_value post_max_size 16M
Make sure you have an 'index.html' file in the subdirectories
'/var/www/html/LGI/repository', '/var/www/html/LGI/inc',
'/var/www/html/LGI/interfaces', '/var/www/html/LGI/servers'
'/var/www/html/LGI/resources', '/var/www/html/LGI/daemon',
'/var/www/html/LGI/tools', '/var/www/html/LGI/privatekeys',
'/var/www/html/LGI/src', '/var/www/html/LGI/bin',
'/var/www/html/LGI/python', '/var/www/html/LGI/specs' and
'/var/www/html/LGI/scheduler' with the content:
! No browsing allowed !
For high performance, make sure you scale the Apache MPM settings to your
needs taking the amount of RAM your server has into account:
http://httpd.apache.org/docs/2.2/mod/prefork.html
http://www.ibm.com/developerworks/linux/library/l-tune-lamp-2.html
It is also advised to set the net.ipv4.tcp_fin_timeout to 15s or less with
the sysctl command. Make this setting permanent by adding it into the file
/etc/sysctl.conf. This setting will make sure that keep-alive connections,
when closed by Apache, do not linger to long until the client finaly closes
the socket.
Finally check your Apache configuration with the 'apachectl configtest' command.
!!!!! WORD OF CAUTION !!!!!
The above configuration of Apache seems superfluous but ensures a secure
environment. Please be very carefull when diverting from the above
suggested configuration. Web application security is a complicated matter
and a lot of care has been taken to ensure a secure default configuration
that will work out-of-the box mostly tuned for performance.
!!!!! WORD OF CAUTION !!!!!
- Now create a MySQL user and database for your project with the database
name equal to your project name "exampleprojectname":
mysqladmin -u root password "yourmysqlrootpassword"
mysqladmin -u root -p create "exampleprojectname"
echo 'GRANT ALL PRIVILEGES ON exampleprojectname.* to
"examplemysqluser"@"localhost" IDENTIFIED BY "examplemysqluserpasswd"'
| mysql -u root -p mysql
mysql -u examplemysqluser -p exampleprojectname < /var/www/html/LGI/LGI.db
For high performance reasons you might want to set the the following options
in /etc/my.cnf for MySQL if you have enough resources available:
table_cache = 512
max_connections = 128
query_cache_size = 128M
key_buffer_size = 128M
sort_buffer_size = 128M
query_cache_type = 1
thread_cache_size = 64
innodb_log_buffer_size = 16M
innodb_flush_method = O_DIRECT
innodb_fast_shutdown = 1
innodb_flush_log_at_trx_commit = 0
innodb_log_archive = 0
innodb_buffer_pool_size = 2048M
innodb_additional_mem_pool_size = 8M
innodb_file_io_threads = 32
innodb_lock_wait_timeout = 50
innodb_thread_concurrency = 32
See the LGI.pdf document for more details on scaling and performance.
- Insert your project server entry into the database by using the database
management tool:
/var/www/html/LGI/tools/ManageDB add resources allowed exampleprojectname
localhost examplemysqluser examplemysqluserpasswd
Enter resource name: apache@exampleserver.somewhere.org
Enter resource url: https://exampleserver.somewhere.org/LGI
Enter certificate file: /var/www/html/LGI/certificates/exampleserver.crt
Enter project server flag: 1
Enter servers to update: any
Just make sure you set the servers to be updated to 'any'. This will
make sure that this insert is recorded as update nr 1 and when adding
slave servers later on, everything will be automatically synchronized.
- Make sure you edit the project web server configuration file
'/var/www/html/LGI/inc/Config.inc':
$Config[ "SERVER_URL" ] = "https://exampleserver.somewhere.org/LGI";
$Config[ "SERVER_NAME" ] = "apache@exampleserver.somewhere.org";
$Config[ "SERVER_SSL_CERTIFICATE_FILE" ] = "../certificates/exampleserver.crt";
$Config[ "SERVER_SSL_KEY" ] = "../privatekeys/exampleserver.key";
$Config[ "SERVER_SSL_CA_CERTIFICATE_URL" ] =
"https://exampleserver.somewhere.org/LGI/certificates/exampleCA.crt";
$Config[ "SERVER_SSL_CA_CERTIFICATE_FILE" ] = "../certificates/exampleCA.crt";
$Config[ "MYSQL_URL" ] = "localhost";
$Config[ "MYSQL_USER" ] = "examplemysqluser";
$Config[ "MYSQL_PASSWD" ] = "examplemusqluserpasswd";
$Config[ "MYSQL_DEFAULT_DATABASE" ] = "exampleprojectname";
$Config[ "REPOSITORY_DIRECTORY" ] = "/var/www/html/LGI/repository";
$Config[ "REPOSITORY_SERVER_NAME" ] = $Config[ "SERVER_NAME" ];
$Config[ "REPOSITORY_URL" ] = $Config[ "SERVER_URL" ]."/repository";
- Edit the /var/www/html/LGI/scheduler/scheduler.php file if more than one
project is configured. Just uncomment the required array_push lines to
add your projects to the list:
array_push( $Projects, "secondprojectname" );
array_push( $Projects, "thirdprojectname" );
Each project has an independant database on the web server, by creating a
second database, as described above, a second project has been setup.
Edit the /var/www/html/LGI/scheduler/check_running script to use the correct
LGI_ROOT directory to run the scheduler. Now you can start the scheduler on
the background like:
/var/www/html/LGI/scheduler/check_running
You can also add a crontab entry to automatically check if the scheduler is
running once per hour:
MAILTO=""
1 * * * * /var/www/html/LGI/scheduler/check_running
Your project servers has been setup now and can be used!
NOTE: For RHEL 5 and 6 bases system, .spec files are present in the specs
directory to build .rpm files. The rpm automatically installs a working
project LGI but the MySQL and python APC optimisations detailed above
have not been implemented into the rpm installation yet!
Part III : Howto setup a second slave project server
----------------------------------------------------
After having successfully setup your first LGI master project server yourself,
adding slave servers is fairly straightforward. Follow the same procedure on
the slave server as was described in part II for the master server, but now
there is no need to setup a CA. You only have to generate a server key and a
certificate request for the slave server, and obviously sign that request with
your CA certificate. Also keep in mind that for the slave server Apache
configuration an extra certificate is needed with the commonname set to the
correct fully quallified hostname.
However, at the point where you insert the server entry into the fresh database
in part II, use that command now to insert the MASTER server data into the
slave database:
(on the slave server)
wget https://exampleserver.somewhere.org/certificates/exampleCA.crt
-O /var/www/html/LGI/certificates/exampleCA.crt
wget https://exampleserver.somewhere.org/certificates/exampleserver.crt
-O /var/www/html/LGI/certificates/exampleserver.crt
/var/www/html/LGI/tools/ManageDB add resources allowed exampleprojectname
localhost exampleslavemysqluser exampleslavemysqluserpasswd
Enter resource name: apache@exampleserver.somewhere.org
Enter resource url: https://exampleserver.somewhere.org/LGI
Enter certificate file: /var/www/html/LGI/certificates/exampleserver.crt
Enter project server flag: 1
Enter servers to update: any
Again be sure to set the servers to be updated to 'any'. This will make sure
that the first update of this slave and of the master server are synchronized.
All other updates from the master server (including the addition of this
slave below) will be automatically picked up by this slave once the scheduler
runs.
Now insert the slave server details into the master server database:
(on the master server)
wget https://exampleslaveserver.somewhere.org/certificates/exampleslaveserver.crt
-O /var/www/html/LGI/certificates/exampleslaveserver.crt
/var/www/html/LGI/tools/ManageDB add resources allowed exampleprojectname
localhost examplemysqluser examplemysqluserpasswd
Enter resource name: apache@exampleslaveserver.somewhere.org
Enter resource url: https://exampleslaveserver.somewhere.org/LGI
Enter certificate file: /var/www/html/LGI/certificates/exampleslaveserver.crt
Enter project server flag: 2
Enter servers to update: any
Make sure you use the correct server flag value '2' and set the servers to be
updated to 'any'. This will make sure that all other slave servers will also
pick up the change and updates from the database. It will also make sure that
your slave server currently being setup will load it's certificate into it's
database through an update automatically. Just continue with configuring the
slave servers config file and scheduler as described in part II. When done, the
two servers are linked together and the schedulers ensure that any updates
pending in the 'updates' table of the master server database are transfered
to the slave server. Users can now use both the slave and the master server
independantly to submit jobs. The resource daemon will automatically request
work from all servers of the project.
Part IV: Updating and maintaining project servers:
--------------------------------------------------
By default on each project server, the basic interface is active and
any user is allowed to have only two jobs in the database. If you want
certain users or groups of users to be able to submit more, you can add
that user or group with a new limit for a certain application. One
should do that with the ManageDB tool:
/var/www/html/LGI/tools/ManageDB add users allowed exampleprojectname
localhost examplemysqluser examplemysqluserpasswd
Enter user name: theusername
Enter application: hello_world
Enter job limit: 10
Enter servers to update: any
The ManageDB tool can be used to perform most project management tasks. If
you want, you can also configure it with default settings (see script itself)
to avoid long command with usernames and passwords:
# some default settings...
MYSQL_HOST="localhost"
MYSQL_USER="examplemysqluser"
MYSQL_PASSWD="examplemysqluserpasswd"
MYSQL_DB="exampleprojectname"
To find out what can all be managed; just run the tool without any parameters:
/var/www/html/LGI/tools/ManageDB
ManageDB {list|add|del} {users|groups|resources} {allowed|denied} [DB [HST
[USR [PWD]]]]
Also read the LGI documentation in the docs directory to understand what
limits can be set and what meaning they have. In general a limit value of 0
means there is no limit; a positive limit value means a maximum total number
of jobs limit and a negative limit is a maximum number of queued or running
jobs in the database. None the less please read the documentation to
understand which tables have precedence over what.
A new resource is also easily added now:
/var/www/html/LGI/tools/ManageDB add resources allowed
Enter resource name: user@resource
Enter resource url: user@resource
Enter certificate file: resource.crt
Enter project server flag: 0
Enter servers to update: any
Here a new group is added on the slave server only:
/var/www/html/LGI/tools/ManageDB add groups allowed
Enter group name: newgroup
Enter application: any
Enter job limit: -10
Enter servers to update: apache@exampleslaveserver.somewhere.org
Or a user is denied on the master server only:
/var/www/html/LGI/tools/ManageDB add users denied
Enter user name: baduser
Enter application: any
Enter servers to update: apache@exampleserver.somewhere.org