Fedora: Adding services to the startup sequence

Today I was wondering if there is a utility that allows you to control the services that are loaded at the startup (e.g. I wanted to add vsftpd/sshd/mysqld to the automatic startup sequence). I found a command line utilities: chkconfig and ntsysv. I briefly checked the first (I will describe it later on) but actually used the ntsysv (but it seems that at the end it uses chkconfig to add the service to a current runlevel) to add the startup services - you can see what does it look like on the attached screenshot. If you would like to use the ntsysv for a specific runlevel:

[root@krystianekb init.d]# ntsysv --level 5 -> for runlevel 5
[root@krystianekb init.d]# ntsysv --level 235 -> for runlevels 2,3,5

For a services that come with the Fedora, like vsftpd, mysqld, sshd etc. it is only a matter of marking the service to be started or not started during boot. However I have created my own service (for starting the JBoss), which I was able to control with the service command but it was not seen by the ntsysv and was not compatible with the chkconfig. I found in the man page of chkconfig about one cool feature, namely that one can specify within the service (own script in the init.d directory) for which level this service could be available with what start and stop priorities. So I have added the following line to the top section of my startup script:

# chkconfig: 235 90 15

The first column identifies the runlevels: 2,3,5, second column the start priority: here 90 and the last one the stop priority: here 15. The following sequence of command demonstrates how to add a service for runlevels 2,3,5 and how does the ntsysv and chkconfig work together:

[root@krystianekb init.d]# chkconfig --add jboss
[root@krystianekb init.d]# chkconfig --list jboss
jboss 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@krystianekb init.d]# ntsysv --level 235 [root@krystianekb init.d]# chkconfig --list jboss
jboss 0:off 1:off 2:on 3:on 4:off 5:on 6:off

Fedora: Setting up FTP server

At work I heard about a utility called systemtap (a similar tool to DTrace but for Linux OS) and I wanted to check it on the similar to the production environment (RHEL). I decided to use Fedora as the closest one to the RHEL (RHEL was only available as 6.0 Beta when the article was written). The installation on the vmware went without any problems -> these came afterwards. First thing that came to me was that I need a FTP server in order to upload things onto my Fedora system but out of the box my Fedora 13 installation did not have it enabled. I would divide the procedure into two major steps:

1) installation and firewall setup - follow instruction on wiki page. To summarize - one needs to install the package (yum install vsftp) and setup the firewall rules so that the traffic is passed to the ftp port

2) configuration of access rights - in my case step 1 was still not enough. Each login access attempt as a normal user ended up with error:

500 OOPS: cannot change directory

In order to solve it I needed to change the SELinux boolean value of ftp_home_dir:

# getsebool -a | grep ftp
allow_ftpd_anon_write --> off
allow_ftpd_full_access --> off
allow_ftpd_use_cifs --> off
allow_ftpd_use_nfs --> off
ftp_home_dir --> off
ftpd_connect_db --> off
httpd_enable_ftp_server --> off
sftpd_anon_write --> off
sftpd_enable_homedirs --> off
sftpd_full_access --> off
sftpd_write_ssh_home --> off
tftp_anon_write --> off
# setsebool -P ftp_home_dir on

# getsebool -a | grep ftp
allow_ftpd_anon_write --> off
allow_ftpd_full_access --> off
allow_ftpd_use_cifs --> off
allow_ftpd_use_nfs --> off
ftp_home_dir --> on
ftpd_connect_db --> off
httpd_enable_ftp_server --> off
sftpd_anon_write --> off
sftpd_enable_homedirs --> off
sftpd_full_access --> off
sftpd_write_ssh_home --> off
tftp_anon_write --> off
#

Afterwards I was able to login/create directories/upload files via FTP as a normal OS user ;)

JEE : JBoss 5.1.0 GA and JDK 1.6

The recommended JDK for the standard downloadable JBoss 5.1.0 GA binaries is JDK 5. So if you have strange exception while invoking the SOAP WS (as I got - enclosed below) you probably forgot to check the releaseNotes and did not do the follow the guidelines from there:

from 5.0.0.GA
JBossAS 5.0.0.GA can be compiled with both Java5 & Java6. The Java5 compiled binary is our primary/recommended binary distribution. It has undergone rigorous testing and can run under both a Java 5 and a Java 6 runtime. When running under Java 6 you need to manually copy the following libraries from the JBOSS_HOME/client directory to the JBOSS_HOME/lib/endorsed directory, so that the JAX-WS 2.0 apis supported by JBossWS are used:
jbossws-native-saaj.jar
jbossws-native-jaxrpc.jar
jbossws-native-jaxws.jar
jbossws-native-jaxws-ext.jar

The files are locatedt in the common/lib/ subdirectory. After copying them to lib/endorsed and of course restarting JBoss the presented below exception was gone:

2010-08-26 23:11:34,073 ERROR [org.jboss.ws.core.jaxws.SOAPFaultHelperJAXWS] (http-127.0.0.1-8080-2) SOAP request exception
java.lang.UnsupportedOperationException: setProperty must be overridden by all subclasses of SOAPMessage
at javax.xml.soap.SOAPMessage.setProperty(Unknown Source)
at org.jboss.ws.core.soap.SOAPMessageImpl.(SOAPMessageImpl.java:87)
at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:215)
at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:193)
at org.jboss.wsf.stack.jbws.RequestHandlerImpl.processRequest(RequestHandlerImpl.java:455)
at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:295)
at org.jboss.wsf.stack.jbws.RequestHandlerImpl.doPost(RequestHandlerImpl.java:205)
at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:131)
at org.jboss.wsf.common.servlet.AbstractEndpointServlet.service(AbstractEndpointServlet.java:85)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Unknown Source)

There is also a jdk6 version available for download on the sourceforge.

Java - JSR181 and jaxws-api maven problem

Today I tried to create a WS that would be used for a purpose of mass provisioning operations. The ideas was to transfer the file as the attachement to the normal SOAP message (to avoid xml tag overhead) - in my case I decided to check MTOM (Message Transmission Optimization Mechanism) and JBoss as the runtime environment (version 5.1.0 with Tomcat web container). I found a manual describing howto enable the MTOM for a WS in JBoss and decided to start the implementation. For that purpose I used EclipseIDE + maven for building the packages and here the first problem occured namely it seems to be a dependency problem between jsr181-api (WebService, SOAPBinding annotations) and jaxws-api (BindingType annotation). In my case I was able to use the first one in version 1.0 MR (MaintenanceRelease) while the jaxws-api in version 2.1-1 required the 1.0 version, which led to the following problem:

Missing artifact javax.jws:jsr181:jar:1.0:compile

The problem basically is that there is no packaging available for this artifact - probably due to some licensing restrictions.

1) Fortunately there is a workaround that helped in my case - to use the geronimo-jaxws_2.1_spec instead of jaxws-api :) And it solved my compilation problems.

2) I found another solution to the previously described problem:
- I added the http://download.java.net/maven/2 repository
- I increased the version of jaxws-api 2.2.1

Moreover the version 2.2.1 comes with JSR224 annotations, which contain the @MTOM annotation for web services supporting the MTOM.

Network: IPv4 address space utilization forecasts - part 2

I decided to find some time slot today to continue on the forecasts issue and indeed as expected the exponential trend line seems to be much closer to the data allocation one. The "doom" date for the IPv4 address space using this trend line is about one year earlier than in case of linear approximation - 1st of March 2013. However the same remarks apply as for the linear trned line (see Linear Trend Line forecasts).

Approximation/trend line fitting was based on the least squares fitting algorithm (taken from: http://mathworld.wolfram.com/LeastSquaresFittingExponential.html). The chart comparing two trend lines is presented below:

Network: IPv4 address space utilization forecasts

Today I have found to extend my Ruby scripts to make some forecasts about the "doom" date - the date in which there will be no IPv4 prefixes available for allocations. The trend line has been approximated using the y=ax+b line (using the formulas from http://www.fourmilab.ch/hackdiet/www/subsubsection1_4_1_0_8_4.html) and extended until the total number of available prefixes is reached. The forecasted date is actually in the near future, namely end of February 2014. As one can see the approaximation is not perfect (probably the exponential approximation would fit better here) since recently the allocation line goes much higher than the trend line but this was for me the starting point. I will still observe the allocations from different registries over time and publish the results on the blog (with possible corrections to the forecasts).

One only has to take into account few things like:

• the current growth is above the trend line (mainly due to the contribution from the APNIC registry, for which the number of allocations is recently grownig much faster than in the past - one has to observe the progress in the future) - which might mean that the IPv4 prefix exhaustion date might be even closer
• there is still some number of prefixes currently reserved for the IANA but that might be released for allocations in the future
• the number of broadcasted prefixes in BGP is much lower than the allocated ones (the chart has been prepared on the basis of the number of prefixes allocated by each registry)
  

Network: BGP vs RIR IPv4 assignments

I have compared the values obtained from each registry against the prefixes announced in BGP (BGP potaroo IPv4 stats) and the results are presented on the chart. As far as the biggest registry is concerned (ARIN) the number of "active" prefixes (broadcasted in BGP) is significantly lower (at level of 60%) comparing to the number of assigned prefixes. Utlization rate of the other rirs is on a quite high level (for the smallest ones there are one or two prerfixes not announced in BGP). In the article I enclose the chart and the table presenting the values.

Registry	BGP announced	Allocated	Percentage
Afrinic	1	2	50%
Apnic	34	43	79%
Arin	59	97	60%
Arin	59	97	60%
Lacnic	5	7	71%
Ripencc	32	40	80%

Network: IPv4 vs IPv6 statistics

In my master thesis I did some research on the usage of the IPv4 address space and decided to continue check that topic right now. I have created a ruby application that parsed the statistics published by each RIR and calculate the overall number of /8 being assigned to each of them. Additionally a group of special purpose addresses reserved by IANA has been identified and placed on the charts.

The chart enclosed on the left presents the distribution of /8 prefixes (status on 22.08.2010). As expected the biggest number of /8 assignment is to the ARIN registry (~40% of the total number of IPv4 /8 prefixes). The numbers of prefixes for APNIC and RIPE are very closed together - although last year RIPE was ahead of APNIC. The important information is that there is still 13% (32 prefixes) available to be assigned for further IPv4 development.

A very valuable information

is how the utilization of the IPv4 prefixes looked over time. There is a figure, which shows the statistics retrieved from the RIRs from th 2003 till now (22.08.2010). One can easily notice that for all RIRs the number of assigned IPv4 prefixes increases. As far as the biggest contributor ARIN is concerned, the increase is very small. The highest increased in the last months can be identified for the APNIC RIR, which last year was still below RIPE and from beginning of this year has overcome it and became number two RIR in terms of /8 prefix utilization.

Figure presenting the total number of prefixes used shows the overall IPv4 /8 prefix statistics (including the reserved space for IANA) over time. As one can see the number is constantly increasing and it seems righ t now that eventually it will reach the total number of available IPv4 /8 prefixes. The author thinks that based on this chart the date of the exhaustion of the IPv4 address space could be estimated (one of the next steps for the author).

NEXT STEPS:

1. Approximate the total IPv4 prefix utilization to make an estimate of the possible date of the exhaustion of the IPv4 address space

2. Verify the number of assigned prefixes against the broadcasted ones (BGP) - hopefully statistics from http://bgp.potaroo.net could be used

Ruby - sending mails with attachements

Recently I have worked on the ruby script that would do some processing and afterwards would send an email containing the results nad charts (generated with gruff).

At first I tried the plain Net:SMTP and it was sufficient for my needs until I reached the attachements part. With the low level API it was very difficult to make it working (I reached the state that the attachements were sent with the email but they were corrupted). Fortunately I found on the web mailfactory gem (http://rubyforge.org/projects/mailfactory/, can be installed using gem install mailfactory) that helps the programmer to construct the content of the email that will be send with the Net:SMTP mechanisms later on. I really recommend it for its simplicity and ease of use. Below I enclose some example to show show to construct basic message with attachements:

def createInitialMail(subject, msg, from, to)
mail = MailFactory.new
mail.to = to
mail.from = from
mail.subject = subject
mail.html = msg
return mail
end

mail = createInitialMail subject, msg, from, to
mail.attach filename

...

Net::SMTP.start(host, port, host, account, password, :plain) do smtp
to_address = mail.to
smtp.send_message mail.to_s, from, to_address
smtp.finish
end

The createInitialMessage method is used in the example to build the main mail object. I marked with green colour the place how add attachements to the message. At the end there is an example how to send the previously constructed message using the Net::SMTP.

Ruby - rmagick gem on windows

I have been using Ruby 1.9.1 and unfortunately the precompiled binary rmagick-win32 gem is available only for 1.8.6. In order to use it on windows one needs to recompile the sources and install the gem. Fortunately after checking several different proposals on how to do that I found one that worked for me - it can be found here: http://www.waydotnet.com/blog/2010/02/rmagick-on-ruby-1-9-1-i386-mingw32-work-d/

My configuration is:

identify -version
Version: ImageMagick 6.6.3-7 2010-08-14 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2010 ImageMagick Studio LLC
Features: OpenMP
ruby -v
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-mingw32]

The OS in my case is Windows 7, 64-bit.

Please do not forget (as I did) to replace the types (xml file from imagemagick), otherwise you will experience problems with the examples from the rmagick gem (e.g. watermark.rb).